Patent ID: 12198225

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one of skill in the art that the inventive concepts may be practiced without one or more of these specific details.

System Overview

FIG.1illustrates a computing device100configured to implement one or more aspects of various embodiments. In one embodiment, computing device100includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. Computing device100is configured to run a training engine122and an execution engine124that reside in a memory116.

It is noted that the computing device described herein is illustrative and that any other technically feasible configurations fall within the scope of the present disclosure. For example, multiple instances of training engine122and execution engine124could execute on a set of nodes in a distributed system to implement the functionality of computing device100.

In one embodiment, computing device100includes, without limitation, an interconnect (bus)112that connects one or more processors102, an input/output (I/O) device interface104coupled to one or more input/output (I/O) devices108, memory116, a storage114, and a network interface106. Processor(s)102may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU. In general, processor(s)102may be any technically feasible hardware unit capable of processing data and/or executing software applications. Further, in the context of this disclosure, the computing elements shown in computing device100may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executing within a computing cloud.

I/O devices108include devices capable of providing input, such as a keyboard, a mouse, a touch-sensitive screen, and so forth, as well as devices capable of providing output, such as a display device. Additionally, I/O devices108may include devices capable of both receiving input and providing output, such as a touchscreen, a universal serial bus (USB) port, and so forth. I/O devices108may be configured to receive various types of input from an end-user (e.g., a designer) of computing device100, and to also provide various types of output to the end-user of computing device100, such as displayed digital images or digital videos or text. In some embodiments, one or more of I/O devices108are configured to couple computing device100to a network110.

Network110is any technically feasible type of communications network that allows data to be exchanged between computing device100and external entities or devices, such as a web server or another networked computing device. For example, network110may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and/or the Internet, among others.

Storage114includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid state storage devices. Training engine122and execution engine124may be stored in storage114and loaded into memory116when executed.

Memory116includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof. Processor(s)102, I/O device interface104, and network interface106are configured to read data from and write data to memory116. Memory116includes various software programs that can be executed by processor(s)102and application data associated with said software programs, including training engine122and execution engine124.

In some embodiments, training engine122trains a machine learning model to learn nonlinear global (e.g., across an entire shape) and local (e.g., in the vicinity of a point or region within a shape) correlations across points or regions in faces, hands, bodies, and/or other three-dimensional (3D) shapes. Execution engine124executes one or more portions of the machine learning model to generate and/or reconstruct additional shapes. More specifically, the machine learning model includes a transformer-based neural network that can represent geometric shape detail in a topology-independent manner with arbitrary spatial resolution. Consequently, the machine learning model can be used in various applications related to generating, interpolating, and/or reconstructing various shapes, as described in further detail below.

Transformer-Based Topology Independent Shape Model

FIG.2is a more detailed illustration of training engine122and execution engine124ofFIG.1, according to various embodiments. As mentioned above, training engine122and execution engine124operate to train and execute a transformer200that represents a domain of statistically plausible 3D shapes in a nonlinear manner. For example, training engine122and execution engine124could use transformer200to represent statistically plausible variations on human or animal faces, bodies, and/or body parts. In another example, training engine122and execution engine124could use transformer200to represent different expressions, postures, or deformations associated with a face, body, body part, or another type of object with a specific identity (e.g., a specific person or animal).

As shown inFIG.2, transformer200represents a given 3D shape as a set of offsets224(1)-224(X) from a set of positions222(1)-222(X) on a canonical shape220, where X represents an integer greater than one. Canonical shape220represents a “template” from which all other shapes are defined. For example, canonical shape220could include a “standard” or “neutral” face, hand, body, or other deformable object. This “standard” or “neutral” representation of the object can be generated by averaging or otherwise aggregating the points across multiple (e.g., hundreds or thousands) of different versions of the object. If transformer200is used to represent expressions or deformations of an object associated with a specific identity (e.g., a specific person's face), canonical shape220could represent the object in a “neutral” or “default” expression or posture.

In one or more embodiments, canonical shape220is defined as a continuous surface in two-dimensional (2D) or 3D space. Each of positions222(1)-222(X) (which is referred to individually as position222) can then be sampled from the continuous surface. For example, canonical shape220could be specified as a manifold representing a face, hand, body, or another deformable object. As a result, positions222could be determined by sampling points from the manifold. After a given position222on the manifold is determined (e.g., as a set of 3D coordinates), the position could be stored for subsequent retrieval and use with transformer200.

Canonical shape220may also, or instead, include a “densest” mesh that includes all possible positions222that can be sampled or selected. For example, canonical shape220could include hundreds of thousands to millions of points in an “average” or “standard” representation of a face, hand, body, or another object. Positions222from which other shapes are defined could include some or all of these points in canonical shape220.

Each of offsets224(1)-224(X) (which is referred to individually as offset224) represents a shift in a corresponding position222in canonical shape220. This shift corresponds to a geometric deformation of canonical shape220into a different shape. For example, offsets224could be added to the corresponding positions222in canonical shape220to produce an output shape216that represents a specific identity, expression, or posture exhibited by a face, body, body part, or another object represented by canonical shape220.

As shown inFIG.2, transformer200includes an encoder204and a decoder206. In various embodiments, encoder204and decoder206are implemented as neural networks. Input into encoder204includes canonical shape positions232in canonical shape220, as well as offsets228between canonical shape positions232and corresponding points in a set of target shapes (e.g., one or more training shapes230). For example, canonical shape positions232could include locations or coordinates of points in canonical shape220, and offsets228could include differences between the locations or coordinates of points in canonical shape220and the locations or coordinates of corresponding points in the target shapes. For a given set of canonical shape positions232and a corresponding set of offsets228that represent a target shape, encoder204generates a shape code218that represents the target shape as a variation of canonical shape220.

Input into decoder206includes a given shape code218that is generated by encoder204, interpolated from two or more other shape codes, and/or randomly generated. For example, shape code218could include a Y-dimensional vector (where Y is an integer greater or equal to 1) that is generated by encoder204from a training shape. In another example, shape code218could be calculated as a Y-dimensional vector that lies along a line between two other Y-dimensional vectors that are generated by encoder204from two training shapes230. In a third example, shape code218could include a Y-dimensional vector, where each element in the vector is sampled from a distribution or range of values for that element outputted by encoder204from a set of training shapes230. Input into decoder206also includes a set of positions222in canonical shape220, which can be the same as or different from canonical shape positions232inputted into encoder204. For each position222in canonical shape220inputted into decoder206, decoder206generates a corresponding offset224that denotes a shift in that position222. Offsets224outputted by decoder206are combined with the corresponding positions222in canonical shape220to produce a new set of positions226(1)-226(X) (each of which is referred to individually as position226) in a different shape216.

FIG.3illustrates an exemplar architecture for transformer200ofFIG.2, according to various embodiments. As shown inFIG.3, encoder204includes a position multi-layer perceptron (MLP)306, an offset MLP308, and a number of encoder transformer blocks302. Decoder206includes a position MLP310, an offset MLP312, and a number of decoder transformer blocks304. Each of these components is described in further detail below.

Position MLP306in encoder204converts a set of canonical shape positions232in canonical shape220into a corresponding set of position tokens336. For example, position MLP306could include a series of fully connected layers that map each of canonical shape positions232in canonical shape220to a higher-dimensional position token in a latent space.

Offset MLP308converts a set of offsets228associated with canonical shape positions232in canonical shape220into a corresponding set of offset tokens338. For example, offset MLP308could include a series of fully connected layers that map each of offsets228to a higher-dimensional offset token in a latent space. Each of offsets228represents a displacement or shift in a corresponding canonical shape position in canonical shape220that results in a new position on a target shape316. Thus, if input into encoder204includes M (where M is an integer greater than or equal to 1) canonical shape positions232on canonical shape220and M corresponding offsets228representing target shape316, position MLP306generates M position tokens336from the M canonical shape positions232, and offset MLP308generates M offset tokens338from the M offsets228. This conversion of canonical shape positions232and offsets228into position tokens336and offset tokens338, respectively, allows transformer200to distribute representations of canonical shape positions232and offsets228in a way that optimizes for the shape modeling task.

Position tokens336are concatenated with the corresponding offset tokens338to form a set of concatenated tokens340. Concatenated tokens340represent a “tagging” of each offset token with features related to a corresponding position in canonical shape220. These “tagged” offset tokens338can be used by other components of transformer200to learn both local and global spatial correlations across variations in canonical shape220, as represented by target shape316.

Concatenated tokens340and a shape token342are inputted into a series of encoder transformer blocks302with the same structure and different weights. For example, concatenated tokens340and shape token342could be processed sequentially by a “stack” of N (where N is an integer greater than or equal to 1) encoder transformer blocks302, so that the output of a given encoder transformer block is used as input into the next encoder transformer block. Each encoder transformer block includes a cross-covariance image transformer (XCiT) block with a cross-covariance attention (XCA) layer, a transformer block with a self-attention layer, and/or another type of transformer neural network architecture. The output of the last encoder transformer block includes shape code218, which captures the differences between canonical shape220and target shape316in a manner that is independent of the topology associated with canonical shape positions232and offsets228.

In one or more embodiments, shape token342is an “extra” input token that represents all input target shapes into encoder204. During training of transformer200, shape token342is updated along with parameters of transformer200. Unlike concatenated tokens340, shape token342is not position encoded. Instead, encoder transformer blocks302embed shape token342with information from concatenated tokens340to produce a corresponding output shape code218. Because shape token342is able to represent positional differences between canonical shape220and a corresponding target shape316, other tokens outputted by encoder transformer blocks302from the corresponding concatenated tokens340can be discarded.

Within decoder206, position MLP310converts a second set of positions222in canonical shape220into a corresponding set of position tokens354. For example, position MLP310could include a series of fully connected layers that map each position to a higher-dimensional position token in a latent space. Position MLP310could include the same structure and weights as position MLP306, or position MLP310could differ in structure and/or weights from position MLP306. Further, positions222inputted into position MLP310of decoder206are not required to match canonical shape positions232inputted into position MLP306of encoder204.

Position tokens354are inputted into a series of decoder transformer blocks304with the same structure and different weights. For example, position tokens354could be processed sequentially by a “stack” of O (where O is an integer greater than or equal to 1) decoder transformer blocks304, so that the output of a given decoder transformer block is used as input into the next decoder transformer block. As with encoder transformer blocks302, each of decoder transformer blocks304includes a cross-covariance image transformer (XCiT) block with a cross-covariance attention (XCA) layer, a transformer block with a self-attention layer, and/or another type of transformer neural network architecture. Each of decoder transformer blocks304also uses shape code218to modulate position tokens354and/or a set of tokens from the previous decoder transformer block. The output of the last decoder transformer block includes a set of offset tokens356, one for each position token inputted into decoder transformer blocks304. Decoder transformer blocks304are described in further detail below with respect toFIGS.4A-4B.

Offset tokens356outputted by decoder transformer blocks304are inputted into an offset MLP312to generate a set of offsets224that represent an output shape216. These offsets224can be added to (or otherwise combined with) the corresponding positions222on canonical shape220to produce positions226in the output shape216.

Returning to the discussion ofFIG.2, training engine122trains transformer200using training data214that includes a set of training shapes230. In one or more embodiments, each of training shapes230includes a mesh, point cloud, or another representation of a set of points with known spatial correspondence. For example, training shapes230could include high-resolution 3D scans, motion capture data, and/or other point-based representations of faces, hands, bodies, and/or other objects. Points in training shapes230can vary in topology and/or spatial resolution. Points in training shapes230can also be averaged or otherwise combined into canonical shape220.

Training engine122computes offsets228between points in training shapes230and the corresponding canonical shape positions232in canonical shape220. Next, training engine122inputs offsets228and canonical shape positions232into encoder204to generate a set of training shape codes212representing training shapes230. Training engine122inputs training shape codes212and the same canonical shape positions232into decoder206to generate decoder output210that includes offsets224from canonical shape positions232. Training engine122also performs supervised training that jointly optimizes the weights of encoder204and decoder206and shape token342based on one or more losses208between decoder output210and offsets228of the corresponding training shapes230. For example, training engine122could calculate an L2 loss between offset values in decoder output210and the corresponding ground truth offsets228. Training engine122also uses a training technique (e.g., gradient descent and backpropagation) to iteratively update weights of encoder204and decoder206in a way that reduces subsequent losses208between offsets228calculated from training shapes230in training data214and the corresponding decoder output210.

In some embodiments, training engine122creates and/or trains transformer200according to one or more hyperparameters. In some embodiments, hyperparameters define higher-level properties of transformer200and/or are used to control the training of transformer200. For example, hyperparameters that affect the structure of transformer200could include (but are not limited to) the number of encoder transformer blocks302in encoder204and/or the number of decoder transformer blocks304in decoder206; the number of layers in position MLP306, offset MLP308, position MLP310, and/or offset MLP312; the dimensionality of the feed-forward layers in encoder transformer blocks302, decoder transformer blocks304, position MLP306, offset MLP308, position MLP310, and/or offset MLP312; and/or the dimensionality of position tokens336, offset tokens338, shape token342, shape code218, position tokens354, and/or offset tokens356. In another example, training engine122could train transformer200based on a batch size, learning rate, number of iterations, and/or another hyperparameter that controls the way in which weights in transformer200are updated during training.

After training engine122has completed training of transformer200, execution engine124can execute the trained transformer200to produce a new shape216from a given shape code218and a set of positions222in canonical shape220. For example, execution engine124could obtain a specific shape code218generated by the trained encoder204from a training shape in training data214, generate shape code218by interpolating between two or more shape codes generated from two or more training shapes230, and/or randomly generate shape code218. Next, execution engine124could input shape code218and an arbitrary set of positions222from canonical shape220into decoder206and obtain, as output of decoder206, offsets224from positions222that represent shape216. Execution engine124adds and/or otherwise combines offsets224with the corresponding positions222in canonical shape220to produce a set of positions226in shape216.

In one or more embodiments, training engine122and/or execution engine124use transformer200in a variety of applications related to topology-independent 3D shape modeling, reconstruction, shape deformation, and/or other operations related to the domain of shapes learned by transformer200. First, decoder206can be used to synthesize new shapes (e.g., shape216) that are not in training data214based on shape codes that are not generated from training shapes230in training data214. When a given shape code218is interpolated from or lies between or among two or more shape codes generated from training shapes230, the resulting output shape216includes identity, pose, expression, and/or other visual attributes that are “in between” those training shapes230.

Second, decoder206can be used to generate new shapes based on offsets224from an arbitrary set of positions222in canonical shape220. These positions222can include points that are not included in canonical shape positions232within training data214, positions222that are near and/or far from one another, positions222with arbitrary spatial resolution, and/or positions222in an arbitrary ordering. For example, transformer200could be trained on canonical shape positions232with a certain topology and spatial resolution. Transformer200could be used to generate new shapes with higher spatial resolutions (e.g., double or quadruple the spatial resolution associated with canonical shape positions232in training data214) without retraining on the higher spatial resolutions. Additionally, transformer200would be able to perform super-resolution or upsampling that increases surface details in a given output shape216(e.g., wrinkles in the palm of a hand or a face) when positions222used to generate shape216have a higher resolution than canonical shape positions232in training data214.

Third, transformer200can be used to “complete” a shape when regions or portions of the shape are missing or occluded. For example, canonical shape positions232and offsets228associated with a partial shape (e.g., a mesh of a face with vertices missing from the cheek, chin, upper lip, nose, and/or another region) could be inputted into encoder204to generate a corresponding shape code218. Shape code218and positions222representing the full shape (e.g., vertices representing a mesh of an entire face) could be inputted into decoder206to generate plausible offsets224and corresponding positions226for the entire shape216. In another example, transformer200could be trained to model deformations or variations on a specific shape (e.g., an actor's face). The trained encoder204could be used to convert a small number of canonical shape positions232and corresponding offsets228on the shape (e.g., dozens of landmarks detected using a landmark detection technique or 100-200 markers from a motion capture technique) into shape code218. The trained decoder206could be used to generate a reconstruction of the corresponding shape216that includes a much larger number of positions226(e.g., thousands to tens of thousands of vertices) on an output shape216. The output shape216thus includes canonical shape positions232used to generate shape code218, as well as additional positions226that are determined based on learned correlations with canonical shape positions232.

Fourth, transformer200can be trained on training shapes230obtained via high resolution input scans. The trained transformer200can be used to generate new shapes with the same level of detail. For example, a set of training shapes230could include dozens of facial scans of a single face. Each scan could include hundreds of thousands of vertices capturing detail that includes skin pores and fine wrinkling. Around one million weights in transformer200could be used to represent tens of millions of vertices in the set of training shapes230. A high-quality facial animation of the same person could be produced by interpolating between or across shape codes produced by the trained encoder204from training shapes230. To accommodate memory limitations, encoder204could encode a fixed set of tens of thousands of canonical shape positions232and corresponding offsets228for each of training shapes230. The tens of thousands of points could be randomly sampled during training to cover the entire face. The corresponding shape codes could then be inputted with tens of thousands of other randomly sampled positions222into decoder206to reconstruct the corresponding offsets224. This random sampling of positions222with the same shape code could additionally be repeated to gradually increase the resolution of the corresponding output shape216. Further, this trained transformer200could be used with positions222associated with different topologies and spatial resolutions to tailor the output shape216to different applications.

Fifth, a trained transformer200can be fit to a new output shape216by iteratively optimizing for a corresponding shape code218that minimizes a loss (e.g., losses208) between a set of points on a target shape and a corresponding set of positions226on the output shape216. For example, gradient descent could be used to iteratively update shape code218in a way that reduces an L2 loss between a previously unseen target shape and positions226on the output shape216produced by decoder206from shape code218. The resulting shape code218can then be used to initialize a subsequent optimization step that further refines the output shape216. This type of optimization can also be used to compute different shape codes for different regions of the target shape, which can further improve the quality of fit and expressibility of shape216.

Sixth, the output shape216can be varied by inputting different shape codes into different decoder transformer blocks304. For example, decoder206could include a series of four decoder transformer blocks304. Each decoder transformer block uses a corresponding shape code218to modulate a set of input tokens representing a set of positions222on canonical shape220. As a result, the output of decoder206could be varied by “mixing and matching” up to four different shape codes as input into the four corresponding decoder transformer blocks304. Each permutation of the shape codes inputted into decoder transformer blocks304would result in a different output shape216.

Seventh, transformer200can be constructed and/or trained in a way that disentangles canonical shape220deformations caused by identity changes from canonical shape220deformations caused by expression changes. In particular, shape code218can be split into an “identity” code representing an identity of a subject (e.g., a specific person) and an “expression” code representing an expression (e.g., a specific facial expression), as described in further detail below with respect toFIG.4B. During training, the identity code is constrained to be the same for all expressions of the same subject, while the expression code is varied for each individual expression. The identity and expression codes can additionally be modulated separately to produce corresponding variations in the output shape216. For example, the identity associated with shape216could be varied (including generating new identities not associated with training shapes230) by changing (e.g., randomly sampling, interpolating, etc.) the identity code and fixing the expression code. Conversely, the expression associated with shape216could be varied (including generating new expressions not associated with training shapes230) by changing (e.g., randomly sampling, interpolating, etc.) the expression code and fixing the identity code. Different identity and/or expression codes can also be applied to different regions of a given output shape216. For example, different identity and/or expression codes can be used with positions222on different sides or regions of a face to generate an output shape216that reflects a combination of the corresponding identities and/or expressions.

FIG.4Ais a more detailed illustration of a decoder transformer block in transformer200ofFIG.2, according to various embodiments. As shown inFIG.4A, input into the decoder transformer block includes an input token426and shape code218. Input token426can include a position token (e.g., position tokens354) outputted by position MLP310in decoder206, or input token426can be a token outputted by a previous decoder transformer block in decoder206.

A style MLP402in the decoder transformer block converts shape code218into an output code422. For example, style MLP402could include multiple fully connected layers with nonlinear activations. An affine transformation404is then applied to code422to produce a token424that is the same size as token426.

A modulation406is then applied to both token424and token426to produce a styled token428. For example, modulation406could include a pointwise multiplication that “infuses” information about the shape represented by shape code218into token426representing a position on canonical shape220. Because a separate styled token428is produced for each position token, each styled token428could be produced as a modulation of a corresponding input token426with a potentially different shape code218. This separate modulation of input tokens426with multiple shape codes allows for additional localized shape deformation, in addition to or in lieu of the techniques described above.

After styled token428is generated for each position token, the set of styled tokens representing all input positions222in canonical shape220is fed into an XCiT layer408. XCiT layer408processes each individual styled token428and exchanges information across the set of styled tokens via cross-covariance attention. The output of XCiT layer408includes a separate output token430for each styled token428. This output token430can then be inputted into the next decoder transformer block in decoder206. If output token430is produced by the last decoder transformer block in decoder206, output token430is inputted into offset MLP312to generate an offset (e.g., offsets224) from the corresponding position in canonical shape220. The set of offsets224outputted by offset MLP312from all output tokens430is then applied to the corresponding positions222in canonical shape220to produce the output shape216.

FIG.4Bis a more detailed illustration of a decoder transformer block in transformer200ofFIG.2, according to various embodiments. More specifically,FIG.4Bshows a variation on the decoder transformer block, in which shape code218is represented as a combination of an identity code412and an expression code414.

As shown inFIG.4B, a set of expression blendweights432(or another representation of a facial or another type of expression) is inputted into an expression MLP410to generate expression code414. For example, expression MLP410could include multiple fully connected layers with nonlinear activations that convert expression blendweights432into expression code414. Expression code414is concatenated with identity code412to form shape code218. Shape code218and input token426are then processed by neural network layers implementing style MLP402, affine transformation404, modulation406, and XCiT layer408in the manner described above with respect toFIG.4A.

FIG.5is a flow diagram of method steps for training a transformer, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-2, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, in step502, training engine122executes an encoder neural network that generates one or more shape codes based on one or more sets of positions in a canonical shape and one or more sets of offsets associated with one or more training shapes. For example, training engine122could obtain a representation of each training shape as a set of positions in the canonical shape and a set of ground truth offsets that are applied to the set of positions to produce the training shape. Training engine122could input the positions and ground truth offsets into one or more MLPs in the encoder neural network to produce a set of position tokens and a set of offset tokens. Training engine122could also input a shape token and a concatenation of each position token with a corresponding offset token into one or more encoder transformer blocks. Training engine122could then obtain the shape code as the output of the encoder transformer block(s).

Next, in step504, training engine122executes a decoder neural network that generates one or more sets of offsets based on the shape code(s) and the set(s) of positions in the canonical shape. For example, training engine122could input the same set of positions used by the encoder neural network to generate a shape code for a given training shape into an MLP in the decoder neural network. Training engine122could obtain a set of position tokens as output of the MLP and input the position tokens and the shape code into one or more decoder transformer blocks. Training engine122could then use another MLP in the decoder neural network to convert output tokens produced by the decoder transformer blocks into a set of offsets associated with an output shape.

In step506, training engine122updates parameters of the encoder and decoder neural networks based on a loss between the set(s) of offsets inputted into the encoder neural network and the set(s) of offsets generated by the decoder neural network from the corresponding shape code(s). For example, training engine122could calculate an L2 loss and/or another measure of error between each set of offsets inputted into the encoder neural network and a corresponding set of offsets outputted by the decoder neural network. Training engine122could then use gradient descent and backpropagation to update weights in the encoder and decoder neural networks in a way that reduces the loss.

In step508, training engine122determines whether or not training of the transformer is complete. For example, training engine122could determine that training is complete when one or more conditions are met. These condition(s) include (but are not limited to) convergence in the parameters of the encoder and decoder neural networks, the lowering of the loss to below a threshold, and/or a certain number of training steps, iterations, batches, and/or epochs. While training of the transformer is not complete, training engine122continues performing steps502,504, and506. Training engine122then ends the process of training the transformer once the condition(s) are met.

FIG.6is a flow diagram of method steps for synthesizing a shape, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-2, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, in step602, execution engine124generates a plurality of position tokens based on a plurality of positions in a canonical shape. For example, execution engine124could input each position in the canonical shape into an MLP and obtain a corresponding position token as output of the MLP.

Next, in step604, execution engine124generates a plurality of offset tokens based on one or more shape codes and the position tokens. For example, execution engine124could obtain each shape code as output of an encoder neural network, interpolate between two or more shape codes to generate each shape code, and/or randomly generate each shape code. Execution engine124could also process the position tokens and shape code(s) using a series of decoder transformer blocks to produce the offset tokens. At each decoder transformer block, execution engine124could modulate each position token and/or a corresponding output token from a previous transformer block using one of the shape codes. Execution engine124could then obtain the offset tokens as the output of the last decoder transformer block.

In step606, execution engine124converts the offset tokens into offsets associated with the positions on the canonical shape. For example, execution engine124could input each offset token into an MLP and obtain a corresponding offset as output of the MLP.

Finally, in step608, execution engine124generates a shape based on the offsets and the positions. This shape can include (but is not limited to) a face, hand, body, body part, human, animal, plant, and/or another type of deformable shape. For example, execution engine124could add each offset obtained in step606with a corresponding position on the canonical shape to obtain a different position on the shape. As a result, the shape corresponds to a “deformation” of the canonical shape that reflects a given identity, expression, posture, and/or another distinguishing visual attribute.

In sum, the disclosed techniques use a transformer-based neural network architecture to represent a domain of deformable shapes such as faces, hands, and/or bodies. A canonical shape is used as a template from which various positions can be sampled or defined, and each shape in the domain is represented as a set of offsets from a corresponding set of positions on the canonical shape. An encoder neural network is used to convert a first set of positions in the canonical shape and a corresponding set of offsets for a target shape into a shape code that represents the target shape. A decoder neural network is used to generate an output shape, given the shape code and a second set of positions in the canonical shape. In particular, the decoder network generates a new set of offsets based on tokens that represent the second set of positions and that have been modulated with the shape code. The new set of offsets is then combined with the second set of positions inputted into the decoder network to produce a set of positions in the output shape. The output shape can also be varied by changing the shape code inputted into the decoder neural network, changing the positions inputted into the decoder neural network, using different shape codes to modulate different tokens, and/or using different shape codes at different decoder transformer blocks in the decoder network.

One technical advantage of the disclosed techniques relative to the prior art is the ability to learn both global and location correlations across points on a shape. Accordingly, the disclosed techniques generate more accurate or realistic shapes than conventional approaches that focus on either global or local spatial correlations in modeling shapes. Another technical advantage of the disclosed techniques is that geometric shape detail can be represented in a topology-independent manner with arbitrary spatial resolution. Consequently, the disclosed techniques do not require hand-crafted precomputation of upsampling and downsampling operations to adapt a parametric shape model to different topologies. Further, the disclosed techniques can be used with multiple mesh topologies and resolutions, both during training and shape synthesis. These technical advantages provide one or more technological improvements over prior art approaches.

1. In some embodiments, a computer-implemented method for synthesizing a shape comprises generating a first plurality of offset tokens based on a first shape code and a first plurality of position tokens, wherein the first shape code represents a variation of a canonical shape, and wherein the first plurality of position tokens represent a first plurality of positions on the canonical shape; generating a first plurality of offsets associated with the first plurality of positions on the canonical shape based on the first plurality of offset tokens; and generating the shape based on the first plurality of offsets and the first plurality of positions.

2. The computer-implemented method of clause 1, further comprising executing an encoder neural network that generates a second shape code based on a second plurality of offset tokens associated with a training shape; and updating one or more parameters of the encoder neural network and a decoder neural network, wherein the decoder neural network generates the first plurality of offset tokens, and wherein the one or more parameters are updated based on a loss between a plurality of ground truth offsets associated with the second plurality of offset tokens and a second plurality of offsets outputted by the decoder neural network based on the second shape code.

3. The computer-implemented method of any of clauses 1-2, wherein executing the encoder neural network comprises for each offset token included in the second plurality of offset tokens, inputting a concatenation of the offset token with a corresponding position token into the encoder neural network; and inputting a shape token associated with the second shape code into the encoder neural network.

4. The computer-implemented method of any of clauses 1-3, wherein the first shape code is randomly generated, interpolated between the second shape code and one or more shape codes generated by the encoder neural network based on one or more training shapes, or selected from the one or more training shapes.

5. The computer-implemented method of any of clauses 1-4, wherein the first plurality of offset tokens are generated by one or more neural network layers.

6. The computer-implemented method of any of clauses 1-5, wherein the one or more neural network layers modulate the first plurality of offset tokens based on the first shape code.

7. The computer-implemented method of any of clauses 1-6, wherein the one or more neural network layers comprise a cross-covariance attention layer.

8. The computer-implemented method of any of clauses 1-7, further comprising generating the first shape code based on an identity code that represents an identity associated with the shape and an expression code that represents an expression associated with the shape.

9. The computer-implemented method of any of clauses 1-8, further comprising generating the first plurality of position tokens as a plurality of latent representations of the first plurality of positions on the canonical shape.

10. The computer-implemented method of any of clauses 1-9, wherein the first plurality of offset tokens are converted into the first plurality of offsets via one or more neural network layers.

11. In some embodiments, one or more non-transitory computer readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of generating a first plurality of offset tokens based on a first shape code and a first plurality of position tokens, wherein the first shape code represents a variation of a canonical shape, and wherein the first plurality of position tokens represent a first plurality of positions on the canonical shape; generating a first plurality of offsets associated with the first plurality of positions on the canonical shape based on the first plurality of offset tokens; and generating a shape based on the first plurality of offsets and the first plurality of positions.

12. The one or more non-transitory computer readable media of clause 11, wherein the instructions further cause the one or more processors to perform the steps of executing an encoder neural network that generates a second shape code based on a second plurality of offset tokens and a second plurality of position tokens associated with a training shape; and updating one or more parameters of the encoder neural network and a decoder neural network, wherein the decoder neural network generates the first plurality of offset tokens, and wherein the one or more parameters are updated based on a loss between a plurality of ground truth offsets associated with the second plurality of offset tokens and a second plurality of offsets outputted by the decoder neural network based on the second shape code.

13. The one or more non-transitory computer readable media of any of clauses 11-12, wherein the encoder neural network comprises a sequence of transformer blocks.

14. The one or more non-transitory computer readable media of any of clauses 11-13, wherein the first shape code is generated by the encoder neural network based on a third plurality of offset tokens and a third plurality of position tokens associated with a first portion of the shape.

15. The one or more non-transitory computer readable media of any of clauses 11-14, wherein the second plurality of position tokens represent a second plurality of positions on the canonical shape, and wherein the second plurality of positions is different from the first plurality of positions.

16. The one or more non-transitory computer readable media of any of clauses 11-15, further comprising iteratively updating the first shape code based on a loss between the shape and a target shape.

17. The one or more non-transitory computer readable media of any of clauses 11-16, wherein generating the first plurality of offset tokens comprises modulating the first plurality of offset tokens based on the first shape code; generating a first plurality of output tokens based on the modulated first plurality of offset tokens; modulating the first plurality of output tokens based on a second shape code that is different from the first shape code; and generating the first plurality of offset tokens based on the modulated first plurality of output tokens.

18. The one or more non-transitory computer readable media of any of clauses 11-17, wherein the instructions further cause the one or more processors to perform the step of sampling the first plurality of positions from a continuous surface representing the canonical shape.

19. The one or more non-transitory computer readable media of any of clauses 11-18, wherein the canonical shape comprises at least one of a face, a hand, or a body.

20. In some embodiments, a system comprises one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to generate a first plurality of offset tokens based on a first shape code and a first plurality of position tokens, wherein the first shape code represents a variation of a canonical shape, and wherein the first plurality of position tokens represent a first plurality of positions on the canonical shape; generate a first plurality of offsets associated with the first plurality of positions on the canonical shape based on the first plurality of offset tokens; and generate a shape based on the first plurality of offsets and the first plurality of positions.

Any and all combinations of any of the claim elements recited in any of the claims and/or any elements described in this application, in any fashion, fall within the contemplated scope of the present invention and protection.

The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module,” a “system,” or a “computer.” In addition, any hardware and/or software technique, process, function, component, engine, module, or system described in the present disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.