One embodiment of the present invention sets forth a technique for generating a pose for a virtual character. The technique includes determining a set of joint representations corresponding to a set of joints in the virtual character based on (i) a base pose for the virtual character and (ii) a set of constraints associated with one or more joints included in the set of joints. The technique also includes generating, via execution of a first neural network, a set of updated joint states for the set of joints based on the set of joint representations. The technique further includes generating, based on the set of updated joint states, an output pose that includes (i) a first set of joint positions for the set of joints and (ii) a first set of joint orientations for the set of joints.

BACKGROUND

Field of the Various Embodiments

Embodiments of the present disclosure relate generally to computer vision and machine learning and, more specifically, to pose-aware neural inverse kinematics.

Description of the Related Art

Films, video games, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality (MR) systems, robotics, and/or other types of interactive environments frequently include entities (e.g., characters, robots, etc.) that are posed and/or animated in three-dimensional (3D) space. Traditional techniques for posing an entity involve manually manipulating multiple control handles corresponding to joints (or other parts) of the entity. An inverse kinematics (IK) technique can also be used to compute the positions and orientations of remaining joints (or parts) of the entity that result in the desired configuration of the manipulated joints (or parts).

However, posing and/or animating entities using conventional IK techniques is associated with a number of drawbacks. First, posing an entity via manipulation of control handles is a time-consuming, iterative, and laborious process. This manual process is repeated for a sequence of poses corresponding to movements that are used to animate the entity, which incurs additional time and resource overhead. Second, poses generated via traditional IK techniques can be unnatural or unrealistic.

Recent advancements in machine learning and deep learning have led to the development of neural IK models, which include neural networks that leverage full-body correlations learned from large datasets to compute the positions and orientations of un-manipulated joints of the body based on manipulated handles and/or other sparse control inputs. However, current neural IK models generate a “global” pose from the sparse control inputs, which can cause a pose outputted by a neural IK model to be influenced by the sparse control inputs. For example, a change in sparse control inputs that corresponds to movement in one hand of a virtual character may cause the neural IK model to generate a new pose that “wiggles” the other hand.

The global nature of conventional neural IK predictions further precludes the preservation of previous edits in posing workflows. For example, an artist may use a neural IK model to generate additional poses for a character from a base pose that was initially defined using a tool that uses conventional IK. As the artist manipulates the joints of the character, the neural IK model may change other joints in the character based on the manipulations and “destroy” portions of the base pose that were previously defined using the other tool.

As the foregoing illustrates, what is needed in the art are more effective techniques for performing neural IK.

SUMMARY

One embodiment of the present invention sets forth a technique for generating a pose for a virtual character. The technique includes determining a set of joint representations corresponding to a set of joints in the virtual character based on (i) a base pose for the virtual character and (ii) a set of constraints associated with one or more joints included in the set of joints. The technique also includes generating, via execution of a first neural network, a set of updated joint states for the set of joints based on the set of joint representations. The technique further includes generating, based on the set of updated joint states, an output pose that includes (i) a first set of joint positions for the set of joints and (ii) a first set of joint orientations for the set of joints.

One technical advantage of the disclosed techniques relative to the prior art is the ability to generate new poses that satisfy sparse constraints while selectively preserving certain aspects of a base pose. The disclosed techniques thus avoid issues associated with global pose changes in conventional neural IK models, where manipulating one part of a skeletal structure can result in changes to un-manipulated parts of the skeletal structure. Another technical advantage of the disclosed techniques is the preservation of previous edits in computer-based posing workflows. Consequently, the disclosed techniques allow artists and/or other users to iteratively refine poses without losing previous work, which can reduce latency and/or improve efficiency associated with the use of computer-based posing workflows to pose and/or animate virtual characters. An additional technical advantage of the disclosed techniques is the ability to learn both local and global representations of a pose via the use of a graph neural network with a cross-layer attention mechanism that performs message passing across multiple resolutions associated with the skeletal structure. As a result, output poses generated via the disclosed techniques can be more natural and realistic than poses produced using traditional IK techniques. These technical advantages provide one or more technological improvements over prior art approaches.

DETAILED DESCRIPTION

System Overview

FIG.1illustrates a computing device100configured to implement one or more aspects of various embodiments. In one embodiment, computing device100includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. Computing device100is configured to run a training engine122and an execution engine124that reside in a memory116.

It is noted that the computing device described herein is illustrative and that any other technically feasible configurations fall within the scope of the present disclosure. For example, multiple instances of training engine122and execution engine124could execute on a set of nodes in a distributed system to implement the functionality of computing device100.

In one embodiment, computing device100includes, without limitation, an interconnect (bus)112that connects one or more processors102, an input/output (I/O) device interface104coupled to one or more input/output (I/O) devices108, memory116, a storage114, and a network interface106. Processor(s)102may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (Al) accelerator, any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU. In general, processor(s)102may be any technically feasible hardware unit capable of processing data and/or executing software applications. Further, in the context of this disclosure, the computing elements shown in computing device100may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executing within a computing cloud.

I/O devices108include devices capable of providing input, such as a keyboard, a mouse, a touch-sensitive screen, and so forth, as well as devices capable of providing output, such as a display device. Additionally, I/O devices108may include devices capable of both receiving input and providing output, such as a touchscreen, a universal serial bus (USB) port, and so forth. I/O devices108may be configured to receive various types of input from an end-user (e.g., a designer) of computing device100, and to also provide various types of output to the end-user of computing device100, such as displayed digital images or digital videos or text. In some embodiments, one or more of I/O devices108are configured to couple computing device100to a network110.

Network110is any technically feasible type of communications network that allows data to be exchanged between computing device100and external entities or devices, such as a web server or another networked computing device. For example, network110may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and/or the Internet, among others.

Storage114includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid state storage devices. Training engine122and execution engine124may be stored in storage114and loaded into memory116when executed.

Memory116includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof. Processor(s)102, I/O device interface104, and network interface106are configured to read data from and write data to memory116. Memory116includes various software programs that can be executed by processor(s)102and application data associated with said software programs, including training engine122and execution engine124.

In some embodiments, training engine122and execution engine124operate to train and execute a machine learning model to perform pose-aware neural inverse kinematics (IK). More specifically, training engine122trains the machine learning model using a pose preservation loss that is computed using an output pose generated by the machine learning model, a corresponding base pose inputted into the machine learning model, and/or input control parameters that specify the degree to which positions, orientations, and/or other aspects of the base pose are to be preserved. Execution engine124executes the trained machine learning model to generate new poses that satisfy sparse constraints (e.g., positions and/or orientations of a subset of joints) while selectively preserving certain aspects of an inputted base pose. Training engine122and execution engine124are described in further detail below.

FIG.2is a more detailed illustration of training engine122and execution engine124ofFIG.1, according to various embodiments. As mentioned above, training engine122and execution engine124operate to train and execute a machine learning model208to perform neural IK for a virtual character. For example, training engine122and execution engine124may use machine learning model208to generate an output pose222for a human, animal, robot, and/or another type of articulated object corresponding to the virtual character. This output poses222may include a set of two-dimensional (2D) and/or three-dimensional (3D) joint positions, joint rotations, and/or other representations of joints in the articulated object.

As shown inFIG.2, input into machine leaning model208includes a base pose210, a set of constraints212, and a set of control parameters214. Base pose210includes “default” positions, orientations, and/or other attributes of some or all joints in the virtual character. For example, base pose210may include a previously defined pose for the virtual character, as specified by an artist, a posing tool, a neural IK model, a motion capture dataset, and/or a frame in an animation that includes the virtual character.

Constraints212include changes to and/or deviations from base pose210for one or more joints of the virtual character. For example, constraints212may include positions, orientations, look-at constraints, and/or other types of attributes that are specified via user manipulation of control handles for the joint(s) of the virtual character and/or other user-interface elements.

Control parameters214include values that are used to control the generation of output pose222from base pose210and constraints212. For example, control parameters214may include an orientation preservation parameter in the range of [0,1] that specifies the extent to which the orientations of joints in base pose210should be preserved. Control parameters214may also, or instead, include a position preservation parameter in the range of [0,1] that specifies the extent to which the positions of joints in base pose210should be preserved. Values of the orientation preservation parameter and position preservation parameter may be specified for individual joints, sets of joints (e.g., limbs, body segments, upper body, lower body, etc.), all joints in the virtual character, and/or other groupings of one or more joints in the virtual character.

Given base pose210, constraints212, and control parameters214, machine learning model208generates a new pose218that includes updated positions, orientations, and/or other attributes of some or all joints in the virtual character. Pose218can include attributes of joints that are derived from base pose210and/or constraints212, where the influence of base pose210and/or constraints212on one or more attributes of a given joint is determined based on one or more corresponding control parameters214. Continuing with the above example, a higher value for the orientation preservation parameter may cause the orientations of a corresponding grouping of joints in base pose210to exert a greater influence on the orientations of the same joints within the new pose218. Similarly, a higher value for the position preservation parameter may cause the positions of a corresponding grouping of joints in base pose210to exert a greater influence on the positions of the same joints within the new pose218. In both instances, a greater influence of base pose210on the new pose218may cause one or more joints in the new pose218to deviate from the corresponding constraints212. Conversely, a lower value for a given preservation parameter may cause a corresponding attribute in base pose210to exert less influence on the new pose218and maintain constraints212in corresponding joints within the new pose218.

To generate pose218, machine learning model208converts base pose210, constraints212, and control parameters214into a set of joint vectors216. Machine learning model208uses a set of neural network blocks and/or other components to convert joint vectors216into multiple sets of joint states226for joints in the virtual character. Machine learning model208then converts a final set of joint states226into positions and orientations of the joints within the new pose218. The operation of machine learning model208is described in further detail below with respect toFIG.3.

FIG.3illustrates an example architecture for machine learning model208ofFIG.2, according to various embodiments. As shown inFIG.3, machine learning model208includes a set of encoders302and304, a skeletal transformer306, and a set of decoders308and310. Each of these components is described in further detail below.

Input into encoders302and304includes base pose210, constraints212, and/or control parameters214. For example, base pose210may be represented by a Nj×(3+6) matrix, where each row of the matrix corresponds to one of Njjoints in a virtual character, the position of each joint is specified in three dimensions, and the orientation of each joint is specified in six dimensions. Constraints212may include positional constraints212that specify positions of certain joints, orientation constraints212that specify orientations of certain joints with respect to other joints, and/or look-at constraints212that specify orientations of certain joints with respect to an external target. Positional constraints212may be represented by a Np×3 matrix and a Np×1 vector, where each row of the matrix and each element of the vector corresponds to one of Npjoints with positional constraints212, each row of the matrix specifies a position in three dimensions, and each element of the vector specifies a joint identifier for the joint to which the position applies. Orientation constraints212may be represented by a Nr×6 matrix and a Nr×1 vector, where each row of the matrix and each element of the vector corresponds to one of Nrjoints with orientation constraints212, each row of the matrix specifies an orientation in six dimensions, and each element of the vector specifies a joint identifier for the joint to which the orientation applies. Look-at constraints212may be represented by a Nla×6 matrix and a Nla×1 vector, where each row of the matrix and each element of the vector corresponds to one of Nlajoints with look-at constraints212, each row of the matrix specifies a look-at orientation in six dimensions, and each element of the vector specifies a joint identifier for the joint to which the look-at orientation applies.

Given this input, encoders302and304generate a set of state vectors322and a set of embedding vectors324included in joint vectors216. More specifically, encoder302generates a set of state vectors322that represent positions, orientations, and/or other attributes of joints in the virtual character based on base pose210and constraints212. For example, encoder302may include a fully connected neural network with one hidden layer and/or another type of machine learning architecture. Encoder302may generate, from attributes of the ith joint as specified in base pose210and/or constraints212, a state vector Jstateithat encodes the position and orientation of the joint.

Encoder304generates a set of embedding vectors324that represent identities, constraints212, and/or control parameters214associated with joints in the virtual character. For example, encoder302may include a fully connected neural network with one hidden layer and/or another type of machine learning architecture that generates an embedding vector Jembifor the ith joint in the virtual character. Encoder302may initially generate an intermediate embeddingof a one-hot encoded joint identifier for the joint:

In the above equation, i is a numeric joint identifier for a corresponding joint (e.g., hip, head, right wrist, etc.), and Wembis a learned linear transformation.

Continuing with the above example, additional values related to constraints212and control parameters214may be added to each intermediate embedding to generate a corresponding embedding vector. These additional values may include cox, which is a real value in the range of [0,1] that corresponds to the orientation preservation parameter; cIKi, which is a real value in the range of [0,1] that corresponds to the pose preservation parameter; and cICi, which is a binary value that is set to 1 if the joint has a position, orientational, and/or look-at constraint and to 0 otherwise.

More specifically, the set of embedding vectors324Jembmay be generated by appending additional values related to constraints212and control parameters214to the set of intermediate vectors:

In the above equation, ⊕ is the concatenation operation, and C*=[c*i]i.

Next, skeletal transformer306uses state vectors322and embedding vectors324to iteratively update joint states226for the joints. As shown inFIG.3, skeletal transformer306uses a series of blocks314(1)-314(X) (each of which is referred to individually herein as block314) to exchange information among neighboring joints in the virtual character. Blocks314(1)-314(X) are additionally used to process information associated with three different graphs316(1),316(2), and316(3) (each of which is referred to individually herein as graph316). Each graph316represents a different resolution associated with the skeletal structure of the virtual character. For example, graph316(1) may represent a joint-level skeleton with one node per joint. Graph316(2) may represent a limb-level skeleton that pools joints from graph316(1) into nodes for the hip, spine, and each of the four limbs. Graph316(3) may represent a body-level skeleton that further reduces nodes from graph316(2) into one node each for the upper and lower body.

In one or more embodiments, skeletal transformer306includes a graph transformer neural network that uses attention mechanisms in blocks314and a number of message passing steps to exchange information among neighboring joints in each graph316. Within blocks314, embedding vectors324are used as keys K and queries Q, and state vectors322are used as values V. Attention scores outputted by blocks314are additionally assigned based on control parameters214in embedding vectors324. The output of skeletal transformer306includes a set of state vectors326representing final joint states226of the joints in the virtual character.

FIG.4is a more detailed illustration of block314in skeletal transformer306ofFIG.3, according to various embodiments. As shown inFIG.4, block314includes a skeletal multi-head attention402layer that splits matrices for the queries, keys, and values into multiple sub-matrices. Each sub-matrix of a given matrix is passed through a different attention head to compute an attention score, and multiple attention scores produced by the attention heads in skeletal multi-head attention402are combined into a single attention score. The output of skeletal multi-head attention402for a given joint is then calculated as a sum of values for neighboring joints in a given graph316that are weighted by the corresponding attention scores.

Block314also includes a fully connected network404and a layer normalization406that further process the output of skeletal multi-head attention402for a given joint. Block314further includes a residual link408that adds the input into block314to the output of layer normalization406to produce an updated joint state for the joint.

In some embodiments, the operation of block314is represented by the following:

In the above equations, Jstatetrepresents the set of joint states226at the tth message passing step, MHA denotes skeletal multi-head attention402, anddenotes the output of multi-head attention402. Additionally, FCN denotes fully connected network404, Layer-Norm represents layer normalization406,is the inverse of CIC, and ⊗ is the Hadamard product.

Returning to the discussion ofFIG.3, skeletal transformer306uses multiple layers of graphs316representing different resolutions associated with the skeletal structure of the virtual character to facilitate the propagation of information across the joints of the virtual character. In particular, skeletal transformer306uses a first block314(1) to perform a first set of message passing steps that exchange information among nodes in graph316(1). After the first set of message passing steps is complete, skeletal transformer306uses the output of the first set of message passing steps and the same block314(1) to perform a second set of message passing steps that exchange information among nodes in graph316(2). After the second set of message passing steps is complete, skeletal transformer306uses the output of the second set of message passing steps and the same block314(1) to perform a third set of message passing steps that exchange information among nodes in graph316(3). Skeletal transformer306additionally uses the output of the third set of message passing steps and block314(1) to perform a fourth set of message passing steps that exchange information among nodes in graph316(2). Skeletal transformer306then uses the output of the fourth set of message passing steps and block314(1) to perform a fifth set of message passing steps that exchange information among nodes in graph316(1). Skeletal transformer306then repeats the process with additional blocks314until state vectors326representing final joint states are outputted by block314(X).

In some embodiments, blocks314and graphs316reduce the number of message passing steps performed to converge on a given pose218. For example, skeletal transformer306may perform six message passing steps to exchange information among nodes in graph316(1), four message passing steps to exchange information among nodes in graph316(2), and two message passing steps to exchange information among nodes in graph316(3) instead of a much larger number of message passing steps to exchange information among nodes in a single high-resolution graph (e.g., graph316(1)).

Skeletal transformer306may additionally use various pooling and/or un-pooling functions to mix information between graphs316associated with different resolutions. For example, skeletal transformer306may use masked inter-level Multi-Head Attention blocks314to propagate joint states226associated with nodes from a given graph316to nodes in a different graph316. The mask associated with these blocks314may be designed so that a given node can attend only to itself and corresponding nodes from a different resolution (e.g., one or more nodes in a lower resolution with which the given node is associated, a set of nodes in a higher resolution that are pooled into the given node, etc.). These blocks314additionally allow skeletal transformer306to dynamically assign weights to information from nodes in different layers.

At the beginning of the message passing process, only constrained joints hold information that should be propagated throughout the skeletal structure, as information related to the base pose is already present in the corresponding joint vectors216. Consequently, skeletal transformer306can operate using a node-level mask Mtlthat indicates which nodes hold new information in layer l after block t. At the start of the message passing process, Mt=0jointis the same as CIC, and the limb-level and body-level masks are defined using the following:

In other words, a given node in a lower-resolution graph316is determined to hold information that should be propagated if the given node is associated with another node in a higher-resolution graph316that holds new information.

At the end of every block314, the mask for layer l∈{joint, limb, body} is updated using the following:

In the above equation, Alis the adjacency matrix for nodes in graph316of layer l. Each entry in the mask includes an upper bound of 1 that represents full neighbor influence and prevents message passing from increasing for nodes with degree greater than 1.

State vectors326outputted by skeletal transformer306are processed by a set of decoders308and310. More specifically, decoder308converts state vectors326into positions342of the corresponding joints in pose218, and decoder310converts state vectors326into orientations344of the corresponding joints in pose218. Like encoders302and304, decoders308and310may include fully connected networks with one hidden layer and/or other machine learning architectures.

Returning to the discussion ofFIG.2, training engine122trains machine learning model208using training data204that includes a set of training base poses244and a set of training constraints246. Training base poses244include various poses associated with the virtual character. For example, training base poses244may include poses that are generated using a motion capture technique. These poses may depict a person, animal, robot, and/or another type of articulated object walking, jogging, running, turning, spinning, dancing, strafing, waving, climbing, descending, crouching, hopping, jumping, dodging, skipping, interacting with an object, lying down, sitting, stretching, and/or engaging in another type of action, a combination of actions, and/or a sequence of actions. These poses may be retargeted to a skeleton for the virtual character that includes a certain set of joints. Training base poses244may also, or instead, include poses that are generated and/or edited by artists, animators, and/or other users. Training base poses244may also, or instead, be generated synthetically using computer vision, computer graphics, animation, machine learning, and/or other techniques.

As with constraints212associated with a given base pose210, training constraints246include modifications to and/or deviations from training base poses244. For example, training constraints246may include changes to positions, orientations, look-at constraints, and/or other types of attributes of joints in training base poses244. Training constraints246may be user-specified, randomly generated (e.g., as deviations from one or more attributes in training base poses244that are sampled from one or more corresponding distributions), and/or otherwise determined. Training constraints246may also, or instead, be randomly matched to training base poses244, defined for specific base poses244, and/or otherwise paired with training base poses244.

To generate a training sample in training data204, training engine122may apply a given set of training constraints246to a corresponding training base pose. For example, training engine122may overwrite positions, orientations, and/or other attributes of one or more joints in a given training base pose with corresponding positions, orientations, and/or other attributes specified in a set of training constraints246paired with the training base pose.

A data-generation component202in training engine122determines training control parameters248associated with a given training sample (e.g., a training base pose and a corresponding set of training constraints246). For example, data-generation component202may sample values of a set of orientation preservation parameters and a set of pose preservation parameters from uniform distributions of ranges of values for the parameters.

Data-generation component202also generates a set of training joint vectors250from a training sample that includes a set of training control parameters248, a set of training constraints246, and a training base pose. For example, data-generation component202may use the techniques described above with respect toFIG.3to convert the set of training control parameters248, set of training constraints246, and training base pose into training joint vectors250that include per-joint state vectors322and embedding vectors324.

An update component206in training engine122trains machine learning model208using training joint vectors250generated by data-generation component202from the corresponding training control parameters248, training base poses244, and training constraints246. More specifically, update component206inputs each set of training joint vectors250into machine learning model208. Update component206also executes machine learning model208to produce corresponding training output222that represents a predicted pose for the virtual character. Update component206computes one or more losses224using training output222and training control parameters248, training base pose, and training constraints246used to generate that set of training joint vectors250. Update component206then uses a training technique (e.g., gradient descent and backpropagation) to update model parameters220of machine learning model208in a way that reduces losses224. Update component206repeats the process with additional training joint vectors250and training output222until model parameters220converge, losses224fall below a threshold, and/or another condition indicating that training of machine learning model208is complete is met.

In some embodiments, losses224include a pose preservation loss that is computed between a given training base pose and a corresponding set of training output222generated by machine learning model208from that training base pose. For example, the pose preservation loss may include the following representation:

In the above equation,denotes the pose preservation loss, y and R denote joint positions and orientations in the training base pose, respectively, and y′ and R′ denote joint positions and orientations in training output222, respectively. The pose preservation loss is computed by summing a first term that corresponds to anloss between y and y′ and a second term that corresponds to an orientation geodesic loss between R and R′. The termMt=0jointcorresponds to the inverse of the binary mask introduced in Equation 5 and is used to avoid penalizing joints associated with training constraints246. The orientation preservation parameters CFKand pose preservation parameters CIKare used to weight the pose preservation loss according to the amount of the training base pose to be preserved in training output222.

In one or more embodiments, the pose preservation loss is combined with additional losses into an overall loss

In the above equation,=∥ŷ−y′∥22corresponds to anloss computed between joint positions y′ in training output222and joint positions ŷ for the same joints as specified in training constraints246, and=arc cos[(tr(R′T{circumflex over (R)})−1)/2] corresponds to a geodesic loss that is computed between joint orientations R′ in training output222and joint orientations {circumflex over (R)}for the same joints as specified in training constraints246.=arc cos[·Ĝj13dj] corresponds to a look-at loss geodesic loss that is computed using a unit-length vectorpointing at the external target in world space, a direction vector djfor a joint, a predicted global transform matrix Ĝj, and a global predicted look-at direction represented by Ĝj13dj, where Ĝj13=Ĝj[1:3,1:3].

After training of machine learning model208is complete, execution engine124uses the trained machine learning model208to generate new poses for the virtual character, where each new pose218is derived from a corresponding base pose210, set of constraints212, and set of control parameters214. For example, execution engine124may use a set of encoders in machine learning model208to convert a given base pose210, set of constraints212, and set of control parameters214into a corresponding set of joint vectors216. Execution engine124may use a graph neural network and/or attention mechanisms in machine learning model208to iteratively update a set of joint states226for joints in the virtual character based on joint vectors216and a hierarchy of resolutions associated with a skeletal structure for the virtual character. Execution engine124may then use a set of decoders in machine learning model208to convert a final set of joint states226into a corresponding new pose218. As discussed above, pose218may preserve and/or combine attributes of joints from base pose210and constraints212based on control parameters214that represent the level of influence base pose210and/or constraints212should have on those attributes.

After a new pose218is generated by machine learning model208, execution engine124uses forward kinematics230to convert pose218into a final output pose232that enforces predefined bone lengths for the virtual character. For example, execution engine124may apply forward kinematics230to pose218as a sequence of rigid transformations that use per-joint offset vectors to update the positions and/or orientations of joints in pose218based on positions and orientations of the joints in a resting pose for the virtual character. Each offset vector may represent a bone length constraint for the corresponding joint and specify a displacement of the joint with respect to a parent joint when the rotation of the joint is zero.

After output pose232is generated, execution engine124may generate a representation of the virtual character in output pose232. For example, execution engine124may output a skeleton, rendering, and/or another visual representation of the virtual character in output pose232. Execution engine124may also, or instead, incorporate output pose232into one or more frames of an animation of the virtual character.

In one or more embodiments, values of base pose210, constraints212, and control parameters214are iteratively updated within a workflow for posing and/or animating the virtual character. For example, an artist, animator, and/or another user may import, into the workflow, a “default” pose, manually generated pose, motion capture data, and/or another previously defined pose as an initial base pose210for the virtual character. The user may also use control handles and/or other user-interface elements to specify constraints212on the positions, orientations, and/or other attributes of one or more joints in the virtual character. The user may further specify control parameters214that indicate that joint positions and/or orientations in base pose210should not be preserved (e.g., due to a lack of relationship between base pose210and a target pose to be attained). The user may then trigger the execution of machine learning model208within the workflow to generate a corresponding output pose232that quickly incorporates changes represented by constraints212. The user may repeat the process with the generated output pose232as a new base pose210for the virtual character and updated constraints212that represent changes to be made to the new base pose210. As the generated output pose232is iteratively refined, the user may update control parameters214and/or remove constraints212to reduce the deviation of each output pose232from a corresponding base pose210. The user may additionally associate a given output pose232with a frame of an animation that includes the virtual character and use that output pose232as a starting base pose210for the next frame of the animation.

In one or more embodiments, poses outputted by machine learning model208are used to generate animations, virtual characters, and/or other content in an immersive environment, such as (but not limited to) a VR, AR, and/or MR environment. This content can depict virtual worlds that can be experienced by any number of users synchronously and persistently, while providing continuity of data such as (but not limited to) personal identity, user history, entitlements, possession, and/or payments. It is noted that this content can include a hybrid of traditional audiovisual content and fully immersive VR, AR, and/or MR experiences, such as interactive video.

FIG.5is a flow diagram of method steps for generating a pose for a virtual character, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-2, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, in step502, training engine122and/or execution engine124determine a base pose, set of constraints, and set of control parameters associated with a virtual character. For example, training engine122and/or execution engine124may receive the base pose as a predefined pose for the virtual character. Training engine122and/or execution engine124may also obtain constraints related to the positions, orientations, and/or look-ats of one or more joints in the virtual character. Training engine122and/or execution engine124may additionally select and/or receive the control parameters as values that indicate the extent to which the position and/or orientation of each joint in base pose should be preserved.

In step504, training engine122and/or execution engine124convert the base pose, constraints, and control parameters into joint representations of a set of joints in the virtual character. For example, training engine122and/or execution engine124may use one or more encoder neural networks generate, for each joint in the virtual character, a joint embedding that encodes a joint identifier for the joint, control parameters associated with the joint, and a binary value indicating whether or not the joint is constrained. Training engine122and/or execution engine124may also use the encoder neural network(s) to generate, for each joint in the virtual character, an initial joint state that encodes the position, orientation, and/or other attributes of the joint from the base pose and/or constraints.

In step506, training engine122and/or execution engine124iteratively update a set of joint states for the joints based on the joint representations. For example, training engine122and/or execution engine124may use a graph transformer neural network to perform message passing among the joints and/or between the joints and one or more lower-resolution representations of the skeletal structure of the virtual character. Each message passing step may involve using a block in the graph transformer neural network to update the joint states based on attention scores and/or joint states from a previous message passing step.

In step508, training engine122and/or execution engine124convert the updated joint states into an output pose. Continuing with the above example, training engine122and/or execution engine124may use one or more decoder neural networks to decode final joint states outputted by the graph transformer neural network into positions and orientations of the joints. Training engine122and/or execution engine124may additionally perform a forward kinematics step that updates the positions and/or orientations of the joints in a way that enforces bone lengths in the virtual character.

In step510, training engine122and/or execution engine124determine whether or not to train a machine learning model using the output pose. For example, training engine122and/or execution engine124may determine that the encoder, graph transformer, and/or decoder neural networks are to be trained using the output pose if the output pose is generated during a training process associated with the encoder, graph transformer, and/or decoder neural networks; the output pose is flagged as unnatural, unrealistic, and/or otherwise suboptimal by a user, and/or another condition associated with training of the encoder, graph transformer, and/or decoder neural networks is met.

If training engine122and/or execution engine124determine that the machine learning model is to be trained using the output pose, training engine122performs step512, in which training engine122computes a set of losses based on the output pose, base pose, and/or constraints. These losses may include a pose preservation loss associated with preservation of the base pose in the output pose. These losses may also, or instead, be used to preserve one or more constraints in the output pose.

In step514, training engine122updates parameters of the machine learning model based on the losses. For example, training engine122could use a training technique (e.g., gradient descent and backpropagation) to update neural network weights of the encoder, graph transformer, and/or decoder neural networks in a way that reduces the loss(es).

If training engine122and/or execution engine124determine in step510that the output pose should not be used to train the machine learning model, training engine122and/or execution engine124skip steps512and514and proceed to step516from step510.

In step516, training engine122and/or execution engine124determine whether or not to continue generating poses. For example, training engine122and/or execution engine124may determine that poses should continue to be generated during training of the machine learning model, during execution of a workflow for posing and/or animating the virtual character, and/or in another environment or setting in which poses for the virtual character are to be generated. If training engine122and/or execution engine124determine that poses should continue to be generated for the virtual character, training engine122and/or execution engine124repeat steps502,504,506,508,510,512, and/or514to continue generating new output poses for the virtual character and/or training the virtual character using the new output poses. Training engine122and/or execution engine124also repeat step516to determine whether or not to continue generating poses. During step516, training engine122and/or execution engine124may determine that poses should not continue to be generated once training of the machine learning model is complete, execution of the workflow for posing and/or animating the virtual character is discontinued, and/or another condition is met.

In sum, the disclosed techniques perform pose-aware neural inverse kinematics (IK) using a machine learning model that generates new poses for entities in two-dimensional (2D) and/or three-dimensional (3D) space while preserving characteristics of a base pose. The operation of the machine learning model is conditioned on both sparse constraints (e.g., positions and/or orientations of a subset of joints) and a base pose, which allows the machine learning model to generate new poses that satisfy the sparse constraints while selectively preserving various aspects of the base pose.

The machine learning model is trained using a pose preservation loss that is computed between an output pose generated by the machine learning model and a corresponding base pose. The output pose can further be adjusted via input control parameters that specify the degree to which positions, orientations, and/or other aspects of the base pose are to be preserved.

The machine learning model includes a set of encoder neural network layers that encode identities, positions, and orientations of joints in a skeletal structure for an entity. The machine learning model also includes a graph transformer neural network with a cross-layer attention mechanism that simultaneously performs message passing at multiple resolutions (e.g., joint level, limb level, body level, etc.) associated with the skeletal structure based on the encoded identities, positions, and orientations. The machine learning model further includes a set of decoder neural network layers that decode the final encodings outputted by the graph neural network into positions and orientations of the joints. A forward kinematics step is used to convert the positions and orientations outputted by the machine learning model into updated positions and orientations of the joints that are consistent with the lengths of bones in the skeletal structure.

One technical advantage of the disclosed techniques relative to the prior art is the ability to generate new poses that satisfy sparse constraints while selectively preserving certain aspects of a base pose. The disclosed techniques thus avoid issues associated with global pose changes in conventional neural IK models, where manipulating one part of a skeletal structure can result in changes to un-manipulated parts of the skeletal structure. Another technical advantage of the disclosed techniques is the preservation of previous edits in computer-based posing workflows. Consequently, the disclosed techniques allow artists and/or other users to iteratively refine poses without losing previous work, which can reduce latency and/or improve efficiency associated with the use of computer-based posing workflows to pose and/or animate virtual characters. An additional technical advantage of the disclosed techniques is the ability to learn both local and global representations of a pose via the use of a graph neural network with a cross-layer attention mechanism that performs message passing across multiple resolutions associated with the skeletal structure. As a result, output poses generated via the disclosed techniques can be more natural and realistic than poses produced using traditional IK techniques. These technical advantages provide one or more technological improvements over prior art approaches.1. In some embodiments, a computer-implemented method for generating a pose for a virtual character comprises determining a set of joint representations corresponding to a set of joints in the virtual character based on (i) a base pose for the virtual character and (ii) a set of constraints associated with one or more joints included in the set of joints; generating, via execution of a first neural network, a set of updated joint states for the set of joints based on the set of joint representations; and generating, based on the set of updated joint states, an output pose that includes (i) a first set of joint positions for the set of joints and (ii) a first set of joint orientations for the set of joints.2. The computer-implemented method of clause 1, further comprising training the first neural network using (i) a first loss that is computed between the first set of joint positions and a second set of joint positions included in the base pose and (ii) a second loss that is computed between the first set of joint orientations and a second set of joint orientations included in the base pose.3. The computer-implemented method of any of clauses 1-2, further comprising training the first neural network based on one or more additional losses associated with the set of constraints.4. The computer-implemented method of any of clauses 1-3, wherein the first loss is further computed based on a first set of control parameters associated with preservation of the second set of joint positions in the output pose and the second loss is computed based on a second set of control parameters associated with preservation of the second set of joint orientations in the output pose.5. The computer-implemented method of any of clauses 1-4, wherein determining the set of joint representations comprises generating, via execution of a second neural network, a first set of embeddings associated with a set of identities for the set of joints; determining, based on the base pose and the set of constraints, (i) a second set of joint positions for the set of joints and (ii) a second set of joint orientations for the set of joints; and converting, via execution of a third neural network, the second set of joint positions and the second set of joint orientations into a second set of embeddings for the set of joints.6. The computer-implemented method of any of clauses 1-5, wherein converting the set of joint representations into the set of updated joint states comprises generating the set of updated joint states based on the set of joint representations and a set of message-passing iterations.7. The computer-implemented method of any of clauses 1-6, wherein generating the pose comprises converting, via execution of one or more additional neural networks, the set of updated joint states into the first set of joint positions and the first set of joint orientations; and updating the first set of joint positions and the first set of joint orientations based on a rest pose for the virtual character.8. The computer-implemented method of any of clauses 1-7, wherein the set of constraints comprises at least one of a positional constraint, an orientation constraint, or a look-at constraint.9. The computer-implemented method of any of clauses 1-8, wherein the first neural network comprises a set of cross-layer attention blocks associated with a plurality of resolutions for a skeletal structure of the virtual character.10. The computer-implemented method of any of clauses 1-9, wherein the first neural network comprises a graph neural network.11. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising determining a set of joint representations corresponding to a set of joints in a virtual character based on (i) a base pose for the virtual character and (ii) a set of constraints associated with one or more joints included in the set of joints; generating, via execution of a first neural network, a set of updated joint states for the set of joints based on the set of joint representations; and generating, based on the set of updated joint states, an output pose that includes (i) a first set of joint positions for the set of joints and (ii) a first set of joint orientations for the set of joints.12. The one or more non-transitory computer-readable media of clause 11,wherein the operations further comprise training the first neural network using a first loss that is computed based on the first set of joint positions, a second set of joint positions included in the base pose, and a first set of control parameters associated with preservation of the second set of joint positions in the output pose.13. The one or more non-transitory computer-readable media of any of clauses 11-12, wherein the operations further comprise further training the first neural network using a second loss that is computed based on the first set of joint orientations, a second set of joint orientations included in the base pose, and a second set of control parameters associated with preservation of the second set of joint orientations in the output pose.14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein determining the set of joint representations comprises generating a set of joint embeddings included in the set of joint representations based on (i) a set of identities for the set of joints, (ii) a set of control parameters associated with preservation of the base pose in the output pose, and (iii) the set of constraints; and determining, based on the base pose and the set of constraints, a set of initial joint states corresponding to (i) a second set of joint positions for the set of joints and (ii) a second set of joint orientations for the set of joints.15. The one or more non-transitory computer-readable media of any of clauses 11-14, wherein converting the set of joint representations into the set of updated joint states comprises computing a set of attention scores based on the set of joint representations; and generating the set of updated joint states based on the set of attention scores.16. The one or more non-transitory computer-readable media of any of clauses 11-15, wherein the set of attention scores is further computed based on a plurality of graphs corresponding to different resolutions associated with the set of joints.17. The one or more non-transitory computer-readable media of any of clauses 11-16, wherein the set of attention scores is further computed based on a set of masks associated with the one or more joints.18. The one or more non-transitory computer-readable media of any of clauses 11-17, wherein generating the output pose comprises converting, via execution of one or more additional neural networks, the set of updated joint states into the first set of joint positions and the first set of joint orientations; and updating the first set of joint positions and the first set of joint orientations via a forward kinematics technique.19. The one or more non-transitory computer-readable media of any of clauses 11-18, wherein the first neural network comprises a graph transformer neural network.20. In some embodiments, a system comprises one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform operations comprising determining a set of joint representations corresponding to a set of joints in a virtual character based on (i) a base pose for the virtual character and (ii) a set of constraints associated with one or more joints included in the set of joints; generating, via execution of a first neural network, a set of updated joint states for the set of joints based on the set of joint representations; and generating, based on the set of updated joint states, an output pose that includes (i) a first set of joint positions for the set of joints and (ii) a first set of joint orientations for the set of joints.