ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematics

1 INTRODUCTION . Modeling human pose and learning pose representations have received increasing attention recently due to their prominence in applications , such as computer graphics and animation ( Harvey et al. , 2020 ; Xu et al. , 2020 ) ; immersive augmented reality ( Facebook Reality Labs , 2021 ; Capece et al. , 2018 ; Lin , 2019 ; Yang et al. , 2021 ) ; entertainment ( McDonald , 2018 ; Xpire , 2019 ) ; sports and wellness ( Rosenhahn et al. , 2008 ; Kim et al. , 2021 ) as well as human machine interaction ( Heindl et al. , 2019 ; Casillas-Perez et al. , 2016 ; Schwarz et al. , 2014 ) and autonomous driving ( Kumar et al. , 2021 ) . In the gaming industry , state-of-the-art real-time pose manipulation tools , such as CCD ( Kenwright , 2012 ) , FABRIK ( Aristidou & Lasenby , 2011 ) or FinalIK ( RootMotion , 2020 ) , are popular for rapid execution and rely on forward and inverse kinematics models defined via non-learnable kinematic equations . Inverse kinematics ( IK ) is the process of computing the internal geometric parameters of a kinematic system resulting in the desired configuration ( e.g . global positions ) of system ’ s joints ( Paul , 1992 ) . Forward kinematics ( FK ) refers to the use of the kinematic equations to compute the positions of joints from specified values of internal geometric parameters . While being mathematically accurate , non-learnable IK models do not attempt reconstructing plausible human poses from underconstrained solutions derived from sparse constraints ( e.g . positions of a small subset of joints ) . In this paper we develop a neural modeling approach to reconstruct full human pose from a sparse set of constraints supplied by a user , in the context of pose authoring and game development . We bridge the gap between skeleton-aware human pose representation based on IK/FK ideas and the neural embedding of human pose . Our approach effectively implements a learnable model for skeleton IK , mapping desired joint configuration into predictions of skeleton internal parameters ( local rotations ) , learning the statistics of natural poses using datasets derived from high-quality motion capture ( MOCAP ) sequences . The approach , which we call ProtoRes , models the semantics of joints and their interactions using a novel prototypical residual neural network architecture . Inspired by prototypical networks , which showed that one semantic class can be represented by the prototype ( mean ) of a few examples ( Snell et al. , 2017 ) , we extend it using a multi-block residual approach : the final pose embedding is a mean across embeddings of sparse constraints and across partial pose predictions produced in each block . We show that in terms of the pose reconstruction accuracy , ProtoRes outperforms existing gaming industry tools such as FinalIK , as well as out-of-the-box machine-learning solution based on Transformer ( Vaswani et al. , 2017 ) , which also happens to be 10 times less effective in terms of training speed than the proposed architecture . ∗All authors are with Unity Labs ; correspondence to : B.N . Oreshkin , boris.oreshkin @ gmail.com Finally , we develop user-facing tools that integrate learned ProtoRes pose representation in the Unity game engine ( see Fig . 1 ) . This provides convincing qualitative examples of solutions to the problem of the AI assisted human pose authoring . At the qualitative level , getting traditional workflows to behave the way ProtoRes does would require one to use many techniques in tandem , including IK , FK , layered animation pose libraries , along with procedural rigs encoding explicit heuristics . The process would be highly labor-intensive even for an experienced user while the results would still be of variable fidelity depending on their skill . This is because traditional rigs have no bias towards realistic poses and only allow exploring a limited linear latent space defined by uniform interpolation of a heuristic constraint system . ProtoRes forms a foundation that allows any junior or indie/studio user to bypass these existing complexities and create entirely new workflows for meaningfully exploring learned latent space using a familiar yet far more powerful way . We believe that our model and tools will help speed up the animation process , simplify and democratize game development . 1.1 BACKGROUND Figure 1 : ProtoRes completes full human pose using an arbitrary combination of 3D coordinates , look-at targets and world-space rotations specified by a user . The animation is best viewed in Adobe Reader . We consider the full-body pose authoring animation task depicted in Fig . 1 . The animator provides a few inputs , which we call effectors , that the target pose has to respect . For example , the look-at effector specifies that the head should be facing a specific direction , the positional effectors pin some joints to specific locations in global space and rotational effectors constrain the world-space rotations of certain joints . We assume that the animator can generate arbitrary number of such effectors placed on any skeletal joint ( one joint can be driven by more than one effector ) . The task of the model is to combine all the information provided via effectors and gen- erate a plausible full-body pose respecting provided effector constraints . We define the full-body pose as the set of all kinematic parameters necessary to recreate the appearance of the body in 3D . We assume that each effector can be represented in the space Rdeff , where deff is taken to be maximum over all effector types . Suppose we have 3D position and 6D rotation effectors : deff is 6 . In position effectors , 3 extra values are 0 . We formulate the pose authoring problem as learning the mapping ϒθ : RN×deff → Rdkin with learnable parameters θ ∈Θ . ϒθ maps the input space RN×deff of variable dimensionality N ( the number of effectors is not known apriori ) to the space Rdkin , containing all kinematic parameters to reconstruct full-body pose . A body with J joints can be fully defined using a tree model with 6D local rotation per joint and 3D coordinate for the root joint , in which case dkin = 6J+3 , assuming fixed bone lengths . Given a dataset D= { xi , yi } Mi=1 of poses containing pairs of inputs xi ∈ RN×deff and outputs yi ∈ Rdkin , ϒθ can be learned by minimizing empirical risk : ϒθ = argmin θ∈Θ 1 M ∑xi , yi∈D L ( ϒθ ( xi ) , yi ) ( 1 ) 1.2 RELATED WORK . Joint representations . Representing pose via 3D joint coordinates ( Cheng et al. , 2021 ; Cai et al. , 2019 ; Khapugin & Grishanin , 2019 ) is sub-optimal as it does not enforce fixed-length bones , nor specifies joint rotations . Predicting joint rotations automatically satisfies bone length constraints and adequately models rotations ( Pavllo et al. , 2018 ) , which is crucial in downstream applications , such as deforming a 3D mesh on top of the skeleton , to avoid unrealistic twisting . This is viable via skeleton representations based on Euler angles ( Han et al. , 2017 ) , rotation matrices ( Zhang et al. , 2018 ) and quaternions ( Pavllo et al. , 2018 ) . In this work , we use the two-row 6D rotation matrix representation that addresses the continuity issues reminiscent of the other representations ( Zhou et al. , 2019 ) . Pose modeling architectures . Multi-Layer Perceptrons ( MLPs ) ( Cho & Chen , 2014 ; Khapugin & Grishanin , 2019 ; Mirzaei et al. , 2020 ) and kernel methods ( Grochow et al. , 2004 ; Holden et al. , 2015 ) have been used to learn single pose representations . Beyond single pose , skeleton moving through time can be modeled as a spatio-temporal graph ( Jain et al. , 2016 ) or as a graph convolution ( Yan et al. , 2018 ; Mirzaei et al. , 2020 ) . A common limitation of these approaches is their reliance on a fixed set of inputs , whereas our architecture is specifically designed to handle sparse variable inputs . Pose prediction from sparse constraints . Real-time methods based on nearest-neighbor search , local dynamics and motion matching have been used on sparse marker position and accelerometer data ( Tautges et al. , 2011 ; Riaz et al. , 2015 ; Chai & Hodgins , 2005 ; Büttner & Clavet , 2015 ) . MLPs and RNNs have been used for real-time processing of sparse signals such as accelerometers ( Huang et al. , 2018 ; Holden et al. , 2017 ; Starke et al. , 2020 ; Lee et al. , 2018 ) and VR constraints ( Lin , 2019 ; Yang et al. , 2021 ) . These approaches rely on the past pose information to disambiguate next frame prediction and as such are not applicable to our problem , in which only current pose constraints are available . Iterative IK algorithms such as FinalIK ( RootMotion , 2020 ) have been popular in real-time applications . FinalIK works by setting up multiple IK chains for each limb of the body of a predefined human skeleton and for a fixed set of effectors . Several iterations are executed to solve each of these chains using a conventional bone chain IK method , e.g . CCD . In FinalIK , the end effector ( hands and feet ) can be positioned and rotated , while mid-effectors ( shoulders and thighs ) can only be positioned . Effectors can have a widespread effect on the body via a hand-crafted pulling mechanism that gives a different weight to each chain . This and similar tools suffer from limited realism when used for human full-body IK , as they are not data-driven . Learning-based methods strive to alleviate this by providing learned model of human pose . Grochow et al . ( 2004 ) proposed a kernel based method for learning a pose latent space in order to produce the most likely pose satisfying sparse effector constraints via online constrained optimization . The more recent commercial tool Cascadeur uses a cascade of several MLPs ( each dealing with fixed set of positional effectors : 6 , 16 , 28 ) to progressively produce all joint positions without respecting bone constraints ( Khapugin & Grishanin , 2019 ; Cascadeur , 2019 ) . Unlike our approach , Cascadeur can not handle arbitrary effector combinations , rotation or look-at constraints and requires post processing to respect bone constraints . Permutation invariant architectures . Models for encoding unstructured variable inputs have been proposed in various contexts . Attention models ( Bahdanau et al. , 2015 ) and Transformer ( Vaswani et al. , 2017 ) have been proposed in the context of natural language processing . Prototypical networks ( Snell et al. , 2017 ) used average pooled embedding to encode semantic classes via a few support images in the context of few-shot image classification . Maxpool representations over variable input dimension were proposed by Qi et al . ( 2017 ) as PointNet and Zaheer et al . ( 2017 ) as DeepSets for segmentation and classification of 3D point clouds , image tagging , set anomaly detection and text concept retrieval . Niemeyer et al . ( 2019 ) further generalized the PointNet by chaining the basic maxpool/concat PointNet blocks resulting in ResPointNet architecture . 1.3 SUMMARY OF CONTRIBUTIONS . The contributions of our paper can be summarized as follows . • We define the 3D character posing task and publicly release two associated benchmarks . • We show that a learned inverse kinematics solution can construct better poses , qualitatively and quantitatively , compared to a non-learned approach . • We extend existing architectures with ( i ) semantic conditioning of joint ID and type at the input , ( ii ) novel residual scheme involving prototype subtraction and accumulation across blocks , as opposed to maxpool/concat daisy chain of ResPointNet , ( iii ) two-stage architecture with computationally efficient residual decoder that improves accuracy at smaller computational cost , as opposed to the naive final linear projection approach of PointNet and ResPointNet , and ( iv ) two-stage decoder design . • We propose a novel look-at loss function and a novel randomized weighting scheme combining randomly generated effector tolerance levels and effector noise to increase the effectiveness of multi-task training .

Summary: This paper proposes a DNN-based framework to handle the challenging task of recovering 3D human poses (joint positions and local rotations) from sparse and variable user inputs (joint positions, orientations of a subset of whole body joints). The paper proposes a neural-network architecture with an encoder to extract pose encodings from partial user inputs and a decoder to generate complete poses from the encoded features. The paper further proposes two new datasets to train and evaluate the proposed models. Experimental results demonstrate the superiority of the proposed framework over previous methods.