Year: 2022

Abstract:
Our work focuses on the development of a learnable neural representation of human pose for advanced AI assisted animation tooling. Specifically, we tackle the problem of constructing a full static human pose based on sparse and variable user inputs (e.g. locations and/or orientations of a subset of body joints). To solve this problem, we propose a novel neural architecture that combines residual connections with prototype encoding of a partially specified pose to create a new complete pose from the learned latent space. We show that our architecture outperforms a baseline based on Transformer, both in terms of accuracy and computational efficiency. Additionally, we develop a user interface to integrate our neural model in Unity, a real-time 3D development platform. Furthermore, we introduce two new datasets representing the static human pose modeling problem, based on high-quality human motion capture data, which will be released publicly along with model code.

heading: 1 INTRODUCTION
text: Modeling human pose and learning pose representations have received increasing attention recently due to their prominence in applications, such as computer graphics and animation (Harvey et al., 2020; Xu et al., 2020); immersive augmented reality (Facebook Reality Labs, 2021; Capece et al., 2018; Lin, 2019; Yang et al., 2021); entertainment (McDonald, 2018; Xpire, 2019); sports and wellness (Rosenhahn et al., 2008; Kim et al., 2021) as well as human machine interaction (Heindl et al., 2019; Casillas-Perez et al., 2016; Schwarz et al., 2014) and autonomous driving (Kumar et al., 2021). In the gaming industry, state-of-the-art real-time pose manipulation tools, such as CCD (Kenwright, 2012), FABRIK (Aristidou & Lasenby, 2011) or FinalIK (RootMotion, 2020), are popular for rapid execution and rely on forward and inverse kinematics models defined via non-learnable kinematic equations. Inverse kinematics (IK) is the process of computing the internal geometric parameters of a kinematic system resulting in the desired configuration (e.g. global positions) of system’s joints (Paul, 1992). Forward kinematics (FK) refers to the use of the kinematic equations to compute the positions of joints from specified values of internal geometric parameters. While being mathematically accurate, non-learnable IK models do not attempt reconstructing plausible human poses from underconstrained solutions derived from sparse constraints (e.g. positions of a small subset of joints).
In this paper we develop a neural modeling approach to reconstruct full human pose from a sparse set of constraints supplied by a user, in the context of pose authoring and game development. We bridge the gap between skeleton-aware human pose representation based on IK/FK ideas and the neural embedding of human pose. Our approach effectively implements a learnable model for skeleton IK, mapping desired joint configuration into predictions of skeleton internal parameters (local rotations), learning the statistics of natural poses using datasets derived from high-quality motion capture (MOCAP) sequences. The approach, which we call ProtoRes, models the semantics of joints and their interactions using a novel prototypical residual neural network architecture. Inspired by prototypical networks, which showed that one semantic class can be represented by the prototype (mean) of a few examples (Snell et al., 2017), we extend it using a multi-block residual approach: the final pose embedding is a mean across embeddings of sparse constraints and across partial pose predictions produced in each block. We show that in terms of the pose reconstruction accuracy, ProtoRes outperforms existing gaming industry tools such as FinalIK, as well as out-of-the-box machine-learning solution based on Transformer (Vaswani et al., 2017), which also happens to be 10 times less effective in terms of training speed than the proposed architecture.
∗All authors are with Unity Labs; correspondence to: B.N. Oreshkin, boris.oreshkin@gmail.com
Finally, we develop user-facing tools that integrate learned ProtoRes pose representation in the Unity game engine (see Fig. 1). This provides convincing qualitative examples of solutions to the problem of the AI assisted human pose authoring. At the qualitative level, getting traditional workflows to behave the way ProtoRes does would require one to use many techniques in tandem, including IK, FK, layered animation pose libraries, along with procedural rigs encoding explicit heuristics. The process would be highly labor-intensive even for an experienced user while the results would still be of variable fidelity depending on their skill. This is because traditional rigs have no bias towards realistic poses and only allow exploring a limited linear latent space defined by uniform interpolation of a heuristic constraint system. ProtoRes forms a foundation that allows any junior or indie/studio user to bypass these existing complexities and create entirely new workflows for meaningfully exploring learned latent space using a familiar yet far more powerful way. We believe that our model and tools will help speed up the animation process, simplify and democratize game development.
1.1 BACKGROUND
Figure 1: ProtoRes completes full human pose using an arbitrary combination of 3D coordinates, look-at targets and world-space rotations specified by a user. The animation is best viewed in Adobe Reader.
We consider the full-body pose authoring animation task depicted in Fig. 1. The animator provides a few inputs, which we call effectors, that the target pose has to respect. For example, the look-at effector specifies that the head should be facing a specific direction, the positional effectors pin some joints to specific locations in global space and rotational effectors constrain the world-space rotations of certain joints. We assume that the animator can generate arbitrary number of such effectors placed on any skeletal joint (one joint can be driven by more than one effector). The task of the model is to combine all the information provided via effectors and gen-
erate a plausible full-body pose respecting provided effector constraints. We define the full-body pose as the set of all kinematic parameters necessary to recreate the appearance of the body in 3D.
We assume that each effector can be represented in the space Rdeff , where deff is taken to be maximum over all effector types. Suppose we have 3D position and 6D rotation effectors: deff is 6. In position effectors, 3 extra values are 0. We formulate the pose authoring problem as learning the mapping ϒθ : RN×deff → Rdkin with learnable parameters θ ∈Θ. ϒθ maps the input space RN×deff of variable dimensionality N (the number of effectors is not known apriori) to the space Rdkin , containing all kinematic parameters to reconstruct full-body pose. A body with J joints can be fully defined using a tree model with 6D local rotation per joint and 3D coordinate for the root joint, in which case dkin = 6J+3, assuming fixed bone lengths. Given a dataset D= {xi,yi}Mi=1 of poses containing pairs of inputs xi ∈ RN×deff and outputs yi ∈ Rdkin , ϒθ can be learned by minimizing empirical risk:
ϒθ = argmin θ∈Θ 1 M ∑xi,yi∈D L(ϒθ (xi),yi) (1)

heading: 1.2 RELATED WORK
text: Joint representations. Representing pose via 3D joint coordinates (Cheng et al., 2021; Cai et al., 2019; Khapugin & Grishanin, 2019) is sub-optimal as it does not enforce fixed-length bones, nor specifies joint rotations. Predicting joint rotations automatically satisfies bone length constraints and adequately models rotations (Pavllo et al., 2018), which is crucial in downstream applications, such as deforming a 3D mesh on top of the skeleton, to avoid unrealistic twisting. This is viable via skeleton representations based on Euler angles (Han et al., 2017), rotation matrices (Zhang et al., 2018) and quaternions (Pavllo et al., 2018). In this work, we use the two-row 6D rotation matrix representation that addresses the continuity issues reminiscent of the other representations (Zhou et al., 2019).
Pose modeling architectures. Multi-Layer Perceptrons (MLPs) (Cho & Chen, 2014; Khapugin & Grishanin, 2019; Mirzaei et al., 2020) and kernel methods (Grochow et al., 2004; Holden et al., 2015) have been used to learn single pose representations. Beyond single pose, skeleton moving through time can be modeled as a spatio-temporal graph (Jain et al., 2016) or as a graph convolution (Yan et al., 2018; Mirzaei et al., 2020). A common limitation of these approaches is their reliance on a fixed set of inputs, whereas our architecture is specifically designed to handle sparse variable inputs.
Pose prediction from sparse constraints. Real-time methods based on nearest-neighbor search, local dynamics and motion matching have been used on sparse marker position and accelerometer data (Tautges et al., 2011; Riaz et al., 2015; Chai & Hodgins, 2005; Büttner & Clavet, 2015). MLPs and RNNs have been used for real-time processing of sparse signals such as accelerometers (Huang et al., 2018; Holden et al., 2017; Starke et al., 2020; Lee et al., 2018) and VR constraints (Lin, 2019; Yang et al., 2021). These approaches rely on the past pose information to disambiguate next frame prediction and as such are not applicable to our problem, in which only current pose constraints are available. Iterative IK algorithms such as FinalIK (RootMotion, 2020) have been popular in real-time applications. FinalIK works by setting up multiple IK chains for each limb of the body of a predefined human skeleton and for a fixed set of effectors. Several iterations are executed to solve each of these chains using a conventional bone chain IK method, e.g. CCD. In FinalIK, the end effector (hands and feet) can be positioned and rotated, while mid-effectors (shoulders and thighs) can only be positioned. Effectors can have a widespread effect on the body via a hand-crafted pulling mechanism that gives a different weight to each chain. This and similar tools suffer from limited realism when used for human full-body IK, as they are not data-driven. Learning-based methods strive to alleviate this by providing learned model of human pose. Grochow et al. (2004) proposed a kernel based method for learning a pose latent space in order to produce the most likely pose satisfying sparse effector constraints via online constrained optimization. The more recent commercial tool Cascadeur uses a cascade of several MLPs (each dealing with fixed set of positional effectors: 6, 16, 28) to progressively produce all joint positions without respecting bone constraints (Khapugin & Grishanin, 2019; Cascadeur, 2019). Unlike our approach, Cascadeur cannot handle arbitrary effector combinations, rotation or look-at constraints and requires post processing to respect bone constraints.
Permutation invariant architectures. Models for encoding unstructured variable inputs have been proposed in various contexts. Attention models (Bahdanau et al., 2015) and Transformer (Vaswani et al., 2017) have been proposed in the context of natural language processing. Prototypical networks (Snell et al., 2017) used average pooled embedding to encode semantic classes via a few support images in the context of few-shot image classification. Maxpool representations over variable input dimension were proposed by Qi et al. (2017) as PointNet and Zaheer et al. (2017) as DeepSets for segmentation and classification of 3D point clouds, image tagging, set anomaly detection and text concept retrieval. Niemeyer et al. (2019) further generalized the PointNet by chaining the basic maxpool/concat PointNet blocks resulting in ResPointNet architecture.

heading: 1.3 SUMMARY OF CONTRIBUTIONS
text: The contributions of our paper can be summarized as follows.
• We define the 3D character posing task and publicly release two associated benchmarks.
• We show that a learned inverse kinematics solution can construct better poses, qualitatively and quantitatively, compared to a non-learned approach.
• We extend existing architectures with (i) semantic conditioning of joint ID and type at the input, (ii) novel residual scheme involving prototype subtraction and accumulation across blocks, as opposed to maxpool/concat daisy chain of ResPointNet, (iii) two-stage architecture with computationally efficient residual decoder that improves accuracy at smaller computational cost, as opposed to the naive final linear projection approach of PointNet and ResPointNet, and (iv) two-stage decoder design.
• We propose a novel look-at loss function and a novel randomized weighting scheme combining randomly generated effector tolerance levels and effector noise to increase the effectiveness of multi-task training.

heading: 2 PROTORES
text: ProtoRes, shown in Fig. 2, follows the encoder-decoder pattern, unlike PointNet and ResPointNet that use a linear layer as a decoding mechanism (Qi et al., 2017; Niemeyer et al., 2019). The encoder has to deal with N effectors, whereas the decoder processes the collapsed representation of the pose and is therefore N times more compute efficient. Adding more decoder blocks thus results in accuracy gains at a fraction of compute cost. Below, we describe the rest of architecture in more detail.
User inputs. ProtoRes accepts position (3D coordinates), rotation (6D representation of Zhou et al. (2019)) and look-at (3D target position and 3D local facing direction) effectors. All positions are re-referenced relative to the centroid of the positional effectors to achieve translation invariance. Translation invariance simplifies the handling of poses in global space removing the need in universal reference frame, which is tricky to define. Each effector is further characterized by a positive tolerance value. Smaller tolerance implies that the effector has to be more strictly respected in the reconstructed pose. Moreover, the input includes effector semantics encoded via joint ID, an integer in [0,J), and effector type, indicating a positional (0), rotational (1) or look-at (2) effector. Unlike e.g. PointNet (Qi et al., 2017) acting on dense extensive point clouds, ProtoRes acts on sparse inputs that barely provide enough information about the full pose. Therefore, it is critical to provide the semantic information on whether the given effector affects hands or feet and whether this is a positional or a rotational effector, for example. Type and joint ID variables are embedded into continuous vectors and concatenated with effector data, resulting in the encoder input, a matrix xin ∈ RN×Ein with Ein corresponding to the combined dimension of all embeddings plus 6D effector data and a scalar for tolerance.
Pose Encoder is a two-loop residual network. The first residual loop is implemented inside each block depicted in Fig. 2 (bottom right). The second residual loop shown in Fig. 2 (left) implements the proposed Prototype-Subtract-Accumulate (PSA) residual stacking principle, which we empirically found to outperform ResPointNet’s Maxpool-Concat daisy chain proposed by Niemeyer et al. (2019). Next, we first lay out the encoder equations and then describe the motivation behind them in detail. We assume the encoder input to be x1 = xin ∈ RN×Ein , omitting the batch dimension for brevity, in which case the fully-connected layer FCr,`, with `= 1...L, in the residual block r, r = 1 . . .R, with weights Wr,` and biases ar,` can be conveniently described as FCr,`(hr,`−1)≡ RELU(Wr,`hr,`−1 +ar,`). The prototype layer is defined as PROTOTYPE(x)≡ 1N ∑ N i=1 x[i, :]. The pose encoder is then described as:
xr = RELU[br−1−1/(r−1) ·pr−1], (2) hr,1 = FCr,1[xr], . . . , hr,L = FCr,L[hr,L−1], (3) br = RELU[Lrxr +hr,L], fr = Frhr,L, (4) pr = pr−1 +PROTOTYPE[fr]. (5)
Equations (3) and (4) implement the MLP and the first residual loop. The proposed PSA residual mechanism, described in equations (2) and (5), is motivated by the following. First, it implements the inductive bias that the information in individual effectors is only valuable when it is different from what is already stored in the embedding of entire pose. Equation (2) implements this logic by forcing delta-mode in effectors w.r.t. to the pose embedding, pr−1, from the previous block, which additionally creates another residual loop that should facilitate gradient flow. Second, equation (5) collapses the forward encoding of individual effectors into the representation of the entire pose via prototype, which is known to be very effective at representing information from sparse examples (see e.g. Snell et al. (2017)). Finally, the representation of pose is accumulated across residual blocks in (5), effectively implementing skip connections from very early layers. Distant skip connections implemented via concatenation were shown to be effective in DenseNet (Huang et al., 2017). Concatenation based skipping requires additional computation and is most efficient with convolutional networks, in which kernel size can be traded for feature width while implementing distant skip connections, whereas accumulation is more compute efficient in our context.
Pose Decoder has two blocks: global position decoder (GPD) and inverse kinematics decoder (IKD). Both rely on the fully-connected residual (FCR) architecture depicted in Fig. 2 (right). GPD unrolls the pose embedding generated by encoder into the unconstrained predictions of 3D joint positions. IKD generates the internal geometric parameters (joint rotations) of the skeleton that are guaranteed to generate feasible joint positions after forward kinematics pass.
GPD accepts the encoded pose embedding, b̃0 = pR ∈ RE , and produces 3D position predictions f̃R ∈ R3J of all skeletal joints using the FCR whose r-th block is described as follows:
hr,1 = FCgpdr,1 [b̃r−1], . . . , hr,L = FC gpd r,L [hr,L−1],
b̃r = RELU[Lgpdr b̃r−1 +hr,L], f̃r = f̃r−1 +F gpd r hr,L.
(6)
Since GPD produces predictions with no regard to skeleton constraints, its predictions do not respect bone lengths. For the IKD to provide correct rotations, the origin of the kinematic chain in world space must be given, and GPD conveniently provides the prediction of the reference (root) joint.
IKD accepts input b̂0 ∈ RE+3J , consisting of the concatenation of the encoder-generated pose embedding, pR ∈RE , and the output of GPD, f̃R ∈R3J . Effectively, the draft pose generated by GPD, is used to condition IKD. We show in Section 3 that this additional conditioning improves accuracy. IKD predicts the 6D angle for each joint, f̂R ∈ R6J , and its r-th block operates as follows:
hr,1 = FCikdr,1 [b̂r−1], . . . , hr,L = FC ikd r,L [hr,L−1],
b̂r = RELU[Likdr b̂r−1 +hr,L], f̂r = f̂r−1 +F ikd r hr,L.
(7)
Forward Kinematics (FK) pass, described in detail in Appendix A, applies skeleton kinematic equations to the local joint rotations and global root position produced by IKD. For each joint j, it produces the global transform matrix Ĝ j containing the global rotation matrix, Ĝ13j ≡ Ĝ j[1 : 3,1 : 3], and the 3D global position, ĝ j = Ĝ j[1 : 3,4], of the joint.

heading: 2.1 LOSSES
text: We use three losses to train the architecture in a multi-task fashion. The total loss combines loss terms additively with weights chosen to approximately equalize their magnitude orders.
L2 loss penalizes the mean squared error between the prediction ŷ and the ground truth y: MSE(y, ŷ) = ‖y− ŷ‖22. (8)
L2 loss is used to supervise the GPD as well as the IKD output after FK pass. In the latter case it drives IKD to learn to predict rotations that lead to small position errors after FK.
Geodesic loss penalizes the errors of the IKD’s rotational outputs. It represents the smallest arc (in radians) to go from one rotation to another over the surface of a sphere. The geodesic loss is defined for the ground truth rotation matrix R and its prediction R̂ as (see e.g. Salehi et al. (2018)):
GEO(R, R̂) = arccos [ (tr(R̂T R)−1)/2 ] . (9)
We believe that using a combination of positional and rotational losses is necessary to learn a highquality pose representation. This is especially important when the task is to reconstruct a sparsely specified pose, giving rise to multiple plausible reconstructions. We argue that a model trained to reconstruct both plausible joint positions and rotations is better equipped to solve the task accurately. Empirical evidence presented in Section 3.4 supports this intuition: a model trained on both L2 and Geodesic generalizes better on both losses than models trained only on one of those terms.
Look-at loss, proposed in this paper, enables the “look-at” feature, i.e. the ability to orient a joint to face a particular global position (e.g. having the head looking at a given object). It allows the model to align any direction vector d j ∈ R3 of a joint, expressed in its local frame of reference, towards a global target location t. Given the predicted global transform matrix Ĝ j, look-at loss is defined as:
LAT(t,d j,Ĝ j) = arccos [−−−−→ (t− ĝ j) · Ĝ13j d j ] . (10)
−−−−→ (t− ĝ j) is a unit-length vector pointing at the target object in world space. Ĝ13j , when multiplied by d j, represents the global predicted look-at direction. The look-at loss trains the IKD to produce Ĝ13j consistent with the look-at direction defined by t and d j, both provided as network inputs.

heading: 2.2 TRAINING METHODOLOGY
text: The training methodology involves techniques to (i) regularize model via rotation and mirror augmentations, (ii) learn handling of sparse inputs and (iii) effectively combine multi-task loss terms.
Sparse inputs modeling relies on effector sampling. First, the total number of effectors is sampled uniformly at random in the range [3, 16]. Given the total number of effectors, the effector IDs (one of 64 joints) and types (position, rotation, or look-at) are sampled from the Multinomial without replacement. This induces exponentially large number of effector type and joint permutations, resulting in strong regularizing effects and teaching the network to deal with variable inputs.
Effector tolerance and randomized loss weighting. The motivation behind randomized loss weighting is two-fold. First, we empirically find that when a random weight is multiplicatively applied to the respective loss term and its reciprocal is used as one of the network inputs for the corresponding effector, the network learns to respect the tolerance level. When exposed as a user interface feature, it lets the user control the degree of responsiveness of the model to different effectors. We also discovered that this only works when noise is added to the effector value and the standard deviation of the noise is appropriately modulated by tolerance. For example, the noise teaches the model to disregard the effector completely if the tolerance input value corresponds to the high noise variance regime. Second, we notice that the randomized weighting improves multi-task training and generalization performance. In particular, we observe significant competition between rotation and position losses on our task. The introduction of the randomized loss weighting seems to turn the competition into cooperation as our empirical results suggest in Section 3. We implement the randomized loss weighting scheme via the following steps. For each sampled effector, we uniformly sample Λ ∈ [0,1] treated as the effector tolerance. Given an effector tolerance Λ, noise with the maximum standard deviation σM modulated by Λ (noise models used for different effector types are described in detail in Appendix B) is added to effector data before feeding them to the neural network: σ(Λ) = σMΛη , (11) We use η > 10 to shape the distribution of σ to smaller values. Furthermore, each effector is attached with a randomized loss weight reciprocal to σ(Λ), capped at WM if σ(Λ)< 1/WM:
W (Λ) = min(WM,1/σ(Λ)). (12) Λ drives network input and W (Λ) weighs the loss term affected by the effector.
The detailed procedure to compute the ProtoRes loss based on one batch item is presented in Algorithm 1 of Appendix C and the summary is provided below. First, we sample (i) the number of effectors and (ii) their associated types and IDs. For each effector, we randomly sample the tolerance level and compute the associated noise std and loss weight. Given noise std, an appropriate noise model is applied to generate input data based on effector type as described in Appendix B. Then ProtoRes predicts draft joint positions f̃R, j and local joint rotations R̂ j. World-space rotations and positions Ĝ j for all joints j ∈ [0,J) are computed using FK. We conclude by calculating the individual deterministic and randomized loss terms, whose weighted sum is used for backpropagation.

heading: 3 EMPIRICAL RESULTS
text: Our results demonstrate that (i) ProtoRes reconstructs sparsely defined pose more accurately than existing non-ML IK solution and two ML baselines, (ii) our Prototype-Subtract-Accumulate residual scheme is more effective than the Maxpool-Concat daisy chain of Niemeyer et al. (2019), (iii) twostage GPD+IKD decoding is more effective than IKD-only decoding, (iv) the proposed randomized loss weighting improves multi-task training, (v) joint Geodesic/L2 loss training is synergetic.

heading: 3.1 DATASETS
text: miniMixamo We use the following procedure to create our first dataset from the publicly available MOCAP data available from mixamo.com, generously provided by Adobe Inc. (2020). We download a total of 1598 clips and retarget them on our custom 64-joint skeleton using the Mixamo online tool. This skeleton definition is used in Unity to extract the global positions as well as global and local rotations of each joint at the rate of 60 frames per second (total 356,545 frames). The resulting dataset is partitioned at the clip level into train/validation/test splits (with proportion 0.8/0.1/0.1, respectively) by sampling clip IDs uniformly at random. Splitting by clip makes the evaluation framework more realistic and less prone to overfitting: frames belonging to the same clip are often similar. At last, the final splits retain only 10% of randomly sampled frames (miniMixamo has 33,676 frames total after subsampling) and all the clip identification information (clip ID, meta-data/description, character information, etc.) is discarded. This anonymization guarantees that the original sequences from mixamo.com cannot be reconstructed from our dataset, allowing us to release the dataset for reproducibility purposes without violating the original dataset license (Adobe Inc., 2020).
miniUnity To collect our second dataset we predefine a wide range of human motion scenarios and hire a qualified MOCAP studio to record 1776 clips (967,258 total frames @60 fps). Data collection details appear in Appendix D. Then we create a dataset of a total of 96,666 subsampled frames following exactly the same methodology that was employed for miniMixamo.

heading: 3.2 TRAINING AND EVALUATION SETUP
text: We use Algorithm 1 of Appendix C to sample batches of size 2048 from the training subset, hyperparameters are adjusted on the validation set. The number of effectors is sampled once per batch and is fixed for all batch items to maximize data throughput. We report metrics Ldetgpd−L2, L det ikd−L2, Ldetloc−geo, defined in Appendix E and calculated on the test set, using models trained on the training set. Ldetgpd−L2 is computed only on the root joint. These metrics characterise both the 3D position accuracy and the bone rotation accuracy. The evaluation framework tests model performance on a pre-generated set of seven files containing 6,7 . . .12 effectors respectively. Metrics are averaged over all files, assessing the overall quality of pose reconstruction in scenario with sparse variable inputs. Tables present results averaged over 4 random seed retries and metrics computed every 10 epochs over last 1000 epochs. Additional details and hyperparameter settings appear in Appendix E.

heading: 3.3 KEY RESULTS
text: To demonstrate the advantage of the proposed architecture, we perform two evaluations. First, we compare ProtoRes against two ML baselines in the random effector evaluation setup described in Section 3.2. The first baseline, Masked-FCR, is a brute-force unstructured baseline that uses a very wide J ·3 ·7 input layer (J joints, 3 effector types, 6D effector data and 1D tolerance) handling all effector permutations. Missing effectors are masked with 3 · J learnable 7D placeholders. MaskedFCR has 3 encoder and 6 decoder blocks to match ProtoRes. The second baseline is based on the Transformer encoder (Vaswani et al. (2017), see Appendix G for architecture and hyperparameter settings). The bottom of Table 1, summarizing this study, shows clear advantage of ProtoRes w.r.t. both baselines. Additionally, training Transformer on NVIDIA M40 24GB GPU for 40k epochs of miniMixamo takes 1055 hours (batch size 1024 to fit Transformer in GPU memory), whereas training ProtoRes takes 106 hours. ProtoRes is clearly more compute efficient.
Second, Table 1 (top) compares ProtoRes against a non-ML IK solution FinalIK (RootMotion, 2020), as well as Transformer and Masked-FCR, on a 5-point evaluation benchmark. The 5-point benchmark tests the reconstruction of the full pose from five position effectors: chest, left and right hands, left and right feet. It is chosen, because generating the exponentially large number of FinalIK configurations to process all heterogeneous effector combinations in the random benchmark is not feasible. Note that the 5-point benchmark and random benchmark results are not directly comparable. We can see that all ML methods significantly outperform FinalIK in reconstruction accuracy, ProtoRes being the best overall (please also refer to qualitative analysis results in Appendices J, L.4, L.8). Clearly, ML methods learn the right inductive biases from the data to solve the ill-posed sparse input pose reconstruction problem, unlike the pure non-learnable IK method FinalIK.
Third, qualitative posing results are shown in Fig. 3, demonstrating that visually appealing poses can be obtained with small number of effectors (4 and 7 effectors for the two poses on the right). The left 4 poses demonstrate how pose can be refined successively by adding more effectors. Please refer to Appendix L and supplementary videos for more demonstration examples.

heading: 3.4 ABLATION STUDIES
text: Decoder ablation is shown in Table 2 (top). We keep all hyperparameters at defaults described in Section 3.2 and vary the number of decoder blocks (0 blocks corresponds to a simple linear projection) and compare to the ProtoRes baseline with 3 decoder blocks. We see consistent gain adding more decoder blocks at small compute cost (91, 95, 102, 106 hours of train time on miniMixamo dataset and NVIDIA M40 GPU for 0,1,2,3 blocks). Please see more detailed results in Appendix H.
Prototype-Subtract-Accumulate ablation is presented in Appendix I, in which we compare it against the ResPointNet stacking scheme (Maxpool-Concat daisy chain by Niemeyer et al. (2019)). We show that our stacking scheme is more accurate in a computationally efficient configuration with 3 encoder blocks and allows stacking deeper networks gaining more accuracy.
GPD ablation is shown in Table 2 (middle). We remove GPD and increase IKD depth to 6 blocks to match the capacity of IKD+GPD. Comparing to the baseline, we see that GPD creates consistent gain across metrics and datasets by conditioning IKD with a draft pose.
Ablation of loss terms, shown in Table 2 (middle), studies the effect of (i) removing all L2 loss terms from the output of the FK pass and (ii) removing all Geodesic loss terms from the rotation output of IKD. Interestingly, removing either of the loss terms results in the degradation of all monitored metrics on both datasets. We conclude that jointly penalizing positions with L2 and rotations with Geodesic results in positive synergetic effects and improves the overall quality of pose model.
Randomized loss weighting ablation is shown in Table 2 (bottom). The randomized loss weighting scheme (see Algorithm 1 in Appendix C) is disabled by replacing all randomized loss terms with their deterministic counterparts. For example, Lrndikd−L2 is replaced with ∑ J j=1 MSE(g j, ĝ j). The inclusion of randomized weighting significantly improves generalization performance on all datasets and metrics. Additionally, when L2 weight Wpos increases with disabled randomized weighting, position L2 metrics improve, but at the expense of declining rotation metrics. Therefore, randomized weighting scheme contributes positive effect that cannot be achieved by tweaking the deterministic loss weights.
The limitations of the current work, discussed in detail in Appendix M, include (i) the lack of temporal consistency as we focus on the problem of authoring a discrete pose, (ii) constraints are satisfied approximately, as opposed to the more conventional systems, (iii) exotic poses significantly deviating from the training data distribution may be hard to achieve.

heading: 4 CONCLUSIONS
text: We define and solve the discrete full-body pose authoring task using sparse and variable user inputs. We define and release two datasets to support the development of ML models for discrete pose authoring and animation. We propose ProtoRes, a novel ML architecture which processes a variable number of heterogeneous user inputs (position, angle, direction) to reconstruct a full-body pose. We compare ProtoRes against two strong ML baselines, Masked-FCR and Transformer, showing superior results for ProtoRes, both in terms of accuracy and computational efficiency. We also show that ML models reconstruct full-body poses from sparse user inputs more accurately than existing non-learnable inverse kinematics models. We develop a suite of UI tools for the integration of our model in Unity and provide demos showing how our model can be used effectively to solve the discrete pose authoring problem by the end user. Our results have a few implications. First, our ML based tools will have positive impacts on the simplification and democratization of the game development process by helping a wide audience materialize their creative animation ideas. Second, our novel approach to neural pose representation could be applied in a variety of tasks where efficient and accurate reconstruction of full-body poses from noisy intermittent measurements is important.

heading: Supplementary Material for ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematics

heading: Table of Contents

heading: A Architecture Details 14

heading: B Effector noise model 15
text: B.1 Position effector noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 B.2 Rotation effector noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 B.3 Look-at effector noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

heading: C Training and Evaluation Methodology: Details 16

heading: D Datasets: Details 17
text: D.1 Datasets Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D.2 Datasets Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

heading: E Training and Evaluation Setup: Details 21

heading: F Masked-FCR baseline architecture 24

heading: G Transformer baseline architecture 24

heading: H Ablation of the Decoder: Details 25

heading: I Ablation of the Prototype-Subtract-Accumulate stacking principle 26

heading: J Qualitative comparison to Transformer and FinalIK baselines 27

heading: K The effects of the number and type of effectors 28
text: L Videos and demonstrations 29 L.1 ProtoRes Demo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 L.2 Posing from Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 L.3 Loss Ablation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 L.4 FinalIK Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 L.5 Datasets Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 L.6 Retargeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 L.7 Application to the Quadruped Skeleton . . . . . . . . . . . . . . . . . . . . . . . 30 L.8 Comparison to Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 L.9 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
M Limitations 31

heading: A ARCHITECTURE DETAILS
text: Residual block depicted in Fig. 4 (left) is used as the basic building block of the ProtoRes encoder and decoder.
Decoders The block diagram of the global position and the inverse kinematics decoders used in the main architecture (see Fig. 2) is presented in Fig. 4. The architecture has fully connected residual topology consisting of multiple fully connected blocks connected using residual connections. Each block has residual and forward outputs. The forward output contributes to the final output of the decoder. The residual connection sums the hidden state of the block with the linear projection of the input and applies a ReLU non-linearity.
In the main text we use a convention that the number of layers and blocks in the encoder, as well as in GPD and IKD decoders is the same and is given by L and R respectively. Obviously, using a different number of layers and residual blocks in each of the blocks might be more optimal.
Forward Kinematics pass is applied to the output of the IKD, transforming local joint rotations and global root position into the global joint rotations and positions using skeleton kinematic equations. The FK pass relies on the offset vector o j = [ox, j,oy, j,oz, j]ᵀ and the rotation matrix R j for each joint j. The offset vector is a fixed non-learnable vector representing bone length constraint for joint j. It provides the displacement of this joint with respect to its parent joint when joint j rotation is zero. R j can be naı̈vely represented using local Euler rotation angles α j,β j,γ j:
o j = [ox, j oy, j oz, j ] ; R j = [cosα j −sinα j 0 sinα j cosα j 0 0 0 1 ][ cosβ j 0 sinβ j 0 1 0 −sinβ j 0 cosβ j ][1 0 0 0 cosγ j −sinγ j 0 sinγ j cosγ j ] . (13)
However, we use a more robust representation proposed by (Zhou et al., 2019), relying on vector norm −→u ≡ u/‖u‖2 and vector cross product u×v = ‖u‖‖v‖cos(γ)−→n (γ is the angle between u and v in the plane containing them and −→n is the normal to the plane):
r̂ j,x = −−−−−→ f̂R, j[1 : 3], r̂ j,z = −−−−−−−−−−→ r̂ j,x× f̂R, j[4 : 6], r̂ j,y = r̂ j,z× r̂ j,x, R̂ j = [̂r j,x r̂ j,y r̂ j,z] . (14)
Provided with the local offset vectors and rotation matrices of all joints, the global rigid transform of any joint j is predicted following the tree recursion from the parent joint p( j) of joint j:
Ĝ j = Ĝp( j) [
R̂ j o j 0 1
] . (15)
The global transform matrix Ĝ j of joint j contains its global rotation matrix, Ĝ13j ≡ Ĝ j[1 : 3,1 : 3], and its 3D global position, ĝ j = Ĝ j[1 : 3,4].

heading: B EFFECTOR NOISE MODEL
text: This section describes the details of the NOISEMODEL that is used in Algorithm 1 to corrupt model effector input x[i, :] based on appropriate noise level σ(Λi).

heading: B.1 POSITION EFFECTOR NOISE MODEL
text: If effector type is positional (Ti = 0), i.e. effector i is a coordinate in 3D space, typically corresponding to the desired position of joint Ii in 3D space, we employ Gaussian white noise model:
x[i,1 : 3] = gIi +σ(Λi)εi; x[i,4 : 6] = 0. (16)
Here x[i, :] is the i-th model input, gIi is the ground truth location of joint Ii, σ(Λi) is the noise standard deviation computed based on eq. (20) and εi is a 3D vector sampled from the zero-mean Normal distribution N(0,I).

heading: B.2 ROTATION EFFECTOR NOISE MODEL
text: If effector type is angular (Ti = 1), i.e. effector i is a 6DoF rotation matrix representation, we employ random rotation model that is implemented in the following stages. First, suppose fIi is the ground truth 6DoF representation of the global rotation of joint Ii corresponding to effector i. We transform it to the rotation matrix representation G13Ii using equation (14). Second, we generate the random 3D Euler angles vector εi from the zero-mean Gaussian distribution N(0,σ(Λi)I)1 and convert it to the random rotation matrix Ψi using eq. (13):
Ψi = [cosεi[1] −sinεi[1] 0 sinεi[1] cosεi[1] 0
0 0 1
][ cosεi[2] 0 sinεi[2] 0 1 0
−sinεi[2] 0 cosεi[2] ][1 0 0 0 cosεi[3] −sinεi[3] 0 sinεi[3] cosεi[3] ] . (17)
Third, we apply random rotation to the ground truth matrix, G13′Ii = ΨiG 13 Ii . Finally, we convert the randomly perturbed rotation matrix back to the 6DoF representation:
x[i,1 : 3] = G13′Ii [:,1], x[i,4 : 6] = G 13′ Ii [:,2]. (18)

heading: B.3 LOOK-AT EFFECTOR NOISE MODEL
text: If effector type is look-at (Ti = 2), i.e. effector i is a position of the target at which a given joint is supposed to look, we employ random sampling of the target point along the ray cast in the direction formed by the global rotation of a given joint.
First, we sample the local direction vector di from the zero-mean normal 3D distribution N(0,I) and normalize it to unit length. Second, we sample the distance between the joint and the target object, dt, from the normal distribution N(0,5) folded over at 0 by taking the absolute value. The location of the target object is then determined as ti = gIi +dtdi +σ(Λi)εi. Finally, the output is constructed as follows:
x[i,1 : 3] = ti, x[i,4 : 6] = di. (19)
As previously, εi is a 3D vector sampled from the zero-mean Normal distribution N(0,I).
Algorithm 1 Loss calculation for a single item in the training batch of ProtoRes. Require: R j,G j;N ∼ UNIFORM[3,16] . Ground truth for all joints j ∈ [0,J); number of effectors Ensure: x . Sample inputs
I1, . . . , IN ←MULTINOMIAL({0, . . . ,J−1},N) . Effector IDs T1, . . . ,TN ←MULTINOMIAL({0,1,2},N) . Effector type for i in 1 . . .N do
Λi← UNIFORM[0,1] . Effector tolerance σ(Λi); W (Λi)← σMΛηi ; min(WM,1/σ(Λi)) . Effector noise std and weight x[i, :]← NOISEMODEL(GIi ,σ(Λi),Ti) . Generate noisy effector
end for Predict: f̃R, j, R̂ j,Ĝ j ∀ j based on x Lrndgpd−L2← 1∑Ni=11Ti=0 W (Λi) ∑Ni=11Ti=0 W (Λi)MSE(gIi , f̃R,Ii) . Randomized GPD position loss
Lrndikd−L2← 1∑Ni=11Ti=0 W (Λi) ∑Ni=11Ti=0 W (Λi)MSE(gIi , ĝIi) . Randomized IKD position loss Ldetgpd−L2← ∑Jj=1 MSE(g j, f̃R, j) . Deterministic GPD position loss Ldetikd−L2← ∑Jj=1 MSE(g j, ĝ j) . Deterministic IKD position loss Ldetloc−geo← ∑Jj=1 GEO(R j, R̂ j) . Deterministic local rotation loss Lrndglob−geo← 1∑Ni=11Ti=1 W (Λi) ∑Ni=11Ti=1 W (Λi)GEO(G13Ii ,Ĝ 13 Ii ) . Randomized global rotation loss Ldetlat ← 1∑Ni=11Ti=2 ∑Ni=11Ti=2 LAT(x[i,1 : 3],x[i,4 : 6],Ĝ13Ii ) . Randomized Look-at loss L← WposJ (L rnd gpd−L2+L rnd ikd−L2+L det gpd−L2+L det ikd−L2)+ 1 J (L det lat +L rnd glob−geo+L det loc−geo) . Total loss

heading: C TRAINING AND EVALUATION METHODOLOGY: DETAILS
text: The training methodology involves techniques targeting to (i) regularize model via data augmentation, (ii) learn handling of sparse inputs and (iii) effectively combine multi-task loss terms.
Data augmentation is based on the rotation and mirror augmentations. The former rotates the skeleton around the vertical Y axis by a random angle in [0,2π]. Rotation w.r.t. ground XZ plane is not applied to avoid creating poses implausible according to the gravity direction. Mirror augmentation removes any implicit left- or right-handedness biases by flipping the skeleton w.r.t. the Y Z plane.
Sparse inputs modeling relies on effector sampling. First, the total number of effectors is sampled uniformly at random in the range [3, 16]. Given the total number of effectors, the effector IDs (one of 64 joints) and types (one of 3 types: position, rotation, or look-at) are sampled from the Multinomial without replacement. This sampling scheme produces an exponentially large number of different permutations of effector types and joints, resulting in strong regularizing effects.
Effector tolerance and randomized loss weighting. The motivation behind the randomized loss weighting is two-fold. First, the randomized loss weighting was originally introduced as a binary indicator to force the model to better respect constraints provided as effectors, compared to the joints predicted by the model. Afterwards, we realized that this can be made more flexible by generating a continuous variable representing the tolerance level. This variable can be provided as an input to the network and it can be exposed as a user interface feature to let the user control the degree of responsiveness of the model to different effectors. We also discovered that the latter feature only works when a noise is added to effector value and the standard deviation of the noise is appropriately synchronised with the tolerance. The noise teaches the model to disregard the effector completely if the tolerance input value corresponds to the high noise variance regime.
Second, we observed that the use of the randomized weighting improves multi-task training and generalization performance. Initially, we noticed that increasing the weight of position loss would drive the generalization on the position metric to a better spot, while the rotation metric generalization would be compromised, which is not surprising. This was especially evident when the position loss weight was increased by one or two orders of magnitude. This is a well-known phenomenon when
1Note that in the case of angles, sampling from the Tikhonov (a.k.a. circular normal or von Mises) distribution might be a better idea, but Gaussian worked well in our case.
dealing with multiple loss terms, which we informally call “fighting” between losses (related to the Pareto front, more formally). This effect can be observed when comparing two bottom rows in Table 2. Introducing the randomized loss weighting scheme we observed two things. “Fighting” disappeared, i.e. the randomly generated weights of position effectors varied in a wide range between 1e-1 and 1e5 within a batch, but the fact that some of the weight values are one or two orders of magnitude greater than the baseline position weight of 100, did not lead to the deterioration of the rotation loss. Moreover, the introduction of the randomized loss weighting positively affected the generalization on both position and rotation metrics, which can be assessed by comparing the first row of Table 2 with its bottom rows. This leads us to believe that the randomized loss weighting introduces a sinergy in the multi-task training that is not achievable by simple adjustment of static loss weights. We believe this technique could be more generally applicable to multi-task training, but a more detailed investigation of this is outside of the current scope.
We now describe the technical details behind randomized loss weighting implementation. For each sampled effector, we further uniformly sample Λ ∈ [0,1] treated as effector tolerance. Given an effector tolerance Λ, noise (noise models used for different effector types are described in detail in Appendix B) with variance proportional to Λ is added to effector data before feeding them to the neural network:
σ(Λ) = σMΛη . (20)
We use η > 10 to shape the distribution of σ to smaller values. Furthermore, to each effector is attached a randomized loss weight reciprocal to σ(Λ), capped at WM if σ(Λ)< 1/WM:
W (Λ) = min(WM,1/σ(Λ)). (21)
Λ drives network inputs and is simultaneously used to weigh losses by W (Λ). Thus ProtoRes learns to respect effector tolerance, leading to two positive outcomes. First, ProtoRes provides a tool allowing one to emphasize small tolerance effectors (Λ≈ 0) and relax the large tolerance ones (Λ≈ 1). Second, randomized loss weighting improves the overall accuracy in the multi-task training scenario.
The detailed procedure to compute the ProtoRes loss based on one batch item is presented in Algorithm 1 and the summary is provided below. First, we sample (i) the number of effectors and (ii) their associated type and ID. For each effector, we randomly sample the tolerance level and compute the associated noise std and loss weight. Given noise std, an appropriate noise model is applied to generate input data based on effector type as described in Appendix B. Then ProtoRes predicts draft joint positions f̃R, j, local joint rotations R̂ j, as well as world-space rotations and positions Ĝ j for all joints j ∈ [0,J). We conclude by calculating the individual deterministic and randomized loss terms, whose weighted sum is used for backpropagation.

heading: D DATASETS: DETAILS

heading: D.1 DATASETS DESCRIPTIONS
text: miniMixamo We use the following procedure to create our first dataset from the publicly available MOCAP data available from mixamo.com, generously provided by Adobe Inc. (2020). We download a total of 1598 clips and retarget them on our custom 64-joint skeleton using the Mixamo online tool. This skeleton definition is used in Unity to extract the global positions as well as global and local rotations of each joint at the rate of 60 frames per second (total 356,545 frames). The resulting dataset is partitioned at the clip level into train/validation/test splits (with proportion 0.8/0.1/0.1, respectively) by sampling clip IDs uniformly at random. Splitting by clip makes the evaluation framework more realistic and less prone to overfitting: frames belonging to the same clip are often similar. At last, the final splits retain only 10% of randomly sampled frames (miniMixamo has 33,676 frames total after subsampling) and all the clip identification information (clip ID, meta-data/description, character information, etc.) is discarded. This anonymization guarantees that the original sequences from mixamo.com cannot be reconstructed from our dataset, allowing us to release the dataset for reproducibility purposes without violating the original dataset license (Adobe Inc., 2020).
For miniMixamo our contribution is as follows. Mixamo data is not available as a single file. Therefore, anyone who wants to use the data for academic purposes needs to go through a lengthy process of downloading individual files. Importantly, this step creates additional risks for the reproducibility
of results. We have gone through this step and assembled all files in one place. Furthermore, Mixamo data cannot be redistributed, according to Adobe licensing, which is again a reproducibility risk. However, we do not need the entire dataset for benchmarking on the task we defined. Therefore, we defined a suitable subsampling and anonymization procedure that allowed us to obtain (i) a high quality reproducible benchmark dataset for our task and (ii) a legal permission from Mixamo/Adobe to redistribute this benchmark for academic research purposes. We are extremely grateful to the representatives from Mixamo and Adobe who approved it to facilitate the democratization of character animation. The entire process of creating the benchmark took us a few months of work, which we consider a significant contribution to the research community.
miniUnity To collect our second dataset we predefine a wide range of human motion scenarios and hire a qualified MOCAP studio to record 1776 clips (967,258 total frames @60 fps). Then we create a dataset of a total of 96,666 subsampled frames following exactly the same methodology that was employed for miniMixamo.
The following action scenarios were used to collect MOCAP sequences in miniUnity. The detailed hierarchy of motion scenarios is presented in Table 6. First, the following locomotion types: compass crouch, compass jogs, compass runs, compass walks were collected for female and male subjects under high energy, low energy and injured scenarios. The same locomotion types were collected under neutral energy feminine and neutral energy macho scenarios. Furthermore, under neutral energy generic scenario, for both female and male subjects, we collected following action, object and environment interaction types: archery, bokken fighting, calisthetics, door interactions, fist fighting, food, handgun, hands, knife fighting, locomotion, longsword fighting, phone, place, railing interactions, rifle, seated interactions, shotgun, standings, sword, wall. Among the latter categories, locomotion and handgun had following more detailed subdivisions. Locomotion: compass crawls, compass crouch, compass jogs, compass rifle aim walks, compass rifle crawl, compass rifle crouch, compass rifle jogs, compass rifle runs, compass rifle walks, compass runs, compass walks, rifle idles, walk carry heavy backpack, walk carry heavy sack, walk carry ladder, walk dragging heavy object. Handgun: downwards, level, upwards, verticals. The motion categories existing in miniUnity and miniMixamo can be compared by looking at Tables 6 and 7 respectively.
The key differentiators of the datasets that we release that make them significant contributions toward AI driven artistic pose development are as follows:
• Both miniMixamo (derived from the Mixamo, which is generously provided by Adobe) and miniUnity are collected by professional studio contractors relying on the service of professional actors using high-end MOCAP studio equipment.
• Both datasets are clean and contain data of very high quality. For our dataset, we specifically had to go through multiple cleaning iterations to make sure all the data collection and conversion artifacts are removed. We are very grateful to our contractor for being diligent, detail oriented, and determined to provide the high quality data.
• Both datasets provide data in the industry standard skeleton format compatible with multiple existing animation rigs and therefore making it easy to experiment with the ML assisted pose authoring results in 3D development environments such as Unity. This is in contrast to CMU and AMASS datasets that are collected in heterogeneous environments using non-standard sensor placements.
• Both our datasets provide 64 joint skeletons and contain fine grain hands and feet data, unlike other publicly available datasets.

heading: D.2 DATASETS DESCRIPTIVE STATISTICS
text: The standard deviations of joint positions (in the coordinate system relative to hips joint) and joint local quaternions are presented in Tables 3 and 4.
The distributions of 30 most popular tags across the frames of the original dataset used to build miniMixamo and miniUnity are shown in Figure 5. It is clear that the two dataset cover some common activities such as walking and idling, for example. Additionally, there are numerous categories the two datasets emphasize separately. For example miniMixamo focuses a lot on fighting and handling guns,
whereas miniUnity has more neutral energy activities, provides extensive labeling of male/female poses as well as covers scenarios of handling objects and food.
The distribution of PCA of flattened rotations of all joints of miniMixamo (red) and miniUnity (blue) datasets is shown in Figure 6. Each point corresponds to a pose. It is clear that there are both overlapping areas corresponding to clusters with similar poses and areas with disjoint clusters, showing distinct pose configurations characteristic of the two different datasets.

heading: E TRAINING AND EVALUATION SETUP: DETAILS
text: We use Algorithm 1 of Appendix C to sample batches of size 2048 from the training subset. The number of effectors is sampled once per batch and is fixed for all batch items to maximize data throughput. The training loop is implemented in PyTorch (Paszke et al., 2019) using Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0002. Hyperparameter values are adjusted on the validation set (see Appendix E for hyperparameter settings).
We report Ldetgpd−L2, L det ikd−L2, L det loc−geo metrics calculated on the test set, using models trained on the training set. Ldetgpd−L2 is computed only on the root joint. These metrics characterise both the 3D position accuracy (Ldetgpd−L2, L det ikd−L2) and the bone rotation accuracy (L det loc−geo). They are defined as follows:
Ldetgpd−L2 = MSE(g0, f̃R,0) (22)
Ldetikd−L2 = J
∑ j=1 MSE(g j, ĝ j) (23)
Ldetloc−geo = J
∑ j=1 GEO(R j, R̂ j) (24)
Here f̃R,0 and g0 are ground truth global location of the root joint and its prediction from the GPD, respectively; g j and ĝ j ground truth location of joint j obtained by subjecting the rotation prediction
from IKD to the forward kinematics process; R j and R̂ j are ground truth local rotation matrix of joint j and its prediction obtained from the IKD, respectively.
The evaluation framework tests model performance on a pre-generated set of seven files containing 6, 7, . . . , 12 effectors respectively. Skeleton is split in six zones, with four main zones including each limb, the hip zone and the head zone. In each file, we first sample one positional effector from each main zone. Remaining effectors are sampled randomly from all zones and effector types, mimicking pose authoring scenarios observed in practice. Metrics are averaged over all samples in all files, assessing the overall quality of pose reconstruction in scenario with sparse and variable inputs. All tables present results averaged over 4 random seed retries and metric values computed every 10 epochs over last 1000 epochs, rounded to the last statistically significant digit.
Hyperparameter settings The training loop is implemented in PyTorch (Paszke et al., 2019) using Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0002. We tried to use SGD optimizer
to train the architecture, but it was very difficult to obtain stable results with it. Adam optimizer turned out to be much more suitable for our problem. The learning rate was selected to be 0.0002, which is lower than Adam’s default. Obtaining stable training results with higher learning rates was not feasible. Batch size is selected to be 2048 to accelerate training speed. In practice we observed slightly better generalization results with smaller batch size (1024 and 512). The detailed settings of ProtoRes hyperparameters are presented in Table 5.

heading: F MASKED-FCR BASELINE ARCHITECTURE
text: Masked-FCR is a brute-force unstructured baseline that uses a very wide J ·3 ·7 input layer (J joints, 3 effector types, 6D effector plus one tolerance value) to handle all possible effector permutations. Each missing effector is masked with one of 3 · J learnable 7D placeholders. Masked-FCR has 3 encoder and 6 decoder blocks to match ProtoRes.

heading: G TRANSFORMER BASELINE ARCHITECTURE
text: We implement Transformer baseline using the default Transformer module available from PyTorch https://pytorch.org/docs/stable/generated/torch.nn.Transformer. html. The block diagram of the Transformer baseline architecture is shown in Fig. 7.
We use a standard transformer application scenario in which the transformer source input is fed with the variable length input and the required outputs are queried via target input. In our case the variable length input corresponds to the effector data concatenated with embedded effector categorical variables. The query for the output consists of the embeddings of all joints. Note that the joint embedding is reused both for source and target inputs and both inputs are projected to the internal d model dimensionality of transformer.
The embeddings of joint IDs are used to query the Transformer output, producing one encoder embedding for each of J joints. The J encodings are fed into the 6-block FCR decoder (to match the total number of decoder blocks in ProtoRes) with two heads: one predicting rotation and one predicting unconstrained position. This is similar to the use of Transformer to predict bounding box class IDs and sizes for object detection (Carion et al., 2020). Predictions of rotations and of the root joint are used in the forward kinematics pass, just as in ProtoRes.
Internally, Transformer processes both source and target inputs via self-attention first and then applies the multi-head attention between source and target after self-attention. This results in the output embedding for each skeleton joint that depends on all the input information as well as the learned interactions across all output skeleton joints. The output embedding of the transformer is then decoded to unconstrained position and rotation outputs using a two-headed Fully-Connected residual stack (this is the same architecture as the one used in ProtoRes decoders). Note that this is a well-known Transformer application scheme that has recently been used to achieve SOTA results in object detection, for example (Carion et al., 2020). Table 8 lists the hyperparameter settings for the Transformer baseline. Note that only hyperparameters that are unique to this baseline or different from the ProtoRes defaults appearing in Table 5 are listed.

heading: H ABLATION OF THE DECODER: DETAILS
text: The detailed results of the decoder ablation are shown in Table 9. One block of decoder is much less computationally expensive than one block of encoder. One block of encoder processes N effectors, whereas the decoder deals with the partially defined pose representation collapsed to a vector. Hence the decoder block is N times less expensive. To demonstrate the effectiveness of decoder, we keep all hyperparameters at defaults described in Section 3.2 and vary the number of encoder and decoder blocks in ProtoRes (0 decoder blocks corresponds to a simple linear projection of encoder output). We measure the train time of each configuration on NVIDIA M40 24GB GPU installed on Dell PowerEdge R720 server with two Intel Xeon E5-2667 2.90GHz CPU. The table reveals a few things. First, adding more decoder blocks significantly increases accuracy when the number of encoder blocks is lower (e.g. 3 or 5). When the number of encoder blocks is high (e.g. 7) linear projection provides similar accuracy. Second, using non-trivial decoder is computationally more efficient. For example, the 3 encoder and 3 decoder blocks configuration has comparable accuracy with 5 encoder and 1 decoder blocks, however it is noticeably more compute efficient (5+1 configuration uses 40% more compute time than 3+3).

heading: I ABLATION OF THE PROTOTYPE-SUBTRACT-ACCUMULATE STACKING PRINCIPLE
text: Here we compare the proposed Prototype-Subtract-Accumulate (PSA) residual stacking scheme described in equations (2)-(5) against the ResPointNet (Niemeyer et al., 2019) stacking scheme: Maxpool-Concat Daisy Chain (MCDC). To implement the MCDC stacking proposed by Niemeyer et al. (2019) we setup the experiment as follows.
• Equation (2) is replaced with the concatenation of the output of the previous block, br, with the maxpool of the previous block output along axis 1 (we use batch, effector, channels convention for tensor axes 0,1,2; respectively)
• In equation (4) we only compute br, since fr is not used • Equation (5) is removed
• The final pose embedding is created using the maxpool along axis 1 of br at the last encoder block
• We use decoder with 0 blocks, i.e. only the linear projection at the end to derive both PSA and MCDC results. This is because (i) residual decoder is not part of the original design by Niemeyer et al. (2019), (ii) in this study we focus exclusively on the effects of stacking within the encoder, equalizing all other experimental conditions.
• Hyperparameters of both architectures are taken from Table 5 in Appendix C
We show that our stacking scheme is more accurate and allows stacking deeper networks more effectively. Quantitative results are shown in Table 10. It is clear that the proposed stacking approach provides gain in setups with varying number of encoder blocks. For example, in the case of small number of 3 blocks (computationally efficient setup) our PSA approach is clearly more accurate than MCDC approach of (Niemeyer et al., 2019). Similarly, with 7 blocks (training time is more than two times longer) PSA is again more accurate than MCDC. Moreover, in the case of PSA we see noticeable accuracy improvement while increasing the number of encoder blocks from 5 to 7, whereas MCDC provides almost no additional gain beyond 5 blocks. We conclude that the proposed PSA stacking mechanism is better than MCDC as it provides better accuracy in the computationally efficient configuration and it is significantly more effective at supporting deeper architectures that provide better accuracy in the case of PSA, whereas MCDC accuracy saturates at 5 blocks providing little to no accuracy gain with deeper architectures.

heading: J QUALITATIVE COMPARISON TO TRANSFORMER AND FINALIK BASELINES
text: In this section we provide additional qualitative comparison between ProtoRes and two baselines, the non-learnable baseline FinalIK and the machine learning baseline Transformer. The FinalIK comparison results appear in the top row of Figure 8. The Transformer comparison results appear in the bottom row of Figure 8.
The characteristic feature of FinalIK is that it lacks inductive bias towards realistic poses. Therefore it is very easy to create effector configurations that result in unnatural poses as can be seen in Figure 8. Conversely, creating realistic poses requires significant amount of tuning of the individual joints as most joints of the body act very independently and locally. With FinalIK, it is relatively hard to produce a believable pose with only a few effectors. ProtoRes fills this gap by providing a learned prior for realistic poses. As a result, most of the poses created by ProtoRes look natural, no matter how many effectors are used to steer the pose. In addition, the effector space can be augmented dynamically to capture missing accents, if necessary.
When comparing ProtoRes against Transformer, we can see that Transformer has a less global approach to forming a pose than ProtoRes. This manifests itself in the Tranformer based model having a tendency to create poses in which limbs penetrate each other or the rest of the body (see Figure 8 bottom row, pictures 1,2,3 from the left). Also, Transformer is very good at following the effector inputs and reproducing them in the reconstructed pose — so much that it is willing to sacrifice the overall final pose plausibility at the expense of sticking to the effector, taking the effector guidance very “literally” and missing its re-interpretation given global context. A good example of this behaviour is presented in Figure 8 bottom row, rightmost picture. We see that ProtoRes brings the entire body to the floor and prepares the arms to touch the floor to support the body, all in the attempt to produce a plausible pause of someone looking at the look-at target that is placed relatively low. On the other hand, Transformer takes the look-at effector guidance very literally and locally, willing to break the neck of the pose to look at the provided target and not realizing that the entire body position needs to be changed to accommodate for this rather peculiar configuration of effectors to create a believable pose. In general, our observation is that the Transformer based baseline response to effector inputs tends to be localized. As a consequence, its behaviour has a very pronounced pure
IK flavour to it, at times reminding the behaviour of FinalIK, especially when facing difficult effector configurations. A lot of the time, however, the outputs of ProtoRes and Transformer are comparable and consistent, since both of them develop strong inductive bias towards realistic poses.

heading: K THE EFFECTS OF THE NUMBER AND TYPE OF EFFECTORS
text: The results of the experiment studying the effects of the number of input effectors on the reconstruction accuracy are shown in Figure 9. In this experiment, we sample the inputs to the neural network uniformly at random as in the main text, but now we specifically control the type and the number of effectors that are being provided at the input of the network and we study the evolution of error metrics as a function of the number of provided effectors. A few interesting observations emerge in this experiment. We see that in most cases, errors decrease monotonically as the percentage of joints used as effectors increases, however, for most of them we also observe saturation and the errors do not converge to zero. For the position error and the case of rotation only inputs, we see a U-shape behaviour. Let us reflect on these observations in order.
The case when both position and rotation inputs are provided, forcing error to go down, is intuitively clear, because more effectors provide more information to the input and thus more accurate pose reconstructions are achieved. The saturation of the error may happen for a few plausible reasons, including: (i) the network is trained with effectors whose number is sampled uniformly at random in the range [5,16] (5-25% of joints are used as effectors), clearly above 25% the network is operating in the out-of-distribution regime potentially leading to sub-optimal results, (ii) the network operates as a noisy auto-encoder and it is specifically trained to reconstruct a good, but potentially imperfect pose, therefore we can never expect it to reach perfect reconstruction, even if all joints are provided as inputs. Error saturation is not something we expect to happen for traditional IK systems such as FinalIK, which when provided with enough effectors for a valid pose should be able to recover all parameters of the pose through mathematical equations, resulting in zero errors. At the same time, it is worthwhile noting that (i) the operation with full pose input is a use case neither for FinalIK nor for ProtoRes, in fact the typical use case is to reconstruct full pose from the least number of inputs and (ii) if smaller error is desired with full pose inputs, the network can be trained to address this specific scenario.
The case when only rotation inputs are provided (green curve) is interesting in that we see the rotation errors going down with increasing number of effectors (right pane) and the position errors following
a U-shape (left pane). It appears that the network is able to use more information about rotations to reduce rotation errors, but the task of reconstructing positions from only rotational inputs is too hard. This is not surprising, as rotations of joints bear little information about the absolute position of the skeleton in space, which is necessary to reconstruct the global position of the root joint. To aggravate the situation, errors in the global root joint position will be magnified during the forward kinematics pass used to compute global positions of all joints. It looks like when the network is provided with a relatively small number of rotational inputs (5-25%) it is still able to focus on reconstructing global root joint position, whereas providing more rotational inputs makes errors worse (U-shape on the green line in the left pane). We conjecture that two adverse factors can be at play here: (i) increasingly out-of-distribution operation makes processing large number of rotational inputs less effective and (ii) the network is trained to treat provided effectors as high-priority for reconstruction, therefore it will concentrate on minimizing rotation errors while de-prioritizing positions, eventually rotation errors will decrease at the expense of growing position errors, which is clear comparing green curve in the left and in the right panes. In fact, a similar effect can be observed on the position only effectors curve (blue), with the exception that there is no U-shape on the rotation losses, but there is definitely a lot more saturation in the blue curve on the right (rotation error of position only inputs) than in the blue curve on the left (position error of position only inputs). It looks like position inputs tend to provide more information about rotations to be reconstructed than the rotation inputs do about positions to be reconstructed, which makes sense based on the geometry of the problem.
L VIDEOS AND DEMONSTRATIONS
We provide here descriptions related to each video found in the supplementary materials. The Unity tool that is used to produce demo videos and qualitative pictures shows the raw neural network outputs without applying any post-processing. This is done to ensure fair comparison of different methods and avoid any misleading results due to post-processing. Note that most of the demonstrations are done using a ProtoRes model trained on a large internal dataset not evaluated in this work. One of our demonstrations, described below in Appendix L.5 qualitatively shows some of the most severe impacts of training on ablated datasets.

heading: L.1 PROTORES DEMO
text: The video presents an overview of the integration of ProtoRes as a posing tool inside the Unity game engine, showcasing how different effector types can be manipulated and how they can influence the resulting pose. In this demo, a user-defined configuration is presented in the UI, allowing one to choose which effectors are enabled within that configuration. Note that this configuration could contain more or less effectors of each type, and can be built for specific posing needs. The most generic configuration would present all possible effectors inside each effector type sub-menu.

heading: L.2 POSING FROM IMAGES
text: This video presents screen recordings of a novice user using ProtoRes to quickly prototype poses taken from 2D silhouette images. Note that one can reach satisfactory results in less than a minute in each case, with a relatively low number of manipulations. Note also that fine-tuning the resulting poses can always be achieved by adding more effectors and applying more manipulations.

heading: L.3 LOSS ABLATION
text: This recording shows a setup where different models are used with identical effector setup. Both models use the ProtoRes architecture. On the left, the model uses the total loss presented in Algorithm 1, whereas the right-hand side model uses positional losses only (GPD and IKD positional losses) and both local and global rotational losses, as well as look-at losses are disabled. This demonstration clearly shows how positional constraints, even when respected, do not suffice to produce realistic human poses. Joint rotations have to be modeled as well.

heading: L.4 FINALIK COMPARISON
text: This recording shows a setup where ProtoRes is compared to a full body biped IK system provided by FinalIK RootMotion (2020), with an identical effector setup. Note that FinalIK solves constraints by modifying the current pose, often resulting in smaller changes in the output, when compared to ProtoRes that predicts a full pose at each update. The lack of a learned model of human poses in FinalIK becomes quickly noticeable when manipulating effectors significantly.

heading: L.5 DATASETS COMPARISON
text: In these demonstrations, we showcase how training ProtoRes on different datasets can impact the results. We showcase models trained on the two ablated datasets presented in this work, i.e. miniAnonymous (left) and miniMixamo (right). We also show performance of a model trained on the full Mixamo Adobe Inc. (2020) dataset (center) to qualitatively show how performance can be improved with more training data. In all of these recordings, one can notice differences in the resulting poses, emphasizing the fact that human posing from few effectors, when no extra conditioning signals are used, is an ambiguous task that will be influenced by the training data.
The first sequence makes this fact especially obvious on the finger joints and the head’s look-at direction. The second sequence shows how good data coverage in the training set can significantly impact performance in special or rare effector configurations. Finally, the third sequence shows a similar pattern for look-at targets, where the difference in training data can be noticed in the general posture of the character and the varying levels of robustness with respect to those targets.

heading: L.6 RETARGETING
text: This video shows that ProtoRes trained on one dataset can be successfully applied to multiple characters defined on skeletons on which our model was never trained. Currently, the model is trained on a dataset with a specific skeleton layout with specific bone offsets. In a naı̈ve application scenario, a new skeleton would require retraining the model on a dataset for this specific skeleton, which could be a prohibitively expensive exercise. To overcome this limitation, we show in this video that the model learned using our approach can be retargeted to very different skeletons using conventional retargeting methods via a process that is fully automatic, without changing the model in any way. Thus, our model can be seamlessly used without any retraining with a wide variety of skeletons. In practice this solution allows the use of the same model on characters having similar skeleton in terms of topology and/or morphology and very different in other aspects such as proportions (bone lengths) or bone count. For instance, as our demo shows, it can be used with any humanoid character and still produces valuable results. Transfer of the model across skeletons using more sophisticated ML-based techniques such as fine-tuning, conditioning, domain adaptation is deemed to a be a fruitful future research area, for which our current results form a strong baseline.

heading: L.7 APPLICATION TO THE QUADRUPED SKELETON
text: In this video we demonstrate that ProtoRes can be applied to a completely different type of skeleton (quadruped) and it does not require any additional coding as opposed to FinalIK (no hand-crafted skeleton, effectors, bone chains, pulling, etc). It does require data and re-training however, but does not contain anything specific to humanoids. In fact, we have successfully trained the proposed model on a quadruped (dog) dataset without changing the model or hyperparameters, which demonstrates the generality of the method.

heading: L.8 COMPARISON TO TRANSFORMER
text: In this video we demonstrate qualitative examples demonstrating the basic difference between Transformer and ProtoRes. First, generally speaking, in many cases with challenging effector settings ProtoRes shows much greater robustness to outliers with respect to pose plausibility, whereas in many cases Transformer’s behaviour feels like that of a pure non-learnable IK model. Second, Transformer has a tendency to generate self-intersections between the limbs to satisfy some of the constraints more closely at the expense of the overall pose plausibility. Finally, in response to an extreme look-at constraint Transformer can bend the neck behind the body, which in reality never happens in the data
and would break the neck of a real human. In contrast, ProtoRes keeps the neck naturally oriented when the look-at target is unattainable.
From the theoretical perspective, the fundamental difference between the Transformer framework and the ProtoRes is as follows. ProtoRes creates one global representation of the partially specified pose and then reconstructs the full pose in one shot through a global IKD. On the other hand, Transformer generates embeddings of individual joints for the full output pose, from which a shared IKD reconstructs the local rotation angles for each joint. Transformer has all the ingredients to reproduce processing similar to our approach. However, in reality the Transformer takes a different learning route and (i) learns more localized joint predictions and (ii) learns to more strictly respect local input constraints, even at the expense of creating poses that are not statistically plausible. Combining the global (ProtoRes) and local (Transformer) approaches to derive better hybrid models seems like a promising direction for future work.

heading: L.9 LIMITATIONS
text: The final video shows examples of some specific limitations of the approach that are listed in Appendix M. Namely, we first expose specific consequences of the lack of temporal consistency in our problem formulation, where smoothly moving an effector can cause flickering on some joints, such as the fingers. We also show how between some effector configurations, the character must be flipped completely to stay in a plausible pose, and how it’s possible to place some effectors to reach an invalid pose coming from that flip region of the latent manifold. Finally, we showcase some problematic behaviors that can be caused by extreme look-at targets. In some cases, especially with many other constraints, ProtoRes will tend to produce a plausible pose that will not respect the look-at constraint. In other cases, the extreme look-at target may cause an unrealistic pose, e.g. by causing the character to have an impossible neck rotation. It is interesting to note how invalid poses from look-at effectors tend to happen more often than from other effector types with novice users. We hypothesize that the plausible region of a look-at target, given a current character pose, is less intuitive to grasp than for other effector types. Indeed, the current pose of the character seems to guide more precisely the placement of positional and rotational effectors than look-at effectors, leading more often to configurations outside of the training distribution for look-at effectors.

heading: M LIMITATIONS
text: The limitations of our work can be summarized as follows:
• Constraints are not satisfied exactly, as opposed to the conventional systems. The limitation can be observed in the demo video 4 Final IK comparison.mp4, which compares ProtoRes and FinalIK side-by-side. For example, at seconds 18-21 we can see the chest position is not satisfied exactly by ProtoRes and the generated left foot position traverses the vicinity of the constraint as the user changes the position of the left foot. Another example like this can be seen at seconds 35-41, in which the left foot effector is not satisfied exactly by ProtoRes. It is interesting that for the rather peculiar configuration of effectors, FinalIK produces outputs that tend to severely twist the joints, perhaps beyond the capabilities of an average human. At the same time ProtoRes tends to trade the precision of following individual effectors with the plausibility of the overall pose. This is the price to pay for the ability of the model to inject the data-driven inductive bias that can be used to reconstruct pose from very sparse inputs. This could be mitigated using a conventional solver on top of the trained model. In this case, the model will produce a globally plausible pose, whereas the solver will only do the final pass to strictly satisfy certain constraints. Also, to provide additional flexibility in solving some of the constraints more strictly than the others, our model provides an effector tolerance mechanism that can help the user trade off the strictness of satisfying certain effectors vs. some others.
• Lack of temporal consistency. Our work solves the problem of creating a discrete pose. Therefore, it is limited in how it can be applied to modify an underlying smooth animation clip. For example, we can see flickering of joints (especially fingers) when effectors follow an underlying smooth animation (the finger embedding space is not smooth and has a high
ambiguity). This happens to a smaller degree with the head when it is not constrained with look-at or rotation inputs.
• Exotic poses significantly deviating from the the training data distribution (a common ML/DL problem) may be hard to achieve. For example, the Lotus yoga pose is very hard to achieve with small number of effectors. Extreme or rare effector configurations may not be respected. Extreme look-at targets may not be followed or can cause artifacts in the resulting pose.
• Some effector displacement can cause a complete flip in the final pose as it makes more sense to be e.g. left-oriented or right-oriented to reach a hand position. This is normal, but we can sometimes reach ”in-between” poses on the boundary of the hand effector that causes the flip, leading to weird poses
• We also noted a limitation as ”aiming” poses (holding something in the hands). For example, Finger poses are generally wrong w.r.t. to a gun without additional finger constraints. It may be cumbersome to place hands + look-at for each aiming pose/angle?
• Runtime. In its current state, the model allows interactive real-time rate (about 100 FPS). This is very good for the primary application area of the model in the interactive pose design. However, the current model cannot be used for runtime applications such as driving charachters directly in real-time games, because it would consume too high of a time budget (about 10ms, which is too much to be usable in the game runtime context).
• No contextual input is supported (text description or environment awareness), in particular for finger posing and feet collisions
• Good for realism, but might limit creativity. In particular, no bone stretching support, which is sometimes used by animators to add more expressiveness to non-realistic characters.
• The current model struggles when a large number of fine-grain controls, especially fingers are used simultaneously. Perhaps, a more structured hierarchical approach can be used to enable this functionality.

Review summaries:
This paper develops a system to generate a complete and realistic human pose based on a sparse set of inputs such as look-at direction and end-effector positions. It proposes a prototypical residual network dubbed " ProtoRes" to handle variable input length based on the user-specified parameters. A look-at-loss, randomized weighting scheme, and two datasets based on MoCap are proposed to learn the designed system, and a usable system for authoring realistic human poses is built. Experiments show that the network design and loss outperform baselines such as Transformer and existing Inverse Kinematics methods. The graphical demo built inside Unity also showcases the utility of the learned pose networks. 

This paper focuses on the learnable neural representation of the 3D human pose. A proto-residual network is proposed to construct a full human pose from sparse body part constraints. Several modifications are performed to improve the accuracy of the proposed model, such as look-at loss, two-stage decoder, residual scheme, etc. In the experiment, two new datasets for the static human pose modeling task are conducted and will be released publicly. The experimental results prove the effectiveness of the proposed methods.

This paper proposes a DNN-based framework to handle the challenging task of recovering 3D human poses (joint positions and local rotations) from sparse and variable user inputs (joint positions, orientations of a subset of whole body joints). The paper proposes a neural-network architecture with an encoder to extract pose encodings from partial user inputs and a decoder to generate complete poses from the encoded features. The paper further proposes two new datasets to train and evaluate the proposed models. Experimental results demonstrate the superiority of the proposed framework over previous methods.