Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
Year: 2021

Abstract:
We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

heading: 1 Introduction
text: From a video captured by a single head-mounted wearable camera (e.g., smartglasses, action camera, body camera), we aim to infer the wearer’s global 3D full-body pose and interaction with objects in the scene, as illustrated in Fig. 1. This is important for applications like virtual and augmented reality, sports analysis, and wearable medical monitoring, where third-person views are often unavailable and proprioception algorithms are needed for understanding the actions of the camera wearer. However, this task is challenging since the wearer’s body is often unseen from a first-person view and the
⇤Work done at Carnegie Mellon University.
35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
body motion needs to be inferred solely based on the videos captured by the front-facing camera. Furthermore, egocentric videos usually capture the camera wearer interacting with objects in the scene, which adds additional complexity in recovering a pose sequence that agrees with the scene context. Despite these challenges, we show that it is possible to infer accurate human motion and human-object interaction from a single head-worn front-facing camera.
Egocentric pose estimation can be solved using two different paradigms: (1) a kinematics perspective and (2) a dynamics perspective. Kinematics-based approaches study motion without regard to the underlying forces (e.g., gravity, joint torque) and cannot faithfully emulate human-object interaction without modeling proper contact and forces. They can achieve accurate pose estimates by directly outputting joint angles but can also produce results that violate physical constraints (e.g. foot skating and ground penetration). Dynamics-based approaches, or physics-based approaches, study motions that result from forces. They map directly from visual input to control signals of a human proxy (humanoid) inside a physics simulator and recover 3D poses through simulation. These approaches have the crucial advantage that they output physically-plausible human motion and human-object interaction (i.e., pushing an object will move it according to the rules of physics). However, since no joint torque is captured in human motion datasets, physics-based humanoid controllers are hard to learn, generalize poorly, and are actively being researched [36, 57, 51, 52].
In this work, we argue that a hybrid approach merging the kinematics and dynamics perspectives is needed. Leveraging a large human motion database [29], we learn a task-agnostic dynamics-based humanoid controller to mimic broad human behaviors, ranging from every day motion to dancing and kickboxing. The controller is general-purpose and can be viewed as providing low-level motor skills of a human. After the controller is learned, we train an object-aware kinematic policy to specify the target poses for the controller to mimic. One approach is to let the kinematic model produce target motion only based on the visual input [58, 51, 56]. This approach uses the physics simulation as a post-processing step: the kinematic model computes the target motion separately from the simulation and may output unreasonable target poses. We propose to synchronize the two aspects by designing a kinematic policy that guides the controller and receives timely feedback through comparing its target pose and the resulting simulation state. Our model thus serves as a high-level motion planning module that adapts intelligently based on the current simulation state. In addition, since our kinematic policy only outputs poses and does not model joint torque, it can receive direct supervision from motion capture (MoCap) data. While poses from MoCap can provide an initial-guess of target motion, our model can search for better solutions through trial and error. This learning process, dubbed dynamics-regulated training, jointly optimizes our model via supervised learning and reinforcement learning, and significantly improves its robustness to real-world use cases.
In summary, our contributions are as follows: (1) we are the first to tackle the challenging task of estimating physically-plausible 3D poses and human-object interactions from a single front-facing camera; (2) we learn a general-purpose humanoid controller from a large MoCap dataset and can perform a broad range of motions inside a physics simulation; (3) we propose a dynamics-regulated training procedure that synergizes kinematics, dynamics, and scene context for egocentric vision; (4) experiments on a controlled motion capture laboratory dataset and a real-world dataset demonstrate that our model outperforms other state-of-the-art methods on pose-based and physics-based metrics, while generalizing to videos taken in real-world scenarios.

heading: 2 Related Work
text: Third-person human pose estimation. The task of estimating the 3D human pose (and sometimes shape) from third-person video is a popular research area in the vision community [2, 21, 24, 14, 40, 59, 10, 34, 12, 31, 23, 28, 11, 54, 58], with methods aiming to recover 3D joint positions [25, 46, 14, 34], 3D joint angles with respect to a parametric human model [2, 21, 23, 28], and dense body parts [11]. Notice that all these methods are purely kinematic and disregard physical reasoning. They also do not recover the global 3D root position and are evaluated by zeroing out the body center (root-relative). A smaller number of works factor in human dynamics [42, 43, 38, 49, 4] through postprocessing, physics-based trajectory optimization, or using a differentiable physics model. These approaches can produce physically-plausible human motion, but since they do not utilize a physics simulator and does not model contact, they can not faithfully model human-object interaction. SimPoE [58], a recent work on third-person pose estimation using simulated character control, is most related to ours, but 1) trains a single and dataset-specific humanoid controller per dataset; 2) designs the kinematic model to be independent from simulation states.
Input: Video sequence:
Intialization Kinematic Policy
Camera Poses Image Features Initialization Pose Object State
Initialization
Per-step input:
Egocentric human pose estimation. Compared to third-person human pose estimation, there are only a handful of attempts at estimating 3D full body poses from egocentric videos due to the ill-posed nature of this task. Most existing methods still assume partial visibility of body parts in the image [48, 39, 53], often through a downward-facing camera. Among works where the human body is mostly not observable [20, 55, 56, 33], Jiang et al. [20] use a kinematics-based approach where they construct a motion graph from the training data and recover the pose sequence by solving the optimal pose path. Ng et al. [33] focus on modeling person-to-person interactions from egocentric videos and inferring the wearer’s pose conditioning on the other person’s pose. The works most related to ours are [55, 56, 19] which use dynamics-based approaches and map visual inputs to control signals to perform physically-plausible human motion inside a physics simulation. They show impressive results on a set of noninteractive locomotion tasks, but also observe large errors in absolute 3D position tracking–mapping directly from the visual inputs to control signals is a noisy process and prone to error accumulation. In comparison, our work jointly models kinematics and dynamics, and estimates a wider range of human motion and human-object interactions while improving absolute 3D position tracking. To the best of our knowledge, we are the first approach to estimate the 3D human poses from egocentric video while factoring in human-object interactions.
Humanoid control inside physics simulation. Our work is also connected to controlling humanoids to mimic reference motion [36, 37, 5, 57, 52, 15, 6] and interact with objects [5, 30] inside a physics simulator. The core motivation of these works is to learn the necessary dynamics to imitate or generate human motion in a physics simulation. Deep RL has been the predominant approach in this line of work since physics simulators are typically not end-to-end differetiable. Goal-oriented methods [5, 30, 1] does not involve motion imitation and are evaluated on task completion (moving an object, sitting on a chair, moving based on user-input etc.). Consequently, these frameworks only need to master a subset of possible motions for task completion. People, on the other hand, have a variety of ways to perform actions, and our agent has to follow the trajectory predefined by egocentric videos. Motion imitation methods [36, 37, 52, 57, 52] aim to control characters to mimic a sequence of reference motion, but have been limited to performing a single clip [36, 37, 52, 57] or high-quality MoCap [52] motion (and requires fine-tuning to generalize to other motion generators). In contrast, our dynamics controller is general and can be used to perform everyday motion and human-object interactions estimated by a kinematic motion estimator without task-specific fine-tuning.

heading: 3 Method
text: The problem of egocentric pose estimation can be formulated as follows: from a wearable camera footage I1:T , we want to recover the wearer’s ground truth global 3D poses bq1:T . Each pose bq t , (brpos t , brrot t ,bj rot t ) consists of the root position brpos t , root orientation brrot t , and body joint angles bj rot t of the human model. Here we adopt the popular SMPL [27] human model and the humanoid we
use in physics simulation is created from the kinematic structure and mean body shape defined by SMPL. Our framework first learns a Universal Humanoid Controller (UHC) from a large MoCap dataset (Sec. 3.1). The learned UHC can be viewed as providing the lower level muscle skills of a real human, trained by mimicking thousands of human motion sequences. Using the trained UHC, we learn our kinematic policy (Sec. 3.2) through dynamics-regulated training (Sec. 3.3). At the test time, the kinematic policy provides per-step target motion to the UHC, forming a closed-loop system that operates inside the physics simulation to control a humanoid. The result of the UHC and physics simulation is then used as input to the kinematic policy to produce the next-frame target motion, as depicted in Fig. 2. As a notation convention, we usee· to denote kinematic quantities (obtained without using physics simulation),b· to denote ground truth quantities, and normal symbols without accents to denote quantities from the physics simulation.

heading: 3.1 Dynamics Model - Universal Humanoid Controller (UHC)
text: To learn a task-agnostic dynamics model that can be tightly integrated with a kinematic model, we design our controller’s state space to only rely on the current simulated pose q
t and target posed bq
t+1 and remove all phase or sequence-level information found in prior arts [57, 36, 37]. This design allows us to train on an order of magnitude larger dataset of human motion [29] with only pose information and significantly improve our models’ ability to mimic diverse and unseen motions. Formally, we model controlling a humanoid to follow a reference motion bq1:T as a Markov Decision Process (MDP) defined as a tuple M = hS,A, Pphysics, R, i of states, actions, transition dynamics, reward function, and a discount factor. The state S, reward R, and transition dynamics Pphysics are provided by the physics simulator, while action A is computed by the policy ⇡UHC. At each timestep t, the agent in state st takes an action sampled from the policy ⇡UHC(at|st, bqt+1) while the environment generates the next state st+1 and reward rt. We employ Proximal Policy Optimization (PPO) [41] to find the optimal policy ⇡⇤UHC that maximizes the expected discounted return E[ P T t=1 t 1 rt].
State. The state st , (qt, q̇t) of the humanoid contains the character’s current pose qt and joint velocity q̇
t . Here, the state st encapsulates the humanoid’s full physical state at time step t. It only
includes information about the current frame (q t , q̇ t ) and does not include any extra information, enabling our learned controller to be guided by a target pose only.
Action. The action at specifies the target joint angles for the proportional derivative (PD) controller [45] at each degree of freedom (DoF) of the humanoid joints except for the root (pelvis). We use the residual action representation [15]: qd
t = bq t+1 + at, where qdt is the final PD target, at is the output of the control policy ⇡UHC, and bqt+1 is the target pose. The torque to be applied at joint i is: ⌧ i = kp (qd
t q t ) kd q̇ t where kp and kd are manually specified gains and is the
element-wise multiplication. As observed in prior work [57, 58], allowing the policy to apply external residual forces ⌘
t to the root helps stabilizing the humanoid, so our final action is at , ( eqdt ,⌘t).
Policy. The policy ⇡UHC(at|st, bqt+1) is represented by a Gaussian distribution with a fixed diagonal covariance matrix ⌃. We first use a feature extraction layer Ddiff(bqt+1, qt) to compute the root and joint offset between the simulated pose and target pose. All features are then translated to a root-relative coordinate system using an agent-centic transform T AC to make our policy orientationinvariant. We use a Multi-layer Perceptron (MLP) as our policy network to map the augmented state T AC q t , q̇ t , bq t+1, Ddiff(bqt+1, qt) to the predicted action at.
Reward function. For UHC, the reward function is designed to encourage the simulated pose q t to better match the target pose bq t+1. Since we share a similar objective (mimic target motion), our reward is similar to Residual Force Control [57].
Training procedure. We train our controller on the AMASS [29] dataset, which contains 11505 high-quality MoCap sequences with 4000k frame of poses (after removing sequences involving human-object interaction like running on a treadmill). At the beginning of each episode, a random fixed length sequence (300 frames) is sampled from the dataset for training. While prior works [52, 51] uses more complex motion clustering techniques to sample motions, we devise a simple yet empirically effective sampling technique by inducing a probability distribution based on the value function. For each pose frame bq
j in the dataset, we first compute an initialization state sj1:
sj1 , bq j ,0 , and then score it using the value function to access how well the policy can mimic the sequence bq j:T starting from this pose: V (s j
1, bqj+1) = vj . Intuitively, the higher vj is, the more confident our policy is in mimicing this sequence, and the less often we should pick this frame. The
probability of choosing frame j, comparing against all frames J in the AMASS dataset, is then P (bq
j ) = exp( vj/⌧)PJ
i exp( vi/⌧) where ⌧ is the sampling temperature. More implementation details about
the reward, training and evaluation of UHC can be found in Appendix C.

heading: 3.2 Kinematic Model – Object-aware Kinematic Policy
text: To leverage the power of our learned UHC, we design an auto-regressive and object-aware kinematic policy to generate per-frame target motion from egocentric inputs. We synchronize the state space of our kinematic policy and UHC such that the policy can be learned with or without physics simulation. When trained without physics simulation, the model is purely kinematic and can be optimized via supervised learning; when trained with a physics simulation, the model can be optimized through a combination of supervised learning and reinforcement learning. The latter procedure, coined dynamics-regulated training, enables our model to distill human dynamics information learned from large-scale MoCap data into the kinematic model and learns a policy more robust to convariate shifts. In this section, we will describe the architecture of the policy itself and the training through supervised learning (without physics simulation).
Scene context modelling and initialization. To serve as a high-level target motion estimator for egocentric videos with potential human-object interaction, our kinematic policy needs to be objectaware and grounded with visual input. To this end, given an input image sequence I1:T , we compute the initial object states eo1, image features 1:T , and camera trajectory eh1:T as inputs to our system. The object states, eot , (eoclst , eopost , eorott ), is modeled as a vector concatenation of the main objectof-interest’s class eocls
t , 3D position eopos t , and rotation eorot t . eot is computed using an off-the-shelf
object detector and pose estimator [17]. When there are no objects in the current scene (for walking and running etc.), the object states vector is set to zero. The image features 1:T contains crucial information about the wearer’s movement and is computed using an optical flow extractor [44] and ResNet [13]. Since visual input can be noisy and modern smartglasses and bodycams are often equipped with built-in SLAM or Visual Inertial Odometry (VIO) [50, 9] capabilities, we utilize this additional data modalitiy and compute the 6DoF camera pose from input Images. Using an off-the-shelf VIO method [16], we extracts camera trajectory as: eht , (eh pos t , eh rot t ) (position eh pos t and orientation eh rot t
) . Notice that the camera trajectory eht information is crucial and significantly improve the performance of our framework as shown in our ablation studies (Sec. 4.2).
To provide our UHC with a plausible initial state for simulation, we estimate eq1 from the scene context features 1:T , eo1:T ,and eh1:T . We use an Gated Recurrent Unit (GRU) [7] based network to regress the initial agent pose eq1. Combining the above procedures, we obtain the context modelling and initialization model ⇡initKIN: [eq1, 1:T , eo1, eh1:T ] = ⇡initKIN(I1:T ). Notice that to constrain the ill-posed problem of egocentric pose estimation, we assume known object category, rough size, and potential mode of interaction. We use these knowledge as a prior for our per-step model for pose estimation.
Training kinematic policy via supervised learning. After initialization, we use the estimated first-frame object pose eot and human pose eqt to initialize the physics simulation. All subsequent object movements are a result of human-object interation and simulation. At each time step, we use a per-step model ⇡stepKIN to compute the next frame pose based on the next frame observations: we obtain an egocentric input vector ect through the agent-centric transformation function ect = T AC(eqt, t+1, got+1, eht+1) where c̃t , (er0rott ,ej rot t , t+1, eo0t+1, eh 0 t+1) contains the current agentcentric root orientation er0rot t , joint angles ej rot t , image feature for next frame t+1, object state eo 0 t+1, and camera pose eh 0 t+1. From ect, the kinematic policy ⇡ step KIN computes the root angular velocity ewt, linear velocity evt, and next frame joint rotation ej rot t+1. The next frame pose is computed through a finite integration module T finite with time difference t = 1/30s:
e!t, evt,ej rot t+1 = ⇡ step KIN(ect), eqt+1 = T finite(e!t, evt,ej rot t+1, eqt). (1)
When trained without physics simulation, we auto-regressively apply the kinematic policy and use the computed eq
t+1 as the input for the next timestep. This procedure is outlined at Alg. 1. Since all mentioned calculations are end-to-end differentiable, we can directly optimize our ⇡initKIN and ⇡ step KIN
Algorithm 1 Learning kinematic policy via supervised learning.
1: Input: Egocentric videos I and paired ground truth motion dataset bQ 2: while not converged do 3: M SL ; . initialize sampling memory 4: while M not full do 5: I1:t random sequence of images I1:T from the dataset I 6: eq1, 1:T , eo1, eh1:T = ⇡initKIN(I1:T ) . compute scene context and initial pose 7: for i 1...T do 8: ect T AC(eqt, t+1, eot+1, eht+1) . compute agent-centric input features 9: eqt+1 T finite(⇡ step KIN(ect), eqt)
10: store (eqt, bqt) into memory M SL 11: end for 12: end while 13: ⇡stepKIN,⇡ init KIN supervised learning update using data collected in M SL for 10 epoches. 14: end while
through supervised learning. Specifically, given ground truth bq1:T and estimated eq1:T pose sequence, our loss is computed as the difference between the desired and ground truth values of the following quantities: agent root position (brpos
t vs erpos t ) and orientation (brrot t vs errot t ), agent-centric object position
(bo0pos t vs eo0pos t ) and orientation (bo0rot t vs eo0rot t ), and agent joint orientation (bj rot t vs ej rot t
) and position (bj pos t vs ej pos t , computed using forward kinematics):
LSL = TX
i=1
kerrott brrott k2+kerpost br pos t k 2+keo0rott bo0rott k2+keo0post bo 0pos t k 2+kej rot t bj rot t k 2+kej pos t bj pos t k 2. (2)

heading: 3.3 Dynamics-Regulated Training
text: To tightly integrate our kinematic and dynamics models, we design a dynamics-regulated training procedure, where the kinematic policy learns from explicit physics simulation. In the procedure described in the previous section, the next-frame pose fed into the network is computed through finite integration and is not checked by physical laws: whether a real human can perform the computed pose is never verified. Intuitively, this amounts to mentally think about moving in a physical space without actually moving. Combining our UHC and our kinematic policy, we can leverage the prelearned motor skills from UHC and let the kinematic policy act directly in a simulated physical space to obtain feedback about physical plausibility. The procedure for dynamics-regulated training is outlined in Alg. 2. In each episode, we use ⇡initKIN and ⇡ step KIN as in Alg. 1, with the key distinction being: at the next timestep t+ 1, the input to the kinematic policy is the result of UHC and physics simulation q t+1 instead of eqt+1. qt+1 explicitly verify that the eqt+1 produced by the kinematic policy can be successfully followed by a motion controller. Using q t+1 also informs our ⇡ step KIN of the current humanoid state and encourages the policy to adjust its predictions to improve humanoid stability.
Dynamics-regulated optimization. Since the physics simulation is not differentiable, we cannot directly optimize the simulated pose q
t ; however, we can optimize q t through reinforcement learning
and eq t through supervised learning. Since we know that bq t is a good guess reference motion for UHC, we can directly optimize eq
t via supervised learning as done in Sec. 3.2 using the loss defined in Eq. 2.
Since the data samples are collected through physics simulation, the input q t is physically-plausible and more diverse than those collected purely through auto-regressively applying ⇡stepKIN in Alg. 1. This way, our dynamics-regulated training procedure performs a powerful data augmentation step, exposing ⇡stepKIN with diverse states collected from simulation.
However, MoCap pose bq t is imperfect and can contain physical violations itself (foot-skating, penetration etc.), so asking the policy ⇡stepKIN to produce bqt as reference motion regardless of the current humanoid state can lead to instability and cause the humanoid to fall. The kinematic policy should adapt to the current simulation state and provide reference motion eq
t that can lead to poses
similar to bq t yet still physically-plausible. Such behavior will not emerge through supervised learning and require trial and error. Thus, we optimize ⇡stepKIN through reinforcement learning and reward maximization. We design our RL reward to have two components: motion imitation and dynamics self-supervision. The motion imitation reward encourages the policy to match the computed camera
Algorithm 2 Learning kinematic policy via dynamics-regulated training.
1: Input: Pre-trained controller ⇡UHC, egocentric videos I , and paired ground truth motion dataset bQ 2: Train ⇡initKIN, ⇡ step KIN using Alg. 1 for 20 epoches (optional). 3: while not converged do 4: M dyna ; . initialize sampling memory 5: while M dyna not full do 6: I1:t random sequence of images I1:T 7: q1 eq1, 1:T , eo1, eh1:T ⇡initKIN(I1:T ) . compute scene context and initial pose 8: s1 (q1, q̇1) . compute intial state for simulation 9: for i 1...T do 10: ct T AC(qt, t+1, eot+1, eht+1) . compute agent-centric features using simulated pose qt 11: eqt+1 ⇠ T finite(⇡ step KIN(ct), qt) . sample from ⇡ step KIN as a guassian policy 12: st (qt, q̇t) 13: st+1 Pphysics(st+1|st,at), at ⇡UHC(at|st, eqt+1) . phyics simulation using ⇡UHC 14: qt+1 st+1, rKINt reward from Eq. 3 . extract reward and qt+1 from simulation 15: store (st,at, rt, st+1, bqt, eqt+1) into memory M dyna 16: end for 17: end while 18: ⇡stepKIN Reinforcement learning updates using experiences collected in M dyna for 10 epoches. 19: ⇡initKIN, ⇡ step KIN Supervised learning update using experiences collected in M dyna for 10 epoches. 20: end while
trajectory eht and MoCap pose bqt, and serves as a regularization on motion imitation quality. The dynamics self-supervision reward is based on the insight that the disagreement between eq
t and q t
contains important information about the quality and physical plausibility of eq t : the better eq t is, the easier it should be for UHC to mimic it. Formally, we define the reward for ⇡stepKIN as:
rt = whpe 45.0(khpost eh pos t k 2) + whqe 45.0(khrott eh rot t k 2) + wgtjve 0.005(kj̇rott ċ j rot t k 2)+
w gt jr e 50.0(kjrott bj rot t k 2) + wdynajr e 50.0(kjrot ejrott k 2) + wdynajp e 50.0(kjpost ej pos t k 2) ,
(3)
whp, whq are weights for matching the extracted camera position eh pos t and orientation eh rot t ; wgt jr , w gt jv are for matching ground truth joint angles bj rot and angular velocities ċ j rot t . wdynajr , w dyna jp are weights for the dynamics self-supervision rewards, encouraging the policy to match the target kinematic joint angles ej rot t and positions ej pos t to the simulated joint angles jrot t and positions jpos t
. As demonstrated in Sec. 4.2, the RL loss is particularly helpful in adapting to challenging real-world sequences, which requires the model to adjust to domain shifts and unseen motion.
Test-time. At the test time, we follow the same procedure outlined in Alg 2 and Fig.2 to roll out our policy to obtain simulated pose q1:T given a sequence of images I1:T . The difference being instead of sampling from ⇡stepKIN(ct) as a Guassian policy, we use the mean action directly.

heading: 4 Experiments
text: Datasets. As no public dataset contains synchronized ground-truth full-body pose, object pose, and egocentric videos with human-object interactions, we record two egocentric datasets: one inside a MoCap studio, another in the real-world. The MoCap dataset contains 266 sequences (148k frames) of paired egocentric videos and annotated poses. It features one of the five actions: sitting down on a chair, avoiding obstacles, stepping on a box, pushing a box, and generic locomotion [55] (walking, running, crouching) recorded using a head-mounted GoPro. Each action has around 50 sequences with different starting position and facing, gait, speed etc. We use an 80–20 train test data split on this MoCap dataset. The real-world dataset is only for testing purpose and contains 183 sequences (50k frames) of an additional subject performing similar actions in an everyday setting wearing a head-mounted iPhone. For both datasets, we use different objects and varies the object 6DoF pose for each capture take. Additional details (diversity, setup etc.) can be found in Appendix D.
Evaluation metrics. We use both pose-based and physics-based metrics for evaluation. To evaluate the 3D global pose accuracy, we report the root pose error (Eroot) and root-relative mean per joint position error [23] (Empjpe). When ground-truth root/pose information is unavailable (for real-world
dataset), we substitute Eroot with Ecam to report camera pose tracking error. We also employ four physics based pose metrics: acceleration error (Eacc), foot skating ( FS), penetration (PT), and interaction success rate (Sinter). Eacc (mm/frame2) compares the ground truth and estimated average joint acceleration; FS (mm) is defined the same as in Ling et al. [26]; PT (mm) measures the average penetration distance between our humanoid and the scene (ground floor and objects). Notice that our MoCap dataset has an penetration of 7.182 mm and foot sliding of 2.035 mm per frame, demonstrating that the MoCap data is imperfect and may not serve as the best target motion. Sinter is defined as whether the objects of interest has been moved enough (pushing and avoiding) or if desired motion is completed (stepping and sitting). If the humanoid falls down at any point, Sinter = 0. For a full definition of our evaluation metrics, please refer to Appendix B.
Baseline methods. To show the effectiveness of our framework, we compare with the previous state-of-the-art egocentric pose estimation methods: (1) the best dynamics-based approach EgoPose [56] and (2) the best kinematics-based approach PoseReg, also proposed in [56]. We use the official implementation and augment their input with additional information ( eot and eht) for a fair comparison. In addition, we incorporate the fail-safe mechanism [56] to reset the simulation when the humanoid loses balance to ensure the completion of each sequence (details in Appendix B).
Implementation details. We use the free physics simulator MuJoCo [47] and run the simulation at 450 Hz. Our learned policy is run every 15 timesteps and assumes all visual inputs are at 30 Hz. The humanoid follows the kinematic and mesh definition of the SMPL model and has 25 bones and 76 DoF. We train our method and baselines on the training split (202 sequences) of our MoCap dataset. The training process takes about 1 day on a RTX 2080-Ti with 35 CPU threads. After training and the initialization step, our network is causal and runs at 50 FPS on an Intel desktop CPU. For more implementation details and network architecture, please refer to Appendix B and C.

heading: 4.1 Results
text: MoCap dataset results. Table 1 shows the quantitative comparison of our method with the baselines. All results are averaged across five actions and all models have access to the same inputs. We observe that our method, trained either with supervised learning or dynamics-regulated, outperform the two state-of-the-art methods across all metrics. Not surprisingly, our purely kinematic model performs the best on pose-based metrics, while our dynamics-regulated trained policy excels at the physics-based metrics. Comparing the kinematics-only models we can see that our method has a much lower (79.4% error reduction) root and joint position error (62.1% error reduction) than PoseReg, which shows that our object-aware and autoregressive design of the kinematic model can better utilize the provided visual and scene context and avoid compounding errors. Comparing with the dynamicsbased methods, we find that the humanoid controlled by EgoPose has a much larger root drift, often falls down to the ground, and has a much lower success rate in human-object interaction (48.4 % vs 96.9%). Upon visual inspection in Fig. 3, we can see that our kinematic policy can faithfully produce human-object interaction on almost every test sequence from our MoCap dataset, while PoseReg
and EgoPose often miss the object-of-interest (as can be reflected by the large root tracking error). Both of the dynamics-based methods has smaller acceleration error, foot skating, and penetration; some even smaller than MoCap (which has 2 mm FS and 7mm PT). Notice that our joint position error is relatively low compared to state-of-the-art third-person pose estimation methods [23, 24, 28] due to our strong assumption about known object of interest, its class, and potential human-object interactions, which constrains the ill-posed problem pose estimation from just front-facing cameras.
Real-world dataset results. The real-world dataset is far more challenging, having similar number of sequences (183 clips) as our training set (202 clips) and recorded using different equipment, environments, and motion patterns. Since no ground-truth 3D poses are available, we report our results on camera tracking and physics-based metrics. As shown in Table 1, our method outperforms the baseline methods by a large margin in almost all metrics: although EgoPose has less foot-skating (as it also utilizes a physics simulator), its human-object interaction success rate is extremely low. This can be also be reflected by the large camera trajectory error, indicating that the humanoid is drifting far away from the objects. The large drift can be attributed to the domain shift and challenging locomotion from the real-world dataset, causing EgoPose’s humanoid controller to accumulate error and lose balance easily. On the other hand, our method is able to generalize and perform successful human-object interactions, benefiting from our pretrained UHC and kinematic policy’s ability to adapt to new domains and motion. Table 1 also shows a success rate breakdown by action. Here we can see that “stepping on a box" is the most challenging action as it requires the humanoid lifting its feet at a precise moment and pushing itself up. Note that our UHC has never been trained on any stepping or human-object interaction actions (as AMASS has no annotated object pose) but is able to perform these action. As motion is best seen in videos, we refer readers to our supplementary video.
4.2 Ablation Study
To evaluate the importance of our components, we train our kinematic policy under different configurations and study its effects on the realworld dataset, which is much harder than the MoCap dataset. The results are summarized in Table 2. Row 1 (R1) corresponds to training the kinematic policy only with Alg. 1 only and use UHC to mimic the target kinematic motion as a post-processing step. Row 2 (R2) are the results
of using dynamics-regulated training but only performs the supervised learning part. R3 show a variant trained without the estimated camera pose from VIO. Comparing R1 and R2, the lower interaction success rate (73.2% vs 80.9%) indicates that exposing the kinematic policy to states from the physics simulation serves as a powerful data augmentation step and leads to a model more robust to real-world scenarios. R2 and R4 show the benefit of the RL loss in dynamics-regulated training: allowing the kinematic policy to deviate from the MoCap poses makes the model more adaptive and achieves higher success rate. R3 and R4 demonstrate the importance of intelligently incorporating extracted camera pose as input: visual features
t can be noisy and suffer from domain shifts, and
using techniques such as SLAM and VIO to extract camera poses as an additional input modality
can largely reduce the root drift. Intuitively, the image features computed from optical flow and the camera pose extracted using VIO provides similar set of information, while VIO provides a cleaner information extraction process. Note that our kinematic policy without using extracted camera trajectory outperforms EgoPose that uses camera pose in both success rate and camera trajectory tracking. Upon visual inspection, the humanoid in R3 largely does not fall down (compared to in EgoPose) and mainly attributes the failure cases to drifting too far from the object.

heading: 5 Discussions

heading: 5.1 Failure Cases and Limitations
text: Although our method can produce realistic human pose and human-object interaction estimation from egocentric videos, we are still at the early stage of this challenging task. Our method performs well in the MoCap studio setting and generalizes to real-world settings, but is limited to a predefined set of interactions where we have data to learn from. Object class and pose information is computed by off-the-shelf methods such as Apple’s ARkit [17], and is provided as a strong prior to our kinematic policy to infer pose. We also only factor in the 6DoF object pose in our state representation and discard all other object geometric information. The lower success rate on the real-world dataset also indicates that our method still suffers from covariate shifts and can become unstable when the shift becomes too extreme. Our Universal Humanoid Controller can imitate everyday motion with high accuracy, but can still fail at extreme motion. Due to the challenging nature of this task, in this work, we focus on developing a general framework to ground pose estimation with physics by merging the kinematics and dynamics aspects of human motion. To enable pose and human-object interaction estimation for arbitrary actions and objects, better scene understanding and kinematic motion planning techniques need to be developed.

heading: 5.2 Conclusion and Future Work
text: In this paper, we tackle, for the first time, estimating physically-plausible 3D poses from an egocentric video while the person is interacting with objects. We collect a motion capture dataset and real-world dataset to develop and evaluate our method, and extensive experiments have shown that our method outperforms all prior arts. We design a dynamics-regulated kinematic policy that can be directly trained and deployed inside a physics simulation, and we purpose a general-purpose humanoid controller that can be used in physics-based vision tasks easily. Through our real-world experiments, we show that it is possible to estimate 3D human poses and human-object interactions from just an egocentric view captured by consumer hardware (iPhone). In the future, we would like to support more action classes and further improve the robustness of our method by techniques such as using a learned motion prior. Applying our dynamics-regulated training procedure to other vision tasks such as visual navigation and third-person pose estimation can also be of interest.
Acknowledgements: This project was sponsored in part by IARPA (D17PC00340), and JST AIP Acceleration Research Grant (JPMJCR20U1).

Review summaries:
This paper proposes a method for pose estimation from first-person camera view. The work builds on a universal controller that learns to imitate a wide variety of behaviors. A kinematic policy is then either trained fully via supervised learning or in a hybrid manner, using a combination of supervised learning and reinforcement learning. The reinforcement learning optimization consists of a motion imitation and a physics-based objective. Specifically, the universal controller’s output is utilized to regularize the kinematic policy to become more physically plausible. The main contributions are a universal controller and the dynamics-regulated training procedure for egocentric pose estimation in human-object interaction scenarios. 

This paper looks at the problem of estimating physically-plausible 3D human motion and environment interactions from an egocentric RGB video. It proposes to combine the advantages of kinematic and dynamics-based approaches: first, a physics-based dynamics policy (UHC) is trained to imitate motion capture data; then a kinematic policy is trained to predict a target pose for UHC based on RGB and object & camera pose inputs. Experiments show that the combination of dynamics and kinematics outperforms either individually by a large margin, and that the method can reasonably predict a small set of person-object interactions. 

In this paper, the authors presents a new method for egocentric physically-plausible 3d human motion estimation. Specially, they designed a general-purpose humanoid controller (UHC) and a dynamics-regulated kinematic policy that can be directly trained and deployed inside a physics simulation. The proposed universal humanoid controller is trained on large-scale Mocap datasets and is for general-purpose. In the experiments, the authors evaluate their proposed method on both Mocap dataset and their collected real dataset. The authors show good performance improvements compared with PoseReg and EgoPose, as well as good imitation success rate compared with DeepMimic on Human3.6M.

This paper presents a method for estimating the egocentric 3D pose of a person moving in a scene. The main idea is to build a dynamics-driven human motion controller which is provided target poses to mimic by a kinematics estimator and information about scene objects. Experiments show that improved performance compared to previous approaches.