Dynamics-regulated kinematic policy for egocentric pose estimation
Authors: Zhengyi Luo
Ryo Hachiuma
Ye Yuan
Kris M. Kitani
Venue: NeurIPS 2021 Poster (2021)
Keywords: Egocentric human pose estimation
Physics Simulation
augmented reality
humanoid control
kinematics and dynamics

Abstract: We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
Zhengyi Luo
Ryo Hachiuma
Ye Yuan'
Kris Kitani
Carnegie Mellon University
Keio University
https://zhhegyiluo .gihhu.. ho/projject//kinpooo
Input: Egocentric video
Output: interaction
Reference:
Third-person view
Figure 1:
From egocentric videos, we infer physically-plausible 3D human pose and human- object interaction.
Abstract
We propose method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamicx-regulate train- ing. At each timestep, a kinematic model is used to provide target pose using video evidence and simulation state. Then, prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose in- structed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factor- ing in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, demonstrate for the first time, the ability to estimate prrsically-plausibbe interactions using single wearable camera. We evaluate egocentric pose estimation method both controlled laboratory settings and real-world scenarios.
3D human-object
Introduction
From video captured by single head-mounted wearable camera (e.g., smartglasses, action camera, body camera), we aim to infer the wearer's global 3D full-body pose and interaction with objects in the scene, as illustrated in Fig. 1. This important for applications like virtual and augmented reality, sports analysis, and wearable medical monitoring, where third-person views are often unavailable and proprioception algorithms are needed for understanding the actions of the camera wearer. However, this task is challenging since the wearer's 's body is often unseen from first-person view and the
Work done at Carnegie Mellon University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

body motion needs to be inferred solely based on the videos captured by the front-facing camera. Furthermore, egocentric videos usually capture the camera wearer interacting with objects in the scene, which adds additional complexity in recovering pose sequence that agrees with the scene context. Despite these challenges, we show that it is possible to infer accurate human motion and human-object interaction from single read-worn front-facing camera.
Egocentric pose estimation can be solved using two different paradigms: (1) a kinematics perspective and (2) a dynamics perspective. Kinematics-based approaches study motion without regard to the underlying forces (e.g., gravity, joint torque) and cannot faithfully emulate human-object interaction without modeling proper contact and forces. They can achieve accurate pose estimates by directly outputting joint angles but can also produce results that violate physical constraints (e.g. foot skating and ground penetration). Dynamics-batt -baiics---ddddd approaches, or physics shariss--bdddd approaches, study motions that result from forces. They map directly from visual input to control signals of a human proxy (humanoid) inside physics simulator and recover 3D poses through simulation. These approaches have the crucial advantage that they output weassically-plasiiii
human motion and human-object interaction (i.e.. pushing an object will move it according to the rules of physics). However, since no joint torque is captured in human motion datasets, physics-based humanoid controllers are hard to learn, generalize poorly, and are actively being researched [36, 57, 51, 52].
In this work, we argue that hybrid approach merging the kinematics and dynamics perspectives is needed. Leveraging a large human motion database [29], we learn a task-agnostic dynamics based humanoid controller to mimic broad human behaviors, ranging from every day motion to dancing and kickhoxing. The controller is general-purpon and can be viewed as providing low-level motor skills of a human. After the controller learned, we train an object-aware kinematic policy to specify the target poses for the controller to mimic. One approach is to let the kinematic model produce target motion only based on the visual input [58, 51, 56]. This approach uses the physics simulation as a post-processing step: the kinematic model computes the target motion separately from the simulation and may output unreasonable target poses. We propose to synchronize the two aspects by designing a kinematic policy that guides the controller and receives timely feedback through comparing its target pose and the resulting simulation state. Our model thus serves as a high-level motion planning module that adapts intelligently based the current simulation state. In addition, since our kinematic policy only outputs poses and does not model joint torque, it can receive direct supervision from motion capture (MoCap) data. While poses from MoCap can provide initial-guest of target motion, our model can search for better solutions through trial and error. This learning process, dubbed dynamics-regulater training, jointly optimizes our model via supervised learning and reinforcement learning, and significantly improves its robustness to real-world use cases.
In summary, our contributions are as follows: (1) we are the first to tackle the challenging task of estimating mhysically-plausible 3D poses and human-object interactions from a single front-facing camera; (2) learn general-purpose purpose humanoid controller from large MoCap dataset and can perform a broad range of motions inside a physics simulation; (3) we propose a synamics--egulatee training procedure that synergizes kinematics, dynamies, and scene context for egocentric vision; (4) experiments on a controlled motion capture laboratory dataset and a real-world dataset demonstrate that our model outperforms other state-of-the-arr methods on pose-based and physics-basee based metrics, while generalizing to videos taken in real-world scenarios.
2 Related Work
Third-person human pose estimation. The task of estimating the 3D human pose (and sometimes
shape) from third-person video is popular research area in the vision community [2, 21, 24, 14, 40, 59, 10, 34, 12, 31, 23, 28, 11, 54, 58), with methods aiming to recover 3D joint positions [25, 46, 14, 34], 3D joint angles with respect to parametric human model [2. 21, 23, 28], and dense body parts [11]. Notice that all these methods are purely kinematic and disregard physical reasoning. They also do not recover the global 3D root position and are evaluated by zeroing out the body center (reot-relative). A smaller number of works factor in human dynamics [42, 43, 38, 49, 4] through postprocessing, physics- based trajectory optimization, or using a differentiable physics model. These approaches can produce utilize a physics simulator and does not model contact, they can not faithfully model human-object interaction. SimPoE [58], recent work on third-person pose estimation using simulated character control, most related to ours, but 1) trains a single and dataset-specifee humanoid controller per dataset; 2) designs the kinematic model to be independent from simulation states.
plaauubbe human motion, but since they do not

---

Input: Video sequence
I;
Fr
Inputt Video sequence:
/
Fr
(srr
Intadiation Polloy
Intralication Pose R
Onject
Comera Poses h
Festures $1.
Ø1
Inmalizatioo
Per-stop input:
Oiel
Tacle Sug dett featl
Ageniccniine Transfomnaion
Kinematic Pallcy
Per-aley
Pinzs
kinemadee pasa
Aggenticertiic inpurt
Stiuuistion State:
4T
Intialize a
(.) sleaahha
Ruse Lot
Simustted yostes 9.+1 Oc+1
Physios
Action
01
Univernal Harmanotd
Controttrr
Per-Step updale
Output: pysically-ppausilae poses
Outpun: Physically-plaussinn plausible poses:
Date
Leurradic Module
Compuittton Medale
Figure 2: Overview of dynamics-regulated kinematic policy. Given egocentire video I initialization module "KIN (Irr) computes the first-frame object state o1 human pose 71, camera poses h and image features 1I.7. The object state Ø1 and human pose q1 used to initialize the step phsycis simulation. At each time step, roll out per-step kinematic policy TKIN together with the Universal Humanoid Controller to output physically -laauublle pose q, inside a physics simulator.
miit
Egocentric human pose estimation. Compared to third-person human pose estimation, there are only handful attempts at estimating 3D full body poses from egocentric videos due to the ill-posed nature of this task. Most existing methods still assume partial visibility of body parts in the image [48, 39, 53]. often through a downward- facing camera. Among works where the human body is mostly not observable [20, 55, 56, 33], Jiang al. [20] use kinematics- baaed approach where they construct a motion graph from the training data and recover the pose sequence by solving the optimal pose path. Ng et al. [33] focus modeling person-to-perseee interactions from egocentric videos and inferring the wearer's pose conditioning on the other person's 's pose. The works most related to ours are [55, 56, 19] which use dynamics-basee based approaches and map visual inputs control signals to perform poysically-plassbble human motion inside physics simulation. They show impressive results on a set of noninteractive locomotion tasks, but also observe large errors in absolute 3D position tracking-mapping directly from the visual inputs to control signals is noisy process and prone to eиror accumulation. In comparison, our work jointly models kinematics and dynamics, and estimates a wider range of human motion and human-object interactions while improving absolute 3D position tracking. To the best of our knowledge, we are the first approach estimate the 3D human poses from egocentric video while factoring in human-object interactions.
Humanoid control inside physics simulation. Our work is also connected to controlling humanoids to mimic reference motion [36, 37, 5, 52, 15, 6] and interact with objects [5, 30] inside a physics simulator. The core motivation of these works to learn the necessary dynamics to imitate or generate human motion in a physics simulation. Deep RL has been the predominant approach in this line of work since physics simulators are typically methods [5, 30, 1] does not involve motion imitation and are evaluated on task completion (moving an object, sitting on chair, moving based on user-input etc.). Consequently, these frameworks only need to master a subset of possible motions for task completion. People, on the other hand, have a variety of ways perform actions, and our agent has to follow the trajectory predefined by egocentric videos. Motion imitation methods [36, 52, 57, 52] aim to control characters to mimic sequence of reference motion, but have been limited to performing a single clip [36, 37, 52, 57] or high-quality MoCap [52) motion (and requires fine-tuning to generalize to other motion generators). In contrast, our dynamics controller general and can be used to perform everyday motion and human-object interactions estimated by a kinematic motion estimator without task-specific fine-tuning.
end-to-end differetiable. Goal-oriented
3 Method
The problem of egocentric pose estimation can be formulated as follows: from a wearable camera footage I1,T, we want to recover the wearer's ground truth global 3D poses 91T Each pose
(P got coot, ,r. -J, consists of the root position po root orientation r.p and body joint angles J. of the human model. Here we adopt the popular SMPL [27] human model and the humanoid
7.
3

---

use in physics simulation is created from the kinematic structure and mean body shape defined by SMPL. Our framework first learns Universal Humanoid Controller (UHC) from a large MoCap dataset (Sec. 3. The learned UHC can be viewed as providing the lower level muscle skills of real human, trained by mimicking thousands of human motion sequences. Using the trained UHC. we learn our kinematic policy (Sec. 3.2) through dynamics-regllated training (Sec. 3.3). At the test time, the kinematic policy provides per-step target motion to the UHC, forming a closed-loop system that operates inside the physics simulation to control humanoid. The result of the UHC and physics simulation is then used as input to the kinematic policy to produce the next-frame target motion, as depicted in Fig. 2. As notation convention, we use - to denote kinematic quantities (obtained without using physics simulation), - to denote ground truth quantities, and normal symbols without accents to denote quantities from the physics simulation.
3.1 Dynamics Model - Universal Humanoid Controller (UHC)
To learn task-agnostic dynamics model that can be tightly integrated with kinematic model, we design our controller's s state space to only rely on the current simulated pose q; and target posed q and remove all phase or sequence-level information found in prior arts [57, 36, 37j. This design allows us to train on an order of magnitude larger dataset of human motion [29] with only pose information and significantly improve our models' ability to mimic diverse and unseen motions. Formally, model controlling a humanoid to follow a reference motion T1.T as a Markov Decision Process (MDP) defined as a tuple M = (S, A, Pphyaess R, 7) of states, actions, transition dynamics, reward S.wewrd
function, and a discount factor. The state S. reward Rt and transition dynamics Pphysics are provided
by the physics simulator, while action A computed by the policy TUHC. At each timestep t, the agent in state s, takes an action sampled from the policy TUHC (aec(er (a. (t. 9.+1) while the environment generates the next state St+11 and reward rz. We employ Proximal Policy Optimization (PPO) [41] to find the optimal policy Tühc. that maximizes the expected discounted return //==1>"~>>
State. The state 8t - (q qe) of the humanoid contains the character's ' current pose q, and joint velocity q,- Here, the state 8 encapsulates the humanoid's full physical state at time step t. It only includes information about the current frame (gr, q.) and does not include any extra information, enabling learned controller to be guided by target pose only.
Action. The action a, specifies the target joint angles for the proportional derivative (PD) controller [45] at each degree of freedom (DoF) of the humanoid joints except for the root (pelvis). We use the residual action representation [15]: gf = T + ar where qf the final PD target, a, is the output of the control policy TUHC. and 4c+1 is the target pose. The torque to be applied at joint
i is: -' = k" (q. q.) kd q, where k" and kd are manually specified gains and is the dementtwwise multiplication. As observed in prior work [57, 58), allowing the policy to apply external residual forces 7 to the root helps stabilizing the humanoid, our final action is a, (Ağ m.).
Policy. The policy Tunc(a, St, 1 ) is represented by a Gaussian distribution with a fixed diagonal covariance matrix Σ. We first use a feature extraction layer Dan q.) to compute the root and joint offset between the simulated pose and target pose. All features are then translated to a root-relative coordinate system using an agent-centic transform T to make our policy orientation- invariant. We use a Multi-layer Perceptron (MLP) as our policy network to map the augmented state TAC (Ie, dr. T:41, Dar( (..1. )x)) to the predicted action az.
Reward function. For UHC, the reward function designed to encourage the simulated pose qe to better match the target pose 9.+1. Since we share a similar objective (mimic target motion), our reward is similar to Residual Force Control [57].
Training procedure. We train our controller on the AMASS [29] dataset, which contains 1 11505 high-quality MoCap sequences with 4000k frame of poses (after removing sequences involving human-object interaction like running on a treadmill). At the beginning of each episode, a random fixed length sequence (300 frames) sampled from the dataset for training. While prior works [52, 51] uses more complex motion clustering techniques to sample motions, we devise a simple yet empirically effective sampling technique by inducing probability distribution based the value function. For each pose frame 9 in the dataset, we first compute an initialization state s/:
s A q 0), and then score it using the value function to access how well the policy can mimic the sequence T starting from this pose: V(s) 71)1) = Uy. Intuitively, the higher vy is, the more confident our policy is in mimicing this sequence, and the less often we should pick this frame. The

---

probability of choosing frame j. comparing against all frames J in the AMASS dataset, is then P(O,) Σ exp(-v./1)
exp(-e,/T) where 7 is the sampling temperature. More implementation details about
the reward, training and evaluation of UHC can be found in Appendix C.
3.2 Kinematic Model - Object-aware Kinematic Policy
To leverage the power of our learned UHC, we design an auto-regressive and object-aware kinematic policy to generate per-frame target motion from egocentric inputs. We synchronize the state space of our kinematic policy and UHC such that the policy can be learned with or without physics simulation. When trained without physics simulation, the model is purely kinematic and can be optimized via supervised learning; when trained with physics simulation, the model can be optimized through a combination of supervised learning and reinforcement learning. The latter procedure, coined Donumics-regulater training, enables our model to distill human dynamics information learned from large-scale MoCap data into the kinematic model and learns a policy more robust to convariate shifts. In this section, we will describe the architecture of the policy itself and the training through supervised learning (without physics simulation).
Scene context modelling and initialization. To serve as a high-level target motion estimator for egocentric videos with potential human-objes interaction, our kinematic policy needs to be object- aware and grounded with visual input. To this end, given an input image sequence I1: we compute the initial object states 01. image features I:T, and camera trajectory h1rr as inputs to system. The object states, o A (o,te of o, ). is modeled as a vector concatenation of the main object- of-interest's s class gole 3D position 0. and rotation 0 O, is computed using an off-the-sher object detector and pose estimator [17]. When there are objects in the current scene (for walking and running etc.), the object states vector is set to zero. The image features $1. contains crucial information about the wearer's movement and is computed using an optical flow extractor [44] and ResNet [13]. Since visual input can be noisy and modern smartglasses and bodycams are often equipped with built-in SLAM or Visual Inertial Odometry (VIO) [50, 9] capabilities, we utilize this additional data modalitiy and compute the 6DoF camera pose from input Images. Using an pos off-the-shelf VIO method [16], we extracts camera trajectory as: h, & (h2 hr (position h,
pos p-rot
and orientation rot Notice that the camera trajectory h, information is crucial and significantly improve the performance of our framework as shown in our ablation studies (Sec. 4.2).
To provide our UHC with plausible initial state for simulation, we estimate T from the scene context features Φ1.T ond h..T. We use an Gated Recurrent Unit (GRU) [7] based network to regress the initial agent pose q1 Combining the above procedures, we obtain the context modelling and initialization model rinni: KIN- Iq II.T, 01, h Λ KIN (I Notice that to constrain the ill-posed problem of egocentric pose estimation, we assume known object category, rough size, and potential mode of interaction. We use these knowledge as a prior for our per-step model for pose estimation.
Training kinematic policy via supervised learning. After initialization, we use the estimated first-frame object pose and human pose q, to initialize the physics simulation. All subsequent object movements are a result of human-object interation and simulation. At each time step, we use per-step model Talep to compute the next frame pose based on the next frame observations: we obtain an egocentric input vector e₁ through the agent-centric transformation function é,
T AC qg. Φ o,+1, h441) where č₁ - (F -rrot
+3: - 0' 1+1, h141) contains the current agent- centric root orientation ptrat joint angles J. iimage feature for next frame Φ object state 0'
and camera pose h From č₁ the kinematic policy π step
computes the root angular velocity wi. linear velocity Üt, and next frame joint rotation Jr+1- The next frame pose is computed through a finite integration module T finite with time difference 81 = 1/30s:
wr, 8., J.-1
step KIN
(ē:),
ge+1 Tome (H., G. J..1. 1+ 9c).
When trained without physics simulation, we auto-regressively apply the kinematic policy and use the computed T as the input for the next timestep. This procedure is outlined at Alg. 1. Since all mentioned calculations are end-to-end differentiable, we can directly optimize our T KIN and π
init step
5

---

Algorithm Learning kinematic policy via supervised learning.
1: Input: Egocentric videos and paired ground truth motion dataset Q
2: while not converged do
3: Ms. <-D 0
> initialize sampling memory
4: while M not full do
5: I1a + random sequence of images Fi. from the datasct I
6:
T1: : h1.r TIN (TI:))
b compute scene context and initial pose
7:
for 1...T do
C. - (1: PLL1. 01+1, h.41))41)
compute agent-centric input features
9:
9++1 T imite ( KIN (c 9.)
10:
store (q:: 7.) into memory sLL
11: end for
12: end while
13: TT "KKIN: RKIN supervised learning update using data collected in Ms. for 10 epoches.
end while
through supervised learning. Specifically, given ground truth QI:T and estimated Q1:7 pose sequence, our loss is computed as the difference between the desired and ground truth values of the following quantities: agent root position (F. vs T and orientation (F) T agent-centric object position <ott,
Cf gos vs o and orientation (0, hul vs o, ) and agent joint orientation (g. vs Jr and position pos -pos
Jz computed using forward kinematies)::
CsL
77 Or:
+1 P| 1 || !0 12+ |
10 p0
-0 1 115 29 11
2001
+15
Spps 2000
Jt 1I? (2)
3.3 Dynamics-Regulate Training
To tightly integrate our kinematic and dynamics models, we design a rlynanics-rgulltte training procedure, where the kinematic policy learns from explicit physics simulation. In the procedure described in the previous section, the next-frame pose fed into the network is computed through finite integration and is not checked by physical laws: whether a real human can perform the computed pose never verified. Intuitively. this amounts to mentally think about moving actually moving. Combining our UHC and our kinematic policy, we can leverage the prelearned motor skills from UHC and let the kinematic policy act directly in simulated physical space to obtain feedback about physical plausibility. The procedure for dynamics-regulatee training is outlined in Alg. 2. In each episode, we use π next timestep + 1, the input to the kinematic policy is the result of UHC and physics simulation 4-+1 instead of 4-+1- qu+1 explicitly verify that the 4-+1 produced by the kinematic policy can step be successfully followed by motion controller. Using q:-1 also informs our KIN of the current humanoid state and encourages the policy adjust its predictions to improve humanoid stability.
physical space without
imat KIN and π KIN as Alg. with the key distinction being: at the
nynamics-reullaed optimization. Since the physics simulation is not differentiable, we cannot directly optimize the simulated pose q,: however, and q. through supervised learning. Since we know that T, is a good guess reference motion for UHC, we can directly optimize q, via supervised learning as done in Sec. 3.2 using the loss defined in Eq. 2. Since the data samples are collected through physics simulation, the input g, is mtysscalll--lausibbe step
optimize q, through reinforcement learning
and more diverse than those collected purely through auto-regressively applying TKIN in Alg. 1. This way, our dynamies-regulated training procedure performs a powerful data augmentation step, step with diverse states collected from simulation.
exposing π KIN
However, MoCap pose qe is imperfect and can contain physical violations itself (foot-skating, penetration etc.), so asking the policy KIN to produce q, as reference motion regardless of the current humanoid state can lead to instability and cause the humanoid to fall. The kinematic policy should adapt to the current simulation state and provide reference motion q, that can lead to poses similar to q, yet still physically and require trial and error. Thus, we optimize π KIN through reinforcement learning and reward maximization. We design our RL reward to have two components: motion imitation and dynamics self-supervision. The motion imitation reward encourages the policy to match the computed camera
Such behavior will not emerge through supervised learning step

---

Algorithm 2 Learning kinematic policy via dynamics-regulated training.
1: Input: Pre-trained controller πтUhC. egocentric videos I. and paired ground truth motion dataset Q
Train wini dhe KIN using Alg. for 20 epoches (optional). 3: while not converged do
4: M f- 0
> initialize sampling memory
5: while Maym not full do
I1 <- random sequence images Li:T
7:
71 71 P1.r 01, h1T f- KKN (1:T)
compute scene context and initial pose b compute intial state for simulation
81 + (9..
9:
for i <- 1...T do
10:
4- Txc(q:: Pc+1: h441)
compute agent-centric features using simulated pose 4, b sample from step as a guassian policy
11:
4:+1 ~ T inite (c:):
12:
8. 4, 9,)
13:
8 Pipssies (8L4 sc, a.). a, Tunc 8t, 9c+1) phyies simulation using тTCC extract reward and Z141 from simulation
14:
4.+1 8tt1, KIN &- reward from Eq.
15:
store (st, rr, 8141: q. 1.41) into memory M ayma
16:
for
17:
end while step
18:
T "KIN
<- Reinforcement learning updates using experiences collected in M ayna for 10 epoches. xt KIN, π sep Supervised learning update using experiences collected Mayn for 10 epoches.
19:
20: end while
trajectory h and MoCap pose q,, and serves as a regularization on motion imitation quality. The dynamics self- susersuperviioon reward is based the insight that the disagreement between q; and qe contains important information about the quality and physical plausibility of q the better de is, the easier it should be for UHC to mimic it. Formally, we define the reward for as:
stop
R.th IP) + Whg 45.0([1nm** .en" Te- Whpe 45.0(|| -
w 0..005(10 eD (e. 19+
w 30.0(1 S " + dynu 50.0 + dyna 50.00115" -.. 11)) Wij
ip
Whp. Why are weights for matching the extracted camera position hi and orientation h 1ot w
Jv
dyna dyna are weights are for matching ground truth joint angles 3 and angular velocities j.s W "
for the dynamics self-supervision rewards, encouraging the policy to match the target kinematic joint
poox angles Jr and positions Je to the simulated joint angles jtt and positions J! As demonstrated in Sec. 4.2, the RL loss is particularly helpful in adapting to challenging real-world sequences, which requires the model to adjust to domain shifts and unseen motion.
-fott
Test-time. At the test time, we follow the same procedure outlined in Alg 2 and Fig.2 to roll out our policy to obtain simulated pose 9ı.r given sequence of images Ji.-- The difference being instead of sampling from TINN (c)) as Guassian policy, we use the mean action directly.
Experiments
Datasets. As no public dataset contains synchronized ground-truie truth full-body pose, object pose, and egocentric videos with human-object interactions, we record two egocentric datasets: one inside MoCap studio, another in the real-world. The MoCap dataset contains 266 sequences (148k frames)
of paired egocentric videos and annotated poses. It features one of the five actions: sitting down on chair, avoiding obstacles, stepping on a box, pushing a box, and generic locomotion [55] (walking, running, crouching) recorded using head-mounted GoPro. Each action has around 50 sequences with different starting position and facing, gait, speed etc. We use an 80-20 train test data split on this MoCap dataset. The real-world dataset only for testing purpose and contains 183 sequences (50k frames) of an additional subject performing similar actions in an everyday setting wearing a different objects and varies the object 6DoF pose for
iPhone. For both datasets, each capture take. Additional details (diversity, setup etc.) can be found in Appendix D.
Evaluation metries. We use both pase-based and physics- baasec metrics for evaluation. To evaluate the 3D global pose accuracy, we report the root pose error (Fam) and root-relative mean per joint position error [23] (Empipe). When ground-truth root/pose information is unavailable (for roal-world
7

---

Poshing
MoCap
Ours
Mesing
EgoPase
Uhe CRject
Cheec
PoseReq
the
Mene
McCap
Ours
EgoPose
Cettt
PoseRegg
int the
Figure 3: Results of egocentric pose and human-object interaction estimation from the MoCap datset.
dataset), substitute Exoot with Eçam to report camera pose tracking error. We also employ four physics based pose metrics: acceleration error (Fac), foot skating (S), penetration (PT), and interaction success rate (Sinte). Face (mm/frame²) compares the ground truth and estimated average joint acceleration; FS (mm) is defined the same in Ling et al. [26]; PT (mm) measures the average penetration distance between our humanoid and the scene (ground floor and objects). Notice that our MoCap dataset has penetration of 7.182 mm foot sliding of 2.035 mm per frame, demonstrating that the MoCap data imperfect and may not serve as the best target motion. Sintrr is defined as whether the objects of interest has been moved enough (pushing and avoiding) or if desired motion completed (stepping and sitting). the humanoid falls down at any point, Sinter = For a full definition of evaluation metrics, please refer to Appendix B.
Baseline methods. To show the effectiveness of our framework, we compare with the previous state-of-the-art egocentric pose estimation methods: (1) the best dynamics-based approach EgoPose [56] and (2) the best kinematics-ban bassed approach PoseReg, also proposed in [56]. We use the official implementation and augment their input with additional information (ō, and h2) for a fair comparison. In addition, we incorporate the fail-safe mechanism [56] to reset the simulation when the humanoid loses balance to ensure the completion of each sequence (details in Appendix B).
Implementation details. We use the free physics simulator MuloCo [47] and run the simulation at 450 Hz. Our learned policy is run every 15 timesteps and assumes all visual inputs are at Hz. The humanoid follows the kinematic and mesh definition of the SMPL model and has 25 bones and 76 DoF. We train our method and baselines on the training split (202 sequences) our MoCap dataset. The training process takes about day RTX 2080-T with 35 CPU threads. After training and the initialization step, our network is causal and runs at 50 FPS on an Intel desktop CPU. For more implementation details and network architecture, please refer to Appendix B and C.
4.1 Results
MoCap dataset results. Table shows the quantitative comparison of our method with the baselines. All results averaged across five actions and all models have access the same inputs. We observe that our method, trained either with supervised learning or dynamics-regulated, outperform the two Wate-of-the-ee art methods across metrics. Not surprisingly, our purely kinematic model performs the best on pose-based metrics, while our dynamics-regulater trained policy excels at the physics-based metrics. Comparing the kinematics-only models can see that our method has a much lower (79.4% eтror reduction) root and joint position error (62. 1% еттоr reduction) than PoseReg, which shows that our object-aware and autoregressive design of the kinematic model can better utilize the provided visual and scene context and avoid compounding errors. Comparing with the dynamics- based methods, we find that the humanoid controlled by EgoPose has a much larger root drift, often falls down to the ground, and has much lower success rate in human-object interaction (48.4 48.4 % vs 96.9%). Upon visual inspection Fig. 3, we can see that our kinematic policy can faithfully produce human-object interaction on almost every test sequence from our MoCap dataset, while PoseReg

---

Table 1: Quantitative results on pose and physics based metries on the MoCap and real-world Dataset.
MoCap dataset
Method
Physics
Sinee 1
Froot 4
Empiie 4
Face 4
FS +
PT +
PoseReg
X
0.857
87.680
12,981
8.566
42.153
Kin poly: supervised learning (ours)
0.176
33.149
6.257
5.579
10.076
EgoPose
48.4%
1.957
139.312
9.933
2.566
7.102
Kin Kin_poly: reguutcc-regullted (ours)
96.9% %
0.205
40.443
7.064
2.474
0.686
world dataset
Method
Physics
Suier?
Eqam J
FS L
PT L
Per class success rute Siner t
PoscReg
1.260
6.181
50.414
Kin_poly: supervised learning (ours)
0.491
5.051
34.930
Sit
Push
Avoid
Step
EgoPose
9.3%
1.896
2.700
1.922
7.93%
6.81% 90.9%
4.87%
0.2% 74.2%
Kin_poly denamics-regulated (ours)
92.3%
0.476
2.742
1.229
98.4%
100%
and EgoPose often miss the object-of-intersst (as can be reflected by the large root tracking error). Both of the mnamics-basede methods has smaller acceleration error, foot skating, and penetration; some even smaller than MoCap (which has 2 mm FS and 7mm PT). Notice that our joint position error is relatively low compared to state-of-the-ar due to our strong assumption about known object of interest, its class, and potential human-object interactions, which constrains the ill-posed problem pose estimation from just front-facing cameras.
hurd-person pose estimation methods [23, 24, 28)
Real-world dataset results. The real-world dataset is far more challenging, having similar number of sequences (183 clips) as our training set (202 clips) and recorded using different equipment, environments, and motion patterns. Since no ground-truth 3D poses are available, we report our results on camera tracking and physics-based metrics. As shown in Table 1, our method outperforms the bascline methods by a large margin in almost all metrics: although EgoPose has less foot-skating (as it also utilizes a physics simulator), its human-object interaction success rate is extremely low. This can be also be reflected by the large camera trajectory error, indicating that the humanoid is drifting far away from the objects. The large drift can be attributed the domain shift and challenging locomotion from the real-world dataset, causing EgoPose's 's humanoid controller to accumulate error and lose balance easily. On the other hand, our method is able generalize and perform successful human-object interactions, benefiting from our pretrained UHC and kinematic policy's 's ability to adapt new domains and motion. Table also shows success rate breakdown by action. Here we can see that "stepping on a box" is the most challenging action as it requires the humanoid lifting its feet a precise moment and pushing itself up. Note that our UHC has never been trained on any stepping or human-object interaction actions (as AMASS has no annotated object pose) but is able to perform these action. As motion is best seen videos, we refer readers to our supplementary video.
4.2 Ablation Study
To evaluate the importance of our components, we train our kinematic policy under different configurations and study its effects the real- world dataset, which is much harder than the MoCap dataset. The results are summarized in Table 2. Row 1 (R1) corresponds to training the kinematic policy only with Alg. 1 only and use UHC to mimic the target kinematic motion as post-processing step. Row (R2) are the results of using dynamics-regulatee
Table 2: Ablation study of different components of framework.
Component
Metric
SL
Dynarreg
RL
VIO
Sime:
Ecan J
FS..
PT
73.2%
0.611
4.234
1.986
80.9%
0.566
3.667
4,490
x
54.1%
1.129
7,070
5,346
92.3%
0.476
2.742
1.229
training but only performs the supervised learning part. R3 show a variant trained without the estimated camera pose from VIO. Comparing RI and R2, the lower interaction success rate (73.2 vs 80.9% indicates that exposing the kinematic policy to states from the physics simulation serves as powerful data augmentation step and leads to a model more robust to real-world scenarios. R2 and R4 show the benefit of the RL loss in Synammic-regulatte
training: allowing the kinematic policy to deviate from the MoCap poses makes the model more adaptive and achieves higher success rate. R3 and R4 demonstrate the importance of intelligently incorporating extracted camera pose as input: visual features o can be noisy and suffer from domain shifts, and using techniques such as SLAM and VIO to extract camera poses as an additional input modality

---

can largely reduce the root drift. Intuitively, the image features computed from optical flow and the camera pose extracted using VIO provides similar set of information, while VIO provides a cleaner information extraction process. Note that our kinematic policy withoul using extracted camera trajectory outperforms EgoPose that uses camera pose in both success rate and camera trajectory tracking. Upon visual inspection, the humanoid in R3 largely does not fall down (compared to in EgoPose) and mainly attributes the failure cases to drifting too far from the object.
5 Discussions
5.1 Failure Cases and Limitations
Although our method can produce realistic human pose and human-object interaction estimation from egocentric videos, we are still at the early stage of this challenging task. Our method performs well in the MoCap studio setting and generalizes to real-world settings, but is limited to a predefined set of interactions where we have data to learn from. Object class and pose information is computed by off-the-shelf methods such as Apple's S ARkit [17]. and is provided as a strong prior to our kinematic policy to infer pose. We also only factor in the 6DoF object pose in our state representation and discard all other object geometric information. The lower success rate on the real-world dataset also indicates that our method still suffers from covariate shifts and can become unstable when the shift becomes too extreme. Our Universal Humanoid Controller can imitate everyday motion with high accuracy, but can still fail extreme motion. Due the challenging nature of this task, in this work, we focus on developing a general framework to ground pose estimation with physics by merging the kinematics and dynamics aspects of human motion. To enable pose and human-object interaction estimation for arbitrary actions and objects, better scene understanding and kinematic motion planning techniques need to be developed.
5.2 Conclusion and Future Work
In this paper, we tackle, for the first time, estimating phyyicaall--plasiibe 3D poses from an egocentric video while the person is interacting with objects. We collect a motion capture dataset and real-world dataset develop and evaluate our method, and extensive experiments have shown that our method outperforms all prior arts, We design a dynamics-regulated kinematic policy that can be directly trained and deployed inside physics simulation, and we purpose a general-purpose humanoid controller that can be used in physics-basee estimate 3D human poses and human-object interactions from just an
vision tasks easily. Through our real-world experiments,
we show that it is possible egocentric view captured by consumer hardware (iPhone). In the future, we would like to support more action classes and further improve the robustness of our method by techniques such as using a learned motion prior. Applying our dynamics-regulatte training procedure to other vision tasks such as visual navigation and third-person pose estimation can also of interest.
Acknowledgements: This project was sponsored in part by IARPA (D17PC00340), and JST AIP Acceleration Research Grant (JPMJCR20UI).
References
[1] Kevin Bergamin, Simon Clavet, Daniel Holden, and J. Forbes. Drecon. ACM Transactions on Graphics (TOG), 38:1 11, 2019.
[2] Federica Bogo, A. Kanazawa, Christoph I Lassner, Gehler, J. Romero, and Michael J. Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In The Furopean Conference on Computer Vision (ECCV), 2016.
[3] Nicolò Bonettini, Edoardo Daniele Cannas, Sara Mandelli, Luca Bondi, Paolo Bestagini, and Stefano Tubaro. Video face manipulation detection through ensemble of cnns. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5012-5019, 2021.
[4] M. Brubaker, L. Sigal, and David J. Fleet. Estimating contact dynamics. 2009 IEEE 12th International Conference on Computer Vision, pages 2389-2396, 2009.
[5] Yu- Ye-Wei Chao, Jimei Yang. Weifeng Chen, and Jia Deng. Learning to sit: Synthesizing human-char interactions via hierarchical control. ArXiv, abs/1908 009999923........1 2019.
10

---

[6] N. Chentanez. M. Müller, M. Macklin, Viktor Makoviychuk, and S. Jeschke. Physics-bused motion capture imitation with deep reinforcement learning. Proceedings the 1ihh Annual International Conference on Motion, Interaction, and Games, 2018.
[7] Kyunghyun Cho, B. V. Merrienhoer, Çaglar Güllgehre, Dzmitry Bahdanau, Fethi Bougares, Halger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-dccoder for statistical machine translation. ArXiv, abs/1406. 1078, 2014.
[8] Erwin Coumans and Yunfei Bai. Pybullet, python module for physies simulation for games, robotics and machine learning. http://pybulles
2016.
[9] J. Engel, J. Sturm, and D. Cremers. Semi-dense visual odometry for monocular camera. In IEEE International Conference Computer Vision (ICCV), pages 1449-1456, Los Alamitos, CA, USA, Dec. 2013, IEEE Computer Society.
[10] Georgios V. Georgakis, Ren Li, Srikrishna Karanam, Terrence Chen, Jana Kosecka, and Ziyan Wu. Hierarchical kinematic human mesh recovery. ArXiv, abs/2003.04255 2020.
[11] Riza Alp Güler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7297-7306. 2018
[12] I. Habibie, W. Xu, D. Mehta, G. Pons-Moll, and C. Theobalt. In the wild human pose estimation using explicit 2d features and intermediate 3d representations. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10897-10906, Jun. 2019.
[13] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[14] Mir Rayat Imtiaz Hossain and J. Little Exploiting temporal information for 3d human pose estimation. In The European Conference on Computer Vision (ECCV), 2018
[15] Soo hwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Lee. Learning predict-and-simulat policies from unorganized human motion data. ACM Transactions on Graphics (TOG), 38:1 - 11, 2019.
[16] Apple Inc. Estimating camera pose with arkit. arkit/ arcamera, 2021. Accessed: 2021-03-16.
[17] Apple Inc. Scanning and detecting 3d objects with arkit. https://developer... waw.ellpper.pppllee...p apple. pople.ccmm com/ cessed: 2021-03-16.
httpcccannnnnnnnnnnneeeeeeeeeeee wew.eeeetinnddd.bp..e... 2021. Ac-
[18] Catalin lonescu. Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(7))3325-133 13399, jul 2014.
[19] Mariko Isogawa, Yuan, Matthew O'Toole, and Kris M Kitani. Optical non-line-of-sight physics-bac based 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7013-7022, 2020.
[20] Hao Jiang and Kristen Grauman. Seeing invisible poses: Estimating 3d body pose from egocentric video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3501-3509, Jun. 2016.
[21] A. Kanazawa, Michael J. Black, D. Jacobs, and Jitendra Malik. End-to-end recovery human shape and pose. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7122-7131, 2018.
[22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCI.R, 2015.
[23] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inference for human body pose and shape estimation. 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5252-5262, 2020.
[24] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Danilidis.. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2252 22212 2019.
[25] Sijin Li and Antoni B. Chan. 3d human pose estimation from monocular images with deep convolutional neural network. In ACCV, 2014.
[26] Hung Yu Ling, Fabio Zinno, George H. Cheng, and M. V. Panne. Character controllers using motion vaes. ACM Transactions Graphics (TOG), 39:40:1 40:12, 2020.

---

[27] M. Loper. Naureen Mahmood, J. Romero, Gerard Pons-Moll, and Michael J. Black. Smpl: skinned multi-person linear model. ACM Trans. Gruph., 34:248:1-248:16 2015.
[28] Zhengyi Luo, S. Alireza Golestanch, and Kris M. Kitani. 3d human motion estimation via motion compression and refinement. In Proceedings of the Asian Conference Computer Vision (ACCV), November 2020.
[29] Naureen Mahmood, N. Ghorbani, N. Troje, Gerard Pons-Moll, and Michael J. Black, Amass: Archive of motion capture as surface shapes. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 5441 15400, 2019.
[30] J. Merel, S. Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, T. Erez, Greg Wayne, and N. Heess. Reusable neural skill embeddings for vision-guided whole body movement and object manipulation. ArXiv, 0bs/19111006333 2019.
[31] Gycongsik Moon, Juyong Chang, and Kyoung Mu Lec. Camera distance-awarr top-down approach for 3d multi-person pose estimation from single rgh image. In IEEE Conference on International Conference on Computer Vixion (ICCV), pages 10112-0 Oct. 2019.
[32] Gyeongsik Moon and Kyoung Mu Lee. 12l-meshnet: Image-to mixeel prediction network for accurate 3d human pose and mesh estimation from a single rgh image. ArXiv, abs/2008..03 033113,,, 2020.
[33] Evonne Ng. Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. CoRR, abs/a/a4.0/9999900 2019.
[34] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in vidco with temporal convolutions and semi- supervipe training. In Conference Computer Vision and Pattern Recognition (CVPR), Jun. 2019.
[35] X. Peng. M. Chang. Grace Zhang, P. Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compossitionl policies. In NeurlPS, 2019.
[36] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37(4):14::1114::4, 2018.
[37] Xue Bin Peng. Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Stv: Reinforcement learning of physical skills from videos. ACM Trans. Graph., 37(6), November 2018.
[38] Davis Rempe, L. Guibas, Aaron Hertzmann, Bryan C. Russell, R. Villegas, and Jimei Yang. Contact and human dynamics from monocular video. In SCA, 2020.
[39] Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafutdinow, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, and Christian Theobalt. Egocap: Egocentric marker-less motion capture with two fisheye cameras. ACM Trans. Graph., 35(6), November 2016.
[40] Grégory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[41] John Schulman, F. Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. ArXiv, abs//177.006444 2017.
[42] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Patrick Pérez, and Christian Theobalt. Neural monocular 3d human motion capture. ACM Transactions on Graphics, 40(4), aug 2021.
[43] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and C. Theohalt. Physeap: Physically plausible monocular motion capture in real time. ACM Trans. Graph., 39:235:1-235:16. 2020.
[44] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2018
[45] J. Tan, K. Liu, and G. Turk. Stable proportionallddrriatiive controllers. IEEE Computer Graphics and Applications, 31(:)34 44, Jul. 2011.
[46] Bugra Tekin, Isinsu Katircioglu, M. Salzmann, Vincent Lepetit, and P. Fua. Structured prediction of 3d human pose with deep neural networks. ArXiv, abs/1605. 081.0. 2016.
[47] E. Todorov, T. Erez, and Y. Tassa Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026- 5033, Oct. 2012.
12

---

[48] Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino, xr-egopose: Egocentric 3d human pose from hmd camera. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). pages 7728-7738, Oct. 2019.
[49] M. Vondrak, L. Sigal, J. Hodgins, and O. C. Jenkins. Video-based 3d motion capture through biped control. ACM Transactions on Graphics (TOG). 31:1 2012.
[50] S. Wang, R. Clark, H. Wen, and N. Trigoni. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In IEEE International Conference on Robotics and Automation (ICRA), pages 2043-2050, 2050, May 2017.
[51] Tingwu Wang, Yunrong Guo, Maria Shugrina, and S. Fidler. Unicon: Universal neural controller for physics-based character motion. ArXiv, 0.0s/2011.11110 2020.
[52] Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. ACM Trans. Graph., 39(4), 2020.
[53] Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer, Helge Rhodin, Pascal Fua, Hans-Peter Seidel, and Christian Theobalt. Mo"Cap" Real-time mobile 3d motion capture with cap-mounted fisheye camera. IEEE Transactions on Visualization and Computer Graphics, pages 1-1, 2019.
[54] Yuanlu Xu, S. Zhu, and Tony Tung. Denserac: Joint 3d pose and shape estimation by dense render- and-compare. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7759-7769, 2019.
[55] Ye Yuan and Kris Kitani. 3d ego pose estimation via imitation learning. In The European Conference on Computer Vision (ECCV), Sep. 2018.
[56] Ye Yuan and Kris Kitani. Ego-pose estimation and forecasting as real-time pd control. In IEEE International Conference Computer Vision (ICCV), pages
Oct. 2019.
[57] Ye Yuan and Kris Kitani. Residual force control for agile human behavior imitation and extended motion synthesis. In Advances in Neural Information Processing Systems, 2020.
[58] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[59] Hongwen Zhang, Jie Cao, Guo Lu, Wanli Ouyang, and Z. Sun. Learning 3d human shape and pose from dense body parts. ArXiv, abs/1912.13344, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Due to space constraints, lim- itation is briefly discussed the conclusion section of the main paper and more rigorously discussed in the supplement.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Provided in the supplemantry materials.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We train our methods multiple times using different seeds to verify its effectiveness and the results are consistent across multiple runs. Analysis about the stochasticity of our method is included in the supplement.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "r2sdo2j-EOq", "reviewer": "Reviewer_ZmMv", "summary": "This paper proposes a method for pose estimation from first-person camera view. The work builds on a universal controller that learns to imitate a wide variety of behaviors. A kinematic policy is then either trained fully via supervised learning or in a hybrid manner, using a combination of supervised learning and reinforcement learning. The reinforcement learning optimization consists of a motion imitation and a physics-based objective. Specifically, the universal controller’s output is utilized to regularize the kinematic policy to become more physically plausible. The main contributions are a universal controller and the dynamics-regulated training procedure for egocentric pose estimation in human-object interaction scenarios. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths ###\n\nThe paper is written in a clear manner and simple to follow. It introduces the new setting of egocentric pose estimation with objects in the scene. The main idea of using a physics-based policy directly to guide the training of the kinematics-based model is novel. In prior works [1][2][3], the policy was only used to correct the kinematics model. \n\n### Weakness ###\n\nOverview of the main concerns, which are detailed in the paragraphs below:\n1. Improper evaluation of the universal controller.\n2. Missing insights into the data and the strong performance of the pose estimation, which perform better than state-of-the-art pose estimation from third-person view on established benchmarks, despite the much harder task. \n3. Generalization of the learned tasks is not properly evaluated.\n\n\n**1. Universal controller**\n\nFirst of all, learning robust universal controllers is a challenging task and is known to suffer from instabilities, especially if it has to replicate unseen motions. There is a branch of research [4][5][6] with the sole focus on building controllers that can scale to larger datasets, but have only achieved learning on a subset of AMASS (CMU ~ 2k motion sequences). The proposed method builds on a universal controller that is trained on AMASS (~11k sequences). Hence, the claim of a universal controller that can scale to an order of magnitude larger motion database than state-of-the-art needs to be properly evaluated on its own, since it would mark a substantial improvement in the direction of imitation learning from motion capture. As stated by the authors, the policy is able to execute a wide variety of motion “ranging from dancing to kickboxing” (cf. L.40) and therefore needs to be empirically evaluated on the full AMASS dataset to substantiate this claim. The current controller is only tested on the relatively simple motions contained in H36M (walking, standing, etc.). Furthermore, it is important to see how well it adapts to noisy estimates, because at the beginning of training the dynamic-regulated model, the kinematics policy will likely produce random residuals and hence noisy target poses. \n\n**2. Missing insights into data and performance**\n\nThe universal controller is then utilized in the downstream task of egocentric pose estimation. Since the task is ill-posed, estimating pose from egocentric video without top-down view is extremely difficult. The results, namely the mean per joint position error, which is very low (~30-40mm), may indicate that there is very little variation between different motion sequences and overfitting to the training data (which is likely similar to the test data). For instance, since there is no way to tell how the occluded arms are moving, such a small error is only possible if the deviation of motion between the sequences in the data is very small. The authors should provide: 1) statistical data on their dataset to be able to better assess their quantitative results (e.g., data on per joint trajectories, pose diversity against available 3D datasets), 2) a proper discussion of the very small reported errors on the test data and 2) the per joint error to be able to get better insights into the performance of the model.     \nThis seems even more evident when looking at the supplementary video, where it appears that the agent is always moving in the same way (movement of arms, gait, speed, etc.). Since the authors already provide the statistical data of the speed in the datasets, it would be good to see whether the agent actually learns to adapt to different speeds or just overfits to one single motion.\n\n**3. Generalization**\n\nAs illustrated in the details of the dataset (Appendix D), the interaction objects are always positioned in the same location for both datasets, and hence it would be important to conduct experiments on generalization. For instance, how would the method fare if the objects’ locations were slightly different? Otherwise, the learned policy is likely to overfit to single object instances and not be useful for downstream applications. This should ideally be tested and at least be discussed in the paper. Furthermore, although the trajectory analysis provided in Figure 5 indicates a slight variation in the facing direction towards the object, it seems that the agent is mostly facing the interaction objects and needs to walk straight to reach it (cf. supplementary video). It would be interesting to see whether the agent can act in the scene without being right in front of the object of interest and facing towards it. \n\n\n### Other Comments and Technical Questions ###\n- The training procedure of the universal controller proposed for sampling sequences likely impose quite a large computational overhead since it has to run all frames of AMASS (4000k) through the value function. How often is this distribution recomputed? Moreover, is there a mistake in the notation of the initialization states, or what is the reason that the target state is the same as the input state and not the next state?\n\n- Did the authors run experiments without the redundancy of information in the state space of the universal controller (e.g. to have joint angles in axis angle and quaternion representation or the difference between joint position and target joint position in world and agent-centric coordinates) or what was the incentive behind overloading the state space?\n\n- Do the authors employ any technique, such as early stopping, when training the universal controller and the dynamics-regulated kinematic policy to avoid learning from failure cases? \n\n- The notation in Appendix C Policy Network architecture seems to be inconsistent. The quaternion difference $\\ominus$ and the minus seem to be used for angle-axis difference and vice-versa, at least according to the dimensions provided. \n\n- Will the authors release the dataset in case of publication to foster further research?\n\n- A potential missed citation is [7]. The authors use a differentiable physics model to correct their kinematics model for pose reconstruction.\n\n[1] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih, “Simpoe: Simulated character control for 3d human pose estimation”, CVPR, 2021\n\n[2] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. 2020, “PhysCap: physically plausible monocular 3D motion capture in real time”, ACM Transactions on Grap., 2020.\n\n[3] Davis Rempe, Leonidas J Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang, “Contact and human dynamics from monocular video”, ECCV 2020\n\n[4] Jungdam Won, Deepak Gopinath, and Jessica Hodgins, “A scalable approach to control diverse behaviors for physically simulated characters”, ACM Trans. Graph., 2020.\n\n[5] ] Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler, “Unicon: Universal neural controller for physics-based character motion”, arxiv, abs/2011.15119, 2020.\n\n[6] Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess, “Neural probabilistic motor primitives for humanoid control”, ICLR, 2019\n\n[7] Soshi Shimada, Vladislav Golyanik,  Weipeng Xu, Patrick Pérez, and Christian Theobalt, “\"Neural PhysCap\" Neural Monocular 3D Human Motion Capture with Physical Awareness”, ACM Trans. Graph., 2021.\n\n### Post rebuttal\nI appreciate the author's extensive answer to my concerns and questions. In the light of how the concerns were addressed, I'm willing to raise my score to 6. What I would like to see in the final version of the paper in case of acceptance is 1) a thorough discussion of the limitations (which is missing in the current main part of the manuscript) 2) the requested evaluations of the universal controller and the dataset statistics. 3) Clarification of how the pose metrics were obtained with respect to failing sequences. It seems that such good numbers for the pose can only be achieved if the sequences are ended for episodes deemed unsuccessful.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the limitations and societal impact is discussed properly.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "NhJP0Pdd8fR", "reviewer": "Reviewer_cAMH", "summary": "In this paper, the authors presents a new method for egocentric physically-plausible 3d human motion estimation. Specially, they designed a general-purpose humanoid controller (UHC) and a dynamics-regulated kinematic policy that can be directly trained and deployed inside a physics simulation. The proposed universal humanoid controller is trained on large-scale Mocap datasets and is for general-purpose. In the experiments, the authors evaluate their proposed method on both Mocap dataset and their collected real dataset. The authors show good performance improvements compared with PoseReg and EgoPose, as well as good imitation success rate compared with DeepMimic on Human3.6M.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is well written and easy to follow. The authors presents an interesting approach for egocentric physically-plausible 3d human motion estimation. \n+ This paper is a re-submission of their AAAI work. The authors addressed the major concerns from their previous submission in method novelty compared with [Ye et al, ICCV'19], [Jungdam et al, TOG'20], performance significance and real-world dataset scale. \n+ The proposed general-purpose humanoid controller and the designed dynamics-regulated kinematic policy is quite interesting. \n+ The authors conduct quite some experiments to validate the method effectiveness and elaborate the implementation and experiment details well.\n+ Code is attached in supplementary.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall this paper is well prepared. I found a few limitations in current method.\n- Most of the testing data are quite simple and generally belong to walking/sitting poses. From the results reported in supplementary video and material, it seems the proposed methods cannot handle challenging poses well. It would be interesting to see how much the proposed method could work on more challenging poses. The authors could further expand their pose categories to more non-upright poses (e.g., working out, yoga, lying down) and this could probably give us more insights about the boundaries of the proposed method.\n- I found the human-object interaction part is a little bit confusing. The objects are not reconstructed and tracked from the input video and we could see sometimes the object state in real-world video does not match it in simulation. What is the assumption here? If we wanna model real-world object interaction, how would you further leverage the current approach?\n- Real-world testing data and experiments are still a little bit limited. The evaluation on Human3.6M is quite interesting. It would be interesting to dive deeper and analyze more about the failure cases and how the proposed policy could potentially improve to handle such failure cases.\n- From the results I noticed the walking looks a little bit robotic and not quite like natural walking gait. Could you explain what causes this and how would you improve to make the movement more natural?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "CeHXgp55ImC", "reviewer": "Reviewer_c7KC", "summary": "This paper presents a method for estimating the egocentric 3D pose of a person moving in a scene. The main idea is to build a dynamics-driven human motion controller which is provided target poses to mimic by a kinematics estimator and information about scene objects. Experiments show that improved performance compared to previous approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would like to start with several things that I liked about the paper.\n\n- The problem being solved is important and useful. Combining kinematics and dynamics is of increasing interest in the human pose estimation community. This paper takes it further by operating on egocentric data.\n- The paper is well written, motivated and the experiments are comprehensive.\n- The problem formulation is interesting and novel.\n\nIn the following, I focus on questions, comments, and suggestions for improvement.\n\n- The introduction makes says (lines 33) that dynamics-based methods use physics simulators, but this is not always true. Many methods model dynamics explicitly and may not use a dynamics simulator.\n\n- The introduction does not discuss the objects in much detail. It would have been nice to do that as this is a part of the proposed method.\n\n- In the method section, I was lacking motivation for why certain components were used. For instance, what are the differences between the proposed controller vs. other methods that try to achieve the same thing as MotionVAE? The proposed approach appears fine but lacks insight and is more complex than other methods.\n\n- In section 3.2, similarly for the kinematics model: Why not use a simple neural network to predict poses autoregressively?\n\n- Line 183: How are object states encoded? How many objects are supported and does this generalize to unseen objects?\n\n- For experiments, this might be a useful dataset to work on. The paper states no dataset fits but perhaps this could?\n\nhttps://ps.is.tuebingen.mpg.de/publications/grab-2020\n\nOverall, the paper presents some interesting ideas which I would like to see published. I encourage the authors to address the above points in their rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in the supplementary, but I would have liked to see this discussion in the main paper. The paper discusses broader discusses.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2CzNcKrOe8S", "reviewer": "Reviewer_7PSv", "summary": "This paper looks at the problem of estimating physically-plausible 3D human motion and environment interactions from an egocentric RGB video. It proposes to combine the advantages of kinematic and dynamics-based approaches: first, a physics-based dynamics policy (UHC) is trained to imitate motion capture data; then a kinematic policy is trained to predict a target pose for UHC based on RGB and object & camera pose inputs. Experiments show that the combination of dynamics and kinematics outperforms either individually by a large margin, and that the method can reasonably predict a small set of person-object interactions. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\nThe problem of physically-plausible 3D human motion (from ego or even 3rd person) is challenging, and even more so when considering environment and object interactions (which hasn't even been done much for 3rd person). Though some assumptions are made, the proposed approach makes a great first step towards solving this problem, producing impressive qualitative video results.\n\nThe technical approach leverages multiple good ideas that make it a compelling contribution. First, combining dynamic and kinematic approaches in a way that they are aware of each other, and so the dynamics policy can regularize the kinematic one. And second the notion of first learning a low-level physical motion controller on a diverse set of motions, and then leveraging this with a higher-level model for a problem with less available good data. The standalone dynamics policy (UHC) could also be useful for other applications. \n\nExperiments report an extensive set of evaluation metrics that measure the kinematic, physical, and interaction accuracy of the proposed method. Quantitative and qualitative improvement is clear compared to SOTA kinematic-only & dynamics-only baselines and an ablation that naively uses dynamics to post-process a kinematics prediction, showing that the combination is advantageous. \n\nThe method also achieves a high success rate when estimating (a limited set of 4) human-object interactions, showing it is not limited to non-interacting motions.\n\nWeaknesses:\n-\nThe method makes a few limiting assumptions in order to make the problem feasible - these are understandable considering the difficulty and novelty of the problem. First, it assumes that accurate detections and poses of interacting objects are feasible from the ego view, and the ego-pose pipeline overfits to a fairly small set of actions/objects (sit, push, avoid, step). In many of the supp video examples, the object is not visible from the camera view so these detections would be difficult (I'm not sure exactly how they were done in the real-world data?) - then the method needs to rely on given prior knowledge of the 3D environment or overfitting to the image/camera pose features during interactions. Second, the method relies on accurate estimation of camera pose, which although is \"optional\" significantly affects performance as in Table 2. The ability of the method to generalize so well to the real-world dataset (with very different image features), suggests it relies heavily on this input. It would be helpful to see qualitative examples of the ablation that doesn't use camera pose; I'm also curious to see results without any image inputs, only using camera pose, would it still do well?\n\nA few small technical issues/questions. First, the PD controller gains must be manually specified - these could instead be learned as in SimPOE. Second, on L199, how is the kinematic policy expected to predict next frame pose/velocities (at $t+1$) without any inputs corresponding to the next frame? Is all necessary future information contained in the image features through optical flow? Why not use the object/camera pose and image features at $t+1$ as well? Otherwise, it seems to be blindly sampling one possible future rather than reconstructing the observations.\n\nLastly, it's clear that combining kinematics+dynamics is useful, but the experiments do not justify why the dynamics policy must be pre-trained. Could Alg 2 be used to train both the dynamics and kinematic policy jointly? I suspect this will result in much worse performance - I like the idea of pre-training dynamics on a much wider set of motions before using it for the specific ego-estimation task -  but this baseline would empirically justify the pre-trained general UHC.\n\nQuestions/Suggestions:\n-\nL211: why is object position supervised in Eqn (2)? How is object position deviating from the initial detections if no physics are used in the regular supervised setting?\n\nL288 says the method is causal and runs at 50 FPS, but the initial pose estimation from $\\pi_{KIN}^{init}$ uses all future timesteps. How is this done in a real-time setting? Also in Alg 2, L7: how is the initial velocity $\\dot{q}_1$ calculated? Is this predicted along with the pose by the initialization model?\n\nWill the mocap and real-world datasets be released publicly?\n\nThe mocap dataset contains three different subjects, but it's not clear how the proposed method can handle differing body shapes and proportions since they are not given as input. Is the method currently simply overfitting to the heights of these three people?\n\nNames like \"Kin_poly: supervised learning (ours)\" in Table 1 are a bit cumbersome, maybe consider giving a short, catchy name to the method variations.\n\nTypos:\n- Alg 1, L7 and Alg 2, L8: what is $n$? Should this be $t \\leftarrow 1 \\dots T$?\n- Alg 2 , L14: does $\\tilde{q}_{t+1}$ also need to be stored to compute the reward in Eq (3)?\n- L228 differentialbe -> differentiable\n- L288 casual -> causal\n- Table 1 real-world data: shouldn't kin_poly supervised have an X in the \"Physics\" column?\n- L310: phsycis->physics\n\n=====================================\n\nAfter Author Response:\n-\n\nI would like to thank the authors for their detailed responses to my and other reviewers' concerns and questions. After reviewing these responses and other reviews, I have kept my initial rating of accept.\n\nAdding the updated quantitative evaluation of the UHC on AMASS to the paper/supp (using a __held out__ split of AMASS rather than the training data) and including more qualitative results on difficult motions make this contribution stronger and will be useful to the community, especially with code released. Furthermore, the idea of using UHC as physically-plausible regularization for inverse problems like pose estimation will be useful beyond ego-centric pose tracking.\n\nAs mentioned by the authors, the small set of motions and interactions in the ego-centric pose estimation task contribute to the very low errors, so the good performance on such an ill-posed task must be, at least in part, due to overfitting in some sense. Also, the requirement of the initialization procedure to acquire accurate object and head-camera poses further limits the generalizability of the current approach. Telling from Table 2 in the paper and followup experiments provided in the response wrt camera pose, I would encourage authors not to call the camera pose __optional__ since it is clearly important to good performance and admittedly the \"framework mainly infer pose from egocentric motion\".\n\nHowever, I feel these limitations are understandable considering the difficulty of the new problem, and the results do still support the claim that dynamics-regulated training with UHC is useful. Moreover, the current formulation of the problem could be essentially seen as motion infilling [1] with only head pose observed and added object interactions to account for. This in itself is an open problem [2] with applications to e.g. animation in VR. The authors might consider discussing this in the revised paper, as I think the ideas they introduce are more generally useful to problems involving inferring physically-plausible motion from limited observations - not just ego-pose estimation.\n\n[1] Convolutional Autoencoders for Human Motion Infilling, Kaufmann et al., 3DV 2020.\n\n[2] LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals, Yang et al., Eurographics 2021.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed in the supp document and video - perhaps some of this discussion could be moved to the main paper. \n\nThe \"Broader societal impact\" section discusses valid privacy concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Originally there was a slight disagreement among the reviewers, with this paper receiving 3 positive ratings and 1 negative one. The reviewers acknowledge the interest of the studied scenario and that the empirical results are convincing. They nonetheless proposed several ways to improve the paper, by clarifying several aspects of the method and of the results, as well as discussing some additional works. The authors addressed these points in their feedback and managed to convince the most negative reviewer to raise their score. We therefore believe that this paper can be accepted to NeurIPS but strongly encourage the authors to incorporate their feedback in the paper for the final version.
Decision: Accept (Poster)