Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for pose estimation from first-person camera view. The work builds on a universal controller that learns to imitate a wide variety of behaviors. A kinematic policy is then either trained fully via supervised learning or in a hybrid manner, using a combination of supervised learning and reinforcement learning. The reinforcement learning optimization consists of a motion imitation and a physics-based objective. Specifically, the universal controller’s output is utilized to regularize the kinematic policy to become more physically plausible. The main contributions are a universal controller and the dynamics-regulated training procedure for egocentric pose estimation in human-object interaction scenarios. 
main_review: ### Strengths ###

The paper is written in a clear manner and simple to follow. It introduces the new setting of egocentric pose estimation with objects in the scene. The main idea of using a physics-based policy directly to guide the training of the kinematics-based model is novel. In prior works [1][2][3], the policy was only used to correct the kinematics model. 

### Weakness ###

Overview of the main concerns, which are detailed in the paragraphs below:
1. Improper evaluation of the universal controller.
2. Missing insights into the data and the strong performance of the pose estimation, which perform better than state-of-the-art pose estimation from third-person view on established benchmarks, despite the much harder task. 
3. Generalization of the learned tasks is not properly evaluated.


**1. Universal controller**

First of all, learning robust universal controllers is a challenging task and is known to suffer from instabilities, especially if it has to replicate unseen motions. There is a branch of research [4][5][6] with the sole focus on building controllers that can scale to larger datasets, but have only achieved learning on a subset of AMASS (CMU ~ 2k motion sequences). The proposed method builds on a universal controller that is trained on AMASS (~11k sequences). Hence, the claim of a universal controller that can scale to an order of magnitude larger motion database than state-of-the-art needs to be properly evaluated on its own, since it would mark a substantial improvement in the direction of imitation learning from motion capture. As stated by the authors, the policy is able to execute a wide variety of motion “ranging from dancing to kickboxing” (cf. L.40) and therefore needs to be empirically evaluated on the full AMASS dataset to substantiate this claim. The current controller is only tested on the relatively simple motions contained in H36M (walking, standing, etc.). Furthermore, it is important to see how well it adapts to noisy estimates, because at the beginning of training the dynamic-regulated model, the kinematics policy will likely produce random residuals and hence noisy target poses. 

**2. Missing insights into data and performance**

The universal controller is then utilized in the downstream task of egocentric pose estimation. Since the task is ill-posed, estimating pose from egocentric video without top-down view is extremely difficult. The results, namely the mean per joint position error, which is very low (~30-40mm), may indicate that there is very little variation between different motion sequences and overfitting to the training data (which is likely similar to the test data). For instance, since there is no way to tell how the occluded arms are moving, such a small error is only possible if the deviation of motion between the sequences in the data is very small. The authors should provide: 1) statistical data on their dataset to be able to better assess their quantitative results (e.g., data on per joint trajectories, pose diversity against available 3D datasets), 2) a proper discussion of the very small reported errors on the test data and 2) the per joint error to be able to get better insights into the performance of the model.     
This seems even more evident when looking at the supplementary video, where it appears that the agent is always moving in the same way (movement of arms, gait, speed, etc.). Since the authors already provide the statistical data of the speed in the datasets, it would be good to see whether the agent actually learns to adapt to different speeds or just overfits to one single motion.

**3. Generalization**

As illustrated in the details of the dataset (Appendix D), the interaction objects are always positioned in the same location for both datasets, and hence it would be important to conduct experiments on generalization. For instance, how would the method fare if the objects’ locations were slightly different? Otherwise, the learned policy is likely to overfit to single object instances and not be useful for downstream applications. This should ideally be tested and at least be discussed in the paper. Furthermore, although the trajectory analysis provided in Figure 5 indicates a slight variation in the facing direction towards the object, it seems that the agent is mostly facing the interaction objects and needs to walk straight to reach it (cf. supplementary video). It would be interesting to see whether the agent can act in the scene without being right in front of the object of interest and facing towards it. 


### Other Comments and Technical Questions ###
- The training procedure of the universal controller proposed for sampling sequences likely impose quite a large computational overhead since it has to run all frames of AMASS (4000k) through the value function. How often is this distribution recomputed? Moreover, is there a mistake in the notation of the initialization states, or what is the reason that the target state is the same as the input state and not the next state?

- Did the authors run experiments without the redundancy of information in the state space of the universal controller (e.g. to have joint angles in axis angle and quaternion representation or the difference between joint position and target joint position in world and agent-centric coordinates) or what was the incentive behind overloading the state space?

- Do the authors employ any technique, such as early stopping, when training the universal controller and the dynamics-regulated kinematic policy to avoid learning from failure cases? 

- The notation in Appendix C Policy Network architecture seems to be inconsistent. The quaternion difference $\ominus$ and the minus seem to be used for angle-axis difference and vice-versa, at least according to the dimensions provided. 

- Will the authors release the dataset in case of publication to foster further research?

- A potential missed citation is [7]. The authors use a differentiable physics model to correct their kinematics model for pose reconstruction.

[1] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih, “Simpoe: Simulated character control for 3d human pose estimation”, CVPR, 2021

[2] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. 2020, “PhysCap: physically plausible monocular 3D motion capture in real time”, ACM Transactions on Grap., 2020.

[3] Davis Rempe, Leonidas J Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang, “Contact and human dynamics from monocular video”, ECCV 2020

[4] Jungdam Won, Deepak Gopinath, and Jessica Hodgins, “A scalable approach to control diverse behaviors for physically simulated characters”, ACM Trans. Graph., 2020.

[5] ] Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler, “Unicon: Universal neural controller for physics-based character motion”, arxiv, abs/2011.15119, 2020.

[6] Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess, “Neural probabilistic motor primitives for humanoid control”, ICLR, 2019

[7] Soshi Shimada, Vladislav Golyanik,  Weipeng Xu, Patrick Pérez, and Christian Theobalt, “"Neural PhysCap" Neural Monocular 3D Human Motion Capture with Physical Awareness”, ACM Trans. Graph., 2021.

### Post rebuttal
I appreciate the author's extensive answer to my concerns and questions. In the light of how the concerns were addressed, I'm willing to raise my score to 6. What I would like to see in the final version of the paper in case of acceptance is 1) a thorough discussion of the limitations (which is missing in the current main part of the manuscript) 2) the requested evaluations of the universal controller and the dataset statistics. 3) Clarification of how the pose metrics were obtained with respect to failing sequences. It seems that such good numbers for the pose can only be achieved if the sequences are ended for episodes deemed unsuccessful.


limitations_and_societal_impact: Yes, the limitations and societal impact is discussed properly.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors presents a new method for egocentric physically-plausible 3d human motion estimation. Specially, they designed a general-purpose humanoid controller (UHC) and a dynamics-regulated kinematic policy that can be directly trained and deployed inside a physics simulation. The proposed universal humanoid controller is trained on large-scale Mocap datasets and is for general-purpose. In the experiments, the authors evaluate their proposed method on both Mocap dataset and their collected real dataset. The authors show good performance improvements compared with PoseReg and EgoPose, as well as good imitation success rate compared with DeepMimic on Human3.6M.
main_review: Overall, this paper is well written and easy to follow. The authors presents an interesting approach for egocentric physically-plausible 3d human motion estimation. 
+ This paper is a re-submission of their AAAI work. The authors addressed the major concerns from their previous submission in method novelty compared with [Ye et al, ICCV'19], [Jungdam et al, TOG'20], performance significance and real-world dataset scale. 
+ The proposed general-purpose humanoid controller and the designed dynamics-regulated kinematic policy is quite interesting. 
+ The authors conduct quite some experiments to validate the method effectiveness and elaborate the implementation and experiment details well.
+ Code is attached in supplementary.
limitations_and_societal_impact: Overall this paper is well prepared. I found a few limitations in current method.
- Most of the testing data are quite simple and generally belong to walking/sitting poses. From the results reported in supplementary video and material, it seems the proposed methods cannot handle challenging poses well. It would be interesting to see how much the proposed method could work on more challenging poses. The authors could further expand their pose categories to more non-upright poses (e.g., working out, yoga, lying down) and this could probably give us more insights about the boundaries of the proposed method.
- I found the human-object interaction part is a little bit confusing. The objects are not reconstructed and tracked from the input video and we could see sometimes the object state in real-world video does not match it in simulation. What is the assumption here? If we wanna model real-world object interaction, how would you further leverage the current approach?
- Real-world testing data and experiments are still a little bit limited. The evaluation on Human3.6M is quite interesting. It would be interesting to dive deeper and analyze more about the failure cases and how the proposed policy could potentially improve to handle such failure cases.
- From the results I noticed the walking looks a little bit robotic and not quite like natural walking gait. Could you explain what causes this and how would you improve to make the movement more natural?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for estimating the egocentric 3D pose of a person moving in a scene. The main idea is to build a dynamics-driven human motion controller which is provided target poses to mimic by a kinematics estimator and information about scene objects. Experiments show that improved performance compared to previous approaches.
main_review: I would like to start with several things that I liked about the paper.

- The problem being solved is important and useful. Combining kinematics and dynamics is of increasing interest in the human pose estimation community. This paper takes it further by operating on egocentric data.
- The paper is well written, motivated and the experiments are comprehensive.
- The problem formulation is interesting and novel.

In the following, I focus on questions, comments, and suggestions for improvement.

- The introduction makes says (lines 33) that dynamics-based methods use physics simulators, but this is not always true. Many methods model dynamics explicitly and may not use a dynamics simulator.

- The introduction does not discuss the objects in much detail. It would have been nice to do that as this is a part of the proposed method.

- In the method section, I was lacking motivation for why certain components were used. For instance, what are the differences between the proposed controller vs. other methods that try to achieve the same thing as MotionVAE? The proposed approach appears fine but lacks insight and is more complex than other methods.

- In section 3.2, similarly for the kinematics model: Why not use a simple neural network to predict poses autoregressively?

- Line 183: How are object states encoded? How many objects are supported and does this generalize to unseen objects?

- For experiments, this might be a useful dataset to work on. The paper states no dataset fits but perhaps this could?

https://ps.is.tuebingen.mpg.de/publications/grab-2020

Overall, the paper presents some interesting ideas which I would like to see published. I encourage the authors to address the above points in their rebuttal.
limitations_and_societal_impact: Limitations are discussed in the supplementary, but I would have liked to see this discussion in the main paper. The paper discusses broader discusses.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper looks at the problem of estimating physically-plausible 3D human motion and environment interactions from an egocentric RGB video. It proposes to combine the advantages of kinematic and dynamics-based approaches: first, a physics-based dynamics policy (UHC) is trained to imitate motion capture data; then a kinematic policy is trained to predict a target pose for UHC based on RGB and object & camera pose inputs. Experiments show that the combination of dynamics and kinematics outperforms either individually by a large margin, and that the method can reasonably predict a small set of person-object interactions. 
main_review: Strengths:
-
The problem of physically-plausible 3D human motion (from ego or even 3rd person) is challenging, and even more so when considering environment and object interactions (which hasn't even been done much for 3rd person). Though some assumptions are made, the proposed approach makes a great first step towards solving this problem, producing impressive qualitative video results.

The technical approach leverages multiple good ideas that make it a compelling contribution. First, combining dynamic and kinematic approaches in a way that they are aware of each other, and so the dynamics policy can regularize the kinematic one. And second the notion of first learning a low-level physical motion controller on a diverse set of motions, and then leveraging this with a higher-level model for a problem with less available good data. The standalone dynamics policy (UHC) could also be useful for other applications. 

Experiments report an extensive set of evaluation metrics that measure the kinematic, physical, and interaction accuracy of the proposed method. Quantitative and qualitative improvement is clear compared to SOTA kinematic-only & dynamics-only baselines and an ablation that naively uses dynamics to post-process a kinematics prediction, showing that the combination is advantageous. 

The method also achieves a high success rate when estimating (a limited set of 4) human-object interactions, showing it is not limited to non-interacting motions.

Weaknesses:
-
The method makes a few limiting assumptions in order to make the problem feasible - these are understandable considering the difficulty and novelty of the problem. First, it assumes that accurate detections and poses of interacting objects are feasible from the ego view, and the ego-pose pipeline overfits to a fairly small set of actions/objects (sit, push, avoid, step). In many of the supp video examples, the object is not visible from the camera view so these detections would be difficult (I'm not sure exactly how they were done in the real-world data?) - then the method needs to rely on given prior knowledge of the 3D environment or overfitting to the image/camera pose features during interactions. Second, the method relies on accurate estimation of camera pose, which although is "optional" significantly affects performance as in Table 2. The ability of the method to generalize so well to the real-world dataset (with very different image features), suggests it relies heavily on this input. It would be helpful to see qualitative examples of the ablation that doesn't use camera pose; I'm also curious to see results without any image inputs, only using camera pose, would it still do well?

A few small technical issues/questions. First, the PD controller gains must be manually specified - these could instead be learned as in SimPOE. Second, on L199, how is the kinematic policy expected to predict next frame pose/velocities (at $t+1$) without any inputs corresponding to the next frame? Is all necessary future information contained in the image features through optical flow? Why not use the object/camera pose and image features at $t+1$ as well? Otherwise, it seems to be blindly sampling one possible future rather than reconstructing the observations.

Lastly, it's clear that combining kinematics+dynamics is useful, but the experiments do not justify why the dynamics policy must be pre-trained. Could Alg 2 be used to train both the dynamics and kinematic policy jointly? I suspect this will result in much worse performance - I like the idea of pre-training dynamics on a much wider set of motions before using it for the specific ego-estimation task -  but this baseline would empirically justify the pre-trained general UHC.

Questions/Suggestions:
-
L211: why is object position supervised in Eqn (2)? How is object position deviating from the initial detections if no physics are used in the regular supervised setting?

L288 says the method is causal and runs at 50 FPS, but the initial pose estimation from $\pi_{KIN}^{init}$ uses all future timesteps. How is this done in a real-time setting? Also in Alg 2, L7: how is the initial velocity $\dot{q}_1$ calculated? Is this predicted along with the pose by the initialization model?

Will the mocap and real-world datasets be released publicly?

The mocap dataset contains three different subjects, but it's not clear how the proposed method can handle differing body shapes and proportions since they are not given as input. Is the method currently simply overfitting to the heights of these three people?

Names like "Kin_poly: supervised learning (ours)" in Table 1 are a bit cumbersome, maybe consider giving a short, catchy name to the method variations.

Typos:
- Alg 1, L7 and Alg 2, L8: what is $n$? Should this be $t \leftarrow 1 \dots T$?
- Alg 2 , L14: does $\tilde{q}_{t+1}$ also need to be stored to compute the reward in Eq (3)?
- L228 differentialbe -> differentiable
- L288 casual -> causal
- Table 1 real-world data: shouldn't kin_poly supervised have an X in the "Physics" column?
- L310: phsycis->physics

=====================================

After Author Response:
-

I would like to thank the authors for their detailed responses to my and other reviewers' concerns and questions. After reviewing these responses and other reviews, I have kept my initial rating of accept.

Adding the updated quantitative evaluation of the UHC on AMASS to the paper/supp (using a __held out__ split of AMASS rather than the training data) and including more qualitative results on difficult motions make this contribution stronger and will be useful to the community, especially with code released. Furthermore, the idea of using UHC as physically-plausible regularization for inverse problems like pose estimation will be useful beyond ego-centric pose tracking.

As mentioned by the authors, the small set of motions and interactions in the ego-centric pose estimation task contribute to the very low errors, so the good performance on such an ill-posed task must be, at least in part, due to overfitting in some sense. Also, the requirement of the initialization procedure to acquire accurate object and head-camera poses further limits the generalizability of the current approach. Telling from Table 2 in the paper and followup experiments provided in the response wrt camera pose, I would encourage authors not to call the camera pose __optional__ since it is clearly important to good performance and admittedly the "framework mainly infer pose from egocentric motion".

However, I feel these limitations are understandable considering the difficulty of the new problem, and the results do still support the claim that dynamics-regulated training with UHC is useful. Moreover, the current formulation of the problem could be essentially seen as motion infilling [1] with only head pose observed and added object interactions to account for. This in itself is an open problem [2] with applications to e.g. animation in VR. The authors might consider discussing this in the revised paper, as I think the ideas they introduce are more generally useful to problems involving inferring physically-plausible motion from limited observations - not just ego-pose estimation.

[1] Convolutional Autoencoders for Human Motion Infilling, Kaufmann et al., 3DV 2020.

[2] LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals, Yang et al., Eurographics 2021.


limitations_and_societal_impact: Limitations are addressed in the supp document and video - perhaps some of this discussion could be moved to the main paper. 

The "Broader societal impact" section discusses valid privacy concerns.
review_title: null

Meta-review: Originally there was a slight disagreement among the reviewers, with this paper receiving 3 positive ratings and 1 negative one. The reviewers acknowledge the interest of the studied scenario and that the empirical results are convincing. They nonetheless proposed several ways to improve the paper, by clarifying several aspects of the method and of the results, as well as discussing some additional works. The authors addressed these points in their feedback and managed to convince the most negative reviewer to raise their score. We therefore believe that this paper can be accepted to NeurIPS but strongly encourage the authors to incorporate their feedback in the paper for the final version.
Decision: Accept (Poster)