Embodied Scene-aware Human Pose Estimation
Authors: Zhengyi Luo
Shun Iwase
Ye Yuan
Kris M. Kitani
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D human pose estimation
Physics simulation
Embodied AI

Abstract:  We propose embodied scene-aware human pose estimation where we estimate 3D poses based on a simulated agent's proprioception and scene awareness, along with external third-person observations. Unlike prior methods that often resort to multistage optimization, non-causal inference, and complex contact modeling to estimate human pose and human scene interactions, our method is one-stage, causal, and recovers global 3D human poses in a simulated environment. Since 2D third-person observations are coupled with the camera pose, we propose to disentangle the camera pose and use a multi-step projection gradient defined in the global coordinate frame as the movement cue for our embodied agent. Leveraging a physics simulation and prescanned scenes (e.g., 3D mesh), we simulate our agent in everyday environments (library, office, bedroom, etc.) and equip our agent with environmental sensors to intelligently navigate and interact with the geometries of the scene. Our method also relies only on 2D keypoints and can be trained on synthetic datasets derived from popular human motion databases. To evaluate, we use the popular H36M and PROX datasets and achieve high quality pose estimation on the challenging PROX dataset without ever using PROX motion sequences for training. Code and videos are available on the project page.

Embodied Scene-aware Human Pose Estimation
Zhengyi Luo'
Shun Iwase 1- Ye Yuan' !
Kris Kitani
Carnegie Mellon University
https: https://zhengyinne wwwpeeeennwl.oggguuu....... ho/ http///o/tt///ennoieeeeeeeeeeeee poseeeeddppppeeeeee
Figure I: From a monocular video input in known scene, we estimate 3D absolute body pose by controlling
a simulated character to match 2D observation in sequential manner.
Abstract
We propose embodied scene-aware human pose estimation where we estimate 3D poses based simulated agent's proprioception and scene awareness, along with external third-person observations. Unlike prior methods that often resort multistage optimization, non-causal inference, and complex contact modeling to estimate human pose and human scene interactions, our method is one-stage, causal, and recovers global 3D human poses in simulated environment. Since 2D third-person observations are coupled with the camera pose, we propose to disentangle the camera pose and use multi-step projection gradient defined in the global coordinate frame the movement cue for embodied agent. Leveraging a physics simulation and prescanned scenes (e.g., 3D mesh), we simulate our agent in everyday environments (library, office, bedroom, etc.) and equip agent with environmental sensors to intelligently navigate and interact with geometries of the scene. Our method also relies only on 2D keypoints and can be trained synthetic datasets derived from popular human motion databases. To evaluate, we use the popular H36M and PROX datasets and achieve high quality pose estimation on the challenging PROX dataset without ever using PROX motion sequences for training. Code and videos are available the project page.
Introduction
We consider 3D human pose estimation setting where calibrated camera is situated within a prescanned environment (with known 3D mesh geometries). Such a setting has a wide range
Equal Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

of applications in mixed reality telepresence, animation, and gaming. at home robotics, video surveillance, etc., where a single fixed RGB camera is often used to observe human movement. However, even under such constraints, severe occlusion, low-textured garments, and human-scene interactions often cause pose estimation methods to fail catastrophically. Especially when estimating the glohal 3D human pose, tfte-of-thheart methods often resort to expensive multistage batch optimization [6], motion prior or infilling [31, 48], and depth information to recover reasonable global translation and orientation. Consequently, these methods have limited use cases in time-critical applications. Multistage optimization can also suffer from failure of convergence, where the optimizer cannot find good solution due to occlusion, numerical instabilitty or poor initialization.
The effective use of scene information is also critical for human pose estimation. Unfortunately, the lack of datasets with paired ground-truth truth 3D body pose and annotated contact labels makes it difficult to incorporate scene information to estimate pose. Thus, popular methods often resort to 1) minimizing the distance between predefined contact points in the body and the scene [6]. 2) using vertex-based PROXimity [49] as a proxy scene representation, and 3) semantic labels as affordance cues [17]. These methods, while effective in modeling human-scene interactions, are expensive to compute and heavily rely on a batch optimization procedure to achieve good performance. More complex scenes are also harder to model in batch optimization framework, as the number of contact constraints increases significantly as the number of geometries increases.
We hypothesize that embodiment may be a key element in estimating accurate absolute 3D human poses. Here, we define embodiment as having a tangible existence a simulated environment in which the agent can act and perceive. In particular, humans move according to their current body joint configuration (proprioception), movement destination (goal), and an understanding of their environment's 's physical laws (scene awareness and embodiment). Under these guiding principles, we propose embodied scene-aware human pose estimation as an alternative to optimization and regression-basee based methods. Concretely, we estimate poses by controlling physically simulated agent equipped with envirommental sensors to mimic observations from third-person a streaming fashion. At each timestep, our agent receives movement goals from a third-person observation and computes its torque actuation based on (1) current agent state: body pose, velocities, etc., (2) third fard-person observation, and (3) environmental features from the agent's 's point of view. Acting in physics simulator, our agent can receive timely feedback about physical plausibility and contact to compute actions for the next time frame. Although there exist prior methods that also use simulated characters, we draw a distinction in how the next frame pose is computed. Previous work first regress the kinematic pose [4, 46] without considering egocentric features such as body and environmental states. As a result, when the kinematic pose estimation methods fail due to occlusion, these simulated agents will receive unreasonable target poses. Our method, on the other hand, predicts target poses by comparing the current body pose with third-person observations. When third-person observations are unreliable, the target poses will be extrapolated from environmental cues and current body states. Thus, our method is robust to occlusion and can recover from consecutive frames of missing body observations. Our method is also capable of estimating pose for long-term (several minutes to perpetual). in sub-real-time (~~ 10 fps), while creating physically realistic human scene interactions.
To achieve this, we propose a multi-step projection gradient (MPG) to provide movement cues to embodied agent from 2D observations. At each time step, MPG performs iterative gradient descent starting from the current pose of the humanoid to exploit the temporal coherence of human movement and better avoid local minima. MPG also directly provides temporal body and root movement signals our agent in the world coordinate frame. Thus, our networks does not need take the 3D camera pose as input explicitly. To control a simulated character, we extend the Universal Humanoid Controller (UHC) and the kinematic policy proposed in previous work [21] to support the control of humanoids with different body shapes in environments with diverse geometries. Finally, we equip our agent with an occupancy--oe Since only 2D keypoint coordinates are needed by our method to estimate body motion, our model can be learned from purely synthetic data derived from popular motion databases.
environmental sensor learn about affordances and avoidance.
To summarize, our contribution is as follows: 1) we propose embodied scene-aware human pose estimation, where we control a simulated scene-aware humanoid to mimic visual observations and recover accurate global 3D human pose and human-scene interactions; 2) we formulate a multi-step projection gradient (MPG) to provide a camera-agnostie movement signal for an embodied agent to match 2D observations; and 3) we achieve state-of- tftte-of-the-att result on the challenging PROX dataset while our method remains sub-real-time. causal, and trained only on synthetic data.
2

---

2 Related Work
Local and Absolute 3D Human Pose Estimation. is popular to estimate the 3D human pose in
the local coordinate system (relative to the root) [I, 11, 12, 13, 14, 15, 16, 20, 24, 25, 27, 30, 37, 39] where the translation is discarded and the rotation is defined in the camera coordinate frame. Among these methods, some focus lifting 2D keypoints to 3D [1, 27], while others use template 3D human mesh (the SMPL [19] body model and its extensions [26, 32]) and jointly recover body shape and joint angles. Among these methods, some [15, 37, 46] make use of iterative optimization to better fit 2D keypoints. SPIN [15] devises an ppaimizationiin-th-lloo fitting procedure to provide better starting point for pose regressor. NeuraIGD [37] and SimPOE [46] both utilize learned gradient update to iteratively refine body pose and shape. where optimization starts from either zero pose (NeuralGD) or estimated kinematic pose (SimPOE). Compared to these methods, our MPG starts the computation process from the previous time-step pose to leverage the temporal coherence in human motion. While estimating pose in the local coordinate frame is useful in certain applications. it is often desirable to recover the pose in the absolute/glote space to fully leverage the estimated pose for animation or telepresence. However, recovering the absolute 3D body pose that includes root translation is challenging due to limb articulation, occlusion, and depth ambiguity. Among the work that attempt to estimate the absolute human pose [24, 27, 30, 33, 43), most take a two-stage approach where the root-relative body pose is first regressed from a single image and then root translation is estimated from image-level features. Compared to these methods, we leverage physical laws and motion cues to autoregressively recover both root movement and body pose.
Scene-aware Human Pose and Motion Modeling. The growing interest in jointly modeling human pose and numan-scene interactions has led to a growing popularity in datasets containing both human and scenes [7, 23, 41]. In this work, we focus on the PROX [7] dataset, where a single human is recorded moving in daily indoor environments such as offices, libraries, bedrooms, etc. Due to the diversity of motion and human- -cenn interaction (sitting on chairs, lying on beds, etc.) and lack of ground-truth training data, the PROX dataset remains highly challenging. Consequently, state-of-the-an methods are mainly optimization-based [7, 31, 47] while leveraging the full set of observations (scene scans, camera pose, depth). HuMoR [31] and LEMO [47] achieves impressive result by using large-scale human motion datasets to learn strong prior of human movement to constrain or infill the missing human motion. However, multistage optimization can often become unstable due to occlusion, failures in 2D keypoint detections, or erroneous depth segmentation.
Physics-based humanoid control for pose estimation. Physical laws guard every aspect of human motion and there is a growing interest in enforcing physics prior in human pose estimation [4, 9, 21, 35, 36, 42, 44, 45, 46). Some methods prefer to use physics as postprocessing step and refine initial kinematic pose estimates [35, 36, 42], while ours follows the line of work where a simulated character is controlled to estimate pose inside physics simulation [4, 9, 21, 46). Although both approaches have their respective strong suites, utilizing a physics simulation simplifies the process of incorporating unknown scene geometries and can empower real-time pose estimation applications. Among the simulated character control-based methods, Neural Mocon [9] utilizes sampling-basee control and relies on the learned motion distribution from the training split of the dataset to acquire quality motion samples. SimPOE [46] first extracts the kinematic pose using fff-the-shelf method [13] and then uses simulated agent for imitation. PoseTrriplet [4] proposes using a self-sur supervied framework to co-evolve a pose estimator, imitator, and hallucinator. Compared to ours, SimPoE and PoseTrripet both rely on kinematic pose estimator as backhones [13, 27] and require well-observed 2D keypoints for reliable pose estimates. They also require lengthy policy training time and do not model any scene geometries. Our work extends Kin Kin_pof, iin_poly [21], where simulated character is used to perform first-person pose estimation. While Kin_poly uses egocentric motion as the only motion cue, we propose a Multistep Projection Gradient to leverage thrid-person observations. Compared to their object-pose based scene feature, we use an occupancy-basee scene feature inspired by the graphics community [5, 38] to better model scene geometries.
3 Method
Given a monocular video Jir taken using a calibrated camera known environment, our aim is to estimate, track, and simulate 3D human motion in metric space. We denote the 3D human
3

---

1
Observation
Ihe
Projection
Updited
Minsnztoooo gose
70 keypo nos
Projection
Malti-atop Projection Gredient (MPG)
Propriocaption
Camera Projectio
Prkyen dv)
Plyskca Semulation
St
Scene awarnsss
<RkaNd AKKiN( Ap, P..)
9:+1
4-11
Ginemaha Policy
Estimated Target Post
Continlllrr
4/+1
Figure 2: Our embodied human pose estimation framework: given 2D keypoint features, our model computes the next- next-time frame pose based on third-person 2D observation (multi-step projection gradient Φ.), proprioception features (body state s,), and scene (occupancy 0z).
pose as q, - (r., rT. 0z), consisting of the 3D root position rT. the orientation rß and the angles of the body joints 0r. Combining 3D pose and velocity, we have the character's state 8 & (q,, q.)- As notation convention, we use to represent kinematic poses (without physics simulation), to denote ground-trum quantities from Motion Capture (MoCap), and normal symbols without accents for values from the physics simulation. In Sec 3.1, will first summarize our extension to the Universal Humanoid Controller [21] to support controlling people with different body proportions. Then, in Sec.3.2, will describe the state space of our character's control policy: the multi-step projection gradient (MPG) as a feature and occupancy based scene feature. Finally, we will detail the training procedure to learn our embodied pose estimator using synthetic datasets in See 3.3. Our pose estimation pipeline is visualized at Fig.2.
3.1 Preliminaries: Physics-based Character and Universal Humanoid Controller (UHC)
To simulate people with varying body shapes, we first use an automatic simulated character creation process similar to SimPoE [46] to generate human bodies with realistic joint limits, body proportions and other physical properties based on the popular human mesh model SMPL [19]. In SMPLL, body joints are recovered from mesh vertices and joint regressor, so the length of each limb may vary according to the body pose. This is undesirable as varying the length of the body limbs physically inaccurate. Recovering body joints by first computing large number of vertices (6890) followed by joint regressor is also time-cossuming. Therefore, we fix the limb lengths using those calculated from the T pose (setting the joint angles to zero) and use forward kinematics to recover the 3D location of the body joints J, from joint angles. Given the shape of the body 33 and the pose q we define the forward kinematics function as FK,(qg) Jr.
To learn a universal humanoid controller that can perform everyday human motion with different body types, we follow the standard formulation in physics-based character and model controlling humanoid for motion imitation as Markov Decision Process (MDP) defined as tuple M (S, T. R, 7) of states, actions, transition dynamics, reward function, and discount factor. The state & E S, reward r E RR and the transition dynamics T are determined by the physies simulator, while the action E AA is given by the control policy Tunc(a, |8, q:41, P). Here we extend the Universal Humanoid Controller proposed Kin_poli [21] and condition the control policy on the current simulation state 8c, the reference motion T., and the body shape B. We employ Proximal
Policy Optimization (PPO) [34] to find the optimal control policy TUhC to maximize the expected
discounted reward IE ΣT 1
>r For more information on the UHC network architecture and training procedure, please refer to Kin pony [21] and our Appendix.
3.2 Embodied Scene-Aware Human Pose Estimation
To estimate the global body pose from the video observations, we control a simulated character to follow the 2D keypoint observations in sequential decision manner. Assuming known camera

---

pose and scene geometry, we first use an off-the-shelf 2D keypoint pose estimator, DEKR [3], to extract 2D keypoints kp; at a per-frame level. Then, we use HuMoR [31] to extract the first-frame 3D pose do and body shape /3 from the RGB observations as initialization. Given qu and B. we initialize our simulated character inside the pre-scanned scene, and then we attempt to estimate poses which match observed 2D keypoints in a streaming fashion.
Multi-step Projection Gradient (MPG) as a Motion Feature. To match the motion of the 3D simulated humanoids to 2D keypoint observations, it is essential to provide 3D information about how poses should be modified in the global coordinate frame (coordinate frame of the scene geometry and the 3D humanoid). As 2DD keypoints are coupled with camera pose (the same body pose corresponds to different 2D keypoints based on the camera pose), it is problematic to directly input 2D observations without a proper coordinate transform. To avoid this issue, prior arts that use simulated character control [4, 46] often estimate the kinematic human pose in the camera coordinate frame using kinematic pose estimator. However, conducting physies simulation using the camera coordinate frame is challenging, since the up-vector of the camera is not always aligned with the direction of gravity. As a result, manual alignment of the character's up up-vector to the gravity direction is often needed for successful simulation [4].
To decouple the pose of the camera from the pose estimation pipeline, we propose using a multi-step projection gradient (MPG) feature as an alternative to estimated kinematic pose as an input to the policy. Intuitively, the MPG is modified (learned) sum of 2D projection loss gradients that encodes how much the 3D pose should be adjusted for to fit the next frame 2D keypoints. Given the current global state of the humanoid a, and the camera projection function Team: we define MPG Φ as:
K
Oprs d: kpp pp..kpppp):eeee 9: k+1 pq:
@Cooi 0q.
Φ. 4e
K
qi
>
9
g(
qr: (pp441)))41))
==
and Cpnu || (FK, keypoints kp2+1. is the step size of the gradient and & indexes into the total number of gradient &c represents how the root pose and body joints descent steps K. The 2D project loss gradient Dq,
2D kp la is the L2 reprojection loss with respect to the observed 2D
should change to better match the 2D key points of the next frame and contains rich geometric information independent of the camera projection function Team
Due to noise in the detection, occlusion, and depth ambiguity of 2D key points, a single-step 0Lyn can be erroneous and noisy, especially the gradient of root
instantaneous projection gradient āg
translation and orientation. Thus, we augment aye G that is intended to remove noise. In this way. the MPG feature is an aggregation of a collection of modified gradients to compute a more stable estimate of the 2D projection loss gradient.
acms with a geometric and learning- baased transform
To better recover root rotation, we use temporal convolution model (TCN) similar to VideoPose3D [27] and PoseTriplet [4] to regress camera-space root orientation from 2D keypoints. We notice that while TCN may produce an unreasonable pose estimate during server occlusion, its root orientation estimation remains stable. We compute the camera space root orientation using a causal TCN FIEN and transform it into the world coordinate system r! using the inverse camera rotation: "t - (FTCN (kp2-p=411)) We then calculate the difference between the current root rotation rß and rft as an additional movement gradient signal. Given the current 3D pose qf and the projection function Team. we can obtain a better root translation ri by treating the 3D keypoints obtained as fixed rigid body and solving for translation using linear least squares. Combining the FICN model and the geometric transformation step. we obtain the function G to obtain better root translation and orientation gradient. By taking multiple steps, we avoid getting stuck in local minima and obtain better gradient estimates. For more complete formulation of MPG, see our Appendix.
Scene Representation. To provide our embodied agent with an understanding of its environment, we provide our policy with two levels of awarenewrsssss: 1) we simulate the geometry of the scene inside physics simulation, which implicitly provides contact and physics constraints for our policy. As the scanned meshes provided by the PROX dataset often have incorrect volumes, we use semi-automatic geometric fitting procedure to approximate the scene by simple shapes such rectangles and cylinders. The modified scene can be visualized together in Eq.2. For more
5

---

ALGORITHM 1: Learning embodied pose estimator via dynamics- training.
Input: controller suuc- -
dataset
while converged do
Myn t- initialize sampling memory
while M g.o full do
Pu <- sample random sequence human pose
2.D <- Naa (q).))
I/ compute paired 20 projaction.;
+- I...I do
<- (9.
d. + Ix=o 91 poi kpi mei-
11 novenent cues from 20 observations.
ou;
4 f- RKIN(* 4s 0z) +:;
#++1 f- ((4+4 (st, ). ar &- munc se. 9c+1) // physics simulation and agent control using
Tun
#2+1
memory Mays
end
end
end
TAIN
Supervised learning update using experiences collected daaa for epoches.
information about our scene simplification process, please refer the Appendix. We voxel-based occupancy map to indicate the location and size of the scene geometries, similar to [5, 38]. Occupancy provides basic-level scene and object awareness our agent and is robust to changing scene geometries. Concretely, we create cube of 1.8 meters edge length as the sensing range of our agent's surroundings and create a 16 by by evenly spaced grid. At the center of each point on the grid, we sample the signed distance function of the scene F, to obtain how far the sampling point from the geometries of the scene. Then, we threshold the gird feature to provide finer scene information. In time step i, for threshold a 0.05, the value of point p; in the grid can be computed as of, and o, {ob. ):
F.(pi) < 0 F.(m: > (2)
0f
F. F.(pi)/
otherwise
Figure 3: Our modified scene for simulation.
Scene-aware Kinematic Policy, To train and control our embodied agent in the physics simulator, we extend policy network similar to perform third-person pose estimation a real-world scene. Under the guiding principle of embodied human pose estimation, our kinematic policy TKIn computes a residual 3D movement q, based on proprioception s;, movement goal 6,, and scene awareness Og:
[21], with the important distinction that our networks
q TKIN St, &. o.). 4e+1 q. qe-
Specifically, the input q, contains the current body joint configurations, Φ; provides the movement signal per step, while o provides awareness to the agent's 'ss surroundings. Within TKIN, an agent- centric transform TAC is used to transform q:- d, and s, into the agent-centire coordinate system.
The output 7, defined d. A (#. FT 0z), which computes the velocity in root translation i and orientation PT. well the joint angles 0,. Notice that q. computes the residual body movement instead of the full kinematic body pose q and establishes a better connection with the movement feature Φ, During inference, πKKN works in tandem with Tunc and computes the next-step target pose g.+11 for the humanoid controller to mimic. In turn, π KIN receives timely feedback on physical realism and contact information based on the simulated agent state.
3.3 Learning the embodied human pose estimator
Since л πK' relies only on keypoints, we can leverage large-scale human motion databases and train pose estimation network using synthetic data. We use the AMASS [22], H36M [10], and

---

Kin silypplll//// [21] datasets to dynamically generate 2D keypoints and 3D pose pairs for training. At the beginning of each episode, we sample a motion sequence and randomly change its direction and starting position in a valid range, similar to the procedure defined in NeuralGD [37]. Then we use randomly selected camera projection function Team to project 3D keypoints onto 2D images to obtain paired 2D keypoints and human motion sequences. During training, we initialize the agent with the ground-truth 3D location and pose. The H36M and Kin poly dataset contains simple human-scene interactions, such as sitting on chairs and stepping on boxes. To better learn object affordance and collision avoidance, we also randomly sample scenes from the PROX dataset and pair them with motions from the AMASS dataset. Although these motion sequences do not correspond to the sampled scene, the embodied agent will hit the scene geometries while trying to follow the 2D observations. We stop the simulation when the agent deviates too far away from the ground-tree truth motion to learn scene affordance and avoidance. For more details about our synthetic data sampling procedure, please refer to the Appendix.
We train the kinematic policy as a per-step model together with physics simulation, following the dynamics-regulated training procedure in [21] (Alg, 1.). Using this procedure, we can leverage the pre-learned motor skills from the UHC and let the kinematic policy act directly in a simulated physical space to obtain feedback physical plausibility. In each episode, we connect the UHC and the kinematic policy using the result of the physics simulation qt-1 to compute the input features to the kinematic policy. Using 9.+1 verifies that q, can be followed by our UHC and informs our KIN of the current humanoid state to encourage the policy to adjust its predictions to improve stability. Here, only train our KIN using supervised learning, as third hhird-person pose estimation is more well-posed problem and requires less exploration than estimating pose from only egocentric videos [21]. For each motion sequence, our loss function is defined as L1 distance between the predicted kinematic pose and the ground truth plus a prior loss using a learned motion transition prior Pa and the encoder qs from HuMoR [31]. The prior term encourages the generated motion to be smoother:
wna llq. q.l1i + Uprior DKI (g6 q. (=((=()
(4)
Experiments
Implementation details. We use the MuJoCo [40] physics simulator and use modified scene geometries as environments. The training process or our embodied pose estimator takes around 2 days on a RTX 3090 with 30 CPU threads. During inference, our network is causal and runs at 10 FPS on an Intel desktop CPU (not counting the time for 2D keypoint pose estimation, which runs at around 30 FPS [3]). For more details on the implementation, please refer the Appendix.
Datasets. Our Universal Humanoid Controller is trained on the AMASS dataset training split that contains high-quality SMPL parameters of 1 11402 sequences curated from the various MoCap datasets. For training our pose estimator, we motions from the AMASS training split, the Kin_ dataset, and H36M dataset. The Kin Liny dataset contains 266 motion sequences of sitting, stepping, pushing, and avoiding, and the H36M dataset consists of 210 indoor videos with 51 different unique actions. Note that we only use the motion sequences from these datasets and not the paired videos. For evaluation, we use videos from the PROX [7] dataset and the test split of H36M.
Metrics and Baselines. We compare with a series of sfate-of-the art methods HyperlK [16), Me- TRAbs [33], VIBE [13], and SimPOE [46] in the H36M dataset. On the PROX dataset, we compare with the HuMoR [31], LEMO [47], and HyberlK with RootNet [24]. We use the official imple- mentation of the above methods. We use pose-based and plausibility-based metrics for evaluation. To evaluate pose estimation when there is ground-truth -ruth MoCap pose, we report the absolute/global and root-relative mean per joint position error: A-MPJPE and MPJPE, On the PROX dataset, since there is ground truth 3D pose annotation, we report: success rate SR (whether the optimization procedure converges and whether the simulated agents lose balance), one-way average chamfer distance from the point cloud (ACD, acceleration (Accel.c), and scene penetration frequency (Freq.) and distance (Pen.). For a full definition of our evaluation metrics, please refer to the Appendix.
4.1 Results
7

---

Foom
For MBGds
Poo 500 Duer
Fox S00ffe
Poice 100 Spmndars
Fer
Figure 4: Visualization of our pose estimation result on the PROX and H36M dataset, along with our modified scenes. For each scene, we approximate the scene geometries with simple shapes.
Results on H36M. We report the pose estimation on the H36M dataset in Table 1. Here, we can see that our method achieves a reasonable pose es- timation result on both root-relative and global pose metrics. Notice that our experiment with the H36M dataset meant be a sanity check on our pose estimation framework and not a fair comparison. Although our method does not obtain the state-of-the-arr result on H36M, we want to point out that 1) our method is trained on purely synthetic data and has never seen detected 2D keypoints from H36M videos, 2) do not use any image-level information and only relies on small set (12 in total) of 2D keypoints to infer global pose. Under such conditions, is especially challenging when the person's movement direction is parallel to the forward direction of the camera (the person is moving along the depth direction in the image plane). We also evaluate a different subset 3D keypoints, as our of forward kinematics FK3 causes joint definition be different from prior work. For more clarification and visualization on the H36M evaluation, please refer to the Appendix and project site.
Table 1: Our pose estimation result the H36M dataset. *Since our method does not directly use any image-level feature, uses a smaller set of 2D keypoints for observation, and evaluates on a different set of 3D keypoints, it is not fair comparison against SOTA methods.
features
A-MPIE
PAMPPIPE
MeTRAbs [33]
223.9
40.3
$2.7
VIBE [16]
613
43.1
SimPoB [16]
56.7
41.6
HybelK RootNet
2040
105
72.1
Table 2: Evaluation of plausibility on the PROX dataset. Since PROX has ground truth annotation. we employ ACD (one-way Average Chamfer Distance) between the segmented point cloud and predicted body mesh as a proxy metric for pose estimation plausibility, formulated in HuMoR[31]. Notice that the ACD values correspond to different number sequences based on the success rate (out of 60 sequences from PROX). Since all of the metrics can only approximate plausibility, encourage our readers to refer to our project website for visualization all 60 sequences on PROX of method. Time(s) is the per-frame average runtime evaluated RTX 2080 Super. Note that only HybrlK [16] w/ RootNet [24] and ours are not multi-stage optimization based methods (indicated by the coloum "Opti.") and offers relatively fair comparison.
Ground
Scene
Phys.
Opt.
Accel.
Freg.
Freq.
Time(s)
PROX [7]
98.3%
181.1
157.4
10.5% %
109.1
0.7
46.8 %
75.1
RGB-D
HuMoR [3]]
100%
325.5
3.4
2.4%
113.6
0.1
47.5
13.72
173.2
PROX 171
90.0%
>1000
381.6
4.6%
> 1000
0.39 g
36.7
47.64
HuMoR [31]
/
100%
311.5
3.1
0.1%
38.1
0.3%
36.4
11.73
HybxiK RooiNa (24]
1000
310.6
4,7%
Ours
98.3%
171.7
9.4
16.6
Results on PROX. Table shows our result in the PROX dataset where we compare against SOTA multi-stage batch optimization based methods: LEMO [47] and HuMoR [31]. We also report the
8

---

Ours (Causal)
Hybrik-Peee -RGB
RGBD
20 Keypoints
(Causal)
(Muti-stage Optimization)
IMuti-stage Optimization [Mutt-ssaaee Optimization
Figure Comparision with SOTA methods. Last two rows are failure cases.
result using HyperlK with RootNet [16] to show that severe occlusion and human scene interaction on the PROX dataset is also challenging to SOTA regression methods. From Table 2, we can see that our method achieves the best ACD (chamfer distance) without using any depth information and is competitive in all other metrics against both RGB and RGB-D based methods. Note that LEMO and HuMoR directly optimize the chamfer distance and ground penetration during optimization, while method relies only on 2D keypoints and laws of physics to achieve plausible pose estimates. Methods with exceedingly high ACD values (> 1000) can be attributed to unstable optimization or unreliable depth estimation, which causes the humanoid to fly away for sequences.
In Fig.4, we visualize our results each of the scene from the PROX dataset. Fig.5 compares our result with SOTA methods. We can see that our method can estimate accurate 3D absolute poses that align with the 3D scene. Compared to batch multistage optimization methods that utilize RGB-D input, PROX [7] and LEMO [47], method can achieve comparable results while running in near-

---

real-time. Compared to the regression- based method, HybrIK [16] with RootNet [24], our method can estimate much more accurate root translation and body pose. In terms of physical plausibility, optimization-base methods often perform well but can still exhibit floating and penetration. While optimization methods can take over a minute per-frame, catastrophic failure can still occur (as shown in the fourth row). For video quantitative comparison, please refer to our project website.
4.2 Ablation
In Table 3, we ablate the main components of our framework. Row (R1) corresponds not using our modified scene geometry and uses the mesh provided by the PROX dataset; row 2 corresponds to not using any scene awareness (voxel features); R2 uses MPG with only one time-step, which results in nosier gradients; R3 trains a model without using the geometric translation step in G; R4 does not use the TCN model FICN in G: R5 does not leverage modified 3D scenes and uses the original scene scans. RI shows that our modified geometry leads to better overall performance, as the scanned mesh does not correspond well to the actual scene. R2 demonstrates that scene awareness further improves our result, where our agent learns important affordance information about chairs and sofas to improve the success rate. R3 highlights the importance of taking multiple gradient steps to obtain better gradients. When comparing R4 and R5 with R6, we can see that the learned function G is essential to obtain a better root translation and orientation, without which the agents drift significantly.
Table 3: Ablation study of different components of our framework.
Grand
Secre
w/ Modided Sccce w/ Scace Awareness
MGP wio multi step MFG w/s Geomeukk
MPG wis TCN
SR
ACD
Accel.
Freg. J
Pen.
Free. J
Pen...
15.0%%
152.1
0.3
100
25.4
18.4
65%
0.0%
0.0%
96.798
126.0
0.0%
34.8
0.5%
91.7%
604.6
05.09
121.3
5 Discussions
Failure Cases. We show the failure cases of our proposed method in the last two rows of Fig. 5. Although our method can estimate human pose and human-scene interaction from the PROX dataset with a high success rate, there is still much room for improvement. We simplify all humanoids and scene geometries as rigid bodies and do not model any soft materials such as human skin, cushions, or beds. As a result, our sitting and lying motion can often result in jittery motion, as contact occurs only in very small surface area on the simulated body and hard surfaces. Due to the nature of physies simulation, our simulated humanoid sometimes gets stuck on narrow gaps between chairs and tables, or when lying beds. In these scenarios, method can often recover quickly by jolting out of the current state and trying to follow the target kinematic pose. Nevertheless, if the drift of global translation is severe many 2D keypoints are occluded, agent may lose balance. Our causal formulation may also lead to jittery motion when 2D keypoint observations jitter. Currently, our agent has no freedom to deviate from the 2D observations and always tries to match the 2D motion even when it is stuck. More intelligent control modeling will be needed to improve these scenarios.
Conclusion and Future Work. We introduce embodied scene-aware human pose estimation, where we formulate pose estimation in known environment by controlling simulated agent to follow 2D visual observations. Based on the 2D observation, its current body pose, and environmental features, embodied agent acts in simulated environment to estimate pose and interact with the environment. multi-step projection gradient (MPG) is proposed to provide global movement cues to the agent by comparing the current body pose and target 2D observations. The experimental result shows that our method, trained only on synthetic datasets derived from popular motion datasets, can estimate accurate global 3D body pose the challenging PROX dataset while using only RGB input. Future work includes providing a movement prior for the embodied agent to intelligently sample body motion based scene features to better deal with occlusion.
Acknovledgements: This project is sponsored in part by the JST ATP Acceleration Research Grant (JPMICR20UI).
10

---

References
[1] Y.-F. Cheng, B. Yang, B. Wang, Y. Wending, and R. T. Tan. Occlusion -wware networks for 3d human pose estimation in video. 2019 IEEE/CVF International Conference on Computer Vision
(ICCV), pages 723-732, 2019.
[2] K. Cho, B. V. Merrienboer, Çaglar Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and
Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine
translation. ArXiv, abs/ 140/1 2014.
[3] Z. Geng, K. Sun, B. Xiao. Z. Zhang, and J. Wang. Bottom-up human pose estimation via disentangled keypoint regression. 202/ IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). pages 14671-14681, 2021.
[4] K. Gong, B. Li. J. Zhang, T. Wang. J. Huang, M. B. Mi, J. Feng. and X. Wang. Posetriplet:
Co-evolving 3d human pose estimation, imitation, and hallucination under self-supervision.
ArXiv, abs/2203. 15625, 2022.
[5] M. Hassan, D. Ceylan, R. Villegas, J. Saito, J. Yang. Y. Zhou, and M. Black. Stochastic scene-aware motion prediction. 2021 IEEE/CVF International Conference on Computer Vision
(ICCV), pages 11354-11366 11364, 2021.
[6] M. Hassan, V. Choutas, D. Tzionas, and M. J. Black. Resolving 3d human pose ambiguities with
3d scene constraints. 2019 IEEE/CVF International Conference on Computer Vision (ICCV),
pages 2292, 2019.
[7] M. Hassan, V. Choutas, D. Tzionas, and M. J. Black. Resolving 3D human pose ambiguities
with 3D scene constraints. In International Conference on Computer Vision, Oct. 2019.
[8] E. Hedlin, H. Rhodin, and K. M. Yi. A simple method to boost human pose estimation accuracy by correcting the joint regressor for the human3 6.mmn...mmmmr dataset. ArXiv, abs/2205, 000066, 2022.
[9] B. Huang. L. Pan, Y. Yang. J. Ju, and Y. Wang. Neural mocon: Neural motion control for physically plausible human motion capture. ArXiv, abs/2203. 14065, 2022.
[10] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattem Analysis and Machine Intelligence,
jul 2014.
[11] A. Kanazawa, M. J. Black, D. W. Jacobs, and Malik. End-to-end recovery of human shape and pose. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7122-7131, 2018.
[12] A. Kanazawa, J. Y. Zhang, P. Felsen, and J. Malik. Learning 3d human dynamics from video. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5607-5616, 2019.
[13] M. Kocabas, N. Athanasiou, and M. J. Black. Vibe: Video inference for human body pose and shape estimation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5252-5262, 2020.
[14] M. Kocabas, C.-H. P. Huang, J. Tesch, L. Muller, O. Hilliges, and M. J. Black. Spec: Seeing people in the wild with an estimated camera. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1 11015-1 1105,, 2021.
[15] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis. Learning reconstruct 3d human pose and shape via model-fitting in the loop. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2252-2262222
2019.
[16] J. Li, C. Xu, Z. Chen, S. Bian, L. Yang. and C. Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3383-3393, 2021.
[17] X. Li, S. Liu, K. Kim, X. Wang, M.-H. Yang, and J. Kautz. Putting humans in a scene: Learning affordance in 3d indoor environments. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1
2019.
[18] T.-Y. Lin. M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.
Microsoft coco: Common objects in context. In ECCV, 2014.
[19] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: a skinned multi-
person linear model. ACM Trans. Graph., 34:248:1-228:111 2015,
[20] Z. Luo, S. A. Golestaneh, and K. M. Kitani. 3d human motion estimation via motion compres-
sion and refinement. In ACCV, 2020.
[21] Z. Luo, R. Hachiuma, Y. Yuan, and K. Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. In Advances in Neural Information Processing Systems, 2021. [22] N. Mahmood, N. Ghorbani, N. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture as surface shapes. 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pages 5441-5450, 2019.
11

---

[23] A. Monszpart, P. Guerrero, D. Ceylan, E. Yumer, and N. J. Mitra. imapper: Interaction cerracttoo-guidedd joint scene and human motion mapping from monocular videos. ArXiv, abs/ 806.0788. 2019. [24] G. Moon, J. Y. Chang, and K. M. Lee. Camera distance-awand top-down approach for 3d multi-
person pose estimation from single rgb image. 2019 IEEE/CVF International Conference on
Computer Vision (ICCV), pages 101 101441 2019.
[25] G. Moon and K. M. Lee. 121-meshnet: Image-to to-lxell prediction network for accurate 3d human pose and mesh estimation from a single rgb image. ArXiv, abs/2008. 0030008...77,,.. 2020.
[26] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975-10985, 2019.
[27] D. Pavllo, C. Feichtenhofer. D. Grangier, and M. Auli 3d human pose estimation in video with temporal convolutions and remi-supervised training. 2019 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pages 7745-7754, 2019.
[28] X. Peng. M. Chang. G. Zhang, P. Abbeel, and S. Levine. Mcp: Learning composable hierarchical
control with multiplicative compositional policies. In 2019.. 2019.
[29] X. B. Peng. P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep
reinforcement learning of physics-based character skills, ACM Trans. Graph., 37:143:1-143:14
2018.
[30] N. Reddy, L. Guigues, L. Pischulini, J. Eledath, and S. G. Narasimhan. Tessetrack: End-to-end learnable multi-person articulated 3d pose tracking. 2021 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pages 15185-15195, 2021.
[31] D. Rempe, Birdal, A. Hertzmann, Yang, S. Sridhar, and L. J. Guibas. Humor: 3d human motion model for robust pose estimation. In International Conference on Computer Vision
(ICCV), 2021.
[32] J. Romero, D. Tzionas, and M. Black. Embodied hands: Modeling and capturing hands and bodies together. ArXiv, Abs/2201.02610., 2022.
[33] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe. MeTRAbs: metric-scale truncation-robust heatmaps for absolute 3D human pose estimation. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(1):16-3) 2021.
[34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. ArXiv, abs/1707 1000000677, 2017.
[35] S. Shimada, V. Golyanik, W. Xu, P. Pérez, and C. Theobalt. Neural monocular 3d human motion
capture. ACM Transactions on Graphics, 40(4), aug 2021.
[36] S. Shimada, V. Golyanik, W. Xu, and C. Theobalt. Physcap: Physically plausible monocular 3d motion capture in real time. ACM Trans. Graph. 39:235:1-235: 2020.
[37] J. Song. X. Chen, and O. Hilliges. Human body model fitting by learned gradient descent.
ArXiv, abs/2008..00 00474, 2020.
[38] S. Starke, H. Zhang, T. Komura, and Saito. Neural state machine for character-scene
interactions. ACM Transactions on Graphics (TOG), 38:1 14, 2019.
[39] Y. Tian, H. Zhang, Y. Liu, and L. Wang. Recovering 3d human mesh from monocular images:
A survey. ArXiv, abs//203.01 2029222,, 2022.
[40] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026-5033,
Oct. 2012.
[41] Z. Wang, I.. Chen, S. Rathore, D. Shin, and C. C. Fowlkes. Geometric pose affordance: 3d
human pose with scene constraints. ArXiv, has/1905.07718, 2019.
[42] K. Xie, T. Wang. U. Iqbal, Y. Guo, S. Fidler, and F. Shkurti. Physics-based human motion
estimation and synthesis from videos. 202/ IEEE/CVF International Conference on Computer
Vision (ICCV), pages 11512-11521, 2021.
[43] Y. Yuan, U. lqbal, P. Molchanov, K. Kitani, and J. Kautz. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[44] Y. Yuan and K. Kitani. 3d ego-pose estimation via imitation learning. In The European Conference on Computer Vision (ECCV), Sep. 2018
[45] Y. Yuan and K. Kitani. Ego-pose estimation and forecasting as real-time pd control. In IEEE International Conference on Computer Vision (ICCV), pages 10082-10092, Oct. 2019.
[46] Y. Yuan, S.E. Wei, T. Simon, K. Kitani, and J. Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[47] S. Zhang, Y. Zhang, F. Bogo, P. Marc, and S. Tang. Learning motion priors for 4d human body capture in 3d scenes. In International Conference on Computer Vision (ICCV), Oct. 2021.
12

---

[48] S. Zhang. Y. Zhang, F. Bogo. M. Pollefeys, and S. Tang. Learning motion priors for 4d human body capture in 3d scenes. ArXiv, abs/2108. 103999 2021. [49] S. Zhang. Y. Zhang, Q. Ma, M. J. Black, and Tang. Place: Proximity learning of articulation
and contact in 3d environments. 2020 International Conference 3D Vision (3DV), pages 642-651, 2020.
13

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec.5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the supplementary material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code included in supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Included in supplemental material
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "gCyMjZFnoI7", "reviewer": "Reviewer_wM8A", "summary": "In this paper, the authors propose a scene-aware pose estimation framework based on the concept of embodiment. Different from previous work with multistage optimization, non-causal inference, and complex contact modeling for the scene-aware pose, this method is simple(only one stage) and recovers robust scene-aware pose in the simulated environment. Besides, to simulate good poses, they disentangle the camera pose and use a multi-step projection gradient defined in the global coordinate frame as the movement cue for our embodied agent.\nThis method only needs 2D observation and can be trained on synthetic datasets for real-world pose estimation. They achieve the state-of-the-art on PROX dataset without using any training data in this dataset.", "questions": "1) It would be better to discuss how the contact information models in this paper. I think it is one of the most important differences between this work and other simulation-based mocap methods. \n2) Does this method need to train new agents for different motion sequences?", "limitations": "Please follow the weakness.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The main contribution--simulating scene-aware humanoid to mimic visual observation and recover 3D poses in the real-world environment is significant. I believe it will be a good baseline for the scene-aware pose estimation.\n+ This method achieves promising results on the challenging poses of PROX dataset.\n+ The MGP is reasonable on 3D pose recovery and achieves significant improvements shown in ablation studies.\n+ This paper is well written and easy to follow.\nWeakness:\n-The simplified scene representation causes some failure cases in the poses of sitting or laying.\n- The contact modeling between motion and scene is a little bit unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "fTq_pmTWJLA", "reviewer": "Reviewer_aaAC", "summary": "This paper aims to perform single-camera pose estimation. It uses 2D key points and then uses a 3d simulated scene that the person is within, to enable an improvement in the pose estimation. The approach is tested in H36M and PROX datasets", "questions": "For the  MPG whatabout if temporal prediction neural networks such as LSTMs or transformers were used instead?\nCould a further dataset demonstrate the performance better than H36M", "limitations": "none stated", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe work doesn't require complex contact modelling or multi-stages of optimization processes, it is a single inference step.\nThe work proposes a temporal gradient projection to smooth the estimation over time.\nThe work is able to perform well on the POX dataset\n\n\n\nWeaknesses\nThe H36M dataset is ill-suited for scene modelling as there are few objects within the foreground and seems to perform poorly and is distracting for the paper\nThe MPG is heavily based on the work of [3]\nThere is very limited experimental discussion around the work of the PROX dataset, making it hard for a reader to fully appreciate the performance of the approach\nThere is a disconnect between the 2D keypoint detection and the 3D/depth \"perfect\" generation of the scene, ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cVRMx7QTUy", "reviewer": "Reviewer_SrSt", "summary": "The paper proposes a method for estimating 3D human pose given a monocular RGB video observing a human moving through a scene tha has been previously reconstructed (i.e. a scene for which a 3D mesh reconstruction is available).  The method is based on physical simulation of the human moving through a geometrically simplified version of the scene.  The simulation is driven by a multi-step projection gradient connecting 2D pose keypoints to the controller that drives the humanoid pose in simulation.\n\nThe experiments evaluate the proposed method as well as ablations against a breadth of methods from prior work on two datasets: PROX and H36M.  Performance is evaluated in terms of pose accuracy (mainly joint error metrics where ground truth pose is available) and pose plausibility (mainly distance to scene geometry, interpentretation frequency and distance etc.). The results show that the approach is competitive with prior work that uses image features on the H36M dataset, and mostly outperforms prior work on the PROX dataset.", "questions": "I would like the authors to respond to the point regarding scene simplification.", "limitations": "Limitations are described in the last section of the paper, mainly focusing on the simplification of human bodies and scene geometry to rigid bodies.  As far as I can tell, there is no discussion of potential negative societal impacts (despite a statement pointing to the supplement in the checklist).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The physics-based formulation is novel and offers a complemetary approach to the 3D human pose estimation problem statement relative to most prior work.  The fact that it can provide strong performance is quite impressive given the limited amount of input information that is given compared to other methods\n+ The paper is well-written and presents a breadth of ablations that analyze the impact of different components of the method on performance\n\nWeaknesses\n- The approach relies on a \"semi-automatic\" simplification of the 3D scene geometry to address failure modes due to reconstruction noise.  I would have liked to see a more detailed discussion of this issue and also of the actual process for creating these simplified scenes.  This is an important detail as it constitutes a significant modification to the input available to the presented method relative to prior work.  Ideally, there would be an \"ablation\" that would report results based on the original (un-simplified) scene geometry to concretely measure the impact and value of this semi-automatic step", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b9WKbfEp1CF", "reviewer": "Reviewer_fkcV", "summary": "The paper presents a 3d human pose framework based on both simulators and 2d/3d human estimators. It recovers both global 3D human poses and local human poses. The methods can be applied to daily activities such as PROX dataset.", "questions": "As I understand, the geometry in PROX is soft, is that reasonable to fully treat the sofas as hard/always flat? should we consider the deformation during the interaction of the geometries?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIt sounds novel to me as I am not a simulator guy but a learning and 3d person guy. Using these simulators naturally will help generate lots of physics priors and encode the physics into the learning process.\nI may not provide enough evidence for the novelty of the simulation part.\n\nWeaknesses:\nI did not see the comparison with \"Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture\", if so, what is the difference, can you work on the dataset they are working on?\n\nI think the performance is somewhat unsatisfying, we can see several metrics in Table 2 is not SOTA.\n\nSeveral typos: \nline 287:  scene penetration frequency Freq. and distance Pen..\nline 204: keytpoints\nline 202: rotationrR ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept