Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos
Authors: Cuong Le
Viktor Johansson
Manon Kok
Bastian Wandt
Venue: NeurIPS 2024 poster (2024)
Keywords: human motion
dynamics
optimal state
kalman filter
physics-based
Primary area: machine_vision

Abstract: Human motion capture from monocular videos has made significant progress in recent years. However, modern approaches often produce temporal artifacts, e.g. in form of jittery motion and struggle to achieve smooth and physically plausible motions. Explicitly integrating physics, in form of internal forces and exterior torques, helps alleviating these artifacts. Current state-of-the-art approaches make use of an automatic PD controller to predict torques and reaction forces in order to re-simulate the input kinematics, i.e. the joint angles of a predefined skeleton. However, due to imperfect physical models, these methods often require simplifying assumptions and extensive preprocessing of the input kinematics to achieve good performance. To this end, we propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting, inspired by a neural Kalman-filtering approach. We develop a control loop as a meta-PD controller to predict internal joint torques and external reaction forces, followed by a physics-based motion simulation. A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion, resulting in an optimal-state dynamics prediction. We show that this filtering step is crucial to provide an online supervision that helps balancing the shortcoming of the respective input motions, thus being important for not only capturing accurate global motion trajectories but also producing physically plausible human poses. The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics, compared to state of the art. The code is available on https://github.com/cuongle1206/OSDCap.

Optimal-state Dynamies Estimation for Physics-based Human Motion Capture from Videos
Cuong Le Viktor Johansson' Manon Kok and Bastian Wandt'
DDparrmment of Electrical Engineering, Linköping University, Sweden Delft Center for Systems and Control, Delft University of Technology, The Netherlands
Noisy kinematics
Optimal-State Estimation
Physics
Figure OSDCap an optimal-state dynamics estimation (cyan) based on two streams input motion, kinematics-based pose estimation from videos (top-left), and physics-based simulation by a meta-PD controller (bottom-left). The predicted motion is poysically-plausible contains reduced high-frequency noise, while retaining highly accurate global position.
Abstract
Human motion capture from monocular videos has made significant progress in recent years. However, modern approaches often produce temporal artifacts, e.g. in form of jittery motion and struggle achieve smooth and physically plausible motions. Explicitly integrating physics, in form internal forces and exterior torques, helps alleviating these artifacts. Current state-of-the-art approaches make use an automatic PD controller to predict torques and reaction forces in order to re-simulate the input kinematics, i.e. the joint angles of predefined skeleton. How- ever, due to imperfect physical models, these methods often require simplifying assumptions and extensive preprocessing of the input kinematics achieve good performance. To this end, we propose novel method to selectively incorporate the physics models with the kinematies observations an online setting, inspired by neural Kalman Kalman-filterig approach. We develop a control loop meta-PD controller to predict internal joint torques and external reaction forces, followed by physics-based motion simulation. A recurrent neural network is introduced realize Kalman filter that attentively balances the kinematics input and sim- ulated motion, resulting in an optimal-state dynamics prediction. We show that this filtering step is crucial to provide an online supervision that helps balancing the shortcoming of the respective input motions, thus being important for not only capturing accurate global motion trajectories but also producing physically plausible human poses. The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics, compared to state of the art. The code is available
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1 Introduction
Three-dimensional human motion estimation is a mtandinan and challenging research goal in computer vision, particularly in monocular scenarios due to inherent depth ambiguities. Previous approaches have incorporated kinematic priors, e.g. by enforcing smoothness, maintaining bone length constancy, or imposing symmetry constraints. However, due to inconsistencies of frame-wise predictions, these solutions do not necessarily lead to physically plausible motions. This has led to the emergence of a new research direction that combines traditional 3D motion estimation with physical models of the human skeleton. Instead of directly predicting a human pose, these approaches estimate the internal joint torques and exterior forces that drive the motion. Consequently, physics 50 21).
simulators are employed to obtain the resulting motion 37 38
However, since simulators are never perfect representations of the real world, they introduce inevitable errors, where the complex human body was never fully modelled, only approximation by rigid body dynamics [3] Moreover, measurements, including "ground truth" recordings, are inherently noisy. To tackle these problems, we propose OSDCap, state-aware architecture that combines a differentiable physical simulation with our novel neural Kalman filtering approach. Fig. [ shows our reconstructed poses predicted from noisy kinematics estimates as well as the estimated dynamics from video.
OSDCap is an online filtering and dynamics estimation that can be trained in an end-to-end manner. In detail, our approach consists of two steps, starting from a noisy kinematics reconstruction obtained by an off-the-shelf video-based 3D human pose estimator: 1) simulation branch that estimates joint torques using PD controller and computes the resulting motion, 2) an adaptive filtering stage that combines the output of the simulation stage and the rideo-based kinematics input to produce refined motion. We follow prior work that utilizes the meta-PD algorithm for torque calculation [38] 21 to simulate plausible motion. However, the effectiveness of the PD algorithm heavily relies on the choice of the P and D gains 38 and on an accurate model of the human kinematic chain, which is generally unknown. Moreover, the measurements from the monocular 3D kinematics pose estimator contain a large amount of noise, which ultimately leads to inaccurate predictions. Shimada et al. [38] mitigate these problem by introducing an additional offset term into the PD controller. While this approach still produces reasonable output motion it is neither physically explainable nor consistent with PD controllers in control theory. We aim to solve this problem at the root by taking inspiration from control theory and propose solution for processing the imperfect PD calculation by learnable Kalman filtering method [36]. The proposed Kalman filter takes the simulated motions and the noisy 3D pose estimation inputs, combines them, and produces optimal state prediction as the output. The Kalman filter effectively refines the PD controller-based simulated motions into more plausible and realistic motions. While the Kalman filter fixes inaccurate kinematic measurements from the 3D pose estimator does not take different weight distributions in the human body parts into account. We calculate an initial weight distribution the inertia matrix for an average human body shape. However, as for the skeletal structure, these are only approximations that lead to inaccurate simulations. We mitigate this issue by predicting an inertia bias matrix in each time step which is added to the initial inertia matrix.
We demonstrated the 3D reconstruction performance of our method the popular Human3.6 3.6 [15] dataset, and the newer Fit3D (7] and SportsPose [14 datasets, comparing them with recent of the-art physics-based methods.
In summary, OSDCap introduces a new physics-based human motion and dynamics estimation method leveraging learnable Kalman filter and learnable inertia prediction, that produces plausible motion as well as valuable estimates of exterior forces and internal torques. By offering improved accuracy and interpretability in human motion estimation, OSDCap presents promising step towards bridging the gap between computer vision and the complex physics-based human motion modeling.
2 Related Work
2.1 Kinematics 3D Human Motion Capture
Monocular 3D human motion capture is well-studied line of research, with common approaches that can be roughly divided into two groups, 1) end-to-end approaches that directly predict human poses from images (39 29), and 2) lifting from 2D
[ 26
28
12
4
30 42
2
10
44
22
47
43 31..
Recent work addresses the problem by fitting volumetric models to 2D/3D evidence, aiming to
achieve realistic human motion [27 17 25
19
20
48
45
g 23 53 40) Despite the significant

---

progress, vision-based human 3D pose estimation is still an ill-posed problem, due to the loss of depth information from the monocular setup. Therefore, captured 3D motions often contain different types of implausibility, ranging from unnatural poses, jittering, or unrealistic body artifacts 46 g].
2.2
Physics-based 3D Human Motion Capture
enforce physics as constraints for motion reconstruction, eliminating implausible artifacts created by the monocular estimation, i.e. jittering, ground penetration, and unnatural human poses.
Recent studies 37 50 46 8
21
Motion imitation using reinforcement learning (RL) is popular approach for simulating physically plausible results [32 49 50 33 51]. RL-based methods enforce physics constraints in the reward functions, either from manually -eesige formulas, from physics engines. The bottleneck of RL-based approaches is the low transferability of the learned policies to unseen motions.
Motion optimization is another common approach for physics-based human motion capture. However, optimization problems often require a differentiable framework, thus, instead of relying on non- differentiable physics engines, prior studies <4 337
46 adapt simplified motion equations [5] as a dynamics constraint for simulated motions. More recent approaches [13 9 manage to optimize through non-differentiable simulation using evolutionary optimization methods [I]]. Gärtner et al. 8 implement a differentiable version of PyBullet [3) resulting in an optimizable framework with complex physics engines. However, most optimized motion solutions, similar to RL-based solutions, have limited adaptability to different data distributions, requiring re-optimizing on new sets of action.
Utilizing the generalizability of neural network models in an end-to-end manner is still an open line of research, due to the difficulty of finding physical plausibility patterns from data. Rempe et al. 35 utilizes variational autoencoder architecture for predicting plausible motions, approximating the dynamies simulation by decoder network. This assumption might result in unrealistic force prediction with respect to biomechanics literature. Li et al. (21] utilize the meta-PD controller with learnable parameters for torque prediction, but with an additional compensation term based on root residual forces. Zhang et al. [52] realize transformer-base
autoencoder to refine kinematies input sequences, while integrating physics constrain inside the latent embedding. However, both Li et al. 21 and Zhang et al. (52) make predictions based on the encoding of the full motion sample, i.e. they require knowledge of past and future motions, therefore, limiting the applicability of the method to offline setups, where future information is available. Shimada et al. (38) also
meta-PD controller for calculating the optimal joint torques, which in turn generates a simulated motion matching visual kinematics estimation. Despite the plausibility of the estimated pose, the global precision of the motion in world coordinate is limited and not fully addressed.
We aim to leverage physics based approach (with meta-PD controller) motion data captured by monocular camera systems, in a recursive online setup, and expand the prediction to more complex practical movements such as sports.
3 Method
This section presents our proposed approach OSDCap in detail. We start by creating an average proxy character [B] based on the uniform human configuration from the Mluman3.6M dataset (15). The character approximates a human body by circles and cylinders. Additionally, we leverage the pretrained neural network TRACE [40] obtain initial 3D pose estimation. Without any additional priors, OSDCap aims to predict the joint torques and external forces that drive the proxy character to match the kinematics evidence given by TRACE. Following prior work, employ a neural network that predicts the parameters of a meta-PD) controller which consecutively predicts the joint torques. While related approaches [38 21) stop here, we note that the quality of the motion given by the PD controllers' prediction highly depends on the realism of the sadeos-basce kinematic estimation and the proxy character. Since model can always only be an approximation of the real world, this leads to inaccurate predictions which prior work compensates for by adding an additional offset term to the PD controller. Unfortunately, this not only introduces a non- non-physical assumption but through experimentation we found that this term attributes the major part to the prediction of the PD controller. We aim to maintain the physical plausibility of our approach by introducing a novel filtering approach inspired by neural Kalman filter (36] to refine and update the motion states. Additionally, at each time step, the foot contact states and ground reaction forces are estimated directly from the motion leading to a full description of the system dynamics. Fig. 2 shows overview of our method.
3

---

Adspattion
pass
Syeamaraat
Kaiman Optimaa estimation
Systnm
Haman
Observation
Human pous Quuppst
State Tranaittoning Modo. besod
Prodict stute f1-sy
Wessity
Velocity Opritts
Kattnon gans
OdD
11 stad
GRU
Velcoity update Physios simulation
NN
PD giins
PD dige thm
ic:ga
Physics flgid body dynamics
Extemet
Forward Dynamics
Compostic
Marri
Nowtonian
Og body thin
Nun-lincero
Equarion of Motion
Figure The main pipeline of OSDCap. Our approach consists of one neural network model, OSDNet (orange), and three processing components. OSDNel takes the current system state, estimates a Kalman gain matrix, PD gains, external force and an inertia-biat matrix. The optimal pose estimation performs contains Kalman filter for the current system state and the input kinematics. Yellow refers to the algorithm's 's state vectors and cyan denotes processing operations. The physics priors block (gray) computes the inertia matrix and non-linear forces using the Composite rigid-body algorithm and Inverse dynamics (5) Using the PD algorithm and forward dynamics (Eq. []]. the physics simulation block (green) updates the velocity based on the computed optimal pose and physics priors.
3.1 Preliminaries Rigid Body Dynamics
Similar to previous studies 34 37 38 46), we enforce physics constraints based on Rigid Body Dynamics [5]. inline with the Newtonian equation of motion. For total of N keypoints, the full vector q E R6+3N, encoding the global translation and rotation in the first entries, and internal joint angle states in the remaining 3N entries. q E R6+3N is the corresponding velocity vector. The motion dynamics of the captured human poses should satisfy the Newtonian equation motion, expressed as
human pose is represented
- h(q. q),
(I)
where M(q) ((6++ is the inertia matrix, computed from the character, q e R6+3N is the acceleration, T R6+3N are the internal joint torques, À E R are the external forces, and h(q. q) E R is the non-linear term including gravitational, Coriolis, and centrifugal forces, computed using inverse dynamics with zero acceleration on the proxy character [6. Our goal is to estimate the two vectors T and À that produce plausible motion dynamics.
3.2 Optimal-state Dynamics Capture
The proposed OSDCap consists of three main processing stages: optimal pose estimation, physics priors calculation, and a velocity update based on physics simulation. The optimal pose estimation phase is filtering approach inspired by KalmanNet (36) that estimates an optimal output pose based on the current system state and video-based 3D kinematics inputs. The physics priors calculation computes the current inertia matrix and non-linear forces based on the proxy character from the current system state. The physics simulation phase computes the next velocity state using the PD algorithm and forward dynamics (Eq.) D from the estimated optimal pose.
The required inputs for the optimal pose estimation and physics simulation are estimated by our neural network OSDNet. OSDNer consists of two modules, one predicts Kalman gains for the Kalman filtering, and the other predicts PD gains, external forces and inertia-bias for the physics simulation.
Optimal Pose Estimation. As shown in Fig. 2 the Kalman filtering block takes the current system state and the videos-based kinematies pose inputs. Following traditional Kalman filters the next predict state is computed from the previous state. We define the state transitioning phase as
qr+1jt
qoje + At.
(2)

---

The predicted positional state 4 is the physics consttaeen body pose. The observation matrix H (Fig. 2 maps the predicted states Qt+1|t to an observed simulated positional state. an adaptation matrix to reduce the gap between observed states from videos, and observed states from physics simulation. H and C are optimized along with OSDNer while training, but stay constant during inference. From the current system states, a Kalman state update process is performed as
qr+1|e+1 = Qr+1je + K2 (Cq: Hqj+)je)
(3)
where K2 contains the estimated Kalman gains step t based on the current states and observations. Inspired by [36]. the Kalman gains estimation module is implemented as Gated Recurrent Units, which have the ability to propagate the latent motion dynamics throughout the simulated motion via the hidden states of the GRU. The prediction of Kalman gains requires information about the the system's s state dynamics (36), thus four additional dynamies features need to be feed into OSDNet's GRU input, namely: observation, innovation, forward evolution and forward update. They are calculated as
Aevolution A:lt 9:-1kt11
Aupdate 4:le 9:/t11
Ainnovation Câ Hqu+1j
Aobservation = C. qr-
We modify the original design from [6) due to the practical reasons of our system. In self-occluded scenarios, the noisy input q often contains artifacts such as body deformation and they often last for a period of time (approx. 10 frames). The intermediate difference between q. and q:-1 in the original design [36] is not strong enough to model those artifacts, because they could both contain the same incorrect kinematic estimation. We change the calculation of Aobservation as in Eq. 4 better deal with mis-detection cases that would cause large responses in Lobservation. The pose q: + is the optimal state step and inherits the global translation estimation from kinematics observations while retaining the physical plausibility of the human pose from the physics simulation.
Physies Simulation. The purpose of the physics simulation stage in Fig. i to update the velocity a that best describes the dynamics of the filtering process. Therefore, the estimated pose qr+ 4r+1/r-11 can be used as the target signal for the PD algorithm, calculating the joint torque T, that maps the predict pose Qs+1je to the optimal pose q++1 4+1- The joint torque is predicted by the PD algorithm
(5) where KP, ED are proportional and derivative gains respectively. Inspired by [38. the meta-PD controller was applied at this stage, where Kp, KD are learnable and estimated from OSDNet. By using the filtered optimal pose as the target, no unrealistic temporal filtering or optimization needed to refine the noisy kinematics inputs.
T K k - qe-1je) + K
Additionally; the external forces are also estimated by OSDNet, assuming the source of external forces comes only from contact points and is computed as
-.ipprs J
(6)
where J is Jacobian matrix that maps linear velocity at contact point to rotational velocity of every other joints, Pf and f are the contact probability and the linear force vector at contact c. The three vectors are separately estimated by OSDNet.
Inertia Estimation. Since do not have access the real bone length and mass distribution of the human, there exists a knowledge gap between simulated human character and the real human subject, the inertia tensor computed by the composite rigid-body algorithm is sub-optimal. OSDNet is designed to also estimate an inertia bias term M, that reduces this knowledge gap. The required acceleration to drive the current simulated pose to the next states calculated in Eq. 7
q (M + MT) A: h(q., 4:)),
To update the system state, finite interpolation is applied, using the newly calculated accelcration ğ.. The update process is given by
(8) where 4 is the updated system state that represents the current system dynamics, under physics constraints from gravity and contact forces. The system now proceeds back to the transitioning phase in Eq. 2 creating a closed loop process that works recursively.
4 = ف + at
5

---

3.3 Objective Losses
To reconstruct the optimal state, we define the overall objective loss as a weighted sum of multiple loss functions as
T Σ = T
(w) lig P + +w2 9 + w3 w + wa 1.e 91+11 + wa Ti + I,()):
Pr+ils
where W1 = 0.5, = 0.1, 0.7,W4 w4 = 0.2, wb = 0.4 are weighting factors. The optimal reconstruction losses Lt 1 and L, I. measure the L1 distance between the estimated optimal pose 9(+1/(+1 and its corresponding 3D keypoints (obtained from forward kinematics) with the ground-truth poses 9:+1 and ground-truth 3D keypoints P The supervision for predict pose 11+ is carried out similarly, ensuring the correct behaviour of the physics simulation. Lę is the contact loss, using Binary Cross Entropy measurement between the predicted contact probabilities pi of two feet with
contact binary labels PE We generate the ground truth contact labels for training based on foot-ground distances of ground-truth 3D keypoints. The individual losses are computed
6+3N
L Pt4i|t+1
Σ PPT Pr41j44lll
L gc-iltt1
Σ
9
6-3N
L Petali
P Pettjell
Lt
9
GT
4
(10)
Li
Pi log(p;) + PE) log(1 PE).
By re-introducing a part of the noisy kinematics measurements into the prediction, an additional regularization loss _ beneficial ensure smoothness and plausibility of the output motions. The regularization consists of three objectives: 1) L is the acceleration loss, computed as the absolute difference between q and 1 2) T.rel is the velocity loss, measuring the distance between the first-order difference of ground-truth motion q. and of estimated optimal q-+1, and 3) The friction
loss L, fric encourages the feet to stay in the same position during ground contact. With the regulator weighting of W6 0.14, w7 0.03, ws 0.28, 1 is expressed as
T,
we Tace
+ W7 T.uel
el + wg T.ç fric
2 -4)ll wB
Σ gg. ا || w7 Σ
GT 9
GT [((
Σ
pi (pi+1 p
w6
(11)
4 Experiments
4.1
Datasets
We evaluate our approach on two human motion benchmark datasets. The first and main dataset the popular Human3.6.. dataset [15]. The dataset contains indoor 3D human motion capture
data, including 2D and 3D keypoints, skeleton joint angles, and videos. Seven actors perform 15 different actions. Following previous work [38] 21, the first five subjects (S1, $5, S6, S7, S8) used for training, and the last two (S9, S11) for evaluation. According to [77], only actions that have foot-ground contacts were considered. Details about the selected sequences are found in the supplemental document[C]
The second database is Fit3D 7 Fit3D contains indoor motion capture data for variety of exercises. We split the data by taking samples from the actors (s03, s04, s05, s07, s08, s10) for training, and actors (s09, s s1 for evaluation. inspired by the setup from [37] on Human3.6M.
Since the scene setting from Human3.66 .6M and Fit3D are very similar, we perform an additional evalu- ation on the new dataset SportsPose (14), which consists of video-based sport action sequences with corresponding ground truth 3D keypoints. We use this dataset show oft-of-domain performance, since the 3D kinematics estimator TRACE 44 has not been trained on it.

---

4.2 Implementation Setups
The initial motion observation is generated by TRACE 40) As suggested by [38 8] all extracted motions are down-sampled from 100Hz 50Hz. The samples are aligned to the world origin in the first frame, then split into 100-frame sub-sequences to utilize batch training and evaluation. The proxy character is created with respect to the provided skeleton metadata Human3.6M [15]. including the mean bone lengths and joint angles configuration. The inertia matrix and bias force (including gravitational. Coriolis, and centrifugal forces) are calculated online using RBDL based on the state of the proxy character.
We train OSDNet in an end-to-end procedure. OSDNet consists of three fully-connected layers, followed by six different heads for PD gains pk,, Kp), inertia bias (M'), contact probability (p"), linear external force from the ground (A), and Jacobian matrix (J). These six entries are responsible for the motion simulation phase, following Eq. D and 6 The GRU units in the proposed optimal-state prediction module (ef. Fig. 2 take current system states, additional dynamics features (Fqq. 4. and its hidden state hagu as inputs. The output is the Kalman gain-matrix for the Kalman update process. For a details descriptions of the OSDNet's architecture, please refer to the supplementary document
OSDNer is trained for 15 epochs with a base learning rate of 5e and a batch size of 64. The learning rates from all training processes are scheduled reduce by factor of 10 at epochs 10 and 13. LeakyReLU and Layernorm are used as the activation function and normalization for each linear layer of every module. We also apply training warm-up strategy the first 5 epochs by increasing the learning rate by factor of 2 to the base learning rate at epoch 5. This helps reducing the impact of unstable physics simulation at the beginning of training, mitigating gradient explosion.
4.3 Metrics
There are two standard protocols for the evaluation on Human3.6M 66M [15.. Both of these protocols assess the Mean Per Joint Position Error (MPJPE). This metrie represents the average Euclidean distance between the reconstructed joint coordinates and the provided ground truth 3D keypoints. While the first protocol directly calculates the MPJPE for root-aligned poses, the second protocol initially employs rigid alignment between the poses which is called MPJPE-PA (MPJPE Procrustes Aligned). Since our approach estimates poses in global coordinate system, we additionally calculate the MPJPE-G in global coordinates which is the MPJPE without frame-wise root alignment. In addition to the different variations of the MPJPE, the Percentage of Correct Keypoints (PCK) measures the percentage of predicted joints that are within a distance of 150mm or less from their corresponding determines pose as correct only
ground truth joint. Unlike the PCK, the CPS measurement all its joints are estimated correctly according to threshold value, similar to the PCK. To ensure independence from a specific threshold value, the CPS computes the area under the curve within the 1mm to 300mm threshold range. To evaluate the global translation error, not accounting for the differences between poses, we report the global root position (GRP) error, which calculates the Euclidean distance between only the root joints. We also use the acceleration (Accel) metric from 19 to measure the jitter of the output motions. Accel is computed as the second-order difference between 3D keypoints across all sequence frames.
4.4 Comparison with State of the Art We report the quantitative results of OSDCap and other related work on different metrics in Tab. I Due to the novelty of dynamics-based motion capture the evaluation protocols differ significantly across different approaches. Here, we make an effort of consistently structuring approaches with similar evaluation protocols to achieve a fair comparison. To be comparable possible we follow the most used protocol introduced by Shimada et al. [37]- We outperform all online approaches in MPJPE, PCK and CPS. For the global error MPJPE-G, we improve upon state of the art by a large margin. Notably, DnD (21) achieves a lower MPJPE-PA. However, DnD's 's estimation depends on encoding the full action sequence, extracted from temporal convolutions, assuming significantly more knowledge which is not suitable for an online setting. Moreover, AMASS (2] is used as an additional training data source, thereby, not following the standard protocols for Human3, 6M. SimPoE [50] achieves best smoothness performance on the Accel metric, due to being constrained by a high-frequency physics engine. However, as the discussion in Sec. [ only relying on modeling the physics can lead to sub-optimal human pose quality. IPMAN-R [4] also shows good performance in terms of MPJJEEPP. However, it is a single-image approach that contains physically inspired constraints such as ground penetration, but no dynamics. The MPJ.E-PA, i.c. the MPJPE after
7

---

Methods
Phys. Onl.
MPIPE 4
MPIPE-G 4
MPIPE-PA.
PCK
CPS
GRP
Accel 1
[mmm]
[mmm]
[mm]
[36]
[mm]]
[mm]]
[mm /
Vnect 27
89.6
62.7
85.1
185.
79.4
55.0
231.1
HMR [6]
x
78.9
54.3
88.2
204.3 2
TRACE 40
78.1
152.7
62.5
88.3
169.1
125.9
19.2
VIBE 19
68.6
207.7
43.6
23,4
Gärtner
84.0
143.0
56.0
DiffPhy 8
x
81.7
139.1
55.6
PhysPT
36.7
*DnD 21
x
$2.5
35.5
PhysCap 37
V
97.4
65.1
82.3
182.6
NeurPhys 15
V
76.5
58.2
89.5
Xie al. 40
68.1
85.1
IPMAN-R 41
V
60.7
41.1
SimPol:
41.6
OSDCap
54.8.10.1
132.8 1.5
39.8
0.0.1
95.5
9:01
197.7
1:0.1
119.1
8.4:
Table 1: Quantitative comparison on the Human3. 66M dataset 15 Related methods are separated into
two main categories: kinematics (top) and physics-based (bottom). In addition only [37 38 46 50
retains the online prediction ability of the video-based kinematics estimations. Bold numbers denote the best evaluation score on each metric. Our approach achieves state-of-the-an in MPJPE and PCK
among online approaches, and competitive results on GRP and Accel. Note that *DnD [1 does not follow standard evaluation protocols by using additional training data.
MPIPE 4 [mm]
MPIPE-G
MPIPE-PA &
PCK 1 [%]
CPS
GRP
Accel 1 [mmm /
Datasel
Methods
[mm]
[mm]
[mm]
[mm]
TRACE 40 OSDCap
85.4
131.2
65.2
85.5
166.6
178.1
20.2
FiLD M
58.7
73.8
42.6
96.7
209.4
47.2
8.2
TRACE
361.9
168.1
353.0
15.8
SportsPose 44
OSDCap
71.7
113.6
52.4
68.8
190.0
90.2
10.9
Table 2: Evaluation results Fit3D 7 and SportsPose [14. OSDCap improves the kinematics bascline TRACE by a large margin across all metrics. We fine-tune OSDCap (pretrained on Hu- man3.6M) on SportsPose's ground truth keypoints for additional 15 epochs. Even with very noisy inputs from SportsPose, CSDCap still manage to retain the robust estimation thanks to the Kalman filtering process, especially on global translation metrics (MPJPE-G and GRP).
pose-wise rigid alignment, is reported for completeness. While being a reasonable metric for single- image pose estimation, argue that for physics-based pose estimation, rigid alignments distort the interpretation of the results since they remove all information about global rotation.
We additionally evaluate OSDCap on the more challenging motions in the Fit3D dataset 77 Tab. 2 shows the results. Since Fit3D is recorded in the same setting as Human3.6M 6M, we additionally evaluate on the newer SportsPose [14 dataset to show the generalizability to other motion domains. We improve on the kinematics baseline TRACE by large margin, especially for global metrics as shown by the MPJPE-G and GRP. Fig. illlstraaes the benefits of OSDCap. OSDCap significantly reduces the impact of noisy and inaccurate kinematics input when encountering high depth-uncertaii monocular views, while retaining the correct estimation with respect to the ground truth. The bars on the left represent the predicted Kalman gains, where the y-direction the direction of the optical axis which indicates a predicted low trust in the kinematics prediction and leads the Kalman filter to prefer the physics simulation. Fig. 3b shows an example (side view) OSDCap adjusting unnatural leaning into physically plausible pose by our physics simulation.
from
4.5
Ablation study
4.5.1
Optimal-State Estimation
We conduct an ablation study to verify the impact of the optimal-state estimation process on simulated motions. We sample subset of data consisting of only the first action class of all subjects in the camera view 60457274 from Human3.6N, training. This setup creates suitable challenge to test the proposed method, limiting the types of motion that are seen during training. Tab. shows the original simulated result from straight-foward smoothing methods, PD controller and the improvement by OSDCap.
15). and S11 are for evaluation, and the rest for
8

---

#purams. [mm]]
MPIPB [mmm]]
MPIPE-G 4 [mm]]]]]
MPIPE PA 4
PCK
GRP 1
Accel
Methods
[%)
[mm]]
TRACE TRACE (median)
4D
78.4
153.9
62.7
88.1
128.2
19.7
78.2
62.6
88.2
TRACE (Gaussian)
77.8
162.4
62.4
88.5
126.7
6.5
(only) PD (Geussiaa)
844M
87.7
145.0
67.7
82.7
105.9
6,4
8.4M
77.7
136.0
61.0
86.5
103.2
5.2
OSDCap bias)
6.6M
55.0
111.9
40.0
95.7
94.9
OSDCap
7.2M
54,0
111.0
40.0
95.9
94.8
8.7
Table 3: Ablation study on the impact of OSDNet on a subset of Human 3.6M [15]. Naive methods such as median or Gaussian smoothing cannot help with the plausibility of the pose. Without our Kalman filtering process, the PD controller cannot train and estimate the correct dynamics. We also study the effects of the inertia-bias M and some performance gains has been recorded.
Front view
Top view
Side view
Front view
Ground muth Kinemalics
(a) Incorrect global translation.
(b) Unnatural leaning artifacts.
Figure 3: Qualitative results of OSDCap (cyan) compared to the kinematics input [40] (purple), with corresponding ground truth pose (red). Left: Filtering results of OSDCap
sample from SportsPose [1], where the kinematics estimation is very inaccurate along the camera's 's depth dimension. The Kalman gain the y-axis (optical axis) is greatly decreased due the incorrect translation of the kinematics input. Therefore, the simulated state is preferred. Right: Example from Fit3D (7). with an unnaturally leaning pose caused by depth ambiguities. Unlike Fig. 3al the three poses manually separated apart for better visualization. OSDCap recovers the physically plausible upright pose.
Naive approaches for smoothing the noisy input estimation apply temporal filters such median Gaussian filter. However, simply filtering the signal does not help the motion to become physically plausible, unnatural poses still prevail. As shown in Tab. naive filtering reduces the jitter of the input motions (reduction in Accel measurements), but does not help with any other metrics.
To ensure the plausible physics constraints of the forward dynamics process (unlike [37] (3)] we employ external forces into the calculation, which leads to a much more challenging scenario for the PD controller. This can be observed in Tab. 3 where the PD controller struggles to reconstruct the motions, even with temporal filtering on the input signals and increase the number of parameters. By using our optimal-state estimation module, the PD controller has significantly better performance, leading to the optimal results for online human motion reconstruction.
4.5.2 Comparison to classical Kalman Filter
The biggest challenge of using classical Kalman filter for OSDCap is the tuning of unknown noise covariances of both the kinematic imput TRACE [40] and the simulated result from PD controller. Our choice of learnable Kalman filter (36) relieves from trial-and-erre process of finding the correct noise covariance matrices and achieves the best results. We conducted an additional experiment where we replace our learnable filter by a traditional one, the results are shown in Tab. 4
Assuming noise covariances that are constant over time and equal in all directions, the ratio between the noise covariance of the simulated PD controller (process noise) and the noise covariance of the kinematic input TRACE (measurement noise) governs the quality of Kalman filter estimates. The evaluation results can be seen in Tab. where we use constant noise covariances with ratios 100/1, 10/1,1/1 1/10, 1/100 between process noise and measurement noise. While classical Kalman filter approach increases the result marginally, optimal results are difficult to find.

---

MPJPE
MPIPE-G
MPIPE-PA PA
PCK 1
GRP J
Accel
Method
[mm]
[mm]
[mm]]
[[]]
[mm]]
[mm/ |
TRACE
78.4
153.9
62.7
88.1
128.2
19.7
only
127.4
cKF 100/1
60.9
120.7
43.4
94.3
102.0
7.7
10/1
103.0
cKF_
59.9
117.6
43.2
94.8
100.1
6.5
cKF_ 1/10
63.6
124.0
44.3
93.8
102.7
11.5
<KF 1/100
65.3
132.3
44.0
93.5
110.2
9.7
OSDCap
Table 4: Ablation study on the performance the classical Kalman filtering (cKF) on the ablation set from the Human 3.6m dataset. Due unknown noise covariance matrices, we tested with constant noise covariances with ratios 100/1, 10/1, 10/1,1111 1/1, 1/11,,11 1/10, 1 1/100. The performance of applying Kalman filtering on only the kinematics input TRACE 40 (cKF (cKF_kin_ong) is also conducted.
GD
Priction [mm]]
Velocity [mm/el
Foot-skating [%)
Method
[aum]
[mml]
TRACE OSDCap
2.6
12.5
31.5
22.4
37.0
5.3
8.2
14.6
12.8
15.2
Table 5: Additional physics-bases based measurements for kinematics input TRACE and OSDCap. Because the ground penetration (GP) metric does not correctly reflect the foot-ground contact quality, i.e. floating above the ground is ignored and produces error, we propose using additional ground- distance (GD) metric. For foot-skating, we followed DiffPhy to compute the percentage of frames that contain skating artifacts over the whole sequence.
4.5.3 Additional physics-based metrics
We provide additional metrics for physic-based measurements introduced in Sec. 3.3 The results can be seen in Tab. 5 OSDCap helps refining the input kinematics most of the physics-based metrics. Note that TRACE 40) outperforms our approach in the ground penetration metric. The reason is that most cases the TRACE predictions float above the ground, which gives low penetration error but can be seen as equally bad. Thus, we additionally provide a ground distance metric (GD) to reflect the correct foot foot-grous goot-ground quality during contact. The value is computed the mean absolute vertical differences between foot contact points and ground plane during contact duration, expressed as
1
Pelpe
OSD
GT Pc
(12)
where Pe is the predicted binary label of contact, Pe OSD and Pc are the 3D vertical positions of contact. There are total of six contact points considered, three contacts in each foot accounting
for heel, foot and toe. The joint configuration follow Human 3.6M skeleton [15], with bone length between joints optimized during training and fixed during inference.
Conclusion
This paper presents OSDCap, a new physics-based approach to reconstruct kinematics-based human motion captured from monocular videos. We found that previous approaches relying only physical simulation produce non-optimal motions due to unavoidable imperfections in the physical model and noisy measurements. This led us introduce learnable Kalman filtering for refining implausible motions simulated by PD controller with noisy kinematic evidence as the target. In comparison with related research on physies-based motion capture, the proposed approach achieves state-of-the oo-the--rt results on the Human3.6M, Fit3D, and SportPose datasets, especially on the global estimation of pose trajectories.
Limitations and future work. While taking a step into highly accurate predictions of the full body dynamics, our physical external forces still not comparable directly measuring with mechanical force plates. However, our approach only requires a single camera, e.g. from a smartphone, instead of motion capture studio other expensive hardware, such force plates, estimate meaningful forces. In the future, detailed modeling for the hands, feet, and body shape, will be investigated, targeting more realistic motion reconstruction.
10

---

Acknowledgments and Disclosure of Funding
This research partially supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Foundation, and by the Sensor Al Lab, under the Al Labs program of Delft University of Technology. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
[1] Ching-Hane Chen and Deva Ramanan. 3d human pose estimation 2d pose estimation matching. In CVPR, 2017.
[2] Hai Ci, Chunyu Wang. Xiaoxuan Ma, and Yizhou Wang. Optimizing network structure for 3d human pose estimation. In JCCV, 2019.
[3] Erwin Coumans and Yunfei Bai. Pybullet, python module for physies simulation for games, robotics and machine learning. http://pybuulet. org. 2016-2019.
[4] Haoshu Fang, Yuanlu Xu, Wenguan Wang, Xiaobai Liu, and Song-Cln Lang-Chun2 Zhu. Learning pose grammar to encode human body configuration for 3d pose estimation. In AAAI, 2018.
[5] Roy Featherstone. Dynamies rigid body systems. Rigid Body Dynamics Algorithmss pages 39-64. Springer US, 2008.
[6] Martin L.. Felis. Rhdl: an efficient rigid-body dynamics library using recursive algorithms. Autonomous Robots, 41(2):955- 511, 2017.
[7] Mihai Fieraru, Mihai Zanfir, Silviu-Cristin Pirlca, Vlad Olaru, and Cristian Sminchisescu. Aifit: Automatic 3d human-interpretable feedback models for fitness training. In CVPR, 2021.
[8] Erik Gärtner, Mykhaylo Andriluka, Erwin Coumans, and Cristian Sminchisescu. Differentiable dynamics for articulated 3d human motion reconstruction. In CVPR, pages 13190-13200, 2022.
[9] Erik Gärtner, Mykhaylo Andriluka, Hongyi Xu, and Cristian Sminchisescu. Trajectory optimization for ahysics-based reconstruction of 3d human pose from monocular vidco. In CVPR, pages 13106-133 15, 2022.
[10] Ikhsanul Habibie. Weipeng Xu, Dushyant Mehta, Gerard Pons-Moll, and Christian Theobalt. In the wild human pose estimation using explicit 2d features and intermediate 3d representations. CVPR, 2019.
[11] Nikolaus Hansen. The CMA Evolution Strategy: A Comparing Review, pages 75-102. Springer Berlin Heidelberg, Berlin, Heidelberg. 2006.
[12] Mir Rayat Imtiaz Hossain and James J. Little. Exploiting temporal information for 3d human pose estimation. In ECCV, 2018
[13] Buzhen Huang. Liang Pan. Yuan Yang, Jingyi Ju, and Yangang Wang. Neural mocon: Neural motion control physically plausible human motion capture. CVPR, pages 6417-6426, 2022.
[14] Christian Keilstrup Ingwersen. Christian Mikkelstrup, Janus Nørtoft Jensen, Morten Rieger Hannemose. and Anders Bjorholm Dahl. Sportspose: A dynamie sports pose dataset. IEEE/CVF International Workshop on Computer Vision in Sports, 2023.
[15] Catalin lonescu. Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI, 36(7):1325-1333 2014.
[16] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik, End-to-end recovery of human shape and pose. In CVPR, 2018.
[17] Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In CVPR, 2019.
[18] Jeonghwan Kim, Mi- Mi-Gyeong Gwon, Hyunwoo Park, Hyukmin Kwon, Gi-Mun Um, and Wonjun Kim. Sampling is Matter: Point-guided 3d human mesh reconstruction. In CVPR, 2023.
11

---

[19] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, pages 5253-5263, 2020.
[20] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical- neural inverse kinematics solution for 3d human pose and shape estimation. In CVPR, pages 3383-3393, 2021.
[21] Jiefeng Li, Siyuan Bian, Chao Xu, Gang Liu, Gang Yu, and Cewu Lu. D&d: Learning human dynamics from dynamic camera, In ECCV, 2022.
[22] Shichao Li, Lei Ke, Kevin Pratama, Yu-Wing Tai, Chi-Keung Tang, and Kwang-Ting Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In CVPR, 2020.
[23] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022.
[24] Matthew Loper, Naureen Mahmood, Javier Romero. Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model. ACM TOG, (1:::::2:::122::1111 2015.
[25] Naureen Mahmond, Nima Ghorbani, Nikolaus Troje, Gerard Pons-Moll, and Michael J. Black, AMASS: Archive of motion capture as surface shapes. In ICCV, pages 1442-5451, 2019.
[26] Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, 2017.
[27] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with single rgb camera. ACM TOG, 36(4), 2017.
[28] Francesc NNguuer 3d human pose estimation from single image via distance matrix regression. In CVPR, 2017.
[29] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniliidis Coarse-to-fiee volumetric prodiction for single-image 3d human posc. In CVPR, 2017.
[30] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and cupi-super training. In CVPR, 2019.
[31] Jihua Peng. Yanghong Zhou, and PY Mok. Kipformer: Kinematics and trajectory prior knowledge- enhanced transformer for 3d human pose estimation. In CVPR, pages 1123-1132, 2024.
[32] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Picter Abheel, and Sergey Levine. Sfv: Reinforeement learning of physical skills from videos. ACM TOG, 37()), 2018
[33] Xue Bin Peng. Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM TOG, 41(4), 2022.
[34] Davis Rempe, Leonidas J. Guibas. Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang. Contact and human dynamics from monocular vidco. In ECCV, 2020.
[35] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In CVPR, pages 11488 (1888-1149)) 11499, 2021.
[36] Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adrià López Escoriza, Ruud G. van Sloun, and Yonina C. Eldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70:1 1532-1155,, 2022.
[37] Sashi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Thenbalt. Physcap: Physically plausible monocular 3d motion capture in real time. ACM TOG, 39(6), 2020.
[38] Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Patrick Pérez, and Christian Theobalt. Neural monocular 3D human motion capture with physical awareness. ACM TOG, 40(4), 2021.
[39] Cristian Sminchisescu. human motion analysis in monocular video techniques and challenges. In IEEE International Conference on Video and Signal Based Surveillance, page 76, USA, 2006. IEBE Computer Society.
[40] Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J. Black. TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments. In CVPR, 2023.
12

---

[41] Shashank Tripathi. Lea Müller, Chun-Hao P. Huang, Omid Taheri, Michael J. Black, and Dimitrios Tzionas. 3d human pose estimation via intuitive physics. In CVPR, pages 4713-4725, 2023.
[42] Bastian Wandt and Bodo Rosenhahn, Repnet: Weakly supervised training of an adversarial reprojection
network for 3d human pose estimation. In CVPR, 2019.
[43] Bastian Wandt, Marco Rudolph, Petrissa Zell, Helge Rhodin, and Bodo Rosenhahn. Canonpose: Self- supervised monocular 3d human pose estimation in the wild. In CVPR, 2021.
[44] Jue Wang, Shaoli Huang, Xinchao Wang, and Dacheng Tao. Not all parts created equal: 3d pose estimation by modelling bi-directional dependencies of body parts. In ICCV, 2019.
[45] Yufu Wang and Kostas Daniilidis. Refit: Recurrent fitting network for 3d human recovery. In ICCV, 2023.
[46] Kevin Xie, Tingwu Wang, Umar lqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. Physics-based human motion estimation and synthesis from vidcos. In ICCV, pages 11532-1154, 2021.
[47] Jingwei Xu, Zhenbo Yu, Bingbing Ni, Jiancheng Yang. Xiaokang Yang, and Wenjun Zhang. Deep kinematics analysis for monocular 3d human pose estimation. In CVPR, 2020.
[48] Yingxuan You, Hong Liu, Ti Wang, Wenhao Li, Runwei Ding, and Xia Li. Co-evolution of pose and mesh for 3d human body estimation from vidco. In ICCV, pages 14963 14963-14973, 2023.
[49] Ri Yu, Hwangpil Park, and Jehee Lee. Human dynamics from monocular video with dynamic camera movements. ACM TOG, 40(6), 2021.
[50] Ye Yuan, Shih- En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. In CVPR, pages 7159 7169, 2021.
[51] Ye Yuan, Jiaming Song, Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In ICCV, pages 16010-16021, 2023.
[52] Yufei Zhang, Jeffrey O. Kephart, Zijun Cui, and Qiang Ji. Physpt: Physics-and aware pretrained transformer for estimating human dynamics from monocular videos. In CVPR, pages 2305 2317, 2024.
[53] Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Motionbert: A unified perspective on learning human motion representations. In ICCV, 2023.
13

---

Supplementary Materials
A Network details
Fig. shows the architecture of the proposed OSDNet. The network consists of multiple branches for Kalman gains, PD controller gains, inertia-bias, and extemal force estimations. The Kalman esti- mation module takes the combined inputs from the GRU's and current states embedding, outputting the Kalman gains matrix. The diagonal of Kalman gain matrix initialized to be approximately 0.5 by modifying the bias of the last linear layer. This is due to the instability of the PD branch at the beginning of training, which may cause gradient explosion if the Kalman gains are too low (zero trust in the kinematics stream). The hidden states lGRU of the GRU unit are updated throughout the simulated sequence and used as one of the inputs for the next prediction.
Similar to [88. we scale Kp and KD differently for global translation, global rotation, and joint angles. The initial scaling are [30.0, 14.0, 1.9] for Rp and [1.5, 0.1, 0.05] for Kp. Notice that our initial gains are much lower than (38), because we want to explain the global motion by external reaction forces, avoiding the need for "unrealistic" residual force. These scalings are further optimized along with the training of OSDNet.
OSDNer estimates the inertia-bias matrix M To ensure the symmetric positive definite (SPD) of the inertia matrix, estimate intermediate Massy and compute M M (Masse;)
The Jacobian branch of OSDNer takes the current state embedding as input and outputs the Jacobian matrix that maps end-effector linear velocity to rotational velocity of each joints. The contact and external force branch takes the current feet positions and velocity as additional inputs to the state embedding. The contact branch outputs mapped by sigmoid function to create the contact probability p. The external force branch outputs the linear reaction force for two feet, with the -etis initialized with the weight (9.81 mass) of the proxy character.
The adaptation matrix C and observation matrix H are initialized as identity matrices. We optimize them during the training process, and they are kept constant during inference.
A
STTHI
Agru
K
128
Dynamic features
GRU
M Mase
512
512 512
728
System state
MW
128
128
Jr
128
PE
position velocity
F2
Figure 4: Architecture of the proposed OSDNet. The network consists of hidden layer of size 512 to generate system state's embedding. Based on the state embedding, the inertia-bias matrix Maose PD b
gains Kp, KD, Jacobian matrix J., contact probability Of and external force X, are estimated. The proposed GRU unit with size 128 takes the dynamics features (mentioned in Sec. 3.2) as input, the Kalman gain matrix K, is estimated from the concatenation of GRU and the state embedding. The hidden state fign is continuously updated at each time step. For better estimation of foot- goot-ground contacts and reaction forces, we also feed the feet position and linear velocity as additional inputs.
14

---

we generate B Human body proxy model
The simulated proxy character is created based the body configuration of the SMPL model [24]. Bone lengths and weight distribution are the same as in the Human 3.6M metadata of the 'common human body [5]. Fig. 5 visualization of the proxy character, composed of spheres and cylinders. The bone lengths are treated as extra learnable parameters and optimized along with the model during the training process. During testing, the bone lengths are fixed to the ones learned during training, i.e. no ground truth bone lengths are used when testing.
Figure 5: The simulated proxy character used in the paper. The RBDL library 6 is used extract the inertia matrix M, and bias forces (1q q)-
C Dataset details
As mention in Sec. 4.1 we evaluate our proposed method on Human3.6.
[5], Fit3D [ and
SportsPose [4). For Human3.6M. we follow prior works (37 38, 21 to consider actions that only involve foot-ground contact (S1, S5, S6, S7, S8) for training and (S9, $11) for testing. For 77, we apply the same protocol with only foot-ground available actions
used: (s03, s04, s05, s07, s08, s10) for training and (s09, sl 1) for testing. For SportsPose, we only consider sequences that contain human time step O: (S02, S03, SO5, S06, S07, S08. S09) for fine-tuning and (S12, S13, S14) for evaluation.
TRACE [40] is used extract the kinematics input from the data. All extracted motions are aligned at the origin in the first time step, eliminating the effect of wrongly calibration process. Each action is then equally split into 100-frame sub-sequences, utilizing batch processing for training the OSDNet.
D Contact labels
Since there ground truth contact labels provided in all three datasets 15
our own annotations based on the ground truth keypoints. To create contact labels, ground truth feet 3D positions are considered. inconsistent MoCap sensor placement) above the ground plane and also not moved more than 2cm from the previous frame, is labeled as a valid contact point. Similar protocol is applied for both left and right foot. The foot-ground contacts are modelled directly by the OSDNet and automatic data annotation, using the ground truth 3D poses from the training data set.
foot position is within 10 cm (already compensated for shoes and
E Global rotation
To mitigate the Gimbal lock problem in the original Fuler representation of Human3.6M 15 we convert all root rotation (d.o.f to 6th of the state vector qep)) into quaternions quateje (z, y, w).
15

---

(a) Tennis
(b) Baseball
(c) Football
(d) Volleyball
Figure 6: Example results on SportsPose [1] test data. Here we show four out of five action classes of SportsPose [14] that have foot-ground contacts. Qualitatively OSDCap matches the provided ground truth much better than the kinematics input TRACE.
with the real part the end. Since quaternions not linear representation, the computation of quaternion differences is given as
Aquat,j quatz+1j441 * quat e/c
(13)
Aquat,i is the input error term for the PD controller for computing the corresponding root torque by Eq. SS and Eq. [1] The procedure for finite integration (during the state transitioning stage) from quat state vector qij to the predict state quaet given the system state vector qu in quaternions is expressed as
quat L+tle
guat L.j
+ 0.5( quot * Agft
quat At.
(14)
Computing resources
The proposed pipeline of OSDCap was trained and evaluated the NVIDIA-A100 GPU with 40Gb of memory. In average, OSDCap requires an additional 0.02 second on top of the processing time of the kinematics estimation [40] on each frame. Each ablation study in 4.5 takes 45 minutes to train and evaluate. The full training and testing on Human3.66 3.6M consumes approximately hours, on Fit3D hour, and on SportsPose 15 minutes.
Besides, we also train and evaluate OSDCap on multiple random seed values to demonstrate reproducibility of results. Table preesents the means and standard deviations of the evaluation results across multiple random seeds from 0 to 4. We did not report error bars for every other experiment since it would be too computationally expensive.
16

---

Figure 7: Qualitative results when projecting the SMPL body model from the OSDCap poses back to the input 2D images. The overlayed SMPL models are shown in sparse blue point cloud to maximize the visibility of the input human pose.
G Additional results
One can refer to our additional supplementary material for a better visualization of the OSDCap recon- structions against the kinematies input and ground truth. Some example footage on the challenging SportsPose dataset can be seen in Fig. O
Additional visualization of estimated pose overlayed on 2D input images be found in Fig. 7 There is always a trade-off between the reprojection error and the model- assumptions. In our case the physics simulation other hand, produces more plausible motion as shown in Tab. of the main paper and Tab. 5
stronger assumptions than purely kinematics-basee model. On the
Despite not being training objective during training, the re-projected poses match well with the input humans in the input image as shown in Fig. 7 The slight offset is due to the mis-match bone length between the proxy character and the actual testing human subjects. An adaptive human shape estimation would be investigated the future.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work can be found in the Conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
There is no theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The detailed experiment setting can be found in the experiments section 4 The detailed architecture of the model can be found in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: The paper will provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results in the final version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli///uuu guwde//ddee///
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details can be found in the Experiment section 4.2
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation for the main experiment 4 on Human3. 6M, with five different random seeds. However, we did not do the same for other experiments because would too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources we used are specified in Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://euurpss... httpa//ppw.....ppp/cccccccccaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no obvious negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not have a high risk for misuse such that safeguards are needed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models), used the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets is introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "G7KmWLIz0J", "reviewer": "Reviewer_wqMZ", "summary": "The kinematic motion estimation suffers from inconsistencies of frame-wise predictions while the physics-based method suffers from the gap between the simulator environment and the real-world ground truth. This paper proposes a method to taking advantages of both by connecting them by a learnable kalman filter network. The proposed method OSDCap is a new physics-based human motion and dynamics estimation method. The proposed Kalman filter takes the simulated motions and the 55 noisy 3D pose estimation as inputs, combines them, and produces an optimal state prediction as the 56 output.  It also has a learnable inertia prediction for weights distribution, that produces plausible motion as well as valuable estimates of exterior forces and internal torques.", "questions": "I am overall satisfied with the novelty of the proposed method with no previous work having implemented the straightforward but well-connected idea as presented in this work as far as I know. However, I will need more evidence about the experimental significance to adjust my rating. Or, could the authors clarify the reason of the missing of more recent baseline works? I would need more evidence to recognize that the kalman filter (at least the learnable one in this work does work generalizability  and is more superior than the classic parametric KF).", "limitations": "The authors have discussed the limitations in the paper. They use the widely distributed datasets with human faces but they are controlled in the studio environment and under consent.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is very straightforward and intuitive. Considering the noise pattern of the frame-wise kinematic method (frame independent) and the noise of physics-based method (temporal propagated), the idea matches with the design motivation of kalman filter very well.\n- The presentation is good. I could understand the high-level idea and the technical design without trouble.", "weaknesses": "- Some related works are missing in the experiments, especially considering that kalman filter is widely used for human pose tracking. Moreover, as the authors proposed a learnable module to replace the role of Kalman filter (by predicting the gains), it would be helpful to compare the proposed method and a baseline with the classic kalman filter parametric model.\n- There are some missing baselines, especially the kinematic ones in the Table 1, for example PointHMR (CVPR’2023) and KTPFormer (CVPR2024), considering that the papers are available before the submission of this work, is there any reason that some recent works are not listed in the comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment to our proposed idea and writing quality. For a better interpretation of our work, we clarify and answer the questions of the reviewer below.\n\n**Comparison to a classical Kalman Filter.**\n\nClassical Kalman filters have been used traditionally for all types of tracking problems, including human pose tracking. More recently, the only work we could find that utilizes a Kalman filter is from Buizza et al. [A], where a classical parametric Kalman filter is employed to track 2D keypoints estimated from an off-the-shelf 2D human pose estimators. We are happy to take further suggestions for related work that we might have missed.\n\nWe agree that our learnable Kalman filter should be compared to a classical Kalman filter. Therefore, we conducted an additional experiment where we replace our learnable filter by a traditional one. The biggest challenge of using classical Kalman filter is the tuning of unknown noise covariances of both the kinematic input TRACE and the simulated result from PD controller. Assuming noise covariances that are constant over time and equal in all directions, the ratio between the noise covariance of the simulated PD controller (process noise) and the noise covariance of the kinematic input TRACE (measurement noise) governs the quality of the Kalman filter estimates. The evaluation results can be seen in Table*1 in the PDF file, where we use constant noise covariances with ratios 100/1, 10/1, 1/1, 1/10, 1/100 between process noise and measurement noise. While a classical KF approach helps increase the result marginally, optimal results are difficult to find. Our choice of a learnable Kalman filter relieves us from trial-and-error process of finding the correct noise covariance matrices and achieves the best results.\n\n[A] Buizza et al., Real-Time Multi-Person Pose Tracking using Data Assimilation, WACV, 2020.\n\n**Additional baselines.**\n\nWe will add the suggested baselines and others suggested by reviewer DQhK to the paper. Please refer to our global rebuttal response for references about their performances. We would like to briefly discuss the two suggested approaches in relation to our work.\n\n- PointHMR (CVPR 2023) is a frame-wise mesh and keypoint regression method that can work in an online setting. However, unlike TRACE, PointHMR does not produce global root translation of the estimated poses, therefore, can only be used for MPJPE reference, not for physics-based reconstruction methods.\n\n- KTPFormer (CVPR 2024) predicts 3D keypoints directly instead of 3D joint angles. This might lead to implausibilities in the skeletal configuration, for example in changing bone lengths, and is therefore not transferable to a dynamics model. Since it appears to be the current state of the art in terms of the common metrics like MPJPE we will include it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBdL65Z8cW", "reviewer": "Reviewer_DQhK", "summary": "This paper introduces a novel method to estimate 3D human motion from a single camera, aiming for physical plausibility and accuracy. The authors use a differentiable Kalman-filtering approach in an online setting that balances kinematics estimated from an off-the-shelf algorithm (TRACE) and the physics simulation. They built OSDNet, which consists of fully connected and GRU layers, for the given optimal state human dynamics capture task. They show their method outperforms the existing methods in the same online estimation setting while each technical component was validated through ablation studies.\n\nIn general, the method sounds reasonable and novel. However, there are several things that need to be cleared.", "questions": "[Some are duplicated with the Weakness section]\n– How did you construct a physics-based 3D person model?\n– Why did you use TRACE? IF you value the potential of future use case with moving camera, this needs to be stated.\n– Why do neural PD controllers perform so badly in your test case?\n– I guess the GRU unit is uni-directional, am I correct?", "limitations": "–", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "They propose the approach inspired by the neural Kalman filtering approach. This method seems to be novel.\n\nCompared to their baseline approach (TRACE), they provides significant performance improvement, which is not the case for the other physics-based approaches (if I remember correctly).", "weaknesses": "It is not clear how they constructed the character. They mention that they used metadata of Human3.6M dataset. Does it mean they use the ground-truth bone lengths of the subject?\n– If so, they should state their method with automatic character generation (similar to what SimPoE did) for a fair comparison. It is not clear if their outperformance is coming from the known limb length assumption or the method itself.\n– If not, please state more clearly how they generate the character in the physics simulator.\n\nIt is not clear why they selected TRACE as a default method. From Table 1, TRACE seems to significantly underperform VIBE, which has been widely used as a baseline kinematics estimator (e.g., SimPoE). The major advantages of using TRACE are its capability to track a person's identity and the robustness of the camera motion. However, all training and evaluation were done in a fixed camera coordinate system.\n\nIn the paper, they mention that since TRACE's output is expressed in the first frame global coordinate, their method is agnostic to the initial calibration (line 473). I disagree with this line. Isn't the initial pose of camera essential to know the gravitational direction? Will this method be robust to the camera which is highly tilted? \n\nIn Table 3, conventional PD methods are showing catastrophic performance. I am doubtful about this, maybe the training configuration was not set properly? This is telling us that neural PD control is destroying the performance which does not align with existing literature. \n\nMany of the state-of-the-art video-based kinematics estimation models are not stated in the paper including HybrIK (CVPR 2021), CLIFF (ECCV 2022), PMCE (ICCV 2023), ReFit (ICCV 2023), MotionBERT (ICCV 2023), and HMR2.0 (ICCV 2023). To the best of my knowledge, all these methods (not limited to) have better performance on Human3.6M dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and constructive review. We would like to address questions and clarify the concerns below.\n\n**Proxy character creation.**\n\nFor character construction, we used the Human 3.6M metadata skeleton as the initialization of our character. The bone lengths are treated as extra learnable parameters and optimized along with the model during the training process. During testing, the bone lengths are fixed to the ones learned during training, i.e. no ground truth bone lengths are used when testing.\n\n**Why TRACE?**\n\nThe reason for choosing TRACE as our kinematic estimation input is their additional global translation estimation (w.r.t the first frame), while for example VIBE only predicts root relative poses. We use this global information to enable the integration of physics laws into the system. Moreover, TRACE is a well-established and new (CVPR'2023) baseline. The additional benefit of TRACE being robust to camera motion is an advantage we plan to explore in future work.\n\n**OSDCap estimation is agnostic to initial calibration.**\n\nThe statement in line 473 explains how we preprocess the data. It indeed only removes the translational component of the camera calibration. As correctly stated in the review, the rotational component is important for the physics simulation. OSDCap will still work in the case of highly tilted camera, as long as the camera pose is provided. If a fully automatic calibration process for tilted cameras is desired, an automatic calibration with humans as calibration targets can be employed [B].\n\n[B] Tang et al., CasCalib: Cascaded Calibration for Motion Capture from Sparse Unsynchronized Cameras, 3DV, 2024.\n\n**The performance of the PD-only method.**\n\nWe thank the reviewer for pointing out this potentially misleading table.\n\nThe PD implementations in Table 3 are different from other PD controller-based works (PhysCap, NeurPhys, DnD). While NeurPhys (cf. their implementation on Github) assumes that gravitational forces and external forces are cancelling out, which is not true in the real world, we decide to keep them to maintain physical plausibility. Moreover, to make the PD controllers work, PhysCap, NeurPhys, and DnD add an additional predicted offset term to the controller output which introduces another source of implausibility. By contrast, we maintain the physical plausibility of the simulation and the PD controller which leads to state-of-the-art performance among physics-based approaches. We will clarify this in the final version.\n\n**Additional baselines.**\n\nWe will add the suggested kinematics estimation methods in the comparison. However, similar to what have been done in related physics-based methods such as NeurPhys, DiffPhy or DnD, it is not trivial to directly compare physics-based motion reconstruction to image-based pose estimators, since the latter do not predict the global 3D trajectory in world coordinate, making it impossible to evaluate the global motion quality (MPJPE-G, GRP). We kindly refer to the comments to all reviewers above for a detailed discussion.\n\n**Is GRU uni-directional?**\n\nYes, the GRU unit is uni-directional.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYpnxxKbdN", "reviewer": "Reviewer_BVYQ", "summary": "This work focuses on tackling the problem of single-person motion estimation from a monocular video. Current approaches produce temporal artifacts such as jittering. Most approaches are entirely kinematic while others that combine physics, do it by re-simulating the kinematic inputs by using automatic PD controllers. These methods, however, require simplifying assumptions that compromise the motion realism.This method combines information from two sources to produce an optimal estimate of single-person 3D human motion. One source comes from an off-the-shelf kinematic motion estimation pipeline and the other from a differentiable physics formulation. These two \"measurements\" about the motion are combined in a Kalman-filter to generate an optimal output.  Here, the authors propose to selectively incorporate the physics models with the kinematics observations in an online setting, taking as inspiration the neural Kalman-filter. The method uses a meta-PD controller and a physics-based simulation step. The Kalman filter is realized via a recurrent neural network which aims to balance the kinematic inputs with the simulation. \n\nThe authors propose an end-to-end model for this purpose which is not trivial to accomplish. The method is capable of capturing accurate global trajectories and, at the same time, producing physically plausible human poses.", "questions": "- L38: To be clear, what the authors refer to as a \"differentiable physics simulation\" is the use of rigid body dynamic equations or is it an actually differentiable simulator, e.g.,  Tiny Differentiable Simulator from PyBullet, similar to Gartner et. al?\n- L116: Authors create a proxy character (shown in appendix B). I wonder if it is possible to use the humanoid generated by SimPOE or KinPoly for the same end? This latter humanoid seems to be the most realistic representation of a 3D human body for simulation. I would like what the authors think about this and why they chose this specific form.\n- Sec. 3.2: How are the contacts modeled? Are they modeled directly with the GRU and data annotations or is there also a specific physics formulation to account for these contacts?", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper is very well written and the experiments are well presented which makes the paper easy to follow. \n* Authors present extensive experiments comparing several SoTA methods and include in-domain and out-of-domain test data for these. \n* The setup and the method are sound. \n* I would say that this is the first paper that successfully combines information from kinematic estimates and a differentiable physics simulation step in an end-to-end manner. It is not trivial to refine kinematic estimates with physics simulation (or physics informed estimates) outside the RL framework. It seems that the neural Kalmal filter is a promising direction to bridge the gap between kinematics and physics. I believe that this work is of high significance for the field.", "weaknesses": "### **Presentation**\nThe qualitative results could be better presented as it is sometimes hard to have a good sense of the pose estimated by the kinematic approach (TRACE) both in Fig. 6 and in the supplementary .gif images. The way the poses and the original video are visualized can be improved. First, I would suggest making the kinematic skeleton more visible as it is “obscured” by OSDCap results. Even better, it would be nice to have SMPL visualizations of GT, kinematics and OSDCap as it is presented in Fig. 1. In my opinion, changing visualization styles within the paper can reduce the presentation quality. I also advise the authors to focus on video results with several examples. If possible I would like to see this as part of the rebuttal, if not, then this should be present for the camera ready version of the paper. \n\n### **Physics-based metrics**\nSec 4.3: It would be interesting to show the results for more physics-based metrics other than the Acceleration metric, for example, foot skating and ground penetration, which should be corrected by the physics formulation. As authors are modeling contacts and have physics-based losses (e.g., friction, velocity), it would be interesting to know these metrics in comparison with the SoTA or at least the baseline used (TRACE).\n\n### Minor\n- (Typo) Fig.2: 4th line: performs contains-->contains.\n- (Typo) L295: HMDCap -->OSDCap?\n\n### References\nI would advise the authors to include most recent papers that combine kinematic and physics estimates for pose/motion estimation, either in the introduction or related works. For example:\n- Ugrinovic et. al., “MultiPhys: Multi-Person Physics-aware 3D Motion Estimation” (CVPR 2024).\n- Zhang et. al. “PhysPT: Physics-aware Pretrained Transformer for Estimating Human\nDynamics from Monocular Videos” (CVPR 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very positive assessment and the recognition of our method's novelty. We would like to provide answers and clarifications to the remaining questions of the reviewer.\n\n**Presentation Improvement.**\n\nWe appreciate the suggestions of the reviewer. We will change the visualization to SMPL model for the GT, kinematics and OSDCap prediction in the camera-ready version of the paper, similar to Figure*2 in the rebuttal PDF document. We are also open to any other recommendations to further improving the presentation quality.\n\n**Additional physics-based metrics.**\n\nWe provide additional metrics for physic-based measurements in Table*2 of the rebuttal PDF above. OSDCap helps refining the input kinematics on most of the physics-based metrics. Note that TRACE outperforms our approach in the ground penetration metric. The reason is that in most cases the TRACE predictions float above the ground, which gives a low penetration error but can be seen as equally bad. Thus, we additionally provide a ground distance metric (GD) to reflect the correct foot-ground quality during contact.  The value is computed as the mean absolute vertical differences between foot contact points and ground plane during contact duration, expressed as:\n\n$GD=\\frac{1}{6} \\sum_{c=1}^{6} \\rho_{c} |p_{c}^{OSD} - p_{c}^{GT}|$,\n\nwhere $\\rho_{c}$ is the binary label of contact $c$, $p_{c}^{OSD}$ and $p_{c}^{GT}$ are the 3D vertical positions of contact $c$. There are a total of 6 contact points considered, 3 in each foot accounting for heel, foot and toe.\n\n**Differentiable simulation in OSDCap.**\n\nThe differential simulation in line 38 refers to the usage of rigid body dynamics equations and Euler's integration on a proxy character, not an actual physics engines like TDS or PyBullet.\n\n**Selection of a simple proxy character.**\n\nWe based the calculation of the inertial properties of the proxy character on the Rigid Body Dynamics Library (RBDL) which was shown to be effective by Shimada et al. [PhysCap]. Since the simpler character definition (as for example compared to KinPoly) yielded good results, we decided to stick with it and mainly focusing on implementing our core contribution of the learnable Kalman filter. During experimentation we found that prediction errors mostly occur from insufficient kinematic or dynamic predictions than from using a more sophisticated proxy character. However, in the future we plan to extend this approach to other external forces, e.g. from full body contact, which will strongly benefit from a detailed human body model such as KinPoly.\n\n**How did foot-ground contacts are modelled?**\n\nThe foot-ground contacts are modelled directly by the NN model and automatic data annotation, using the ground truth 3D poses from the training data set.\n\n**Additional baselines.**\n\nThank you for the additional baseline suggestions, MultiPhys and PhysPT. We will include their results and contributions into the related section. We would like to briefly discuss the suggested paper due to their high relevance:\n\n- MultiPhys introduces a method for multi-person physics-based 3D human motion capture that mainly address plausibility of the captured multi-human poses, inside an physics engine and a reinforcement learning framework. This work could bring new insights about inter-person body interaction, and a more detailed proxy character such as KinPoly (as suggested by the reviewer) is needed.\n\n- PhysPT provides a pre-trained transformer model specifically for human dynamics capture. Instead of explicitly defining physical constraints, as in our or related [NeurPhys, DnD], PhysPT realizes the equations of motion as the main training objective, forcing the transformer model to be physics-aware, thereby allowing a violation of physics laws. The pretrained model can be used to refine kinematics input, acting as an NN-based approximation of a physics simulation.\n\n**Typos.**\n\nThanks for catching the two typos. We will correct them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0bJc7wio71", "reviewer": "Reviewer_QEqx", "summary": "The paper presents a novel physics-based human motion capture method that is physically explainable, conforming to the PD control theory and rigid body dynamics. The key designs of the method involve an integration of a kinematic Kalman filter and Newtonian equation-based physics simulation, and learnable Kalman gains, PD gains, external forces and robot inertia biases. The method outperforms existing kinematics-based and physics-based motion capture methods on keypoint accuracies.", "questions": "Typos:\n\n* In Figure 2, \"$q_{t|t-1}$\" -> \"$q_{t+1|t}$\"\n* In the caption of Figure 2, \"performs\" is redundant\n* In Equation 6, \"$\\sum_c^2$\" -> \"$\\sum_{c=1}^2$\"\n* In line 295, \"HMDCap\" -> \"OSDCap\"\n* In line 478, \"the\" is redundant", "limitations": "One limitation is that the human dynamic model is formulated as a connection of circles and cylinders, which neglects the modeling of geometric details and wearings of humans.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The method is physically explainable without unrealistic approximations of the control process and the robot dynamics.\n\n(2) Under the paradigm of using physics simulation to capture human motion, the method provides novel insights about which physical properties should be modeled by neural networks.\n\n(3) The method is superior to previous kinematics-based and physics-based methods in the accuracy of joint predictions.\n\n(4) The writing is clear and easy to follow.", "weaknesses": "(1) The contact modeling only considers foot-ground contact, ignoring full-body contact that commonly appears in human-object and human-scene interaction scenarios. Besides, the contact on each foot is represented as a force vector on a pre-defined contact point, ignoring changes in the contact point and the resultant torque of the contact.\n\n(2) The method updates the inertia matrix $M$ online. However, the inertia matrix is the attribute of the robot and should be fixed values during the whole motion capture process for better physical interpretability.\n\n(3) To fully examine the generalizability of the proposed method, existing physics-based methods should also be compared on datasets Fit3D and SportsPose.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the highly positive evaluation and the constructive feedback to our work. For a better clarification, we provide the answers to the reviewer's questions below.\n\n**Full-body contacts.**\n\nWe agree that full body contact is the next logical step towards a fully environment-aware physical model. Note that our approach (though not integrated in the current version) can easily add an additional force to any part of the body. However, this would require contact detection and estimation of the physical properties (e.g. velocity, weight, softness) of the object which is non-trivial. Recently, this field received more attention [C, D] and we are looking forward to integrating these approaches into our model in the future.\n\nFor a similar discussion on body models that enable to add external forces we kindly refer to the “Selection of a simple proxy character” section with reviewer BVYQ.\n\n[C] Tripathi et al., DECO: Dense Estimation of 3D Human-Scene Contact In The Wild, ICCV, 2023.\n\n[D] Xie et al., Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation, CVPR, 2024.\n\n**Inertia-matrix M.**\n\nFor a robotic arm, the mass distribution often does not change significantly. However, while a constant limb-wise mass distribution is an acceptable approximation for a human, it is not entirely correct. Due to muscle movement and soft tissue deformation the mass distribution changes which has a direct effect on the estimated motion [E, F]. While our inertia matrix intentionally leaves this additional degree of freedom, we agree that its interpretability is limited. Increasing the interpretability of the inertia matrix will be investigated as the next step of this work.\n\n[E] Pai, D.K., 2010. Muscle mass in musculoskeletal models. Journal of biomechanics 43, 2093–2098.\n\n[F] Featherstone, 2008. Dynamics of rigid body systems. Rigid Body Dynamics Algorithms, 39–64.\n\n**Evaluation of related works on Fit3D and SportPose.**\n\nDue to the availability of the implementation code of related physics-based methods, we could not replicate all related physics-based methods on Fit3D and SportPose during the rebuttal phase. We hope that these experiments will be finished during the discussion phase and at the latest we will add the evaluation results of works that provided implementation, such as NeurPhys, to the camera-ready version of the paper.\n\n**Typos.**\n\nThanks for catching the typos. We will correct them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qzNfJCdOpk", "reviewer": "Reviewer_mMU1", "summary": "This paper proposes a method to capture physically plausible human motions from monocular videos. The paper builds on top of NeuralPhys and introduces Kalman filtering to alleviate noise from kinematic motions. The framework combines Kalman filtering and physics simulation to be fully differentiable, thus enabling training in a supervised manner. The experimental results show that the proposed approach can achieve competitive performance.", "questions": "- Do $q_t$, $\\dot{q}_t$ in the output still follow rigid body dynamics as in Eq. 1?\n- The work builds on top of NeuralPhys with additional Kalman filtering. What is the performance of applying Kalman filtering on simulated or kinematic motion only?\n- Humans may penetrate the ground plane due to depth ambiguity. How does the method address penetration between humans and the ground plane?\n- The body shape in this work is fixed. Can the trained models be generalized to different skeleton shapes?", "limitations": "The limitations are discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method uses Kalman filtering to produce targets for PD control, which is more robust to the noise from pure kinematic poses.\n- The framework can be trained in a supervised manner due to its fully differentiable implementation.", "weaknesses": "- The simulation and filtering refine the initial kinematic poses with dynamic constraints, which do not consider the image observations. The simulation may remove some high-frequency motions; thus, the refined poses may not be consistent with 2D poses. The authors should overlay the 3D pose onto 2D images and show more visualization results so that we can evaluate the discrepancy.\n- Although the method predicts contact information, the framework cannot prevent penetrations between humans and the floor since scene dynamics are not considered.\n- The optimal pose is predicted from Kalman filtering, while the velocity is obtained from simulation. In previous works, the final body pose is integrated from simulated velocity as in Eq. 2, which can guarantee physical plausibility. However, the optimal pose in this work is further updated with Kalman filtering, which may not be consistent with simulated velocity. Therefore, the final poses may not obey physical laws.\n- The work relies on external force, which may result in some unnatural states (e.g., body leaning).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We address the concerns and questions from the reviewer in the section below.\n\n**2D overlay of estimated pose on the input image.**\n\nThank you for the suggestion. We plotted the overlay of 3D estimated poses from OSDCap onto the input 2D images in Figure*1. As can be seen from the figure, the re-projected SMPL meshes overlay closely to the human inside the input images. Despite the image-based observations being refined by the Kalman filter, the idea of considering original image observations could potentially benefit the prediction, and will be investigated in future work.\n\n**OSDCap cannot prevent foot-ground penetration and scene dynamics.**\n\nYes, foot-ground contact is not explicitly modeled in OSDCap. However, our contact estimation is able to mitigate it and creates physically plausible motion. Please refer to Table*2 in the attached PDF above for our additional physics-based measurements of OSDCap.\n\n**Is the final pose obeying physics laws?**\n\nOSDCap combines the two data sources and collectively creates the best of both worlds. Naturally, if the GRU model decides that the noise from the physics simulation data is larger than the kinematic input in a specific time step, the resulting human pose $q_t$ and $\\dot{q}_t$ after Kalman filtering might not strictly obey physics laws anymore. There always exists a knowledge gap between simulation and real-world that can cause the simulated result to be sub-optimal under the provided constraints. While the image-based kinematic estimation, despite being jittery and noise, reflects real-world observation and can act as an online compensation for the mistakes made by the simulation.\n\n**Unnatural leaning due to the usage of external force.**\n\nA false external force prediction might result in implausible artifacts such as unnatural leaning. This is one of the motivation for proposing a Kalman filtering approach, where a noisy inputs to the in physics simulation, i.e. false external force prediction, can be monitored and compensated directly by the kinematics input. However, unnatural leaning artifacts often occur more in the kinematic input due to depth uncertainty in the optical axis, especially in scenarios when the person is facing the camera. We show an example of using a physics-based simulation with external forces can actually refine the unnatural pose and prevent unnatural leaning in Figure 3b in the main paper.\n\n**Kalman filter on kinematics or simulation only.**\n\nThe performance of applying Kalman filtering on kinematic motion can only be shown in Table*1 in the rebuttal PDF above. As can be seen, without physics-based knowledge, the Kalman filtering scheme only does smoothing and achieves worse performance than our approach. For a deeper analysis of the classical Kalman filter compare to a learnable one proposed in the paper, we kindly refer to the discussion with reviewer wqMZ.\n\n**Adaptability of trained models on different skeleton shapes.**\n\nThe trained models can be generalized to different skeleton shapes since we are predicting joint angles of the skeletons while training the NN models. Therefore, a motion retargeting to a character with different bone lengths is straight-forward. However, achieving the same performance as in the three used dataset Human3.6M, Fit3D and SportPose for data with a completely different character configuration would require additional training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude towards the reviewers for their helpful reviews and assessment!

We are happy that the reviewers recognize that the proposed method OSDCap for physics-based 3D human motion capture is novel (wqMZ, DQhK, BVYQ, QEqx), of high significance to the field (BVYQ), physically explainable (QEqx), and fully differentiable (mMU1). Reviewer BVYQ comments that our paper is the first to successfully combine kinematics estimation with differentiable simulation in and end-to-end manner, providing a promising direction to bridge the gap between kinematics and physics. Three reviewers (wqMZ, BVYQ, QEqx) agree that the manuscript is well-written and easy to follow. 
In this section, we provide the answers to the common question asked by multiple reviewers. Further on, we address the reviewers' questions individually in the respective sections.

**Additional references.**

Reviewers wqMZ, DQhK and BVYQ pointed out that a comparison to other human pose estimation work is missing. We followed the common practice in the field of physics-based human motion estimation (DiffPhy, NeurPhys, DnD) and compared only to other physics-based approaches in our main experiments, since their objective of creating a plausible motion is the same as ours. However, we agree that for a better interpretation of the performance of physics-based approaches, including traditional approaches for comparison is beneficial and we provide results in the following table for the Human 3.6M dataset.

| Method | Venue | MPJPE | MPJPE-PA | 
|:----------------|:----------------:|:----------------:|:-----------------:|
| HybrIK | CVPR'2021 | 54.4 | 34.5 |
| PMCE | ICCV'2023 | 53.5 | 37.7 |
| PhysPT | CVPR'2024 | 52.7 | 36.7 | 
| ReFit | ICCV'2023 | 48.5 | 32.4 |
| PointHMR | CVPR’2023 | 48.3 | 32.9 |
| CLIFF | ECCV'2022 | 47.1 | 32.7 |
| MotionBERT | ICCV'2023 | 43.1 | 27.8 |
| KTPFormer | CVPR'2024 | 33.0 | 26.2 |

However, while these produce a low MPJPE, most of them do not estimate the global motion and more importantly might not be physically plausible, including strong jitter and floating/ground penetration. By contrast, although our approach sometimes produces worse distance-based metrics (MPJPE, PCK, etc.), it produces significantly more plausible motions as well as mitigating unnatural floating or ground penetration. We think that the novel concept of a learnable Kalman filter that produces state-of-the-art results for physics-based pose estimation outweighs the pure distance-based evaluation of other approaches.

**The accompanying PDF document.**

To fully address reviewers' questions and concerns, we conduct some additional measurements on the output of the proposed method OSDCap. The PDF contains:

- Table*1, evaluation results of using a classical Kalman Filter to combine the physics simulation and the kinematics observation. This table is relevant to our response to Reviewer wqMZ.
- Table*2, additional physics-based metrics of the estimated human motion from OSDCap. The metrics consist of ground penetration (GP), friction loss, velocity loss, and foot skating. Since the GP metric cannot correctly reflect the floating artifact of the estimated pose, we also compute a ground distance metric (GD) to measure the foot-ground contact quality. This table is to address the concerns from Reviewers BVYQ and Reviewer mMU1.
- Figure*1, the overlay re-projected human pose, presented as SMPL model, onto the input 2D images. This figure is in response to Reviewer mMU1 about discrepancy evaluation.
- Figure*2. the 3D visualization of OSDCap's estimated pose, input kinematics TRACE, and the ground-truth pose provided by the Human 3.6M dataset. The figure is created according to the suggestions of Reviewer BVYQ for a better presentation of the results.

Above is the common information that is relevant to our individual responses to all reviewers. We further address each reviewer's concerns respectively in the below sections.

Meta-review: The paper proposes an approach to estimate the 3D pose of a single-person from a monocular camera image. The approach uses a differentiable Kalman-filter that combines a kinematic motion estimator a differentiable physics formulation, which can be trained using a supervised learning approach. Doing so, the approach addresses limitations of previous work (such as jittering). Moreover, the approach outperforms the existing methods in the same online estimation setting while each technical component was validated through ablation studies. 

The reviewers mainly raised issues considering related work and baselines, as well as technical obscurities. However, the authors were able to clarify the open points and provide the missing quantitative results during the rebuttal period. 

Decision:
After a fruitful discussion all reviewers voted for acceptance. I therefore recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.
Decision: Accept (poster)