ImDy: Human Inverse Dynamics from Imitated Observations

## Abstract

Abstract Inverse dynamics (ID), which aims at reproducing the driven torques from human kinematic observations, has been a critical tool for human motion analysis.
However, it is hindered from wider application to general motion due to its limited scalability.
Conventional optimization-based ID requires expensive laboratory setups, restricting its availability.
To alleviate this problem, we propose to exploit the recently progressive human motion imitation algorithms to learn human inverse dynamics in a data-driven manner.
The key insight is that the human ID knowledge is implicitly possessed by motion imitators, though not directly applicable.
In light of this, we devise an efficient data collection pipeline with state-of-the-art motion imitation algorithms and physics simulators, resulting in a large-scale human inverse dynamics benchmark as Imitated Dynamics ( ImDy ).
ImDy contains over 150 hours of motion with joint torque and full-body ground reaction force data.
With ImDy, we train a data-driven human inverse dynamics solver ImDyS(olver) in a fully supervised manner, which conducts ID and ground reaction force estimation simultaneously.
Experiments on ImDy and real-world data demonstrate the impressive competency of ImDyS in human inverse dynamics and ground reaction force estimation.
Moreover, the potential of ImDy(-S) as a fundamental motion analysis tool is exhibited with downstream applications.
The project page is https://foruck.github.io/ImDy .

## 1 Introduction

The rapid progress in human motion capture based on computer vision has made an enormous amount of human motion data available to the research community (Mahmood et al., ; Mandery et al., ).
The accumulation of human motion manages to push motion understanding forward in various tasks, including behavior understanding (Punnakkal et al., ; Shahroudy et al., ) and character animation (Guo et al., ; Tevet et al., ).
However, given the vision-based nature, most current efforts focus only on visible kinematics information.
The invisible factors, especially the dynamic factors, which could carry deeper insights into the underlying production mechanism of human motion, are typically overlooked, such as driven torques and ground reaction forces.
This limits the current motion understanding algorithms from wider applications to domains where physical constraints must be seriously considered, such as robotics, healthcare, and sports training.
To alleviate this, we focus on identifying the driven torques and ground reaction forces for human motion from pure kinematics MoCap data, known as human inverse dynamics (ID).

Human inverse dynamics, as a basic step toward physical motion modeling, has been extensively discussed by the biomechanics community for applications like gait analysis.
A fundamental obstacle is that it could not be measured non-intrusively.
Therefore, computationally expensive optimization-based methods are widely adopted and mature software is developed (Delp et al., ; Damsgaard et al., ; Werling et al., ).
However, accurately measured ground reaction forces are required to ensure a determinate solution, which could be expensive and applicable only in restricted laboratory settings.
Also, the optimization process could be sensitive to small disturbances in either motion capture noises or subject variances.
These make it hard to scale up for wider applications to general motion.
Given the success achieved by data-driven methods in computer vision and natural language processing, deep-learning-based methods are proposed (Zell & Rosenhahn, ; Zell et al., ; Lv et al., ), aiming at scalable human inverse dynamics with only kinematic observations as inputs.

Figure: Figure 1: ImDy pairs diverse SMPL motion data with dynamics including full-body torques and ground reaction forces (GRF) like the right knee GRF for kneeling, which could be hard to achieve under conventional laboratory setups.
Refer to caption: x1.png

Unfortunately, data acquisition becomes a major bottleneck since laboratory setups are still required for ground-truth acquisition.

Given this, we project our sights on the recent progress of Imitation Learning (IL) (Luo et al., ; ), which replicates recorded human motion through fully simulated humanoids with physical control signals, namely, joint torques.
A key insight is that with the goal of kinematics phenomenon imitation, IL might also implicitly imitate the dynamics production mechanism, known as ID.
However, IL is not directly applicable to ID.
Despite the visual resemblances between the recorded and simulated motion, kinematic errors still exist.
These errors could be neglected for kinematic analyses, however, when it comes to dynamic analysis, they could be amplified drastically (Uchida & Seth, ).
Moreover, existing successful IL algorithms are typically based on joint-actuated SMPL (Loper et al., ) avatars, whose physical properties and topology differ from real humans.
To this end, extracting ID knowledge from IL becomes critical.
Here, we adopt the state-of-the-art motion IL algorithm (Luo et al., ) and physics simulator (Makoviychuk et al., ) to imitate recorded motions, extracting the observed kinematic states, joint torques, and the ground reaction forces, resulting in a large-scale human inverse dynamics database named Imitated Dynamics (ImDy) with more than 150-hour human motion.
There are two major merits of ImDy.
First, it is scalable.
Multiple samples could be concurrently collected in the simulator without expensive laboratory setups, which extends the border of ID data acquisition.
As shown in Fig. 1, we could even pair some rather complex motions with ID data, which is hard to achieve in laboratories.
Second, it is holistic.
Beyond the ground reaction force and ID typically recorded in laboratories for previous efforts (Zell et al., ; Mourot et al., ; Han et al., ), the physics simulator enables us to access the GRFs and joint torques of all human body segments, as shown in Fig. 1.

With the accumulated data, we could address the human inverse dynamics in a fully supervised manner.
Given the observed kinematics states that describe a motion transition in a certain period, we train a data-driven solver as ImDyS(olver) to estimate the ground reaction forces and the internal dynamics to drive the transition.
We also devise losses to regulate ImDyS with forward dynamics awareness and motion plausibility constraints.

We demonstrate the efficacy of ImDyS through a wide span of experiments.
First, we evaluate our method on ImDy for a basic performance illustration with simulated ImDy.
Then ImDyS is evaluated on GroundLink (Han et al., ), which contains real-world ground reaction force.
Furthermore, we demonstrate the efficacy of ImDy on the recent real-world human dynamics dataset AddBiomechanics (Werling et al., ).

Our contribution could be summarized as:
(1) We propose a novel pipeline for human inverse dynamics data collection, introducing a large-scale benchmark as ImDy.
(2) Based on ImDy, a data-driven ID solver is instantiated as ImDyS.
(3) Extensive experiments are conducted with analyses of the proposed data-driven methodology, demonstrating the feasibility of ImDyS.

## 2 Background

Conventional Inverse Dynamics.
Inverse dynamics, known as inferring forces/moments from kinematic observations, have been discussed for long in the biomechanics community.
In this literature, it is formulated as an optimization problem: given a representative model of a subject, the joint kinematics over time w.r.t. the subject model, and the external forces imposed on the subject, find the driving torques that produce the observed motion (Uchida & Delp, ).
The Newtonian dynamic equations are involved as

$$ $M(q)\ddot{q}+C(q,\dot{q})+G(q)=J\lambda+\tau,$ (1) $$

where $M(q)$ is the generalized human inertia matrix w.r.t. generalized coordinate $q$, $C(q,\dot{q})$ is the Coriolis and centrifugal forces, $G(q)$ represents gravity, $J$ is the Jacobian matrix mapping external forces $\lambda$ to the generalized coordinates.
Thus, the driven torques $\tau$ could be obtained by minimizing the difference between the left and right terms of Eq. 1.
Mature software based on this has been developed like OpenSim (Delp et al., ), AnyBody (Damsgaard et al., ), and Nimble (Werling et al., ).
In addition, many efforts are made for clinical motion analysis (Fukuchi et al., ; Schreiber & Moissenet, ).
However, these efforts are not as extensively recognized by the computer vision and computer graphics community as expected due to the scalability issue.
Despite the elegant formulation, the efficacy of optimization-based heavily relies on the quality of external force $\lambda$ (like GRF) measurement, whose cost could be non-trivial.
Therefore, most of them focused on limited motion in laboratory settings.
Some resort to wearable devices (Latella et al., ; ) to partially mitigate the limitation.
In addition, fitting the raw captured kinematic observations to a specific human model for joint kinematics could be time-consuming and unstable, even with recent progress on it (Keller et al., ; Werling et al., ).

Learning-based Inverse Dynamics.
With the progress in deep learning, there have been efforts to adopt neural networks to address the human ID problem.
Many efforts focus on lower-body-only (Johnson & Ballard, ; Xiong et al., ) or upper-body-only (Manukian et al., ) inverse dynamics.
More recently, Lv et al. () collected over 1 hour of motion with an optical MoCap system, four force plates, and a pair of pressure insoles.
The ground truth was obtained through optimization and a Gaussian mixture framework was devised.
Zell & Rosenhahn (); Zell et al. (); Zell & Rosenhahn () introduced a predictive dynamics-based human modeling for the acquisition of ground truth. Hundreds of motions were collected and different data-driven techniques were adopted for joint torque regression.
Zell et al. () proposed a weakly supervised method based only on motion for gait analysis.
These efforts were constrained by costly data acquisition in real-world scenarios, resulting in limited data scale.
Very recently, Werling et al. () aggregated multiple existing biomechanics datasets, considerably boosting the data scale.
However, most of the collected sequences contained only regular exercise motion with limited diversity.
Some efforts focused on ground reaction forces such as (Rempe et al., ; Scott et al., ), UnderPressure (Mourot et al., ), and GroundLink (Han et al., ).
Some recent works incorporated inverse dynamics into vision-based markerless MoCap systems.
Shimada et al. () and Li et al. () simultaneously captured motion and joint torques with customized fully differentiated pipelines.
A series of works (Yi et al., ; Gartner et al., ; Gärtner et al., ; Huang et al., ; Wang et al., ) imitated the captured motion in physical simulators with PD controllers and obtained the torques.
However, an inherent problem is the amplification effect from kinematic errors to dynamic errors.
As measured by Uchida & Seth (), only a 2-cm uncertainty of marker placement in a marker-based MoCap system could result in a peak ankle plantarflexion moment of 26.6 $N\cdot m$.
Considering the precision of current markerless MoCap algorithms, the accuracy of the accompanied inverse dynamics could be questionable.
Also, among all these efforts for learning-based inverse dynamics, only a few (Zell et al., ; Zell & Rosenhahn, ; Zell et al., ) were quantitatively evaluated with limited locomotion data.
A scalable benchmark for learning-based inverse dynamics is still not available.

Motion Imitation.
IL for human motion replicates recorded human motion sequences with physically controlled simulated characters, which could be inherently close to ID.
Most early efforts focus on specified usages with limited generalizability (Bergamin et al., ; Peng et al., ; Won et al., ; ; Peng et al., ).
With residual force control (Yuan & Kitani, ), which imposed supernatural forces at the root joint of the humanoid, Luo et al. () generalized to 97% sequences in AMASS (Mahmood et al., ).
Luo et al. () eliminated the supernatural root force and achieved a 98.9% success rate on AMASS with fall-state recovery.
The progress in human motion IL makes it possible to collect human-like motions with full dynamics, shedding new light on the scalable human ID data collection.

## 3 Constructing Imitated Dynamics

ImDy aims to exploit the inherent closeness of inverse dynamics and imitation learning.
Generally, the inverse dynamics (ID) and imitation algorithms (IL) could be abstracted as

$$ $\tau=ID(s_{o}^{t},s_{o}^{t+1}),\ \tau=IL(s_{o}^{t},s_{i}^{t+1}),$ (2) $$

with driven torque $\tau$, timestamp $t$, observed kinematic states $s_{o}$, and the state to imitate $s_{i}$.
Both ID and IL learn the dynamic production mechanism of human motion.
However, IL algorithms are not directly applicable to ID due to the non-equivalence between $s_{o}^{t+1}$ and $s_{i}^{t+1}$.
The errors in kinematics could be magnified in dynamics (Uchida & Seth, ).
This also makes ID algorithms that are deeply coupled with markerless MoCap less reliable.
However, it is possible to extract knowledge from IL for ID.
In this section, we introduce a simple but effective ID data collection pipeline with IL algorithms.
First, the adopted IL algorithm (Luo et al., ) is briefly covered in Sec. 3.1.
Then, the data collection pipeline is introduced in Sec. 3.2.
An overview is given in Fig. 2.

Figure: Figure 2: ImDy construction. We first train a motion imitation policy following Luo et al. (). Then, the policy is adopted to imitate arbitrary motions, with the imitated states recorded as ImDy.
Refer to caption: x2.png

### 3.1 Imitation Learning Basics

A motion imitator $\pi(a^{t}|s_{o}^{t},s_{i}^{t})$ is trained following Luo et al. () to solve the Markov Decision Process $\mathcal{M}=\langle\mathcal{T},\mathcal{S},\mathcal{A},\mathcal{R},\gamma\rangle$.
The transition dynamics $\mathcal{T}$ and states $\mathcal{S}$ are governed by the physics simulator.
For each timestamp $t$, the policy $\pi$ produces action $a^{t}\in\mathcal{A}$ and the reward $\mathcal{R}$, based on state $s\in\mathcal{S}$.
The training goal is maximizing the reward expectation $\mathbf{E}(\sum_{t=1}^{T}\gamma^{t-1}r^{t})$.

Transition.
IsaacGym (Makoviychuk et al., ) is adopted for simulation.
A 24-joint humanoid with SMPL (Loper et al., ) kinematics and physical properties following Luo et al. (; ) is adopted with variable shape parameter $\beta\in\mathbb{R}^{10}$.
Thus, a human pose at timestamp $t$ could be defined as $q^{t}=\{\theta^{t},p^{t}\}$, where $\theta^{t}\in\mathbb{R}^{J\times 6}$ is the joint rotation in the 6d representation (Zhou et al., ) and $p^{t}\in\mathbb{R}^{J\times 3}$ is the 3D joint position.

State.
At timestamp $t$, $s^{t}$ contains the observed $s_{o}^{t}$ and $s_{i}^{t+1}$ to imitate.
$s_{o}^{t}$ is defined in simulator as $s_{o}^{t}=(q_{t},\dot{q}_{t},\beta)$ with 3D body pose $q_{t}$, velocity $\dot{q}_{t}$, and body shape $\beta$.
$s_{i}^{t+1}$ is defined similarly except that it is the reference motion with finite-differentiated velocities.

Action.
All joints but the pelvis are actuated with proportional derivative (PD) controllers, with $a^{t}$ as the PD target.
The torque applied could be calculated as

$$ $\tau^{t}=k^{p}\circ(a^{t}-q^{t})-k^{d}\circ\dot{q}^{t}.$ (3) $$

Reward. The reward is composed of four terms: motion imitation reward for minimizing the difference between the imitated states and the expected states, fail-state recovery reward (Luo et al., ), AMP reward (Peng et al., ), and energy reward to reduce jittering.

Training.
Following PHC, three primitive policies are progressively trained with hard negative mining, two for pure motion imitation, and one for fail-state recovery.
Then, a composer learns to combine the primitives dynamically.
PPO (Schulman et al., ) is adopted to train the policies.

### 3.2 Imitated Data Acquisition

**Table 1: ImDy compared to related human dynamics datasets. Zell et al. () recorded full-body data but simplified the upper body with a single torso segment. All previous efforts contain only GRF for feet (indicated with *), while we include full body GRF.**
| Dataset | #Subj | Duration (h) | Dynamics | Body Repr. | Style |
| --- | --- | --- | --- | --- | --- |
| Zell et al. () | 22 | 0.07 | GRF* & Torques | Partial Skeleton* | Real |
| Scott et al. () | 10 | 7.6 | vertical GRF* | Skeleton | Real |
| UnderPressure (Mourot et al., ) | 10 | 5.5 | vertical GRF* | Skeleton | Real |
| GroundLink (Han et al., ) | 7 | 1.5 | GRF* | SMPL | Real |
| AddBiomechanics (Werling et al., ) | 273 | 57.6 | GRF* & Torques | Rajagopal et al. () | Real |
| ImDy | 435 | 152.3 | GRF & Torques | SMPL | Simulated |

With the imitator $\pi$, we pursue to extract its inherent ID knowledge.
As in Eq. 2, though the imitator-produced $\tau$ is not accurate for $s_{o}^{t}\to s_{i}^{t+1}$ since $s_{i}^{t+1}$ is not guaranteed to reach, $\tau$ is accurate for $s_{o}^{t}\to s_{o}^{t+1}$.
Thus, the idea could be as simple as using $\pi$ to imitate arbitrary motions in the simulator, then collecting all the observed states $s_{o}$, the applied torques $\tau$, and the full-body GRF $\lambda$.

We adopt AMASS (Mahmood et al., ) and KIT (Mandery et al., ) as two major data sources.
Sequences involving humans interacting with objects other than the ground are excluded, resulting in over 50 hours of motion.
All the sequences are re-sampled to 30FPS, with the z-axis as the gravity axis.
Then, the sequences are imitated three times by the two primitive policies and the multiplicative policy with a simulation frequency of 60Hz, resulting in over 150 hours of human motion data with dynamics.
States including $q,\dot{q},\beta$ are recorded in synchronous with the torque $\tau$, all restored in the format of SMPL (Loper et al., ) if possible.
Moreover, GRFs for the whole body are also recorded, resulting in ImDy, a large-scale human motion dynamics dataset.

Detailed statistics of ImDy are demonstrated in Tab. 1.
There are three major advantages.
First, a considerably larger data scale is 100$\times$ compared to previous efforts with full-body dynamics data, covering a wide span of human motion, which could be hard to acquire in laboratory setups.
Second, thanks to the advanced simulator (Makoviychuk et al., ), we could include ground reaction forces for the whole body instead of the two feet only like in previous efforts.
Finally, we represent humans with SMPL (Loper et al., ), increasing availability.

## 4 Learning ImDyS

With the collected ImDy, we could address the human inverse dynamics in a full-supervised manner with a data-driven solver ImDyS.
In Sec. 4.1, we first introduce the formulation of data-driven inverse dynamics.
Then, the proposed data-driven solver is introduced in Sec. 4.2.
The overall pipeline of ImDyS is illustrated in Fig. 3.

Figure: Figure 3: ImDyS overview. Taking a motion transition, ImDyS predicts the internal dynamics and ground reaction forces. Moreover, a prior discriminator is trained with the feature from ImDyS. A two-stage sim2real training curriculum is further designed.
Refer to caption: x3.png

### 4.1 Formulation

Recall the abstraction of ID in Eq. 2, which we rewrite as

$$ $(\tau^{t},\lambda^{t:t+1})=ImDyS(s^{t-w:t+w+1}).$ (4) $$

Given the kinematics states from timestamp $t-w$ to $t+w+1$, ImDyS is required to estimate the internal dynamics $\tau^{t}$ for the transition from $s^{t}$ to $s^{t+1}$ and the ground reaction forces $\lambda$ that the subject bears in timestamp $t$ and $t+1$.

Motion States $s$ could be represented by either SMPL parameters, joint angles, joint coordinates, or marker coordinates.
However, due to the topology divergence, the conversion among SMPL parameters, joint angles, and joint coordinates is non-trivial with limited performances.
To guarantee that ImDyS could be seamlessly adopted to both ImDy and real-world biomechanics data, we adopt marker coordinates as motion state representation for ImDyS.
The state $s^{t}=(m^{t},\dot{m}^{t})$ is composed of marker coordinates $m^{t}$ and finite-differentiated velocities $\dot{m}^{t}$ at timestamp $t$, which are easy to obtain for both ImDy and AddBiomechanics (Werling et al., ).
Two temporal windows before and after the transition with a length of $w$ are included for contextual information.
Notice that human physical properties like height and weight could also be implicitly represented by the markers.
The states are canonicalized w.r.t. the heading direction of $s^{t}$.

Internal Dynamics $\tau$.
For ImDy, the imposed angular momentum $\tau_{am}$ is adopted for dynamics representation.
Notice that in Sec. 3.2, the original sequences are in 30FPS, while the simulation runs at 60FPS.
This means for each motion transition $(s^{t},s^{t+1})$, two torques were applied sequentially, each for $\frac{1}{60}s$.
Predicting both torques is a plausible design choice.
However, the second torque is based on the un-recorded mid-state between $s^{t},s^{t+1}$.
Predicting it involves the forward dynamics from $s^{t}$ to the mid-state, with increased complexity.
To this end, instead of predicting instantaneous torques, we switch to predicting the imposed angular momentum $\tau_{am}\in\mathbb{R}^{(J-1)\times 3}$, the time-accumulation effect of torque, for each motion transition.
Thus, the modeling could stay consistent with proper complexity, only needing to sum the two torques up for $s^{t},s^{t+1}$ and then multiply it with the delta time.
For AddBiomechanics, joint torque $\tau_{jt}$ is adopted for dynamics representation.

Ground Reaction Forces $\lambda$.
Different from previous efforts (Mourot et al., ; Han et al., ; Werling et al., ) with foot GRFs only, we predict full-body GRF $\lambda\in\mathbb{R}^{J\times 3}$ as in Fig. 1.

### 4.2 Data-driven ImDyS

Model architecture.
With the enormous data scale of ImDy, we would like to keep ImDyS simple.
An encoder-head structure is adopted.
$s^{t-w:t+w+1}\in\mathbb{R}^{M\times(2w+2)\times 6}$ is first flattened as $\tilde{s}^{t-w:t+w+1}\in\mathbb{R}^{M\times(12w+12)}$ with window size $w$ and $M$ markers.
Then, a transformer encoder converts $\tilde{s}$ into ID feature $f_{ID}\in\mathbb{R}^{d}$, where $d$ is the feature dimension.
For prediction, we decompose $\tau_{am}$ and $\lambda$ into magnitudes $|\tau_{am}^{t}|,|\lambda^{t:t+1}|$ and direction vectors $\vec{\tau}_{am}^{t},\vec{\lambda}^{t:t+1}$ and predict each of them with a linear head.
$\tau_{jt}^{t}$ is predicted with another linear head.
The final predictions are $\hat{\tau}^{t}_{am}=|\tau_{am}^{t}|\vec{\tau}_{am}^{t},\hat{\lambda}^{t:t+1}=|
\lambda^{t:t+1}|\vec{\lambda}^{t:t+1}$ and $\tau_{jt}^{t}$.

Loss terms.
L1 loss, cosine loss, and L2 loss are adopted to optimized the predicted magnitudes $|\tau_{am}^{t}|,|\lambda^{t:t+1}|$, direction vectors $\vec{\tau}_{am}^{t},\vec{\lambda}^{t:t+1}$, and joint torques $\tau_{jt}$ as $L_{mag},L_{cos},L_{L2}$ respectively.
Besides, a forward dynamics (FD) loss $L_{fd}$ is proposed with an auxiliary FD model to inform the learning with the ID-FD cycle.
The FD model takes $s^{t-w:t},\tau^{t}=(\tau_{am}^{t},\tau_{jt}^{t}),\lambda^{t}$ as input, predicts the next-frame joint angles.
The FD loss is thus computed with cycle consistency as

$$ $L_{FD}=|s^{t+1}-FD(s^{t-w:t},\hat{\tau}^{t},\hat{\lambda}^{t})|.$ (5) $$

Finally, we devise a loss term similar to Peng et al. (), which encourages the ImDy feature $f_{ID}$ to model physically plausible motion transitions.
A linear discriminator takes $f_{ID}$ and outputs a logit indicating whether the motion transition is plausible.
To train the discriminator, besides the positive samples from ImDy and AMASS (Mahmood et al., ), we propose two negative sample generation strategies.
First, $s^{t-w:t+w+1}$ is randomly permuted along the temporal axis.
Second, random Gaussian noises are added on $s^{t-w:t+w+1}$.
Binary cross-entropy loss is adopted as $L_{cls}$.

Sim2Real training curriculum is devised in a simple two-stage manner.
In the first stage, ImDyS is trained on ImDy, with the overall loss as $\mathcal{L}_{s1}=\alpha_{1}L_{mag}+\alpha_{2}L_{cos}+\alpha_{3}L_{FD}+\alpha_{
4}L_{cls}$.
In the second stage, we freeze the encoder and train the linear head for joint torques $\tau_{jt}$.
The loss is calculated as $\mathcal{L}_{s2}=\alpha_{3}L_{FD}+\alpha_{4}L_{cls}+\alpha_{5}L_{L2}$.
Results show that ImDy pre-trained encoder converges fast on AddBiomechanics, indicating that it holds useful knowledge on real-world human dynamics.

## 5 Experiments

### 5.1 Implementation Details

PHC (Luo et al., ) adopted the position-control mode implemented by IsaacGym (Makoviychuk et al., ), where the imposed torque is calculated differently from the naive PD controller and inaccessible.
Therefore, we re-trained the PHC on AMASS (Mahmood et al., ) with the effort-control mode, and a naive PD controller was adopted.
Training the PHC took approximately 10 days, with a success rate on AMASS of 91.3%.
The window size $w$ is set as 2 to keep a short-term motion modeling, which is proven helpful in Sec. 5.3.
The encoder of ImDyS is a three-layer transformer with a dimension of 64, ReLU activation, and LayerNorm.
The loss weights are set as $\alpha_{1}=\alpha_{3}=0.01$, $\alpha_{2}=\alpha_{4}=\alpha_{5}=1$ to maintain all terms at similar numerical scales for training stability.
ImDyS, the prior discriminator, and the FD model are all trained using the AdamW optimizer with a batch size of 2,400 for 140 epochs on ImDy for the first stage.
For the second stage, ImDyS is further tuned on AddBiomechanics for only 10 epochs with the same hyper-parameters.
When generating negative samples for the prior discriminator, the two strategies are randomly adopted with a positive-negative ratio of 1:1.
We split ImDy into a training set of 27,501 sequences and a test set of 3,055 sequences.
All the data collection processes and experiments are conducted on a single NVIDIA RTX3090 GPU.

### 5.2 Evaluation on ImDy

Figure: Figure 4: Qualitative results on ImDy. $\tilde{\ }$ indicates a low-pass filter at 14Hz is applied. A typical gait sample and an arm-waving sample are visualized.
Refer to caption: x4.png

Metric.
We calculate the mPJE (mean Per Joint Error) for $\tau$ and $\lambda$ as

$$ $mPJE_{\tau}=\frac{1}{J}\sum_{j=1}^{J}|\tau_{j}-\hat{\tau}_{j}|_{2},\ mPJE_{ \lambda}=\frac{1}{J}\sum_{j=1}^{J}|\lambda_{j}-\hat{\lambda}_{j}|_{2},$ (6) $$

where $J$ is the number of joints.
The result is further normalized by body weight to align different subjects, with units of $N\cdot m\cdot s/kg$ and $N/kg$.
Specifically, the mPJE for the GRF on both feet $mPJE_{\lambda_{lf}},mPJE_{\lambda_{rf}}$ is also reported.

Baseline.
Few efforts except IL algorithms are feasible as baselines.
To this end, we introduce PHC as a baseline, where the sequences in ImDy are re-imitated by the re-trained PHC.
The imposed angular momentums and the GRF obtained via the re-imitation process are adopted as the baseline predictions.
With this baseline, we demonstrate the amplification effect from the kinematics error to the dynamics error, thus validating the performance of directly adopting IL for ID.

Results.
Quantitative results are shown in Tab. 5.3.
PHC produces an mPJPE of 56.13 mm, which is admirable for kinematics but results in high dynamics errors.
ImDyS demonstrates considerably better performance.
We further visualize two qualitative samples in Fig. 4.
Since the raw data could be jittering, we also filter the predictions with a low-pass filter at 14Hz, denoted as $\tilde{\tau},\tilde{\lambda}$, which helps reveal the general trend of the predictions.
For the gait sample at the left, the imposed angular momentum $\tau$ at the left hip and the left knee are plotted, along with the GRF $\lambda$ at the left toe.
We also plot the error between the predicted values and GT values.
ImDyS manages to faithfully reconstruct $\tau$ for the left knee and hip with minor errors.
Meanwhile, PHC typically produces higher errors due to phase mismatch.
As shown, it tends to lag behind the input motion.
For GRF, ImDyS also produces reasonable predictions.
Besides a typical gait analysis sample, we also demonstrate the performance of ImDyS with an arm-waving motion.
The $\tau$ at directly related body segments including the left thorax and shoulder is visualized.
ImDyS reproduces the dynamic status with better alignment to GT compared to PHC.
Generally, ImDyS produces reasonable ID predictions.
A potential issue is the jittering prediction, which is a consequence of the jittering observations in ImDy.
However, we show that ImDyS could handle real-world smooth observations well even when trained only on jittering ImDyS.
More demonstrations are available in the supplementary video.

### 5.3 Evaluation on GroundLink

Figure: Figure 5: Qualitative results on GroundLink including PHC, GroundLinkNet, and ImDyS. The GRF $\lambda$ for both feet are shown. Surprisingly, ImDyS provides better consistency with the ground truth.
Refer to caption: x5.png

Metrics.
GroundLink (Han et al., ) provides 1.5-hour motion from 7 subjects with GRF.
We adopt subject 7 for evaluation.
mPJEλ at both feet normalized by body weight is reported.

Baselines.
PHC is evaluated similarly to Sec. 5.2.
We also report the performance of GroundLinkNet (Han et al., ).
PHC and ImDyS are not exposed to GroundLink during training, resulting in a zero-shot evaluation for ImDyS and the PHC baseline.
Also, GroundLinkNet operates on 250FPS motion, while ImDyS and the PHC re-imitation baseline only operate on 30FPS motion.
Finally, GroundLinkNet predicts GRF for both feet, while ImDyS and PHC could decouple feet into ankles and toes, and predict GRF separately for each part.
We add up the ankle GRF and the toe GRF as the foot GRF.
All predictions are re-sampled to 30FPS.

Results.
Quantitative results are illustrated in Tab. 5.3.
Surprisingly, both ImDyS and PHC manage to outperform the specifically trained GroundLinkNet.
We attribute this to the enormous scale of AMASS and ImDy, which is much larger than GroundLink.
Moreover, even though ImDyS is trained on simulated ImDy only, it generalizes to real-world data with competitive performance. We visualize the results in Fig. 5, 7.
The PHC re-imitation baseline produces jittering predictions similar to Fig. 4.
GroundLinkNet, though specifically trained on GroundLink, fails to capture the rapid GRF changes in this jumping jack motion, resulting in a relatively flat output.
In contrast, ImDyS surprisingly presents good consistency with GT, and even faithfully reproduces the intense peak GRFs for the left foot for the jumping jack.
Besides, the prediction is not as jittering as in Fig. 4, indicating ImDyS could handle real-world smooth data well.

### 5.4 Evaluation on AddBiomechanics

Metrics. AddBiomechanics (Werling et al., ) is recently proposed with over 50 hours of human dynamics data from 273 subjects.
We adopt the armless part of this dataset. We follow the train/test split in Addbiomechanics and report mPJE for the joint torque normalized by body weight.

Results. A baseline model trained only on AddBiomechanics for 150 epochs with the same architecture as ImDyS is reported to showcase the generalization from ImDy to real-world dynamics. All data are re-sampled to 30 FPS. Quantitative results are illustrated in Tab. 5.3.
ImDyS outperforms the baseline with faster convergence, indicating the efficacy of Imdys in pre-training and mitigating the sim2real gap.
Qualitative results are shown in Fig. 6, 8, where ImDyS shows better alignment with GT and more precise magnitude predictions. More analyses on the relationship between performance, data distribution and quality are in the appendix.

Figure: Figure 6: Joint torque predictions on AddBiomechanics.
Refer to caption: x6.png

### 5.5 Ablation Studies

Different Motion Representations are evaluated on ImDy in Tab. 5.4. Though SMPL and joint-based representations perform better, we adopt marker-based representation for its generality.

Different Loss Terms are evaluated in Tab. 5.4. $L_{FD}$ is proven to contribute more than $L_{cls}$.

Different Window Sizes $w$ are evaluated on AddBiomechanics in Tab. 5.4. ImDyS achieves the best balance between rich contexts and conciseness with $w=2$.

## 6 Discussion

Given the fully simulated nature of ImDy, a reasonable question is the sim2real problem.
ImDy could be unnaturally jittering as in Fig. 4.
Also, the physical properties of the simulated humanoid differ from those of real humans.
Empirically, experiments show that ImDyS generalizes well to real-world data, partially mitigating this gap.
The reason could be threefold.
First, the jitters are unnatural but still physically plausible given that ImDy faithfully preserves consistent information for the simulated physics phenomena.
Second, the small window size of ImDyS prevents it from relying on long-term contexts, where jitters are more salient.
Finally, the enormous scale of ImDy is helpful for generalization.
To further mitigate the sim2real gap with ImDy is a meaningful goal to pursue.
Besides, ImDyS is designed as a first-step baseline to demonstrate the efficacy of ImDy.
Introducing more sophisticated designs to regulate the behavior of ImDyS would be preferable.
Moreover, ImDy only considers GRF, while other external forces are not involved.
Also, interaction with other entities is absent.
Exploration of these would be interesting for future works.

## 7 Conclusion

Leveraging the inherent resemblance between inverse dynamics and imitation learning, we proposed a novel human dynamics dataset ImDy, which contained over 150 hours of human motion paired with full-body driven torques and GRFs from well-developed simulator and imitation algorithms.
Based on ImDy, a data-driven human inverse dynamics solver ImDyS is devised to reconstruct the driven angular momentum and contact forces from kinematic observations.
ImDyS demonstrated impressive performance on both simulated and real-world data.
As a first step toward scalable and easily accessible human inverse dynamics, we hope ImDy can shed new light on the data-driven physical analysis of human motion.

Figure: Figure 7: Extensive visualization on GroundLink. For various motions, ImDyS shows superior alignment with GT compared to specifically trained GroundLinkNet, showcasing the efficacy of ImDy. Especially, the intense peaks are also reproduced by ImDyS.
Refer to caption: x7.png

Figure: Figure 8: Extensive visualization on AddBiomechanics. ImDyS demonstrates superior performance to the baseline, indicating ImDy’s generalization ability.
Refer to caption: x8.png

## Appendix

## Appendix A Applications of ImDyS

In this section, we demonstrate some downstream applications of ImDyS.

Human Work Analysis.
With the predicted $\tau$, we could calculate the work conducted at each joint.
Visualizations are in Fig. 9, reasonably revealing the energy flow during human motion.

Figure: Figure 9: Human work visualization with ImDyS prediction. Green indicates positive work and red indicates negative work.
Refer to caption: x9.png

Motion Assessment.
Another interesting application of ImDyS is based on the discriminator introduced in Sec. 4.2.
Besides facilitating ImDyS learning, it could also assess whether a motion transition is physically plausible as in Fig. 10.
Specifically, we adopt ImDyS to assess the motion generated from MDM Tevet et al. ().
ImDyS reasonably tells when the motion starts to deviate from realism.

Figure: Figure 10: Motion assessment visualization. The motion artifacts are annotated with red with a low indicator value from ImDyS. As shown, ImDyS manages to identify implausible transitions in a kicking motion generated by MDM.
Refer to caption: x10.png

## Appendix B AddBiomechanics Results Analysis

We visualize a failure case on the AddBiomechanics dataset in Fig. 11.
As shown, neither the baseline nor ImDyS manages to faithfully predict the joint torques for the jumping motion.
In the following, we discuss the reasons for the failure.

Figure: Figure 11: Visualization of a failed joint torque prediction case on AddBiomechanics. For the “jumping” motion, the baseline and ImDyS both perform sub-optimally. Neither of them correctly predicts the joint torques.
Refer to caption: x11.png

Figure: Figure 12: AddBiomechanics data distribution. Among all activities, walking and running account for over 75%, indicating that the amount of gait data is large
Refer to caption: x12.png

Data distribution.
Fig. 12 shows the data distribution of AddBiomechanics (Werling et al., ).
As shown, over 75% of the data are either walking, running, or standing, which are extremely limited.
Though ImDyS is empowered with the diverse ImDy, it still requires data to learn the mapping between simulated torques and real torques for non-gait data.
These result in ImDyS’ poor performance when processing non-gait data.
Further mitigating the limited data issue for non-gait motions would be a meaningful goal to pursue.

Figure: Figure 13: Relationship between data quality and model performance differences. Higher residual torque indicates lower data quality with lower reliability of the optimized GT torques. $\Delta$mPJE is the difference between the mPJE of ImDyS and the baseline. #seqs is the number of sequences. With the residual torque increasing, the baseline provides lower mPJE than ImDyS, indicating the baseline overfits low-quality data. Instead, ImDyS, with the knowledge inherited from ImDy, shows less overfitting for these cases.
Refer to caption: x13.png

Figure: Figure 14: Relationship between data quality and ImDyS performance. Higher residual torque indicates lower data quality with lower reliability of the optimized GT torques. The performance of ImDyS degenerates synchronously with data quality.
Refer to caption: x14.png

Data quality.
Besides the distribution, the quality is also limited in AddBiomechanics.
As shown in Fig. 11, joint torques for some joints (like the lumbar) suffer from unstable optimization with jittering results.
According to Werling et al. (), 21.2% of AddBiomechnics are classified with clinical-grade high quality (residual torque $<$ 0.1 * body weight * height).
There exists a 1.6829 Nm/kg average root residual torque of the optimized GTs in AddBiomechanics, which is considerably higher than the mPJE of ImDyS (0.1626 Nm/kg).
We further analyze the relationship between the data quality and the model performances.
We adopt residual torques as an indicator of the data quality and calculate $\Delta$mPJE=mPJEImDyS-mPJEBaseline of sequences with different residual torques.
Notice that higher residual torque indicates lower data quality with lower reliability of the optimized GT torques.
Results are shown in Fig. 13.
As shown, some samples could suffer from bad kinematics fitting (like the unnatural anterior pelvic tilt in Fig. 13), resulting in less reliable GT optimized joint torques.
An interesting phenomenon is that the lower the residual torques are, the better ImDyS performs, which means ImDyS performs better for high-quality samples.
This indicates the baseline might overfit low-quality data with high residual torques.
Instead, ImDyS, with the knowledge inherited from the large-scale diverse ImDy, manages to resist the negative influences from low-quality samples.
We also show how the mPJE of ImDyS changes with data quality in Fig. 14.
As shown, the performance of ImDyS degenerates synchronously with data quality.

Per-Joint Performance Analysis.
It is also noticeable in Fig. 11 that the gap between GT and prediction differs for different joints.
To this end, we further analyze the per-joint performance of ImDyS.
The per-joint mPJE of ImDyS and the per-joint mPJE of ImDyS in each frame for samples with clinical-grade quality (residual torque $<$ 0.1 body weight * height) is demonstrated in Tab. 7.
ImDyS manages to improve the performance on most joints compared to the baseline without ImDy, especially for the hips.
An interesting phenomenon is that ImDyS performs slightly better on the right half of the body.

**Table 7: Per-Joint mPJE of ImDyS for samples with clinical-grade quality.**
|  | Right mPJEτ $\downarrow$ | Left mPJEτ $\downarrow$ |  |  |
| --- | --- | --- | --- | --- |
| Joint Name | ImDyS | Baseline | ImDyS | Baseline |
| Hip Flexion | 0.267 | 0.270 | 0.273 | 0.274 |
| Hip Adduction | 0.196 | 0.212 | 0.198 | 0.199 |
| Hip Rotation | 0.087 | 0.109 | 0.083 | 0.098 |
| Knee | 0.193 | 0.188 | 0.195 | 0.209 |
| Ankle | 0.197 | 0.195 | 0.202 | 0.199 |
| Subtalar | 0.069 | 0.071 | 0.079 | 0.084 |
| MTP | 0.0005 | 0.0006 | 0.0007 | 0.0008 |
| Lumbar Extension | 0.328 | 0.339 | - | - |
| Lumbar Bending | 0.255 | 0.255 | - | - |
| Lumbar Rotation | 0.113 | 0.113 | - | - |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. KIT is also part of the AMASS dataset, but the paper mentions using the AMAS and KIT datasets. Is there any difference?

2. ImDyS is tuned on AddBiomechanics for 10 epochs. How is the performance without fintuning?  

3. I am also curious about whether a better-estimated dynamics is helpful for downstream tasks like human pose estimation.
soundness: 3
strengths: 1. Estimating human dynamics from motion observation is an interesting and meaningful direction.

2. The paper is well-written and easy to follow.

3. The proposed strategy is to make human dynamics look effective.

4. Experiments show better performance than previous methods.
summary: This paper focuses on human inverse dynamics, aiming to estimate the driving torques of body joints from motion observations. While previous methods have primarily targeted lab settings, this paper employs imitation learning within physics simulators. Building on this approach, the authors created a dataset by importing the AMASS and KIT datasets into Isaac Gym, obtaining contact forces and joint torques based on the previous PHC method. Experimental results also demonstrate improved performance compared to the AddBiomechanics benchmark after fine-tuning.
weaknesses: While the proposed method looks promising, I am concerned that its technical contributions may be limited for an ICLR paper. Numerous previous works on physics-aware pose estimation have utilized human dynamics through either non-differentiable physics simulators with reinforcement learning or differentiable dynamics solvers, both of which seem to be more challenging approaches. In these prior works, dynamics estimation often served as an intermediate task, either implicitly or explicitly. In this paper, it seems that only part of the previous methods were adopted, with an increased focus on dynamics estimation. Furthermore, the method employed is also similar to as that of PHC.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Have the authors studied the error in the learned FD model? Is it possible to replace this with a differentiable FD simulation?

I would appreciate it if authors could share visual ablations with various motions, with and without the AddBiomechanics so I can confirm the effect of the sim2real, especially for motions out of distribution from the AddBiomechanics data. It is hard to tell how much jittering still exists just by looking at the mPJE.
soundness: 3
strengths: The proposal is a great data and baseline model to advance the data-driven approach to understanding dynamics in human motions. While the baseline model ImDyS is simple, I see enough novel, especially in the loss design with the cycle consistency using the learned FD model and a discriminator loss on the ID features.
summary: The paper presents a dataset ImDy (Imitaed Dynamics) with 150 hours of physically simulated humanoid characters imitating kinematic human motions with joint torques and full-body ground reaction forces. The paper claims that ImDy data is unique in its scale (100x more than previous ones), includes more general physics parameters such as the ground reaction forces (GRFs) for the entire body, and is represented in SMPL. Using this ImDy, the paper proposes ImDyS (Imitaed Dynamics Solver), a data-driven humanoid inverse dynamics solver. ImDyS is a simple encoder-head design where a transformer encoder maps the pose state to the ID (inverse dynamics) feature, which is used to predict the dynamics using linear heads. The notable loss designs are the use of the FD (forward dynamics) cycle consistency loss with an FD model trained together to predict the pose state from the dynamics, and a discriminator loss on the ID features, similar to AMP [Peng et al. 2021]. To overcome the sim2real issues, the model is trained in two stages, first with the ImDy data, and then with the data from AddBiomechanics while freezing the encoder and training only the linear head for joint torques.
weaknesses: The paper is straightforward and looks good. I appreciate some clarifications in the questions

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1) Connected to the weakness that I pointed above, why does authors' imitation on AMASS have lower success rate than PHC? I would suggest the authors to provide quantitative/qualitative comparisons of two imitation learning schemes (their and PHC), and justify their purpose of the proposed method.

2) It is not clear what can ImDyS bring over the imitation learning. Let's assume we collect kinematics data and want to extract dynamics from it. We can either run ImDyS to estimate dynamics or ImDy policy to imitate the kinematics (where the dynamics will come together). What is the major benefit of using ImDyS in this case? I guess the inference speed but wonder authors' perspective. I would suggest authors to provide comparisons between the proposed IL and ImDyS to highlight the unique values of ImDyS.
soundness: 3
strengths: The paper's strengths can be summarized as follows:

1) They train an imitation policy that can imitate the large-scale human motion capture dataset (AMASS and KIT) which provides the pairs of human kinematics and dynamics with the larger scale than any other publicly available datasets.

2) They train an inverse dynamics solver on the processed data, resulting in the state-of-the-art performance in estimating ground reaction force and dynamics state of the human (on both imitated and real data).

3) Good visual presentation is appreciated
summary: In this paper, authors introduce data-driven inverse dynamics solver (ImDyS) that could estimate dynamic states of the person (including joints torque and ground reaction forces) from the observed kinematics. To train ImDyS in a fully-supervised manner, they first propose the data acquisition to collect scalable human dynamics data following the recently proposed imitation learning methods. 

They benchmarked their method on both the imitated data (ImDy) and real-world data, AddBiomechanics, which has the measured ground reaction forces and 3D kinematics labels. The results showed that the proposed method is more accurate to estimate human dynamics from both the imitated and real datasets. 

Since the authors borrow the concept of imitation learning from the previous work (PHC), I do not see significant novelty in their data acquisition pipeline. However, given their contribution to processing and sharing of the dataset, and the performance of the supervised inverse dynamics module, I think the paper can contribute to the community.
weaknesses: Data acquisition part of this paper does not seem to include significant novelty since most of the part were borrowed from the previous work (PHC, Luo et al.,). I would like authors to elaborate what their imitation learning can offer other than PHC. Can we run PHC to collect kinetic/kinematics pairs of AMASS and KIT data? If so, why do we need to use the proposed method for the dynamics data acquisitions?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Figure 11 visualizes a failed joint prediction case; the plot shows that ground truth torques are quite smooth while the predicted are jittery (before both the ground truth and predicted are jittery). Based on the provided analysis, it is possible the data quality of AddBioMechanics could be the cause here?
soundness: 3
strengths: - The proposed ImDy and ImDyS, as far as I know, is a novel attempt at using simulation and humanoid motion tracking to obtain large datasets for inverse dynamics analysis. The results show that the large-scale dynamics data obtained from simulation could be of use in learning models (ImDyS) that perform well on real human performance data.
- A comprehensive analysis of the proposed ImDyS and the baselines is provided. Since this is a relatively new topic, not many prior arts exist. Nonetheless, ImDyS outperforms the baselines and generalizes to real-world data with competitive performance.
- I appreciate the analysis of failure cases, which analyzed the advantages of ImDyS and plausible causes for failure in predicting the joint torques on the jumping motion.
summary: This paper proposes ImDy, a human inverse dynamics benchmark & dataset that contains 150 hours of motion with joint torque and full-body ground reaction forces obtained from humanoid motion tracking. ImDy provides a scalable approach to obtain more joint-level actuation for human motion as well as ground reaction force grounded in physics simulation. Experiments show that data obtained from ImDy could support learning an inverse dynamics model that has competitive performance on real-world data, showcasing the advantage of having large-scale synthetic dynamics data obtained from simulation.
weaknesses: - Since a dataset is proposed, an analysis of the data composition (like in Figure 12) would be beneficial. Since AMASS also contains many locomotion data, and the PHC with a naive PD controller has a lower success rate, it would be good to see what motions are tracked and included in the dataset.
- ImDys has many details that seem to be omitted. A data flow chart that includes the different losses in the appendix would be helpful. What is the motivation for including the discriminator loss if it does not contribute much to accuracy (as shown in Table 5)? 


Minor:

- I would recommend changing the term "imitation learning" to "motion tracking" as "imitation learning" is quite overloaded nowadays.

Meta-review: Scientific Claims and Findings:
The paper introduces ImDy, a human inverse dynamics dataset and benchmark containing 150 hours of motion with joint torque and full-body ground reaction force data. The key contributions are:
- A novel data acquisition pipeline that leverages imitation learning algorithms to collect large-scale human dynamics data
- A data-driven inverse dynamics solver (ImDyS) trained on this data that can accurately predict joint torques and ground reaction forces
- Demonstration of the model's effectiveness on both synthetic and real-world data (AddBiomechanics dataset)

Strengths:
+ The paper addresses an important gap in scalable human dynamics data collection and analysis.
+ The approach is well-grounded, using established imitation learning methods and physics simulators.
+ The authors provide thorough analysis including ablation studies, failure cases, and real-world validation.
+ The dataset is significantly larger than existing benchmarks (150 hours) and includes comprehensive dynamics information, which would be of interest to the relevant community

Weaknesses:
- The technical novelty in the data collection pipeline is somewhat limited, as it builds heavily on previous work (PHC).
- Some implementation details of ImDyS were initially unclear, though these were addressed during discussion.
- The sim2real gap remains a challenge

The reviewers has unanimously agreed to recommend the paper to be accepted. I agree with their assessment.

Decision: Accept (Poster)