DiffMimic: Efficient Motion Mimicking with Differentiable Physics
Authors: Jiawei Ren
Cunjun Yu
Siwei Chen
Xiao Ma
Liang Pan
Ziwei Liu
Venue: ICLR 2023 poster (2023)
Keywords: Physics-based Animation

Abstract: Motion mimicking is a foundational task in physics-based character animation. However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations. Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting in poor scalability. In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed $\textbf{DiffMimic}$. Our key insight is that DPS casts a complex policy learning task to a much simpler state matching problem. In particular, DPS learns a stable policy by analytical gradients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods. Moreover, to escape from local optima, we utilize an \textit{Demonstration Replay} mechanism to enable stable gradient backpropagation in a long horizon. Extensive experiments on standard benchmarks show that DiffMimic has a better sample efficiency and time efficiency than existing methods (e.g., DeepMimic). Notably, DiffMimic allows a physically simulated character to learn back-flip after 10 minutes of training and be able to cycle it after 3 hours of training, while DeepMimic requires about a day of training to cycle back-flip. More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation in future research. Our code is available at https://github.com/diffmimic/diffmimic. Qualitative results can be viewed at https://diffmimic-demo-main-g7h0i8.streamlitapp.com.

Published as a conference paper at ICLR 2023
DIFFMIMIC: EFFICIENT MOTION MIMICKING WITH DIFFERENTIABLE PHYSICS
Jiawei Ren" Cunjun Yu²
Siwei Chen
Xiao Ma
Liang Pan'
Ziwei Liu
1 S-Lab, Nanyang Technological University
2 School of Computing, National University of Singapore
3 SEA Lab
ABSTRACT
Motion mimicking is a foundational task in physics-based character animation. However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations. Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting poor scalability. In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed DiffMimic. Our key in- sight is that DPS casts complex policy learning task to much simpler state matching problem. In particular, DPS learns a stable policy by analytical gra- dients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods. Moreover, to escape from local op- tima, we utilize an Demonstration Replay mechanism enable stable gradient backpropagation in a long horizon. Extensive experiments on standard bench- marks show that DiftMimic has better sample efficiency and time efficiency than existing methods (e.g., DeepMimic). Notably, DiffMimic allows a physically simulated character learn Backflip after 10 minutes of training and be able cycle it after 3 hours of training. while the existing approach may require about a day of training to cycle Backflip. More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation future research. P
INTRODUCTION
Motion mimicking aims to find a policy to generate control signals for recovering demonstrated motion trajectories, which plays fundamental role in physics-based character animation, and also serves as prerequisite for many applications such as control stylization and skill composition. Although tremendous progress in motion mimicking has been witnessed in recent years, existing methods (Peng et al.) 20188a 8a 202I) mostly adopt reinforcement learning (RL) schemes, which re- quire alternatively learning a reward function and control policy. Consequently, RL-based methods days to imitate one single motion sequence, making their scalability
often take tens of hours notoriously challenging. In addition, RL-based motion mimicking highly relies on the quality of its designed (Peng et al. (2088)) learned (Peng et al. 2021) reward functions, which further burdens its generalization for complex real-world applications.
Recently, differential physics simulator (DPS) has achieved impressive results in many research fields, such as robot control (Xu et al. 2022) and graphics (Li et al., (202)). Specifically, DPS treats physics operators
differentiable computational graphs, and therefore gradients from objectives (i.e., rewards) can be directly propagated through the environment dynamics to control policy func- tions. Instead of alternatively learning between reward functions and control policies, the control policy learning tasks can be resolved in straightforward and efficient optimization manner with
'Equal contribution, listed in alphabetical order.
"Our code is available at http:://ggguubboo/ffffiiimiciiii
Qualitative results be viewed at http://lffiiiiiicccmmmmmn////iit

---

Published as a conference paper at ICLR 2023
Cospriar
L = Ells 82112
Figure I: Overview of our method. Left: DiftMimic formulates motion mimicking as a straight- forward state matching problem and uses analytical gradients to optimize it with off-the-shelf dif- ferentiable physics simulators. The formulation results in simple optimization objective compared to heavy reward engineering in RL-bbased methods. Middle: DiffMimic able mimic highly dynamic skills, e.g. Side-Flip. Right: DiftMimic has a significantly better sample efficiency and time efficiency than state-of-the-art motion mimicking methods. Our approach usually achieves high-quality motion (pose error < 0.15 meter) using less than 2 x 10" samples.
the help of DPS. However, despite their analytical environment gradients, optimization with DPS could easily get into local optima, particularly in contact-rich physical systems that often yield stiff and discontinuous gradients (Freeman et al. 2021 I Suh et al. 2022 Zhong et al.] 2022).. Besides, numerical gradients could also vanish/explode along the backward path for long trajectories.
In this work, we propose DiffMimic, fast and stable motion mimicking method with the help of DPS. Different from RL-based methods that require heavy reward engineering and poor sample efficiency, DiffMimic reformulates motion mimicking as a state matching problem, which could directly minimize the distance between rollout trajectory generated by the current learning policy and the demonstrated trajectory. Thanks to the differentiable DPS dynamics, gradients of the trajec- tory distance can be directly propagated optimize the control policy. As result, DiffMimic could significantly improve the sample efficiency with the first-order gradients.
However, simply utilizing DPS could not guarantee global optimal solutions. In particular, the roll- out trajectory tends to gradually deviate from the expert demonstration and could produce a large accumulative error for long motion sequences, due to the distributional shift between the learning policy and expert policy. To address these problems, we introduce the Demonstration Replay train- ing strategy, which randomly inserts reference states into the rollout trajectory as anchor states to guide the exploration of the policy. Empirically, Demonstration Replay gives a smoother gradient estimation, which significantly stabilizes the policy learning of DiffMimic.
To the best of our knowledge, DiffMimic is the first to utilize DPS for motion mimicking. We show that DiffMimic outperforms several commonly used RL.-based methods for motion mimicking on variety of tasks with high accuracy, stability, and efficiency. In particular, DiftMimmcc allows learning a challenging Backflip motion in only 10 minutes on a single V100 GPU. In addition, we release the DiffMimic simulator standard benchmark to encourage future research for motion mimicking.
2 RELATED WORK
Motion Mimicking. Motion mimicking is technique used produce realistic animations in physics-based characters by learning skills from motion captures (Hong et al.) 2019 Peng et al.,
2018a|
Lee al. 2019). This approach has been applied various downstream tasks physics- based animation including generating new motions by recombining primitive actions (Peng et al.)
2019
Luo et al. 2020). achieving specific goals (Bergamin al. 2019: Park et al.) [0019] Peng al.,
2021
and as a tre-tranning method for general-purpose motor skills (Merel et al. [202 8: Hasenclever
al.
2020 [Peg et al., 2022; Won et al. 2022). The scalability of these tasks can limited by
the motion mimicking process, which a key part of the pipeline in approaches like ScaDiver (Won)
(ct al. 2020). In this work, we demonstrate that the problem of scalability can be addressed using differentiable dynamics.
Speeding Up Motion Mimicking Most motion mimicking works are based on a DRL frame- work (Peng et al. 201 Sa, Bergamin et al. 2019), whose optimization is expensive. Several recent works speed up the DRL process by paremmee searching (Yang & Yin 202T) and constraint Another line of work learns world models achieve end-to-end gradi-
relaxation (Ma al. 2021
2

---

Published as a conference paper at ICLR 2023
Damo Replay
82
8
Reforence Trajectory
$a
Learner Trajectory
Backward
QC
Insert state
Figure 2: Computation graph of DiffMimmcc An example of demonstration replay during the third step demonstrated. The demonstration state in the third step is used replace the simulated state.
ent optimization (Won al., 2022, Fussell et al. 2021 However, learning world models requires extra training and introduces the risk of eπror accumulation. We are the first to use the off-the-sher DPS for motion mimicking, which achieves better sample efficiency than optimized DRL frame- works and requires no additional world models.
Differentiable Physics. Differentiable physics gains traction recently with the emerging differen- tiable physics simulators (DPS) (Hu et al. [219 [2200] Huang et al.) [202] QQao et al. 2021]. It has brought success in many domains and provides a distinctly new approach to control tasks. Super- track (Fussell et al. (021)) proposes to learn world model to approximate the differentiable physics and has achieved promising results. However, its policy horizon is limited during training due to the accumulation of errors in the world model. Various upplications stem from DPS like system identification (Jatavallabhula et al., (021), continuous controls (Lin et al. 2022 Xu et al., (022) in black-box counterparts which
the past few years and demonstrate promising results. Compared learn the dynamics with neural networks (Fussell et al. [202]] Li et al., 2020). differentiable simu- lations utilize physical models provide more reliable gradients with better interpretability. On the other hand, the analytical gradient from DPS suffers from noise or can sometimes be wrong (Zhong al. 2022 Suh et al., 2022) in a contact-rich environment. Non-smoothness or discontinuity in the contact event requires specific techniques to compute the gradient. However, they often introduce noise into the gradient (Zhong et al. 2022). Thus, in this work, we empirically show how to handle the noise in the gradient in contact-rich environment for motion mimicking.
Policy Optimization with Differentiable Physies Simulators. The integration of differentiable simulators in policy optimization has been significant advancement in reinforcement learning. With analytical gradient calculation, the sample efficiency of policy learning significantly im- proved (Mozer 1989 Mora et al.) 2021 Xu et al. 2022). However, the implementation is challenged by the presence of noisy gradients and the risk of exploding or vanishing gradients (Degrave et al. 2017). The policy may also get stuck at sub-optimal points due to the local nature of the gradient. To address these issues, various approaches have been proposed. SHAC (Xu et al. 2022) truncates trajectories into smaller segments to prevent exploding or vanishing gradients, but this requires care- ful design. On the other hand, PODS (Mora et al., (021)) utilizes second-order gradients, leading to monotonic policy improvement and faster convergence compared first-order methods. However, it relies on strong assumptions about the second- -oddr derivatives of the differentiable simulator and sensitive to their accuracy. An alternative approach the Imitation Learning framework with DPS proposed by ILD (Chen et al. 2022). Although it is simple and effective for robot control, it struggles with the exploration of highly dynamic motions. To tackle these challenges, we introduce replay mechanism to enhance its performance.
METHODOLOGY
3.1 A MOTION MIMICKING ENVIRONMENT IN DIFFERENTIABLE PHYSICS ENGINE.
Environment. We build our environment in Brax FFreeman et al. 2021). We design our simulated character following Deep (Peng al. 2018a). The humanoid character has 13 links and 34 degrees of freedom. The character has a weight of 45 kg and a height of 1.62m. Contact is applied to all links with the floor. The environment is accelerated by GPU parallelization. The physics
3

---

Published as a conference paper at ICLR 2023
simulator updates at rate of 480 FPS. The joint limits of the character relaxed to allow smoother gradient propagation. We keep the system configurations like friction coefficients consistent with DeepMimic.
State and Action. States include the position p. rotation q, linear velocity á and angular velocity of all links in the local coordinate. Additionally, following Peng al. (2018a), phase variable e in the range [0, 1] is included in the state to serve as a timestamp. Thus, the state of the environment, 8, contains the aforementioned information, = {p, q, p. 4, Φ). We follow the common practice and use PD controllers to drive the character. Given the current joint angle g, angular velocity ÿ and a target angle q. the torque on the actuator of the joint will be computed as:
T k q) + kalğ 4),
(3.1)
where q = 0, kp and k are manually-specifie gains of the PD controller. We keep kp and ka identical the PD controller used in DeepMimic. A policy network predicts the target angle for the PD controller at each joint. The control policy operates at 30 FPS.
3.2 MOTION MIMICKING WITH DIFFERENTIBLE PHYSICS
Motion Mimicking about matching the policy rollout with the reference motion. While it has a straightforward objective, considering human action can be rich and highly flexible, designing a reward to in- centivize and guide policy learning is not easy. For example, reward functions that
Algorithm 1 DiffMimic
Input: Optimization Iteration I, Episode Length T, Ref-
erence States $ = (81, $2, ) Error Threshold c.
Output: Optimized policy
3: Initialize the stochastic policy as me.
4: for optimization iteration = 1... / do
5: # Roll out trajectories with Expert Replay
tailored to guide policy to learn walk or learn to backflip can be different and this makes reward engineering diffi- cult. We reveal that the mimicking task can be surprisingly easy when analytical gradients can be obtained through DPS.
Initialize s1 + 81
for step = 1... do
T(si, az), mo(ss)) [ as). a mo(aa)))) otherwise.
if [|st - 8:112 <
Set-
8:
end for
The underlying idea DiffMimic is al- low the gradient to directly flow back from the distance between reference motion and policy rollout to the policy through an off- the-shelf DPS. We show the computation graph of DiffMimic in Fig. 2 Compared with existing RL-based approaches, DiffMMmmic directly optimizes the policy with supervised state-matching matching signals and largely eases the reward engineer- ing. Thanks to the analytical gradient, such approach has very high sample efficiency. Prior works (Fussell et al., (02T) explored analytic gradients with a learned differentiable world model which describes the dynamics in latent space. However, the learned world model reveals derlying physics of the system. Consequently, they undermine the effectiveness, generality, and interpretability of the learning with physics-based analytic gradients from DPS, in DiffMimic. In addition, online learning and inference with learned world models remain computationally expen- sive when compared with DPS.
& 10: Update the policy me with analytical gradient VoL.
9:
Compute loss C C= -t ||
11:
end for
12: Return the policy me.
We show in detail how to train the policy with DPS. For each iteration, we initialize the state to the first reference state and then roll out the maximum episode length in batch of parallel environments. Then we compute the wese 1.2 distance between the rollout trajectory and the reference trajectory terms of positions and rotations:
Σ |st 8e112.
(3.2)
|| sell3
1 |
Σ JEJ
1))² wr(g" - و²)² + wo -30 wa(6" م0
where p and أو are the global positions of the j-th joint from rollout and reference, 9¹ and q² are the global rotation of the j-th joint from rollout and reference in 6D rotation representation. The weights
4

---

Published as a conference paper at ICLR 2023
t=7
t=6
t=5
t=4
t=3
t=2
t=1
Figure 3: Illustration of Demonstration Replay, which happens when t=4 where the character is about to fall. The rollout state (in blue) is replaced by the reference state (in red) for the next step.
Wp. Wy, Was and wa only need to be approximately tuned to roughly equalize their magnitudes. Note that we use 6D rotation loss computation since it is advantageous over quaternions gradient- based optimization (Zhou et al. 2019). Without loss of generality, we denote the transition function in the dynamic system as 7 and the next state the system can be obtained from the transition function given the current action a, and character state 8,, 8t+1 T(s, 0₂). ) In DiffMimic, DPS serves as a transition function T that is fully differentiable. Thus, from the loss function, we can directly derive the gradient with respect both the current action a, and state 8t.
dc ÜT( a+)
OC
(
(T (st, as)
ac
0C
(
(OT(s), as)
(3.3)
da,
da,
Üs.
(T (8t,as)
0s,
By doing this recursively, the gradient can be propagated through the whole trajectory.
3.3 DEMONSTRATION REPLAY
Although mimicking with DPS results succinct learning framework. there are three well-known challenges in policy learning with DPS: 1) Exploding/vanishiin gradients with the long horizon; 2) Local minima may cause gradient-ban optimization methods stall; 3) Noisy or wrong gradients.
We observe that the high non-convexity in the motion mimicking task poses severe challenges to analytical gradient-based optimization. Due to the local nature the analytical gradients, the policy can be easily stuck at suboptimal points. As shown Tab. 6 when trying to mimic the Backflip skill, the character learns to use arms to support itself instead of exploring a more dynamic jumping move. In practical implementations, dividing the whole trajectory into smaller cabttrajcctoris for truncated Back-Propagatton- Through-Tiin Time (BPTT) further exacerbates the issue. As shown Tab. (a)-(b), 10-step gradient truncation leads the policy
less favorable local optimal.
Teacher forcing (Williams & Zipser 1989 is a common remedy to escape the local minimum in gradient-based sequential modeling. Teacher forcing randomly replace the states in rollout by the reference states. Given teacher forcing ratio ?y:
((s,, at), a ~ (als;) if 0, b Bernoulli(y) Se+1 { T()r, at ~ (a(a));) otherwise.
(3.4)
However, although the mechanism leads to better global optimal overall, it does not guarantee the character to faithfully mimic the reference motion in each frame. As shown in Tab. 6) although the character's overall trajectory matches the reference, several frames suffer from awkward poses. Fig. shows that a few frames have significantly larger pose errors than other frames.
We propose demonstration-guiie guided exploration to mitigate these challenges in DPS for policy learn- ing to help exploration and encourage faithfully mimicking the same time. The main idea of demonstration-guiee exploration is to replace states in the policy rollout with the corresponding demonstration states when states are too away from the reference. For a threshold
((st, or), a ne(alst) if [|s, s <e ((er, ot), a ~ (a(a)) otherwise.
8111
(3.5)
Since the criterion for choosing which states to replace is based on the performance of the current rollout, the replacing frequency dynamic adjusts itself during the model training.
5

---

Published as a conference paper at ICLR 2023
0.0
9.2
Damteiet 1230
0.5
0.5
00-
180
02-
Senooa
Figure 4: Pose error versus the number of samples. DiffMimic (4s): rollout 4 seconds of Diffmimic for evaluation. DiffMimic (20s): rollout 20 seconds of Diffmimic for evaluation. In general, DiffMimic allows the policy to generate high-quality motions with less than 3.5 x 10 samples. We refer the readers for more results to the appendix.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
For all the experiments, we run the algorithm with one single GPU (NVIDIA Tesla V100) and CPU (Intel Xeon E--2680). Following Peng al. (202)) metric. The average pose error over the whole trajectory of length with J joints is computed between the pose of the simulated character and the reference motion using the relative positions of each joint with respect to the root joint (in units of meters):
the average pose enror the main
Σ tET
Σ JEJ
(oot) (o6 p proot))ls ||2
(4.1)
where pt and p are the position of j-th joint of the simulated motion and reference motion at timestamp in the 3D cartesian space and root refers to the root joint. We mainly compare with DeepMimic (Peng et al. 2018a). Spacetime Bound (Ma et al. [021), and Adversarial Motion Prior (AMP) (Peng al. 2021) Dynamic Time Warping (Sakoe & Chiba 1978) is applied to synchronize the simulated motion and the reference motion following the convention.
Besides mimicking single motion clip, popular benchmark for motion mim- icking to imitate cyclic motions like walking and running, where a single cycle of the motion is provided as the reference motion and the character learns to repeat the motion for 20 seconds. Typ- ically, a curriculum is designed to gradually increase the maximum rollout episode to 20 seconds during training (Yang & Yin 202T Peng al. 2018a). In experiment, remove all bells and whistles and fix the maximum rollout episode to 4 seconds during the training. DiffMimic can produce 20-second long cyclic rollout even though it only sees 4-second rollouts in training. The motion clips are directly borrowed from AMP (Peng et al.) (202I). which are originally collected from combination of public mocap libraries, custom recorded mocap clips, and artist-authored keyframe animations.
Cyclic Motions.
4.2 COMPARISON ON MOTION MIMICKING
In this section, we aim to understand 1) efficiency; 2) the quality of the learned policy of DiffMimic. Following the conventions of previous works (Peng et al. [2018a] 2021 Ma al. (021). count the number of samples required to train the policy to rollout for 20 seconds without falling. The pose error is calculated over the horizon of 20 seconds.
Analytical gradients enhance the sample efficiency in motion mimicking. We show the com- parison between DiffMimic, DeepMimic, and Spacetime Bound sample efficiency in Table 2 DeepMimic is an RL-based algorithm with a careful reward design. Spacetime Bound performs hyperparameter searching for DeepMimic further enhance the sample efficiency. Our results

---

Published as a conference paper at ICLR 2023
Table 1: Pose error comparison in meters. Teyele is the length of the reference motion for a single
cycle. The error is averaged on 32 rollout episodes with maximum length of 20 seconds.
Motion
Tyyck(S)
DeepMimic
AMP
Ours
Back-Flip
1.75
0.076 +: 0.021
0.150
0.028
0.097
0.001
Cartwheel
2.72
0.039 = 0.011
0.067
0.014
0.040 0.007
Crawl
2.93
0.044 + 0.001
0.049 + 0.007
0.037 0.001
Dance
1.62
0.038 0.001
0.055
0.070 0.003
Jog
0.83
0.029 0.001
0.056 =.000
0.031 = 0.002
Jump
1.77
0.033 + 0.001
0.083 0.022
0.025 + 0.000
Roll
2.02
0.072 + 0.018
0.088 +: 0.008
0.061 + 0.007
Run
0.80
0.075 0.015
0.039 + 0.000
Side-Flip
2.44
0.191 0.043
0.124 - 0.012
0.069 + 0.001
Spin- Kick
1.28
0.042 0.001
0.058 + 0.012
0.056 + 0.000
Walk
1.30
0.018 0.005
0.030 + 0.001
0.017 0.000
Zombie
1.68
0.049
0.058 0.014
0.037 0.002
show that DiftMimic constantly outperforms DeepMimic in terms of sample efficiency. The alytical gradients provided by the differentiable simulation allow us to compute policy gradient with a small number of samples while the RL-based algorithm requires large batch to have decent estimate. Compared with Spacetime Bound, DiftMimic is much more stable and con- sistent over various tasks. We notice that Spacetime Bound may require more samples than DeepMimic even for simple tasks like Jump. We show the learning curve of DiffMimic over eight different tasks in Fig. 4 It shows that DiffMimic generally learns high-quality motions (pose error < 0.15) with less than 2e7 samples, even for challenging tasks like Backslip. In terms of the wallclock time, DiftMimic learns to perform Backflip in 10 minutes and learns to cycle it in 3 hours (14.88 10 samples), which can be found in qualitative results.
Sword and Shield
0.6
DiffMimic can learn high-quality motions. The aver- age pose error for 12 different motion tasks is shown in Table [1] DiffMimic outperforms AMP consistently and has comparable performance to DeepMimic. Noticeably, DiffMimic only needs to see the demonstrations of sec- onds to achieve a similar performance of DeepMimic with 20 seconds cyclic rollout, which indicates stable and faith- ful recovery of the reference motion. This further corrobo- rates the efficacy of DiffMimic.
DiffMimic
0.5
0.4
0.3
0.2
0.1
0.0
Wallclock Time (hour)
Analytical gradients accelerate the learning speed in motion mimicking.
Figure 5: Wallelock training time versus pose eтrоr for DiffMimic and AMP. DiffMimic takes half of the training time required by AMP to have comparable result.
generally takes more time to compute analytical gradients than the estimated one, it is natural to compare the wall-clock time for each algorithm. Since the implementation of DeepMimic does not utilize GPU for parallelization, we choose to compare DiftMimic with AMP. The implementation of AMP utilizes the highly parallelized environment, Isaac- Gym (Makoviychuk et al., 2021). Both approaches do not require manual reward design and utilize GPU for acceleration. We compare DiffMimic and AMP wielding sword and a shield to perform a 3.2-second attack move (shown in Appendix Fig. [3). The hyperparameter of AMP is from its official release of code. We show the comparison with spect to wallclock time in Fig. DiffMimic takes half of the training time required by AMP to have a comparable result.
task where humanoid character is
4.3 ABLATION ON TRUNCATION LENGTHI
Training the policy with DPS would suffer from the vanishing/exploding gradients and local opti- mal. Recent research (Xu et al. 2022) points out that such a problem can be mitigated by truncated learning window, which splits the entire trajectory into segments with shorter horizons. we carried experiments to validate whether such an idea be directly applied in motion mimicking with
7

---

Published as a conference paper at ICLR 2023
Number of samples required to roll out 20 seconds without falling (10%). Percentage: change in the fraction of the DeepMimic samples.
Table 2:
Motion
Teyclo(S)
DeepMimic
Spacetime Bound
Ours
Back-Flip
1.75
31.18
41.20 +32.1 132.1%
14.88 -52.2%
Cartwheel
2.72
30.45
17.35 -43.0%
13.92 -54.2%
Walk
1.25
23.80
4.08 -9.55%
7.92 -66.7%
Run
0.80
19.31
4.11 -78.7%
8.16 -57.7%
Jump
1.77
25.65
41.63 477.8%
5.28 -79,4%
Dance
1.62
24.59
10.00 -59.3%
16.56 -32.6%
(a) Reference motion.
(b) Full Horizon Gradient.
(c) Demonstration Replay (Random).
(d) Demonstration Replay (Threshold).
Figure 6: Qualitative results of three variants of DiffMimic with respect to demonstration replay. The policy trained with full horizon gradient fails to jump up. The policy trained with Demonstration Replay (Random) fails to recover the reference motion faithfully.
DPS, Truncation length refers to the horizon over which gradient is calculated. The quantitative results are shown in Fig. 7 Indeed, it is difficult to learn a good policy by propagating the gradient through a whole trajectory that is long. In addition, simply splitting the whole trajectory into seg- ments would worsen the final performance. We hypothesize the naive truncation of the trajectory creates discontinuities in the whole trajectory whereas motions in the trajectory are highly interde- pendent. For example, how flip in mid-air closely relates how the character jumps. We this as strong call for a better strategy to handle these two challenges motion mimicking with DPS.
4.4 ABLATION ON DEMONSTRATION REPLAY
To understand how demonstration replay affects the policy learning of DiftMimic, we compare three different variants of MiffMimie, namely, Demonstration Replay (Random) and Demonstration Replay (Threshold), and Full Horizon Gradient on Backflip and Cartwheel. Demonstration Replay (Random) randomly replaces
the rollout the policy with the demonstration state the same timestamp similar to the teacher forcing (Williams & Zipser 1989). Demonstration Replay (Threshold) inserts the demonstration state based on the pose error. If the pose error between the demonstration state and the simulated state exceeds a threshold c, the simulated state will be replaced by the demonstration state. The Full Horizon Gradient variant backpropagates the gradient through the full horizon without any additional operation. We show the quantitative result in Fig. 7 and the qualitative result in Fig. 6
Demonstration replay helps stabilize policy learning and leads to better performance. Com- pared with the Full Horizon Gradient without demonstration replay, the learning curve with demon- stration replay is much smoother and finally converges to lower pose error as shown in Fig, 7 (c)-(d). We show in Fig. 6 (b), this vanilla version DiffMimic can easily get stuck into local minimum. Instead of learning to backflip, the policy learns to bend down and use arms support itself instead of jumping up. The other two variants, by contrast, both learn to jump and backflip in mid-air successfully.
8

---

Published as a conference paper at ICLR 2023
BackfTpp
Backfip
Backfiip
0.6
0.6
truncio
0.01
0.1
0.5
0.5
0.5
Y- 0.1
rull honizo
0.05
0.3
0.3
0.3
M
0.2
0.2
0.2
0.0
0.12 6.14 ale
0.0
0.0
0.000
0.140.16
Sumples (1w8)
Samples [1+8]
Samples (1w6)
y= 0.01
zon
0.5
y- 0.05
0.5
< 0.2
0.4
0.4
y- 0.1
0.4
< 0.4
herizon
Mew www
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.00 0.04 0.6e 0.10 0.32 a.6
0.02 0.88 0.10 0.14
0.06 0.08 Samples (3e8)
Samptes (1c8)
Samples 11c8)
(b)
Figure 7: (a)-(b) Comparison between Full Horizon Gradient and truncation length of 10. (c)-(d) Comparison between Demonstration Replay (Random) and Full Horizon Gradient. (c)-(f) Compar- ison between Demonstration Replay (Random) and Demonstration Replay (Threshold). The blue dotted line denotes 10 minutes in the corresponding wallclock time.
Backflip
Policy-aware demonstration replay leads a more
0.35
full horizon
faithful recovery of demonstration. We compare Demon- stration Replay (Threshold) with Demonstration Replay
0.25
0 05
(Random) in Fig. 7 (e)-(f) Quantitatively, both variants
yield similar results the hyperparameter is properly tuned.
0.15
However, the behavior of policies trained with these two strategies can be significantly different. In (c) and (d), though both policies learn to backflip successfully, the pol- icy trained with Demonstration Replay (Random) fails to recover the motion frame by frame. We show in Fig- the per-frame pose егror. Even though the average error over the whole trajectory for both variants is similar, Demonstra- tion Replay (Threshold) gives lower maximum per-frame error, which indicates a faithful recovery of the demonstra- tion. This implies that simply minimizing the pose error may not suffice learn policy that tracks the demonstra- tion closely. Finer-grained guidance based on the current performance of the policy required.
0.05
0.00
20 Frame Index
Figure 8: L2 error per frame in the rollout of the final policy. Although Demonstration Replay (Random) duces the overall pose eгror com- pared to full-horizon optimization, the per-frame loss remains large in cer- tain time steps due to the lack of constraint. Demonstration Replay (Threshold) alleviates the issue.
5 CONCLUSION & FUTURE WORKS
In summary, we present DiffMimic utilizes diffcrentiable physics simulators (DPS) for motion mim- icking and outperforms several commonly used RL-based methods in various tasks with high ac- curacy, stability, and efficiency. We believe DiffMimic can serve as a starting point for motion mimicking with DPS and that simulation environments provide exciting opportunities for future research.
While there are many exciting directions explore, there are also various challenges. One such challenge is to enable motion mimicking within minutes given any arbitrary demonstration. With differentiable physics, this would greatly accelerate downstream tasks. In our work, we demonstrate that DiffMimmc can learn challenging motions in just 10 minutes. However, the tasks we evaluated were relatively short and did not involve any interactions with other objects. As the dynamic system becomes more complex with multiple objects, we leave these challenges for future work.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENTS
We have carefully reviewed and adhered to the ICLR Code of Ethics. DifIMimic conducts experi- ments exclusively on humanoid characters, and no experiments involve human subjects. Addition- ally, all experiments are performed using open-sourced materials and are properly cited.
REPRODUCIBILITY STATEMENTS
Our code
is easily accessible and openly available for review at
https://githu..com/diii www.///mmmmmmmi//iiiiiinii Additionally, the reported results can be easily reproduced using the provided code.
ACKNOWLEDGMENT
This research supported by the National Research Foundation, Singapore under its Al Singa- pore Programme (AISG Award No: MISG2-PhD-2021-08-011
and NTU NAP. MOE AcRF Tier (T2EP20221-0012), and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. Drecon: data-driven responsive control of physics-based characters. ACM Transactions On Graphics (TOG), 2019.
Siwei Chen, Xiao Ma, and Zhongwen Xu. Imitation learning via differentiable physics. arXiv preprint arXiv:2206. 04873, 2022.
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis Wyffels. differentiable physics en- gine for deep learning in robotics. Frontiers in Neurorobotics, 2017.
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physies engine for large scale rigid body simulation. arXiv preprint 1IX1:22006.13281, 2021.
Levi Fussell, Kevin Bergamin, and Daniel Holden. Supertrack: Motion tracking for physically simulated characters using supervised learning. ACM Transactions on Graphics (TOG), 2021.
Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. Comic: Com- plementary task learning & mimicry for reusable skills. In International Conference on Machine Learning, 2020.
Scokpyo Hong, Daseong Han, Kyungmin Cho, Joseph S Shin, and Junyong Noh. Physics- based full- body soccer motion control for dribbling and shooting. ACM Transactions on Graphics (TOG), 2019.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simula- tor for soft robotics. Proceedings of IEEE International Conference Robotics and Automation (ICRA), 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: soft-body manipulation benchmark with differentiable physics. In Interna- fional Conference Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2023
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo. Vikram Voleti, Linda Petrini, Mar- tin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable simulation for system identification and visuomotor control. International Conference on Learning Representa- fions (ICLR), 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6988 2014.
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG), 2019.
Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojciech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Trans. Graph., 2022.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning, 2020.
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B. Tenenbaum, David Held, and Chuang Gan. Diff- skill: Skill abstraction from differentiable physics for deformable object manipulations with tools. 2022.
Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. Carl: Con- trollable agent with reinforcement learning for quadruped locomotion. ACM Transactions on Graphics (TOG), 2020.
Li-Ke Ma, Zeshi Yang, Xin Tong, Baining Guo, and KangKang Yin. Learning and exploring motor skills with spacetime bounds. In Computer Graphics Forum, 2021.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nico- las Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. arXiv preprint 1rXiv:1811.000 2018.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev and Stelian Coros. Pods: Policy optimization via differentiable simulation. In Proceedings of the 38th International Conference Machine Learning, 2021.
Michael C. Mozer. focused backpropagation algorithm for temporal pattern recognition. Complex Syst., 1989.
Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. Learning predict-and- simulate policies from unorganized human motion data. ACM Transactions on Graphics (TOG), 2019.
Xue Bin Peng. Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforeement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 2018a
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Re- inforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG), 2018b.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mep: Learning composable hierarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 2019.
11

---

Published as a conference paper at ICLR 2023
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial mo- tion priors for stylized physics- based character control. ACM Transactions on Graphics (TOG), 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large- scale reusable adversarial skill embeddings for physically simulated characters. arXiv preprint DrXiv:2205501906. 2022.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi-body systems. In Advances in Neural Information Processing Systems, 2021.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.0594// 2017.
Reallusion. 3d animation and 2d cartoons made simple, 2022.
Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 1978.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint TrXiv:1707.06344 2017.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In Proceedings of the 39th International Conference on Machine Learning, 2022.
Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, and Matthew Hausknecht. Mocapact: A multi-task dataset for simulated humanoid control. arXiv preprint 2tt.:22208.036633 2022.
Ronald J Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural computation, 1989.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach control diverse behaviors for physically simulated characters. ACM Transactions on Graphics (TOG). 2020.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 2022.
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:2204.077137. 2022.
Zeshi Yang and Zhiqi Yin. Efficient hyperparameter optimization for physics-based character ani- mation. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2021.
Yaofeng Desmond Zhong, Jiequn Han, and Georgia Olympia Brikis. Differentiable physies simula- tions with contacts: Do they have correct gradients w.r.t. position, velocity and control? In ICMI. 2022 2nd Al for Science Workshop, 2022.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2019.
12

---

Published as a conference paper at ICLR 2023
CONTENTS
Introduction
Related Work
3
Methodology
3.1
A Motion Mimicking Environment in Differentiable Physics Engine.
3.2
Motion Mimicking with Differentiable Physics
3.3
Demonstration Replay
Experiments
Experiment Setup
4.2
Comparison Motion Mimicking
4.3
Ablation on Truncation length
Ablation Demonstration Replay
8
5
Conclusion & Future works
Qualitative Results
13
Policy Network
14
Hyper-parameters
14
State and Action Space
14
Learning Curves
15
New Character
16
New Skill
16
H New Training Strategy
16
Analysis on Robustness
17
J Analysis on Sensitivity
18
Analysis on Motion Quality
18
Swording Character
18
A QUALITATIVE RESULTS
We present our qualitative results in the following website: 77h0i8.strammiiapp.comm
https://diffmimic demo-main
13

---

Published as a conference paper at ICLR 2023
It contains the following qualitative results:
Back-Flip 20-second rollout.
Cartwheel 20-second rollout.
Crawl 20-second rollout.
Dance 20-second rollout.
Jog 20-second rollout.
Jump 20-second rollout.
Roll 20-second rollout.
Side-Flip 20-second rollout.
Spin-Kich Kick 20-second rollout.
Walk 20-second rollout.
Zombie 20-second rollout.
Back-Flip 20-second rollout, trained for 2.5 hours.
Back-Flip single cycle rollout, trained for 10 minutes.
B POLICY NETWORK
We use an MLP for the policy network, The network has two hidden layers with sizes 512 and 256. We Swish (Ramachandran al. 2017) for the activation function.
C EYPER-PARAMETE
We Adam optimizer (Kingma & Ba 2014) in training, with learning of 3e-4 that linearly decreases with training iterations. We apply gradient clipping and set the maximum gradient norm 0.3. We the batch size the same as the number of parallel environments.
Cyclic motions. We set the maximum iterations to 5000. The number of environments is set to 200. Acyelic motions. We set the maximum iterations to 1000. The number of environments is set 300.
Expert Replay, We run experiments with two error thresholds, 0.2 and 0.4, and report the better of the two.
D STATE AND ACTION SPACE
Humanoid. We use 6d rotation representation (Zhou et al. 2019) in the state feature. The humanoid has an action size of 28 and state feature size of 193.
Humanoid with Sword and Shield. We set up the Sword and Shield character following ASE (Peng al. 2022). Compared the default humanoid, the new character has additional DOF on the right hand. A sword is attached to its right hand and shield is attached to its left lower arm. The character has an action size of 31. The state feature size 208.
14

---

Published as a conference paper at ICLR 2023
E LEARNING CURVES
Humanoid (Cartwheel)
Humanold (Side (Side.Filp)
DifMimic (206)
DetMMimic (05)
0.8
0.0
mm
DeepMime
1000
0.4
0.4-
0.2
0.2
0.65
0.6
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.6
0.0
10
Samp
160
Samplis
168
Samples
1cB
Humanold (Zombic)
Humanoid (Run)
Humanoid (Walk)
1.0
0.6
0.8
0.4
0.2
0.2
0.0
0.0
0.0
Humanoid (ummp)
Humanoid (Spin-Kic)
Humanold
DilMiimic (20s)
DifMimic (4s)
DezaMimic
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
Samates
1eft
Samoles
165
Sampltss
16B
Humanoid (Crawl)
Humanoid (Roll)
Humanoid (Dance)
2.0
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.0
0.2
0.0
0.0
Figure 9: Pose error versus the number of samples. DiffMimic (4s): rollout 4 seconds of Diffmimic for evaluation. DiffMimic (20s): rollout 20 seconds of Diffmimic for evaluation.
15

---

Published as a conference paper at ICLR 2023
F NEW CHARACTER
We include Ant as new character. We use the trajectory in ILD CChen et al. 2022) as the reference motion, which is obtained from PPO (Schulman al. 2017) agent that learns fast as possible. We achieve a 0.059 meter final pose error. We show the training curve in Figure T0) and the qualitative result on the demo website under the tab "New Character".
forward as
Ant
0.5
0.4
0.3
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Samples (1e8)
Figure 10: Pose error versus the number of samples for the new character, Ant.
G NEW SKILL
We run DiftMimic
skill, 360-degree Jump, from the CMU Mocap Dataset (motion
quence CMU-075_09) The skill has been reported to be challenging for a RL-based method CoMic (Hasenclever al., (2200) in a recent dataset paper (Wagener al., (022). DiftMimic is able to spin in the air and achieve 0.020 meter final pose error. We show the training curve in Figure and the qualitative result on the demo website under the tab "New Skill".
360 360-degree Jump
0.5
0.4
0.3
myy
0.2
0.1
8.00
0.02
0.04
0.06
0.08
0.10
0.12 0.14
Samples (1e8)
Figure 11: Pose error versus the number of samples for the new skill, 360-degree Jump.
H NEW TRAINING STRATEGY
We additionally include the Reference State Initialization (RST) technique (Peng al.) 2018a) into the DiffMimic training. RSI starts the training rollout from a random reference frame instead of the first reference frame. We observe a substantial empirical improvement after employing the new training strategy. The samples required are consistently around 90% less than DeepMimic (Peng et al. 2018a) as shown in Tab. 3 The final pose error has also been slightly improved as shown in Tab. We show the qualitative results on the demo website under the tab "New Training". Notably, using RSI alone or RSI+Early Termination (ET) (Peng et al. 2018a) does not converge as shown in
Figure T2
16

---

Published as a conference paper at ICLR 2023
Table Number of samples required roll out 20 seconds without falling in (10%) with Reference State Initialization (RSI) (Peng al. 2018a). Percentage: change in the fraction of the DeepMimic samples.
Motion
(s)
DeepMimic
Spacetime Bound
Ours
Ours w/ RSI
Back-Flip
1.75
31.18
41.20 +32.1%
14.88 -52.2%
3.82 -87.7%
Cartwheel
2.72
30.45
17.35 -43.0%
13.92 -54.2%
4.72 -84.5%
Walk
1.25
23.80
4.08 -9.55%
7.92 -667%%
1.55 -9.5%
Run
0.80
19.31
4.11 -78.7%
8.16 -57.7%
1.41 -92.7%
Jump
1.77
25.65
41.63 +77.8%
5.28 -79.4%
2.12 -91.%%
Dance
1.62
24.59
10.00 -59.3%
16.56 -32.6%
2.19 -91.1%
Table 4: Pose comparison in meters. Teyele is the length of the reference motion for single cycle. The error is averaged on 32 rollout episodes with maximum length of 20 seconds. The result of Ours+RSI is obtained without DTW.
Motion
(8)
DeepMimic
AMP
Ours
w/o DTW
Ours
Back-Flip
1.75
0.076 + 0.021
0.150 = 0.028
0.097 + 0.001
0.105 = 0.022
0,058 0.015
Cartwheel
2.72
0.039 + 0.011
0.067 = 0.014
0.040 + 0.007
0.040 + 0.007
0,027 0.002
Crawl
2,93
0.044 t 0.001
0.049 + 0.007
0.037 + 0.001
0.037 + 0.001
0,035 0.003
Dance
1.62
0.038 + 0.001
0.055 = 0.015
0.070 + 0.003
0.072 = 0.014
0.042 0.001
Jog
0.83
0.029 1 0.001
0.056 1 0.001
0.031 1 0.002
0.031 1 0.002
0.012.1 0.000
Jump
1.77
0.033 + 0.001
0.083 + 0.022
0.025 0.000
0.031 + 0.002
0.015 0.000
Roll
2.02
0.072 +
0.088 0.008
0.061 0.007
0.083 0.001
0.046 0.005
Run
0.80
0.028 0.002
0.075 0.015
0.039 0.000
0.046 0.000
0.019 0.000
Side-Flip
2.44
0.191 1 0.043
0.124 0.012
0.069 0.001
0.121 1 0.009
0.035 0.001
Spin- Kick
1.28
0.042 + 0.001
0.058 = 0.012
0.056 0.000
0.056 = 0.000
0.036 0.000
Walk
1.30
0.018 = 0.005
0.030 = 0.001
0.017 + 0.000
0.017 = 0.000
0.012 0.000
Zombie
1.68
0.049 T 0.013
0.058 1 0.014
0.037 T 0.002
0.037 1 0.002
0.033.L 0.009
I ANALYSIS ON ROBUSTNESS
We further evaluate DiftMimic's robustness to external forces. A force is applied to the pelvis of the character halfway through motion cycle for 0.2 seconds, and measure the maximum forwards and sideways push each policy can tolerate before falling. DiffMimic achieves comparable robustness to DeepMimic. We show the results in Table S and the qualitative results the demo website under the tab "Robustness".
0.6
Diti4mmic R81)
0.6
RSI
0.6
RSS+ET
0.3
0.3
0.5
0.1
0.0
8.00
0.2
0.64
0.08 0.10 Samples (1e6)
0.06
0.2
0.4
0g
0.32 0.94 0.06 0.08 0.10
012
0.4
08.00
0.02 0.04
0.06
0.08
0.10 012 0.14
Samples (1v6)
Samples (1:8)
(a)
(b)
(c)
Figure 12: Pose emor versus the number samples on Back-Flip using (a) DifMMimic+RII (b) standalone RSI (c) RSI+ET.
17

---

Published as a conference paper at ICLR 2023
Table 5: Maximum forwards and sideways push each policy can tolerate before falling. Each push is applied halfway through a motion cycle to the character's pelvis for 0.2s. F: Forwards push. S: Sideways push. The unit is in Newton.
Motion
DeepMimic (F)
Ours (F)
DeepMimic (S)
Ours (S)
Back-Flip
440
750
100
310
Cartwheel
200
350
470
690
Run
720
500
300
270
Spin-Kick
690
630
600
730
Walk
240
250
300
220
J ANALYSIS ON SENSITIVITY
DiffMimic with inaccurate estimates of physical parameters. Briefly speaking. DifflMmmic is able to learn the control policy well even though the friction coefficient deviates from the original param- eters. We show the results in Table 6 and the qualitative results on the demo website under the tab "Sensitivity"..
Table 6: Sensitivity to the friction coefficient μ., a system parameter. The evaluation is conducted on Zombie Walk, motion heavily impacted by the friction coefficient. μ 1.0 is the standard setting. we additionally evaluate when 0.8 and = 1.2.
Friction
Pose Error (m)
m=0.8
0.047 0.009
1=1.0
0.032 0.009
1=1.22
0.028 0.007
K ANALYSIS ON MOTION QUALITY
Motion quality can be severely affected by small number of large poses, which be reflected by the existing average pose error metric. We propose a Pose Absurdity metric to quantify large-error poses. L2@0001, 0.05, and 0.1 measure the average L2 pose error on the worst 1%, 5%, and 10% frames respectively. Full Horizon Gradient, Demonstration Replay (Random) ratio X, and Demonstration Replay (Threshold) threshold y are studied. Demonstration Replay (Threshold) outperforms other baselines by large margin. We show the results in Table
L SWORDING CHARACTER
Figure 13: Character performing attack move with sword and a shield. The reference motion clip from Reallusion (Reallusion) 2022)
18

---

Published as a conference paper at ICLR 2023
Table 7: Ablation experiment on Full Horizon Gradient, Demonstration Replay (Random) ratio A, and Demonstration Replay (Threshold) threshold with the new Pose Absurdity metric on Back- Flip. L2@0.01, 0.05 and 0.1 measure the average L2 pose error on the worst 1,, 5% and 10% frames respectively. Demonstration Replay (Threshold) significantly improves on the Pose Absur- dity metric.
Method
L2@0.01
L2@0.05
L2@0.1
full horizon
0.357
0.343
0.323
0.01
0.379
0.341
0.318
A = 0.05
0.195
0.177
0.135
0.10
0.359
0.332
0.262
? = 0.1
0.111
0.107
0.095
? 0.2
0.113
0.107
0.093
7 = 0.4
0.110
0.103
0.095
19


Reviews:
[{"review_id": "xR1QcPe8iu", "reviewer": "Reviewer_JNig", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposed a method for motion mimicking using differentiable physics. This manuscript used Brax as the backbone differentiable physics simulator and proposed to use demonstration to enhance the training. This manuscript showed the comparison with several baselines including DeepMimic and AMP, which are well-known as the established methods in this field. These experiments showed better sample efficiency compared to the gradient-free methods. Additionally, this manuscript examined the proposed method by a set of ablation studies of truncation length and demonstration replay.", "summary_of_the_review": "This paper proposed a sensible method for solving motion mimicking using differentiable physics. Their results are backed by extensive experiments on a complex robot. My main complain is on the technical novelty side. However, I think the results can contribute to the community.", "strength_and_weaknesses": "Strength:\n- This manuscript described a feasible pipeline for motion mimicking. From the perspective of differentiable, this is a good task and should be a correct practice of using differentiable physics.\n- The proposed Demonstration Replay looks \n- The authors presented extensive experiments with complex motions.\n\nWeaknesses:\n- My major reservation is about technical novelty. \"Demonstration\" is a developed method for differentiable physics and has shown the power of manipulation and locomotion tasks. The proposed demonstration replay, although effective on the shown tasks, has a marginal gain over the random baseline.\n- Training a controller using differentiable physics can be considered model-based reinforcement learning with a perfect world model. I would expect a model-based RL baseline to demonstrate the claimed \"better sample efficiency\".\n- I admit Humanoid is a complex robot to work on. It will be great to see the proposed method applied to more robots, e.g., ants or cheetahs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pGn88oRvyta", "reviewer": "Reviewer_8KfC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for learning human locomotion policies from human motion data using an off-the-shelf differentiable physics simulator (DPS). Specifically, the method uses the Brax simulator (Freeman et al., 2021) to enable gradient computation of the forward dynamics. A simple joint and angle-based motion matching loss is used to supervise the training. To further improve the stability of training, the paper proposes a technique called demonstration replay, which resets the rollout state to the reference state when it deviates too much. Experiments on human locomotions and acrobatics show the method converges faster than RL-based methods while achieving similar motion quality.", "summary_of_the_review": "The paper shows improved efficiency of motion imitation with a differentiable physics simulator. However, the motion quality improvement is not huge, and the paper didn’t enable learning new motions that RL-based methods can’t. A main concern is also the paper’s lack of novelty. Therefore, I lean towards reject at this point.\n\n--- update ---\nAfter the rebuttal, I'm slightly more positive about this paper. I don't mind seeing the paper accepted. I still think the motion quality needs to be improved (e.g., obvious foot sliding). It is possible the lower quality is due to DPS's approximation of contact to enable differentiability, and it could be an inherent limitation of using DPS.", "strength_and_weaknesses": "**Strength:**\n\n- The method achieves faster convergence by using a differentiable physics simulator compared to RL-based approaches.\n- The paper is generally well-written and easy to follow.\n\n**Weakness:**\n\n- The novelty of the paper is limited. The idea of using a differentiable physics simulator to model human motions has been explored in prior work [1, 2] and in more complex settings, human pose estimation, navigation, path following, etc. There is no new method proposed in this paper besides using a differentiable physics simulator. The demonstration replay seems like an incremental technical contribution, and its benefit is not very significant given the mean pose error stays the same thus traditional approaches perform similarly over the whole motion.\n- From Table 1, we can see that DeepMimic actually outperforms the proposed method in at least 6 motions (Back-Flip, Dance, Run, etc.) where in some cases DeepMimic actually beats the method a lot (0.038 vs 0.070 for Dance), so this method overall has only slightly better performance than an RL-based approach like DeepMimic. Even though the proposed method learns faster due to the use of simulation, achieving better motion quality consistently in the end is more important for motion imitation.\n- From the visualization on the website, the “jog” motion produced by this method seems to be rather unnatural with the character limping, this does not happen for DeepMimic or AMP. There is also some foot sliding in the “roll” motion. Are the artifacts caused by the use of a differentiable physics simulator, which might not enforce the contact constraints as well?\n\n[1] Gärtner, Erik, et al. \"Differentiable Dynamics for Articulated 3d Human Motion Reconstruction.\" CVPR. 2022.\n\n[2] Won, Jungdam, Deepak Gopinath, and Jessica Hodgins. \"Physics-based character controllers using conditional VAEs.\" SIGGRAPH 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T7_SxU4mPq3", "reviewer": "Reviewer_s4rT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A differentiable physics simulator (DPS, Brax in this case), is used to learn an RL policy that\nimitates a given reference motion, as done in DeepMimic. The contributions include (a) better sample\nefficiency, and therefore faster learning, as compared to DeepMimic; (b) the use of \"demonstration\nreplay\" when the motion deviates too far from the reference motions.  This is a type of reset that\nis critical to provide stable gradients and to avoid local minima.\n", "summary_of_the_review": "I see the value of this paper in terms of the pragmatic details that are needed to overcome the basic issues of exploiting differentiable simulators to directly learn policies. Getting these details right is important, as we also see from [Xu et al 2022].  However, the discussion could really be improved. \n\nThe paper claims that DeepMimic rewards are heavily engineered. I believe that the \nDeepMimic imitation reward is quite similar to the loss function used here. \n\nDeepMimic introduces both early termination (ET) and reference state initialization (RSI) \nprecisely to escape local minima of the type described here.  The idea of using a deviation threshold\nto dynamically reset back to the trajectory could also be seen as a combination of ET and RSI.\n\nSection 2, first para, missing bibtex reference e.g., \"?]\"\n\nThe existing work that makes use of differentiable simulators is mentioned, but then more-or-less ignored. \nI'm not clear why. E.g.,   [Fussell et al 2021, Xu et al 2022, Mora et al 2021].\nThe following work also takes advantage of transfer learning when learning to imitate a wide range of motions,\nand to therefore to immediately generalize to a large percentage of new motions:\n\"A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters\"\nIn the long term, this may be more useful than speeding up the single-motion case.\n\nFig 4 shows 3 runs for some methods and only one run for DiffMimic.\nDiffMimic presumably not deterministic, i.e., it still uses a stochastic action policy, \nto help enable encourage robustness. Or is that incorrect?\n\ntext below Figure 4:\n\"Dynamic Time Warping is applied to sync the simulated motion and the reference motion following the convention\"\nWhich convention?  I don't believe that DeepMimic uses DTW.\n\nTable 2:\nThe results might be better stated as a fraction of the DeepMimic time.\n\nHow does the robustness of the learned policies compare to those learned with DeepMimic?\nIs the exploration noise comparable?\n", "strength_and_weaknesses": "Strengths: \n- speeds up learning for an important class of problem (physics-based motion imitation)\n- multiple useful practical insights on window length and resets, i.e., (demonstration replay) that are needed\n  to get DPS-based methods working with complex human motions. This is the first time that I have\n  seen Brax successfully used for this kind of scenario, although [Xu et al 2022] demonstrate results\n  of similar complexity with a simulator of their own.\n\nWeaknesses:\n- mentions of closely related work, but with no deeper discussion on comparing the approaches, \n  or direct experimental comparisons \n  [Fussell et al 2021, Xu et al 2022, Mora et al 2021]\n  as well as \"A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters\"\n  There are differences of the current work with respect to all of these, but much of the value of the paper\n  lies in discussing these differences. I.e., Xu et al 2022 come up with a different solution to the\n  basic challenges, i.e., exploding gradients, local minima.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QrEFcDfBJny", "reviewer": "Reviewer_yUu6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a differentiable physics based framework to solve motion mimicking problems without reinforcement learning. The authors show that their method is much more efficient than reinforcement learning based methods. To solve the local minimum issue caused by the long horizon, the authors proposed an adaptive teacher-forcing mechanism called Demonstration Replay, which adaptively separates the whole trajectory into several sub-trajectories with the ground truth states as the input to aid the training. ", "summary_of_the_review": "This paper proposes a differentiable physics-based framework for motion mocking tasks. The results show that the diff-physics based method is more efficient than RL-based methods. However, replacing the state transitions in RL frameworks with differentiable physics seems pretty straightforward. It would be more impressive to show efforts to decrease the sim-to-real transfer gap using differentiable simulators.", "strength_and_weaknesses": "**Strengths:**\n\nThe evaluation of the proposed method is thorough. The authors evaluated their method on vast types of tasks and compared with several baselines.\n\n**Weakness:**\n\n- How is the data generated?  Is it generated by simulations or motion capture systems?\n- How are the parameters of rigid body dynamics such as friction coefficients/restitution coefficients generated? I would also like to see whether the proposed method is sensitive to physical parameters, i.e., when the estimations of these parameters are not perfect, will the method still gives reasonable results?\n- Please also see the evaluation of novelty below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "503ZAoEnKt1", "reviewer": "Reviewer_a2aF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm for motion mimicking using differentiable physics simulators. With DPS and its gradients, the authors convert a policy learning problem to a much easier state matching problem. Instead of running an entire simulation during training, this paper proposes a Demonstration Replay mechanism to mitigate error accumulation. The Demonstration Replay replaces a simulation frame with the ground truth demonstration if its error is beyond a threshold. \n\nThe authors show that the proposed DiffMimic outperforms previous RL-based motion mimicking methods in terms of sample efficiency.\n", "summary_of_the_review": "This paper provides a new way to solve motion mimicking problems using differentiable physics simulations. The method looks concise and intuitive. The results are also good. It could be better if the paper presentation can be improved and more comparisons can be added. In summary, I think this paper looks fine but can be made more solid.\n", "strength_and_weaknesses": "Strengths\n\nI think the biggest strength of this paper is that it presents a new direction toward motion mimicking problems. Compared to the previous reinforcement learning methods, the gradients from DPS can provide more information to the learning process and thus improve the sample efficiency. \n\nAnother strength is that the proposed approach can achieve reasonably good results (as shown in the supplementary video) for rather complex and dynamic motions. This method can also use constantly lower sampling numbers during the training than Deepmimic.\n\nWeaknesses\n\nOne of my concerns is the generalizability of this method. Since the dynamics and implementation details of the DPS might not be exactly the same as the original demonstration, it is not fully investigated how would the learned policy perform in the original simulation environment. \n\nMoreover, the authors mentioned relaxation in their DPS environment, “The joint limits of the character are relaxed to allow smoother gradient propagation.” I wonder whether the measurement is performed in the same environment as the other comparison methods. How much difference this modification would bring to the learning task?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster