Bi-Level Motion Imitation for Humanoid Robots
Authors: Wenshuai Zhao
Yi Zhao
Joni Pajarinen
Michael Muehlebach
Venue: CoRL 2024 (2024)
Keywords: Humanoid Robots
Imitation Learning
Latent Dynamics Model
TL;DR: We propose a bi-level motion imitation framework based on a novel self-consistent latent dynamics model to improve humanoid learning from MoCap data

Abstract: Imitation learning from human motion capture (MoCap) data provides a promising way to train humanoid robots. However, due to differences in morphology, such as varying degrees of joint freedom and force limits, exact replication of human behaviors may not be feasible for humanoid robots. Consequently, incorporating physically infeasible MoCap data in training datasets can adversely affect the performance of the robot policy. To address this issue, we propose a bi-level optimization-based imitation learning framework that alternates between optimizing both the robot policy and the target MoCap data. Specifically, we first develop a generative latent dynamics model using a novel self-consistent auto-encoder, which learns sparse and structured motion representations while capturing desired motion patterns in the dataset. The dynamics model is then utilized to generate reference motions while the latent representation regularizes the bi-level motion imitation process. Simulations conducted with a realistic model of a humanoid robot demonstrate that our method enhances the robot policy by modifying reference motions to be physically consistent.

Bi-Level Motion Imitation for Humanoid Robots
Wenshuai Zhao', Yi Zhao', Joni Pajarinen', Michael Muchlebach
Aalto University, Finland
2 Max Planck Institute for Intelligent Systems, Germany
A Abstract: Imitation learning from human motion capture (MoCap) data provides a promising way to train humanoid robots. However, due to differences in mor- phology, such as varying degrees of joint freedom and force limits, exact replica- tion of human behaviors may not be feasible for humanoid robots. Consequently, incorporating physically infeasible MoCap data in training datasets can adversely affect the performance of the robot policy, To address this issue, we propose a bi-level optimization aptimizattonn-based imitation learning framework that alternates between optimizing both the robot policy and the target MoCap data. Specifically, we first develop a generative latent dynamics model using novel self-consistent auto- encoder, which learns sparse and structured motion representations while captur- ing desired motion patterns in the dataset. The dynamics model then utilized to generate reference motions while the latent representation regularizes the bi- level motion imitation process. Simulations conducted with realistic model of a humanoid robot demonstrate that our method enhances the robot policy by modi- fying reference motions to be physically consistent'.
Keywords: Humanoid Robots, Imitation Learning, Latent Dynamics Model
Introduction
The use of human motion capture (MoCap) data as reference trajectories offers a promising way to design powerful humanoid robot controllers [1, 2, 3, 4J. After appropriate motion retargeting these close-expert reference trajectories can be directly imitated by robots, reducing the need for extensive reward engineering typically required in reinforcement learning [5, 3]. Existing motion
imitation works either learn the motion styles in a generative adversarial way [6. 2, 7, 3] or directly learn to track the provided motion trajectories [1, 8). While the former method, based on generative adversarial imitation learning (GAIL) [9], avoids the exact definition of similarity between reference motions and robot trajectories, its min- computational formulation usually suffers from unstable learning and sample inefficiency [10, 11]. The latter method, however, can also be problematic be- cause the reference motion is often noisy and physically infeasible for realistic humanoid robots due to embodiment differences such as different force and joint limits between humans and robots [4]. Consequently, including such data may degenerate the policy learning of the robot [4].
The aforementioned issues arising from noisy and physically infeasible reference motion have been mainly studied in the field of motion retargeting [12, 13, 14). For example, in order to create natural motions for various animated characters, researchers pursue retargeting the human MoCap motions into physically consistent motions of new characters, which in our case corresponds to humanoid robots. The common approach used in physics- based retargeting hinges on trajectory optimization with known dynamics the target robot and constraints that arise from the reference trajectories [ (14, 15]. However, the resulting optimization problem is often complex and includes specific domain knowledge. There is therefore an emergent need for learning- based based method that does not rely on an explicit dynamics model while guaranteeing physical consistency the same time. We address this need by proposing the Bi-Level Motion Imitation (BMI) framework.
Project website:
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Our method shares similar bi-level optimization idea with differential optimal control [15] but does not need a prior dynamics model and human-specified constraints. Specifically, BMI first learns a generative latent dynamics model based on a novel self-consistent generative auto-encoder (SCAE) from the reference motions. SCAE regularizes normal auto-encoder training with latent reconstruction error and captures the essential motion patterns with sparse and well-structured latent representations. This enables us sample latent parameters and reconstruct new motions, which are used to train the humanoid robot policy (pre-training step). After pre-training, BMI further finetunes both the decoder and the robot policy as bi-level optimization problem. In this way, the decoder learns to return reference motions that are physically consistent. At the same time, the robot further improves its policy by imitating updated reference motions. We constrain the decoder updates to ensure that the reconstructed motions stay close to the original motions in the latent space, which prevents the decoder from degenerating into trivial motions that are far from the desired motion patterns in the human MoCap data.
We evaluate BMI on the MIT Humanoid Robot [16] in simulation, where we imitate motions from human MoCap data. The experiments first show that the proposed SCAE-based latent dynamics model learns structured motion representations. In the subsequent pre-training, the improved latent representation learned by SCAE also enhances policy learning compared to the baseline latent dy- namics model. Finally, our bi-level fine-tuning with latent space regularization updates the decoder to construct reference motions that physically consistent for the robot and retain the original patterns at the same time. Our experiments show that the robot policy can be further improved by imitating the updated motions.
The key contributions of this paper can be summarized as follows: (i) We propose a self-consistent latent dynamics model that is able to learn sparse and structured representations for human motions. (ii) We propose a bi-level motion imitation framework update the decoder and the robot policy the same time, which enhances the generated motions with physical consistency and closeness the original human MoCap trajectories. (iii) We evaluate method humanoid robot and imitate up to 13 different motions with a single policy. The experiments highlight improved policy learning with the proposed latent dynamics model and bi-level motion imitation framework.
2 Related Work
We first discuss existing reference- based humanoid imitation learning methods. Methods addressing the problem of physically inconsistent reference motions are discussed subsequently.
Humanoid Motion Imitation Imitating from human MoCap data an efficient way for humanoid robots to learn agile and natural-looking skills [1]. Recent works [7, 17] based on generative ad- versarial imitation learning (GAIL) [9, 2] in animation have succeeded in training humanoid robots to track various human motions using large MoCap dataset such AMASS [18]. Nonetheless, the success may be partially attributed to the unrealistic humanoid robot that is used. With up to 69 DoFs, unlimited force, and even assistive external forces [19], the simulated robot is massively overactuated and can, in principle, perfectly track the given reference motions. is therefore un- clear whether the approaches in animation [7, 17] can be transferred to more realistic robots. As the reference motions can be physically infeasible for robots, including them in the training dataset result in sub-optimal mimicking behaviors even complete failure in imitation [13]. The au- thors from [20] train whole-body humanoid controllers that only replicate upper-body movements while the lower body is restricted to track a given forward velocity for the base. An alternation has been proposed in [4] where the infeasible motions are explicitly removed by privileged simulated imitator. Fourier Latent Dynamics (FLD) [8] employs a fallback mechanism to replace the given reference motions with default motions when the reference is far from the training motions.
Physically Consistent Motion Retargeting Motion retargeting describes the process of mapping the human MoCap data target robot configurations such that downstream motion imitation can be performed. While common motion retargeting methods [21, 13] such inverse kinematics-baut

---

methods can generate visually convincing motions, these motions could be physically infeasible for humanoid robots. In order to obtain physically consistent motion retargeting, existing methods are usually formulated as trajectory optimization problems constrained by robot dynamics [22, 12, 14, 15J. For instance, differential optimal control [15] alternatively optimizes the retargeting parameters with manually defined contact constraints and the robot trajectories based on the retargeting as a bi-level optimization problem. However, it is often tedious to model the complex robot dynamics and these methods are therefore hard to generalize across different robots. In contrast, our method is purely data-driven.
3 Preliminaries
Our method involves modifying a latent dynamics model, which maps the motions through an auto- encoder [23] into latent space and back, in order to generate motions for the robot that are physically consistent and at the same time close the desired motion patterns in the original MoCap dataset. However, measuring the closeness between the original trajectory and the generated physically- consistent reference motion for the robot, is challenging [24]. We address this problem by introduc- ing a structured motion representation and incentivizing closeness in the latent space. Our proposed latent dynamics model is inspired by FLD [8]. a structured motion representation method that ex- plicitly enforces the periodicity of motions in the latent space by transforming the learned latent representation into the frequency domain [25].
The structure of FLD is illustrated in Figure 7 in the appendix. We denote given trajectory segment of length H in d-dimensional state space by Ti ($--H+1. .t) RdxH where denotes time and the state at time t. The trajectory segment represents the input to the auto-encoder, where the encoder embeds the original motion trajectory into a latent space with e channels, denoted by 2 E R In order to explicitly account for the periodicity of the motions, FLD builds on earlier work on Periodic Autoencoders (PAEs) [25] and includes a differentiable Fast Fourier Transform (FFT) layer. The FFT layer returns the frequency ft, amplitude a,, and offset b, of the latent motion embeddings, while separate phase de is computed by an additional fully connected (FC) layer and an atan2 operation. This transformation is denoted p:
2t = enc(T:),
(oe, St, at, b.) = =p(z))
where di, ft, b, E Re and enc is the encoder. Particularly, FLD improves PAE with multi-step forward prediction to approximate the subsequent latent vectors by unrolling the latent phase. For local range of subsequent trajectory segments (Tt, T1+1, T++N), }, we assume that the segments share the same latent parameters fi, a, b, while differing only in their phases Φi+i- Furthermore, Φe+z can be approximated by Φeti R the + ifiA,, where A, denotes the time step. This results in,
E'4 p((= + ift ft; 0z, bc),
+ dec((/+)))
where dec is the decoder. We denote by P the embedding reconstruction process from the frequency domain,
& p(0r, fe, ae be) a.sin(2n(ftT + o.)) b.
where T represents known time window with H evenly spaced samples [25]. We note that 2445, 844s are different from &, & as they are approximated by the multi-step forward prediction from the trajectory Tr. This motivates the following loss function that used in FLD,
Σ
-
(4)
FLD
1=U
where a is decay factor and
denotes the Euclidean distance.
4 Method
The proposed method involves three-stage training procedure. (i) In the first stage, learn gen- erative latent dynamics model from the original MoCap data that has been kinematically retargeted
3

---

Trajactory segment
Convolution layers enc
Convolution layers enc
Lalent embedding Rt
Lalent embeddiin &
Lg
FC-BN-Aln2
AC+BN-Alan2
Lis La ... Lx
9:
Sinusoidal reconstruction
Sinuscidal reconstruction
Convolution layers dec
dec
Reconstructed segment 12
Figure Structure of the proposed self-consistent auto-encoder (SCAE). The encoder enc first encodes the original trajectory into latent space žt- Fourier transformation is then applied to to get latent parameters 0, (fr. de, bς) while separate MLP module learns Φt. A sinusoidal function reconstructs the latent embedding &, followed by the decoder dec recovering the input trajectory f. Particularly, we re-input F the encoder to obtain reconstructed latent embedding &. Therefore, SCAE consists of both motion and latent reconstruction losses, as indicated by red arrows. We follow FLD make multi-step predictions and thus the final loss sums Lo, .INN-
to the humanoid. We introduce a self-consistent auto-encoder trained using both reconstruction error and latent regularization, to capture the desired patterns embedded in the noisy kinematic motions more effectively. (ii) The second stage samples latent parameters encoded by the self-consistent dynamics model and then decodes these latent samples into the state space. The decoded states used the reference motions pre-train the robot policy. (iii) We perform bi-level imitation by fine-tuning the policy and updating the decoder at the same time. Crucially, this bi-level optimiza- tion is constrained within the latent space, ensuring that the decoder generates motions that closely adhere to physics- based robot trajectories while preserving the original motion patterns intended for imitation. The following paragraphs explain the three-step procedure in detail.
4.1 Self-Consistent Latent Dynamics
Although FLD learns structured latent representations and shows accurate reconstruction, find that the decoded motions with small reconstruction errors are not guaranteed to stay close to the original motions in the latent space. This means that the learned latent representation overfits to current data and is not robust to noise in the motions. In contrast, with our bi-level motion imitation framework, we introduce latent representation that focuses on the general motion patterns instead of nuances and noise. This is important, since the nuances are likely to change when converted to be physically consistent in the fine-tuning step.
We address the above gap by Self-Consistent Auto-Encoder (SCAE). Specifically, we propose to regularize FLD learning with latent reconstruction error. A similar idea has been applied to VAE [26] but has not been investigated deterministic auto-encoders for motion generation. Fig- ure shows the structure of SCAE, where the reconstructed trajectory 7t fed into the encoder again order to obtain reconstructed latent representation & from the decoded motion Å. We retain the multi-step prediction in FLD and thus our SCAE training loss is
N
NN SCAR
Σ k=0
a
Te4al 31% 12
(5)
where / the coefficient the latent reconstruction error and where we evaluate the loss on the entire dataset. The reconstructed latent representation 8/4 is computed by feeding the reconstructed trajectory #/+ into the encoder, the Fourier transform layer and the sinusoidal reconstruction layer. Note that +++ ; is obtained by the multi-step forward prediction in Equation 2.

---

Convolution layers enc
Latent embedding &
Latent sampler 2, Pz(z)
ACtaN+
FFT
ACtBN+Atan2
FFT
Policy æ
0₂
Sinusoidal reconstruction 2
Sinusoidal reconstruction &
Convolution layers dec
Target ft = {8, H+3, 821
Robot state &t
Figure 2: Bi-level motion fine-tuning (BMI) optimizes both the robot policy and the decoder alter- natively. The learning begins by sampling from the learned latent space p(z) and decoding these latent samples into target reference motions for robot imitation. The decoder's loss function com- prises two components, as indicated by the red arrows: (1) the mean squared error (MSE) between the robot's trajectory and the decoded trajectory, and (2) the latent reconstruction error between the sampled latent embeddings & and the embeddings of the decoded trajectories &.
With perfect decoder, the reconstructed motion Fg exactly the same as the original motion т leading to zero latent reconstruction error | E/+, However, this is usually not achievable. Although 1 - 2l2² generally decreases as the decoder learns to reconstruct the trajectory, experiments show that 18/-, - is not minimized when only optimizing the motion reconsiruc- tion error fi 12 In contrast, due to the latent reconstruction regularization, SCAE enforces the learned latent representation to be consistent with its decoded motions.
4.2 Pre-Training Policy
In this stage, we train our robot policy to track the given reference motions regardless of the fea- sibility of these motions done existing motion imitation works [1, 4]. In contrast to directly sampling trajectories from the original motion dataset train the robot policy, we sample from the latent space of the SCAE and inform the robot policy with the sampled latent parameters as the target motion information. The self-consistent latent dynamics model provides two advantages compared to using the original datasets. (i) We can interpolate latent parameters to generate motion transitions and new motions, as discussed in FLD [8] and PAE [25]; (ii) We observe that a learned latent representation as the tracking goal for the robot more concise with essential motion patterns and focuses less on motion nuances, which beneficial for policy learning.
The policy pre- pre-training procedure is illustrated in Figure 2 without the green are only used in the next fine-tuning stage). For each episode, we sample a set of latent variables 21 from the pre-collected buffer pa (=) during SCAE training. We then obtain (dt, Je, a, b,) = p(z)) by the following FC and FFT layers. Note that instead of taking the learned phase th,, we uniformly sample an initial phase variable do E Re from a fixed range and update de according to the latent dynamics in Equation
modules (these
Φ = - + f-1At,
(fr, a, b.) = 0, = -
(6)
We maintain the same frequency ft. amplitude or, and offset b, for the episode. The latent variables are then used to reconstruct a motion trajectory
14-Ht1,-... 5e) = f = dec(fi ar, be, (b)).
where the most recent state &, serves the target state compute the robot tracking reward the current timestep. The policy is learned using proximal policy optimization [27].
5

---

4.3 Bi-Level Pine-Tuning
This step ensures physical consistency of the reference motions generated by the decoder. Obtaining reference motions that are physically consistent is important as it facilitates policy learning and encourages the robot to learn versatile set of skills, particular when the humanoid robots are under-actuated and have restricted torque limits [7, 20, 4]. We propose convert these unphysical motions into physically consistent ones by bi-level fine-tuning to maximize the benefit of human MoCap data. This represents an important difference from recent works that address this problem by only tracking upper body movements [20] or filtering out the unlearnable motions [4].
Figure shows the structure our bi-level fine-tuning. In this stage, we alternatively optimize the policy and the decoder dec while freezing the convolutional encoder enc and the FC, BN layers. In this way, the decoder is encouraged to generate motions close the robot trajectories, which physically consistent by design. We further regularize the decoder optimization by constraining the generated motions to be close to the original motions in the latent space. This prevents the decoder from generating trivial motions by simply copying the robot, failing to improve the robot policy further. The bi-level optimization problem is formulated as,
min E (aampa()) [| - sel2 + B =, 0oc
0 E arg min (s) [| sel sel
where fee denotes the parameters of the decoder and To, the robot policy with parameters 0z- With the proposed regularized bi-level motion imitation, the decoder is updated to generate motions physically consistent with the robot while retaining the desired motion patterns in the dataset. result, we observed that the robot further improves the policy during this fine-tuning step.
5 Experiments
We evaluate BMI on the MIT humanoid robot [16] in Isaac Gym [28) while keeping the joint and force limits unchanged. We extend the dataset from FLD [8] by including four additional difficult motions, jump, kick. spin-kick, and cross-over [1]. total, we have trajectories from 13 different motions. Our experiments examine both the learned dynamics model and policy performance.
5.1 Latent Dynamics Model Learning
Motion and Latent Reconstruction Figure 3b shows that our method and FLD can reconstruct the original motions with comparable accuracy. However, our method with explicit self-consistens constraints achieves significantly lower latent reconstruction error, i.e., | 2 as shown in Figure 3a. Therefore, the proposed self-consistent regularization improves the latent reconstruction without sacrificing the motion reconstruction accuracy. ablation study on the coefficient of
latent reconstruction loss in the appendix shows that SCAE robust wide range of values.
Latent Recorstruction Error
Motion Reconatruction Error
60
fL
F1G
SCAE
50
SCAE
12
30
60
30
6-
20
4-
2-
10
1.000
3000
4000
5000
1000
2000
3000
4000
5000
Training
Training
Latent reconstruction error
(b) Motion reconstruction error
Figure 3: Reconstruction error during training: (a) The reconstruction error of latent embeddings.
(b) The reconstruction error the original motion states.

---

Learned Phase Timing 8x2
Learned Phase Timing 8x2
(a) FLD
(b) SCAE (Ours)
Figure 4: The figure displays the learned latent phases of four motions. Each circle represents latent channel where the radius the amplitude and the black bar the phase timing. Compared to FLD, SCAE takes fewer frequency components and lower amplitudes to represent the same motion.
(a) Original
(b) FLD
(c) SCAE (Ours)
Figure 5: The figure shows the latent manifolds for 13 motions. Each color corresponds to trajec- tory segment from a motion type. The arrows denote the motion evolution direction. The manifold induced by SCAE shows consistent structures across different motions.
Learned Latent Manifold We visualize the learned latent amplitude and latent phase Φe in eight latent channels, computed in Equation 1, for four motions run, jog, step fast, jump in Figure 4, where each row denotes the same motion. Thanks the latent regularization, learns a much sparser representation than FL.D, SCAE takes fewer frequency components to reconstruct the same motions with most channels' amplitudes around zero.
method
Figure 5 compares the latent structure induced by SCAE with that by FLD, where Figure 5a vi- sualizes the principal components of the original motions. Notably, SCAE demonstrates the most consistent structure across 13 different motions. The circles connecting points with the color represent the primary period of individual motions and each point denotes trajectory segment. The radius of a rough circle means that the high-level features throughout a motion can be constant, such as velocity. frequency, etc. The well-shaped latent manifolds learned by SCAE show that our method successfully captures essential motion patterns.
5.2 Performance of Policy Learning
We both quantitatively and qualitatively compare the policy learned by FLD, SCAE pre-training, and BMI fine-tuning. Since the target reference motions are noisy and sometimes physically inconsistent for the robot, the commonly used mean square еттоr (MSE) from the reference motions is not an ideal performance metric. Instead, we calculate motion-specific quantities to compare the policy performance. Table and Figure show that BMI achieves the longest kicking time and the most stable standing while kicking. We find, perhaps surprisingly, that without further fine-tuning SCAE improves policy learning in the pre-taining stage and achieves the longest jumping time shown in Table 1. We hypothesize that the policy improvement in pre- pre-training is due the change of latent parameterizations used to inform the policy. SCAE learns sparser representation that makes policy

---

Table 1: Results on two selected challenging motions: kick and jump.
Motion (Metric))
FLD
SCAE(Ourr)
BMI(Ours)
Kick (Time (%)
64.4
61.2
71.3
Kick (Height (m))
0.157
0.152
0.164
Jump (Time (%))
32.5
36.2
35.2
0.25
0.125
BMI(Ours) SCAE(Ours)
Height (m))
0.20
FLD
Height
0.15
0.075
0.10
0.050
50
75
100
125
150
175
25
50
75
100
125
150
175
Stop
Step
(a) Kicking foot height when kicking
(b) Standing foot height when kicking
Figure 6: Comparison on the challenging kick task: The left figure shows the height of the kicking foot during one kick trajectory with multiple trials, where both SCAE and BMI outperform FLD in each kick (one mode of the curve). The right figure shows the height the standing foot where BMI and SCAE are more stable with lower height of the standing foot.
learning easier. More qualitative results on the other motions can be found on the website. The videos qualitatively show the improvement via BMI on diverse motions such as cross-over, stride and step. The experiments therefore confirm our hypothesis that by updating the decoder the robot policy can be further improved.
We conduct additional experiments to thoroughly analyze our method, which can be found our website. (i) Two zero-shot sim-to-sim experiments show that the learned policy works well even when the robot added 5kg mass block. (ii) We also visualize the motion changes of the decoded trajectories before and after bi-level fine-tuning. The video displays an increased arm swing in the fine-tuned stride motion, suggesting greater physical consistency with the robot's dynamics. (iii) The learned latent dynamics model potentially function generative model to synthesize new motions. By simply interpolating the latent amplitude and frequency, can generate motions that are kinematically consistent.
6 Limitations & Conclusion
Limitations While the proposed bi-level motion imitation framework alleviates problems arising from physically inconsistent reference motions, the approach relies on decent robot policy in the pre-training stage. Moreover, since the given references obtained by motion retargeting from human MoCap data are not the optimal targets for robot imitation, the choice of metric to quantify robot tracking performance is an open question. We also note that would be beneficial to seale up our method large-scale MoCap dataset such as AMASS [18] and apply the learned policy real-world humanoid robot via sim-to-real techniques [4, 29].
Conclusion This paper presents BMI, a novel bi-level motion imitation framework that minimizes the robot tracking error by alternatively optimizing the robot policy and the motion generation model while being regularized by latent space constraints. Our proposed self-consistent auto-encoder cap- tures the essential motion patterns with sparse and well-structured latent representations, providing a reliable anchor to regularize the decoder to stay close to the desired motion patterns in the dataset. In contrast to existing optimal control methods, BMI addresses the difficulty of including physically inconsistent reference motions in purely data-driven way and is scalable to large-scale human MoCap datasets. Our experiments the realistic MIT humanoid robot show that BMI not only improves the pre-trained policy on challenging tasks but also further stabilizes the learned motions.

---

Acknowledgments
The authors thank the support of the German Research Foundation and the Max Planck Institute for Intelligent Systems, Tuebingen (Germany). Wenshuai Zhao, Yi 7hao, and Joni Pajarinen acknowl- edge funding by the Research Council of Finland (decision numbers 345521, 357301). The authors thank Mlaus-Rudole Kladny for the insightful discussion.
References
[1] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. DeepMimic: Example-guide deep re- inforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
[2] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. AMP: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG), 40(4): 1-20, 2021.
[3] A. Tang, T. Hiraoka, N. Hiraoka, Shi, K. Kawaharazuka, K. Kojima, K. Okada, and M. Inaba. HumanMimic: Learning natural locomotion and transitions for humanoid robot via Wasser- stein adversarial imitation. arXiv preprini arXiv:2309. 14225, 2023.
[4] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04433 2024.
[5] Koenemann, Burget, and M. Bennewitz. Real-time imitation of human whole-body mo- tions by humanoids. In Proceedings of the IEEE International Conference Robotics and Automation, pages 2806 2812, 2014.
[6] J. Merel, Y. Tassa, D. TB, S. Srinivasan, J. Lemmon, Z. Wang, G. Wayne, and N. Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv: 02201, 2017.
[7] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10895-10904, 2023.
[8] C. Li, E. Stanger-Jones, S. Heim, and S. Kim. FLD; Fourier latent dynamics for structured motion representation and learning. arXiv preprint arXiv:2402.1388) 2024.
[9] Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems, 2016.
[10] M. Orsini, A. Raichuk, L. Hussenot, D. Vincent, R. Dadashi, S. Girgin, M. Geist, O. Bachem,
O. Pietquin, and M. Andrychowiez. What matters for adversarial imitation learning? In Advances in Neural Information Processing Systems, 2021.
[11] D. Jung, H. Lee, and S. Yoon. Sample-efficient adversarial imitation learning. Journal of Machine Learning Research, 25(31):1-32, 2024.
[12] G. Bin Hammam, P. M. Wensing, B. Dariush, and D. E. Orin. Kinodynamically consistent motion retargeting for humanoids. International Journal of Humanoid Robotics, 12(04), 2015.
[13] T. Yoon, D. Kang, S. Kim, M. Ahn, Coros, and S. Choi. Spatio-temporal motion retargeting for quadruped robots. arXiv preprint arXiv:2404.11 11557, 2024.
[14] M. Al Borno, L. Righetti, M. J. Black, S. L. Delp, Fiume, and J. Romero. Robust physics- based motion retargeting with realistic body shapes. Computer Graphics Forum, 37:81-92, 2018.

---

[15] R. Grandia, F. Farshidian, E. Knoop, C. Schumacher, M. Hutter, and M. Bächer. DOC: Dif- ferentiable optimal control for retargeting motions onto legged robots. ACM Transactions on Graphics (TOG), 42(4):1-14, 2023.
[16] M. Chignoli, D. Kim, E. Stanger-Jones, and S. Kim. The MIT humanoid robot: Design, motion planning, and control for acrobatic behaviors. In Proceedings of the IEEE-RAS International Conference on Humanoid Robols (Humanoidss, 2021.
[17] Z. Luo, J. Cao, Merel, A. Winkler, Huang, K. Kitani, and W. Xu. Universal humanoid motion representations for physics- based control. arXiv preprint arXiv:3310004582, 2023.
[18] N. Mahmood, N. Ghorbani,
Troje, G. Pons-Moll, and J. Black. AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5442-5451, 2019.
[19] Y. Yuan and K. Kitani. Residual force control for agile human behavior imitation and extended motion synthesis. Advances in Neural Information Processing Systems, 2020.
[20] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 2024.
[21] K.-. Choi and H.-S. Ko. Online motion retargetting. The Journal of Visualization and Com- puter Animation, 11(5):223-233 2000.
[22] Tak and H.-S. Ko. physically-bee bassed motion retargeting filter. ACM Transactions on Graphics (TOG), 24(1):98-117, 2005.
[23] G. Alain and Y. Bengio. What regularized auto-encoders learn from the data data-generating distri- bution. The Journal of Machine Learning Research, 15(1):35633359, 2014.
[24] C. Li, M. Vlastelica, S. Blaes, J. Frey, E. Grimminger, and G. Martius. Learning agile skills via adversarial imitation of rough partial demonstrations, In Proceedings of the Conference Robot Learning, pages 342-352, 2023.
[25] S. Starke, I. Mason, and Komura. DeepPhase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
[26] T. Cemgil, Ghaisas, K. Dvijotham, S. Gowal, and P. Kohli. The autoencoding variational autoencoder. In Advances in Neural Information Processing Systems, 2020.
[27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXXiv1707.0634/4 2017.
[28] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference Robot Learning, pages 91-100, 2022.
[29] L. Smith, Y. Cao, and Levine. Grow your limits: Continuous improvement with real-world RL for robotic locomotion. arXiv preprint arXiv:2310.17634, 2023.
[30] J. Tan, K. Liu, and G. Turk. Stable propertionallderiaaiieccooc controllless IEEE Computer Graph- ics and Applications, 31(4):34-44, 2011.
10

---

A
Appendix
Contents
A.1 Structure of FLD
11
A.2 Pseudo Code of BMI
12
A.3 Experiment Settings
12
A.3.1 Dataset
12
A.3.2 State and Action Spaces
13
A.3.3
SCAE Training
13
A.3.4
Policy Training
14
A.3.5
BMI Training
15
A.4
More Experiment Results
16
A.4.1
Ablation Study on Latent Reconstruction Error
16
A.4.2
More Results Latent Dynamics Model Learning
16
A.4.3
Visualization of Learned Policy
17
A.1 Structure of FLD
The structure of FLD follows PAE [25] using an auto-encoder learn a generative dynamics model, where the encoder and the decoder composed of 1D) convolutional layers. In order enforce the periodicity in the latent manifolds, PAE parameterized each latent channel as sinusoidal function where the amplitude, frequency, and offset are computed by a differentiable Fast Fourier Transform layer while the phase determined with fully connected layer followed by an Atan2 operation. Inspired by the observation that the learned latent frequency, amplitude, and offset by PAE stay nearly constant along the trajectories, FLD improves PAE by combining the structure with a multi- step prediction step in Equation
Trajectory segment T;
T++1
T++
Convolution layers enc
Latent embedding 2t
Lo
FC+BN+Atan2
FFT
L
LN
Φ+
0,
Φc+1, Oc+1
OctN O
Sinusoidal reconstruction
2/11
2. 2
Convolution layers dec
dec
dec
Reconstructed segment ft
Al E+N
2-11
Figure 7: Multi-step forward prediction structure of FL.D.

---

A.2 Pseudo Code of BMI
Algorithm shows the details of BMI training.
Algorithm Bi-Level Motion Imitation (BMI)
Input: SCAE encoder enc and decoder dec, latent parameters of the original motions pa (z)) Pre-trained policy no, based SCAE, initial buffer D
for to K do
Policy Learning:
for i = to MM do
Sample latent targets 2; ~ (z(2)
Extract the target states {$ = dec(p(ft, at, b,, Ø:))
Rollout robot trajectory Ti ~ (T dec, P=)
Collect the trajectory and latent parameter pairs in the buffer D = { Ti) ~ M1}
Update robot policy ne, with PPO or another RL algorithm according to the bottom ob-
jective in Equation 8
end for
Decoder dec Update:
for i = to M2 do
Sample latent parameters and robot trajectories from D
Update the decoder dec according to the upper objective in Equation 8
end for end for
A.3 Experiment Settings
In this section, we provide more detailed experiment settings. We first introduce our dataset and then explain the state and action spaces used SCAE and the policy. Finally, we list the architectures and aaper-parameters used in both the dynamics model learning and policy learning.
A.3.1 Dataset
We use the same dataset as FLD [8] which was originally released in DeepMimic [1]. The human MoCap data were manually processed and retargeted the humanoid robot. Note that even with careful kinematic retargeting the reference motions can be physically inconsistent to the robot dy- namics. Our dataset consists of 13 different motions: run, jog, step fast, jump, spin-kick, back, side left, jog slow, side right, cross-over, kick, stride, step and each motion has 10 trajectories collected from different demonstrations. multiple trials the same action. For example, in one kick trajectory, the demonstrator may contin- uously kick five times as shown in Figure 6a. In total, we have 13 10 240 31200 data points. Each data point corresponds state vector of length 52, where the elements listed in Table (see below).
each trajectory of length 240 steps, the demonstrator performs
Table Elements of one data point (step) the dataset Symbol
Entry
Dimensions
base position
Pb
0:3
base rotation
qb 7:10
3:7
base linear velocity
base angular velocity w
10:13
projected gravity
g
13:16
joint positions
q
16:34
joint velocity
4
34:52
Note that FLD experiments only nine motions: run, jog, step fast, back, side left, jog slow, side right, stride, step, referred to as normal motions, which present mild difficulty for
12

---

the robot to track. However, our experiments include an additional four motions, jump, spin-kick, cross-over, kick, that are significantly more challenging. FLD fails to learn these complex motions satisfactorily without a specifically designed reward function tailored to each individual motion, while our methods show improved performance on the challenging kick and jump with unchanged reward design.
A.3.2 State and Action Spaces
In this section, we introduce the state space used in the latent dynamics model and the observation and action spaces for the robot policy.
State Space of Latent Dynamics Model The state space used in the latent dynamics model is composed of the linear and angular velocities of the robot base v, w in the robot frame, mcasurement of the gravity vector the robot frame, and joint positions q as in Table 3. Note that we use the same setting for both FLD and SCAE.
Table 3: Elements of the state space for latent dynamics model
Entry
Symbol
Dimensions
base linear velocity base angular velocity
v
0:3
w
3:6
projected gravity
g 9:27
6:9
joint positions
4
Observation Space of Robot Policy In addition to the state information used in the latent dy- namics model, the robot observes extra information such as joint velocities ( and its last action a'. Moreover, we provide the latent parameters to the robot as the target motion information. Therefore, observation space is shown as Table 4. Note that we apply domain randomization to the policy training including the observation noise, disturbances of the mass, and disturbances arising from pushing used in FLD [8].
Table 4: Elements of the observation space for robot policy
Entry
Symbol
Dimensions
Noise level
base linear velocity
0:3
0.2
base angular velocity
w
3:6
0.05
projected gravity
9
6:9
0.05
joint positions
9:27
0.01
joint velocities
27:45
0.75
last actions
a'
45:63
0.0
latent phase
sin ø
63:71
0.0
latent phase
COS Ø
71:79
0.0
latent frequency
f
79:87
0.0
latent amplitude
a
87:95
0.0
latent offset
b 95:103
0.0
Action Space of Robot Policy The action space of our robot is of 18 dimensions, which represent
the target positions of 18 joints in the robot. An underlying PD controller [30] is used to compute the torques to drive each joint. The PD gains are set to (30.0, 5.0) for lower body joints and (40.0, 5.0) for upper body joints, respectively.
A.3.3 SCAE Training
We introduce first the architecture of neural networks used in SCAE, which is the same as FLD. Then we list the hyper-parameters for training the latent dynamics model.
13

---

Architecture of SCAE SCAE shares the same architecture as FLD. The architectures of the en-
coder enc and decoder dec are shown in Table 5. BN denotes batch normalization and ELU repre- sents the exponential linear unit.
Table Architecture of the neural networks used in SCAE
Network
Layer
Output size
Kernel size
Normalization
Activation
Convid
64x51
51
BN
ELU
encoder
Convld
64x51
51
BN
ELU
Convld
8x51
51
BN
ELU
phase encoder
Linear
8x2
BN
Atan2
Convld
64x51
51
BN
ELU
decoder
Convid
64x51
BN
ELU
Convld
27x51
51
BN
ELU
Parame-Paraeeters for SCAE Training SCAE uses the same parameerssssserss for training FLD as in Table The extra coefficient of the latent reconstruction regularization used in SCAF, i.e., B Equation 5, set to 1. Adam is used as the optimizer for training the latent dynamics model.
Table Hyper-parameters of SCAE training
Parameter
Symbol
Value
step time seconds
Al
0.02
max training iterations
5000
learning rate
0,0001
weight decay
0.0005
learning epochs
5
mini-batches
latent channels
8
trajectory segment length
multi-step prediction length
N
50
propagation decay
1.0
A.3.4 Policy Training
Architecture of Policy & Value function The neural network architectures of the learning policy and the value function V used in PPO shown in Table 7.
Table 7: Architecture of the neural networks used in policy training
Network
Type
Hidden
Output size
Activation
policy π
MLP
128, 128, 128
18
ELU
value function V
MILP 128, 128, 128
1
ELU
for Policy Training We use Adam as the optimizer for the policy and value function with an adaptive learning rate with a KL divergence target of 0.01. The policy runs at 50 Hz. We parallize 4096 environments in Isaac Gym to collect samples. The summary of the policy training hyper-parameters can be found in Table 8.
Reward Function for Policy Training The reward function used to train the robot policy consists of two categories r = , + n2, where ," denotes the tracking rewards and n represents the regularization rewards. The tracking reward calculates the weighted sum of individual rewards on each dimension bounded [0, 1] with their weights in Table 9,
w.r, r + Wwrw + Wgyg + Wang" Norr + Waam Tymm"
(9)
The reward of each dimension is generally formulated as,
Ti o./d -d1
(10)
14

---

Table 8: Hyper-parameters of policy training
Parameter
Symbol
Value
step time seconds
A
0.02
max training iterations
3000
max episode time seconds
20
learning rate
0.001
steps per iteration
24
learning epochs
5
mini-batches
4
KL divergence target
0.01
discount factor
0.99
clip range
0.2
entropy coefficient
0.01
parallel training environments
4096
where: denotes the in dimension. d, denotes the target value of this dimension while d, represents the reconstructed value. The variable o; denotes a temperature factor for each reward and can be found in Table 10.
Table Weights of the tracking rewards Weight
Wwy
Wg
Wgig
Value
1.0
1.0
1.0
1.0
1.0
Table 10: Temperature factors of the tracking rewards
Weight
Fw
0g
0g
qum
Value
0.2
0.2
1.0
1.0
1.0
The regularization reward is formulated Equation 11, where the weights can be found in Table 11 and each term is detailed as follows:
WarTar or + Wau WarFyT,
(11)
with action rate
ar al
(12)
where a' and a denote the previous and current actions, joint acceleration
d'
Tip
At
(13)
where q' and q denote the previous and current joint velocity, At represents the time step, and with joint torque
TqT = 1
(14)
where T stands for torque.
A.3.5 BMI Training
In the bi-level fine-tuning process, we retain most of the hyperparameters from the pre-training stage. Notable exceptions include the following: (i) We set a lower learning rate for the decoder update compared to the rate used in SCAE training. (ii) To align the magnitudes the latent reconstruction loss and the motion reconstruction loss in Equation 8, we increase the coefficient 3 to 200. The key pyper-parameters used BMI summarized in Table 12. These hyper-parameters may be further tuned for improved results. As this is initial study of bi-level fine-tuning, tested only limited number of syyee-paramettr configurations in experiments.
15

---

Table 11: Weights the regularization rewards
Weight
War
Wqu
War
Value -0.01
-2.5 x 10
-1.0 x 10
Table 12: Hyper- parameetrs of BMI fine-tuning
Parameter
Symbol
Value
coefficient of latent reconstruction loss
200
learning rate for decoder
0.00001
number of mini-batch for decoder max training iteration epochs for decoder
2
50
steps per iteration parallel training environments
24
4096
A.4 More Experiment Results
We show more experiment results this section, including experiments for both the latent dynamics model learning and the policy learning.
A.4.1 Ablation Study on Latent Reconstruction Error
We test range values for SCAE training. The results in Figure 8 show that SCAE is robust to wide range of / values.
Latent Reconstruction Error
Motion Reconstruction Error
8
8
50
40
40
10
30
10
10
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Training Iteration
Training Iteration
(a) Latent reconstruction error w.r.t different 3
(b) Motion reconstruction error w.r.. different 3
Figure 8: The left figure shows that SCAE with small 8 0.1 sufficiently improve the latent reconstruction compared to FLD (3 = 0). The right figure shows that only when 3 = 10, the motion reconstruction error slightly increased. In general, when B (0.1 - 5), SCAE demonstrates similar motion reconstruction as FLD.
A.4.2 More Results on Latent Dynamics Model Learning
Figure 9 compares the learned latent phases across all the 13 motions with different methods. We observe that our method SCAE consistently achieves sparser representations than FLD with fewer frequency components and lower amplitudes.
SCAE learns sparse and well-shaped latent representations. Nonetheless, retains accurate motion reconstruction FLD. As shown in Figure 10, both FLD and SCAE accurately reconstruct the motions, which is also validated by the training loss in Figure 3b.
16

---

Learned Phase Timing 8x2
Learned Phase Timing 8x2
CDCCDF Q
1
O
000
(a) FLD
(b) SCAE (Ours)
Figure 9: Learned latent phases of 13 different motions. From top to bottom, the motions run, jog, step sast, jump, spin-kick, back, side left, jog slow, side right, cross-over, kick, stride, step.
A.4.3 Visualization of Learned Policy
We visualize the motions learned by BMI. In addition to normal motions, such as stride in Fig- ure 11a, which can be effectively learned by FLD, BMI successfully acquires two challenging mo- tions kick and jump which FLD fails. Figure 12a shows that BMI policy can naturally the kicking foot while maintaining the stability of the standing foot. Similarly, Figure 12b illustrates that the robot successfully jumps, with both feet leaving the ground.
However, we note that our policy still struggles with the difficult spin-kick and cross-over motions which are highly dynamic and can significantly influence the robot balance. Consequently, the robot prioritizes maintaining balance over replicating these motion patterns. For example, the robot rarely lifts its kicking foot spin-kick, and the legs do fully cross cross-over, as shown in Figure 13.
17

---

Reconsttructionn (Flattened) 1x1377
Reconstruction (Flattenod)
www.a//lwwwwana/AM/MW//a///// Racomorrccied lage:
MWWMM///wwwaaa/A///////////////A Nacerobbused
Targe:
Aun
"WWM
l89
(a) FLD
(b) SCAE (Ours)
Figure 10: Motion reconstruction performance.
(a) Stride
(b) Back
Figure 11: Normal motions learned by BMI.
18

---

11
(a) Kick
(b) Jump
Figure 12: Challenging motions learned by BMI.
(a) Spin-Kick
(b) Cross-Over
Figure 13: Unsatisfying motions learned by BMI.
19


Reviews:
[{"review_id": "Kf0T52rEfu", "reviewer": "Reviewer_y7N1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a way to handle infeasible human motions for humanoid learning by a bi-level optimization-based imitation learning framework which alternates between optimizing both the robot policy and the target mocap data.", "summary_of_recommendation": "The representation learning design of humanoid learning is novel. My concern is on the scalaibity of this method on 1000x larger motion datasets and sim2real side.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "- I understand \"physical feasible motions\". But what is the definition of physical consistency? I was confused when reading the introduction. \n- \"\\Tau represents a known time window with H evenly spaced samples\". How to get \\Tau for each motion sequence?\n- Introducing the loss of latent representation computed by feeding the reconstructed trajectory seems tricky to me. Because with a large coefficient \\beta, the encoder will just simply converge give the same embedding z for all input motion. The ablation of the \\alpha and \\beta is needed.\n- What is stopping us from seeing a sim2real demo on the MIT humanoid? Is it hardware or RL tuning?\n- The metrics in Table is not sufficient enough. More detailed metrics on more diverse motions are needed.\n\n\nI look forward to discussing with the authors during the rebuttal, and I am open to raise my score if my concerns could be addresed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SlyKztuGNf", "reviewer": "Reviewer_k7pZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a framework for humanoid robot locomotion, to address the challenge of incorporating physically infeasible motion capture (MoCap) data into training datasets. The proposed method consists of three stages: training an auto-encoder, pre-training a policy, and fine-tuning the decoder and the policy. The proposed method named Bi-Level Motion Imitation (BMI) shows better physically feasible motions in simulation environments.", "summary_of_recommendation": "I would recommend reject based on the current techniqual depth and presentations.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsMCrBGTa5", "reviewer": "Reviewer_p8L1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a learning algorithm for self-consistent latent dynamics and an imitation learning method for bilevel fine tuning of policies and latent dynamics. Through an imitation task of 13 different motions of a humanoid robot in simulation, the authors successfully showed that sparse latent phases can be learned by the proposed method and that the latent manifolds are well-formed. The authors also claim that the proposed method improves imitation performance compared to a conventional imitation learning method.", "summary_of_recommendation": "The contribution of the paper is not sufficient: weakness in both quality and clarity of the paper.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "Issues that need to be addressed by the authors are listed in Weaknesses in the Review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach to tackle infeasible human motions in humanoid learning using a bi-level optimization-based imitation learning framework. The framework alternates between optimizing the robot policy and the latent dynamics.

Reviewers found the use of latent motion representation and bi-level optimization to be inspiring. However, before the rebuttal, there were concerns about the absence of real-world experiments and the lack of thorough discussions on the limitations and trade-offs of motion tracking based on latent space. Additionally, the paper lacks detailed explanations for some of the figures, equations, and videos.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns by including new experiments on sim-to-sim transfer and conducting additional analysis/ablations on the bi-level finetuning and the latent parameterization space. The authors also made substantial improvements to the presentation of the work. As a result, all reviewers now rate the paper as a Weak Accept, and the AC agrees that the paper is now much stronger, with the major limitation being the lack of real-world experiments. However, the AC acknowledges the challenges of conducting hardware experiments within the short rebuttal period and appreciates the inclusion of sim-to-sim transfer results.

To further strengthen the paper, the authors should consider Reviewer y7N1’s suggestion to scale to AMASS and transfer the policy to a higher-fidelity simulator like Mujoco or an actual real robot.
Decision: Accept