Bi-Level Motion Imitation for Humanoid Robots

Authors: wenshuai  zhao
yi  zhao
joni  pajarinen
michael  muehlebach
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 18:57:17

Abstract: Imitation learning from human motion capture (MoCap) data provides a promising way to train humanoid robots. However, due to differences in morphology, such as varying degrees of joint freedom and force limits, exact replication of human behaviors may not be feasible for humanoid robots. Consequently, incorporating physically infeasible MoCap data in training datasets can adversely affect the performance of the robot policy. To address this issue, we propose a bi-level optimization-based imitation learning framework that alternates between optimizing both the robot policy and the target MoCap data. Specifically, we first develop a generative latent dynamics model using a novel self-consistent auto-encoder, which learns sparse and structured motion representations while capturing desired motion patterns in the dataset. The dynamics model is then utilized to generate reference motions while the latent representation regularizes the bi-level motion imitation process. Simulations conducted with a realistic model of a humanoid robot demonstrate that our method enhances the robot policy by modifying reference motions to be physically consistent.

Main paper: 
 introduction: 
 The use of human motion capture (MoCap) data as reference trajectories offers a promising way to design powerful humanoid robot controllers [1, 2, 3, 4] . After appropriate motion retargeting these close-expert reference trajectories can be directly imitated by robots, reducing the need for extensive reward engineering typically required in reinforcement learning [5, 3] . Existing motion imitation works either learn the motion styles in a generative adversarial way [6, 2, 7, 3] or directly learn to track the provided motion trajectories [1, 8] . While the former method, based on generative adversarial imitation learning (GAIL) [9] , avoids the exact definition of similarity between reference motions and robot trajectories, its min-max computational formulation usually suffers from unstable learning and sample inefficiency [10, 11] . The latter method, however, can also be problematic because the reference motion is often noisy and physically infeasible for realistic humanoid robots due to embodiment differences such as different force and joint limits between humans and robots [4] . Consequently, including such data may degenerate the policy learning of the robot [4] .The aforementioned issues arising from noisy and physically infeasible reference motion have been mainly studied in the field of motion retargeting [12, 13, 14] . For example, in order to create natural motions for various animated characters, researchers pursue retargeting the human MoCap motions into physically consistent motions of new characters, which in our case corresponds to humanoid robots. The common approach used in physics-based retargeting hinges on trajectory optimization with known dynamics of the target robot and constraints that arise from the reference trajectories [14, 15] . However, the resulting optimization problem is often complex and includes specific domain knowledge. There is therefore an emergent need for a learning-based method that does not rely on an explicit dynamics model while guaranteeing physical consistency at the same time. We address this need by proposing the Bi-Level Motion Imitation (BMI) framework.Our method shares a similar bi-level optimization idea with differential optimal control [15] but does not need a prior dynamics model and human-specified constraints. Specifically, BMI first learns a generative latent dynamics model based on a novel self-consistent generative auto-encoder (SCAE) from the reference motions. SCAE regularizes normal auto-encoder training with a latent reconstruction error and captures the essential motion patterns with sparse and well-structured latent representations. This enables us to sample latent parameters and reconstruct new motions, which are used to train the humanoid robot policy (pre-training step). After pre-training, BMI further finetunes both the decoder and the robot policy as a bi-level optimization problem. In this way, the decoder learns to return reference motions that are physically consistent. At the same time, the robot further improves its policy by imitating updated reference motions. We constrain the decoder updates to ensure that the reconstructed motions stay close to the original motions in the latent space, which prevents the decoder from degenerating into trivial motions that are far from the desired motion patterns in the human MoCap data.We evaluate BMI on the MIT Humanoid Robot [16] in simulation, where we imitate motions from human MoCap data. The experiments first show that the proposed SCAE-based latent dynamics model learns structured motion representations. In the subsequent pre-training, the improved latent representation learned by SCAE also enhances policy learning compared to the baseline latent dynamics model. Finally, our bi-level fine-tuning with latent space regularization updates the decoder to construct reference motions that are physically consistent for the robot and retain the original patterns at the same time. Our experiments show that the robot policy can be further improved by imitating the updated motions.The key contributions of this paper can be summarized as follows: (i) We propose a self-consistent latent dynamics model that is able to learn sparse and structured representations for human motions. (ii) We propose a bi-level motion imitation framework to update the decoder and the robot policy at the same time, which enhances the generated motions with physical consistency and closeness to the original human MoCap trajectories. (iii) We evaluate our method on a humanoid robot and imitate up to 13 different motions with a single policy. The experiments highlight improved policy learning with the proposed latent dynamics model and bi-level motion imitation framework. 
related work: 
 We first discuss existing reference-based humanoid imitation learning methods. Methods addressing the problem of physically inconsistent reference motions are discussed subsequently.Humanoid Motion Imitation Imitating from human MoCap data is an efficient way for humanoid robots to learn agile and natural-looking skills [1] . Recent works [7, 17] based on generative adversarial imitation learning (GAIL) [9, 2] in animation have succeeded in training humanoid robots to track various human motions using a large MoCap dataset such as AMASS [18] . Nonetheless, the success may be partially attributed to the unrealistic humanoid robot that is used. With up to 69 DoFs, unlimited force, and even assistive external forces [19] , the simulated robot is massively overactuated and can, in principle, perfectly track the given reference motions. It is therefore unclear whether the approaches in animation [7, 17] can be transferred to more realistic robots. As the reference motions can be physically infeasible for robots, including them in the training dataset can result in sub-optimal mimicking behaviors or even complete failure in imitation [13] . The authors from [20] train whole-body humanoid controllers that only replicate upper-body movements while the lower body is restricted to track a given forward velocity for the base. An alternation has been proposed in [4] where the infeasible motions are explicitly removed by a privileged simulated imitator. Fourier Latent Dynamics (FLD) [8] employs a fallback mechanism to replace the given reference motions with default motions when the reference is far from the training motions.Physically Consistent Motion Retargeting Motion retargeting describes the process of mapping the human MoCap data to target robot configurations such that downstream motion imitation can be performed. While common motion retargeting methods [21, 13] such as inverse kinematics-based methods can generate visually convincing motions, these motions could be physically infeasible for humanoid robots. In order to obtain physically consistent motion retargeting, existing methods are usually formulated as trajectory optimization problems constrained by robot dynamics [22, 12, 14, 15] . For instance, differential optimal control [15] alternatively optimizes the retargeting parameters with manually defined contact constraints and the robot trajectories based on the retargeting as a bi-level optimization problem. However, it is often tedious to model the complex robot dynamics and these methods are therefore hard to generalize across different robots. In contrast, our method is purely data-driven. 
preliminaries: 
 Our method involves modifying a latent dynamics model, which maps the motions through an autoencoder [23] into latent space and back, in order to generate motions for the robot that are physically consistent and at the same time close to the desired motion patterns in the original MoCap dataset. However, measuring the closeness between the original trajectory and the generated physicallyconsistent reference motion for the robot, is challenging [24] . We address this problem by introducing a structured motion representation and incentivizing closeness in the latent space. Our proposed latent dynamics model is inspired by FLD [8] , a structured motion representation method that explicitly enforces the periodicity of motions in the latent space by transforming the learned latent representation into the frequency domain [25] .The structure of FLD is illustrated in Figure 7 in the appendix. We denote a given trajectory segment of length H in d-dimensional state space by τ t = (s t-H+1 , • • • , s t ) ∈ R d×H , where t denotes time and s t the state at time t. The trajectory segment τ t represents the input to the auto-encoder, where the encoder embeds the original motion trajectory into a latent space with c channels, denoted by z t ∈ R c×H . In order to explicitly account for the periodicity of the motions, FLD builds on earlier work on Periodic Autoencoders (PAEs) [25] and includes a differentiable Fast Fourier Transform (FFT) layer. The FFT layer returns the frequency f t , amplitude a t , and offset b t of the latent motion embeddings, while a separate phase ϕ t is computed by an additional fully connected (FC) layer and an atan2 operation. This transformation is denoted as p:EQUATIONwhere ϕ t , f t , a t , b t ∈ R c and enc is the encoder. Particularly, FLD improves PAE with a multi-step forward prediction to approximate the subsequent latent vectors by unrolling the latent phase. For a local range of N subsequent trajectory segments {τ t , τ t+1 , • • • τ t+N }, we assume that the segments share the same latent parameters f t , a t , b t while differing only in their phases ϕ t+i . Furthermore, ϕ t+i can be approximated by ϕ t+i ≈ ϕ t + if t ∆ t , where ∆ t denotes the time step. This results in,EQUATIONwhere dec is the decoder. We denote by p the embedding reconstruction process from the frequency domain,EQUATIONwhere T represents a known time window with H evenly spaced samples [25] . We note that ẑ′ t+i , ŝ′ t+i are different from ẑt , ŝt as they are approximated by the multi-step forward prediction from the trajectory τ t . This motivates the following loss function that is used in FLD,EQUATION)where α is a decay factor and | • | denotes the Euclidean distance. 
method: 
 The proposed method involves a three-stage training procedure. (i) In the first stage, we learn a generative latent dynamics model from the original MoCap data that has been kinematically retargeted Particularly, we re-input τt to the encoder to obtain reconstructed latent embedding ẑt . Therefore, SCAE consists of both motion and latent reconstruction losses, as indicated by red arrows. We follow FLD to make multi-step predictions and thus the final loss sumsL 0 , • • • , L N .to the humanoid. We introduce a self-consistent auto-encoder trained using both reconstruction error and latent regularization, to capture the desired patterns embedded in the noisy kinematic motions more effectively. (ii) The second stage samples latent parameters encoded by the self-consistent dynamics model and then decodes these latent samples into the state space. The decoded states are used as the reference motions to pre-train the robot policy. (iii) We perform bi-level imitation by fine-tuning the policy and updating the decoder at the same time. Crucially, this bi-level optimization is constrained within the latent space, ensuring that the decoder generates motions that closely adhere to physics-based robot trajectories while preserving the original motion patterns intended for imitation. The following paragraphs explain the three-step procedure in detail.Although FLD learns structured latent representations and shows accurate reconstruction, we find that the decoded motions with small reconstruction errors are not guaranteed to stay close to the original motions in the latent space. This means that the learned latent representation overfits to current data and is not robust to noise in the motions. In contrast, with our bi-level motion imitation framework, we introduce a latent representation that focuses on the general motion patterns instead of nuances and noise. This is important, since the nuances are likely to change when converted to be physically consistent in the fine-tuning step.We address the above gap by a Self-Consistent Auto-Encoder (SCAE). Specifically, we propose to regularize FLD learning with a latent reconstruction error. A similar idea has been applied to VAE [26] but has not been investigated in deterministic auto-encoders for motion generation. Figure 1 shows the structure of SCAE, where the reconstructed trajectory τt is fed into the encoder again in order to obtain a reconstructed latent representation ẑt from the decoded motion τt . We retain the multi-step prediction in FLD and thus our SCAE training loss isEQUATIONwhere β is the coefficient of the latent reconstruction error and where we evaluate the loss on the entire dataset. The reconstructed latent representation ẑ′ t+i is computed by feeding the reconstructed trajectory τ ′ t+i into the encoder, the Fourier transform layer and the sinusoidal reconstruction layer. Note that τ ′ t+i is obtained by the multi-step forward prediction in Equation 2. With a perfect decoder, the reconstructed motion τt is exactly the same as the original motion τ t leading to zero latent reconstruction error | ẑ′ t+i -ẑ′ t+i | 2 . However, this is usually not achievable. Although | ẑ′ t+i -ẑ′ t+i | 2 generally decreases as the decoder learns to reconstruct the trajectory, our experiments show that | ẑ′ t+i -ẑ′ t+i | 2 is not minimized when only optimizing the motion reconstruction error |τ ′ t+i -τ t+i | 2 . In contrast, due to the latent reconstruction regularization, SCAE enforces the learned latent representation to be consistent with its decoded motions.In this stage, we train our robot policy to track the given reference motions regardless of the feasibility of these motions as done in existing motion imitation works [1, 4] . In contrast to directly sampling trajectories from the original motion dataset to train the robot policy, we sample from the latent space of the SCAE and inform the robot policy with the sampled latent parameters as the target motion information. The self-consistent latent dynamics model provides two advantages compared to using the original datasets. (i) We can interpolate latent parameters to generate motion transitions and new motions, as discussed in FLD [8] and PAE [25] ; (ii) We observe that a learned latent representation as the tracking goal for the robot is more concise with essential motion patterns and focuses less on motion nuances, which is beneficial for policy learning.The policy pre-training procedure is illustrated in Figure 2 without the green arrow modules (these are only used in the next fine-tuning stage). For each episode, we sample a set of latent variables z t from the pre-collected buffer p z (z) during SCAE training. We then obtain (ϕ t , f t , a t , b t ) = p(z t ) by the following FC and FFT layers. Note that instead of taking the learned phase ϕ t , we uniformly sample an initial phase variable ϕ 0 ∈ R c from a fixed range and update ϕ t according to the latent dynamics in Equation 2,EQUATIONWe maintain the same frequency f t , amplitude a t , and offset b t for the episode. The latent variables are then used to reconstruct a motion trajectoryEQUATIONwhere the most recent state ŝt serves as the target state to compute the robot tracking reward at the current timestep. The policy is learned using proximal policy optimization [27] .This step ensures physical consistency of the reference motions generated by the decoder. Obtaining reference motions that are physically consistent is important as it facilitates policy learning and encourages the robot to learn a versatile set of skills, in particular when the humanoid robots are under-actuated and have restricted torque limits [7, 20, 4] . We propose to convert these unphysical motions into physically consistent ones by a bi-level fine-tuning to maximize the benefit of human MoCap data. This represents an important difference from recent works that address this problem by only tracking upper body movements [20] or filtering out the unlearnable motions [4] .Figure 2 shows the structure of our bi-level fine-tuning. In this stage, we alternatively optimize the policy π and the decoder dec while freezing the convolutional encoder enc and the FC, BN layers. In this way, the decoder is encouraged to generate motions close to the robot trajectories, which are physically consistent by design. We further regularize the decoder optimization by constraining the generated motions to be close to the original motions in the latent space. This prevents the decoder from generating trivial motions by simply copying the robot, failing to improve the robot policy further. The bi-level optimization problem is formulated as,EQUATIONwhere θ dec denotes the parameters of the decoder and π θπ the robot policy with parameters θ π .With the proposed regularized bi-level motion imitation, the decoder is updated to generate motions physically consistent with the robot while retaining the desired motion patterns in the dataset. As a result, we observed that the robot further improves the policy during this fine-tuning step. 
experiments: 
 We evaluate BMI on the MIT humanoid robot [16] in Isaac Gym [28] while keeping the joint and force limits unchanged. We extend the dataset from FLD [8] by including four additional difficult motions, i.e., jump, kick, spin-kick, and cross-over [1] . In total, we have trajectories from 13 different motions. Our experiments examine both the learned dynamics model and policy performance.Motion and Latent Reconstruction Figure 3b shows that our method and FLD can reconstruct the original motions with comparable accuracy. However, our method with explicit self-consistency constraints achieves significantly lower latent reconstruction error, i.e., | ẑ′ t+i -ẑ′ t+i | 2 , as shown in Figure 3a . Therefore, the proposed self-consistent regularization improves the latent reconstruction without sacrificing the motion reconstruction accuracy. An ablation study on the coefficient β of latent reconstruction loss in the appendix shows that SCAE is robust to a wide range of β values. Learned Latent Manifold We visualize the learned latent amplitude f t and latent phase ϕ t in eight latent channels, computed as in Equation 1, for four motions run, jog, step fast, jump in Figure 4 , where each row denotes the same motion. Thanks to the latent regularization, our method learns a much sparser representation than FLD, as SCAE takes fewer frequency components to reconstruct the same motions with most channels' amplitudes around zero.Figure 5 compares the latent structure induced by SCAE with that by FLD, where Figure 5a visualizes the principal components of the original motions. Notably, SCAE demonstrates the most consistent structure across 13 different motions. The circles connecting points with the same color represent the primary period of individual motions and each point denotes a trajectory segment. The radius of a rough circle means that the high-level features throughout a motion can be constant, such as velocity, frequency, etc. The well-shaped latent manifolds learned by SCAE show that our method successfully captures essential motion patterns.We both quantitatively and qualitatively compare the policy learned by FLD, SCAE pre-training, and BMI fine-tuning. Since the target reference motions are noisy and sometimes physically inconsistent for the robot, the commonly used mean square error (MSE) from the reference motions is not an ideal performance metric. Instead, we calculate motion-specific quantities to compare the policy performance. Table 1 and Figure 6b show that BMI achieves the longest kicking time and the most stable standing while kicking. We find, perhaps surprisingly, that without further fine-tuning SCAE improves policy learning in the pre-taining stage and achieves the longest jumping time as shown in Table 1 . We hypothesize that the policy improvement in pre-training is due to the change of latent parameterizations used to inform the policy. SCAE learns a sparser representation that makes policy learning easier. More qualitative results on the other motions can be found on the website. The videos qualitatively show the improvement via BMI on diverse motions such as cross-over, stride and step. The experiments therefore confirm our hypothesis that by updating the decoder the robot policy can be further improved.We conduct additional experiments to thoroughly analyze our method, which can be found on our website. (i) Two zero-shot sim-to-sim experiments show that the learned policy works well even when the robot is added a 5kg mass block. (ii) We also visualize the motion changes of the decoded trajectories before and after bi-level fine-tuning. The video displays an increased arm swing in the fine-tuned stride motion, suggesting greater physical consistency with the robot's dynamics. (iii)The learned latent dynamics model can potentially function as a generative model to synthesize new motions. By simply interpolating the latent amplitude and frequency, we can generate new motions that are kinematically consistent. 
limitations & conclusion: 
 Limitations While the proposed bi-level motion imitation framework alleviates problems arising from physically inconsistent reference motions, the approach relies on a decent robot policy in the pre-training stage. Moreover, since the given references obtained by motion retargeting from human MoCap data are not the optimal targets for robot imitation, the choice of metric to quantify robot tracking performance is an open question. We also note that it would be beneficial to scale up our method to a large-scale MoCap dataset such as AMASS [18] and apply the learned policy to a real-world humanoid robot via sim-to-real techniques [4, 29] . 
 
 Appendix: This paper presents BMI, a novel bi-level motion imitation framework that minimizes the robot tracking error by alternatively optimizing the robot policy and the motion generation model while being regularized by latent space constraints. Our proposed self-consistent auto-encoder captures the essential motion patterns with sparse and well-structured latent representations, providing a reliable anchor to regularize the decoder to stay close to the desired motion patterns in the dataset.In contrast to existing optimal control methods, BMI addresses the difficulty of including physically inconsistent reference motions in a purely data-driven way and is scalable to large-scale human MoCap datasets. Our experiments on the realistic MIT humanoid robot show that BMI not only improves the pre-trained policy on challenging tasks but also further stabilizes the learned motions.Algorithm 1 shows the details of BMI training.Input: SCAE encoder enc and decoder dec, latent parameters of the original motions p z (z), Pre-trained policy π θπ based on SCAE, initial buffer D for k = 1 to K do Policy Learning:for i = 1 to M 1 do Sample latent targets z i ∼ p z (z) Extract the target states {ŝ t-H+1 , • • • , ŝt } = τt = dec(p(f t , a t , b t , ϕ t )) Rollout robot trajectory τ i ∼ p(π θπ , dec, p z )Collect the trajectory and latent parameter pairs in the bufferD = {(z i , τ i )|i ∼ M 1 }Update robot policy π θπ with PPO or another RL algorithm according to the bottom objective in Equation 8end for Decoder dec Update: for i = 1 to M 2 do Sample latent parameters and robot trajectories from D Update the decoder dec according to the upper objective in Equation 8 end for end forIn this section, we provide more detailed experiment settings. We first introduce our dataset and then explain the state and action spaces used in SCAE and the policy. Finally, we list the architectures and hyper-parameters used in both the dynamics model learning and policy learning.We use the same dataset as FLD [8] which was originally released in DeepMimic [1] . The human MoCap data were manually processed and retargeted to the humanoid robot. Note that even with careful kinematic retargeting the reference motions can be physically inconsistent to the robot dynamics. Our dataset consists of 13 different motions: run, jog, step fast, jump, spin-kick, back, side left, jog slow, side right, cross-over, kick, stride, step and each motion has 10 trajectories collected from different demonstrations. In each trajectory of length 240 steps, the demonstrator performs multiple trials of the same action. For example, in one kick trajectory, the demonstrator may continuously kick five times as shown in Figure 6a . In total, we have 13 × 10 × 240 = 31200 data points. Each data point corresponds to a state vector of length 52, where the elements are listed in Table 2 (see below). In this section, we introduce the state space used in the latent dynamics model and the observation and action spaces for the robot policy.The state space used in the latent dynamics model is composed of the linear and angular velocities of the robot base v, w in the robot frame, measurement of the gravity vector g in the robot frame, and joint positions q as in Table 3 . Note that we use the same setting for both FLD and SCAE. Observation Space of Robot Policy In addition to the state information used in the latent dynamics model, the robot observes extra information such as joint velocities q and its last action a ′ . Moreover, we provide the latent parameters to the robot as the target motion information. Therefore, the observation space is shown as Table 4 . Note that we apply domain randomization to the policy training including the observation noise, disturbances of the mass, and disturbances arising from pushing as used in FLD [8] . Action Space of Robot Policy The action space of our robot is of 18 dimensions, which represent the target positions of 18 joints in the robot. An underlying PD controller [30] is used to compute the torques to drive each joint. The PD gains are set to (30.0, 5.0) for lower body joints and (40.0, 5.0) for upper body joints, respectively.We introduce first the architecture of neural networks used in SCAE, which is the same as FLD.Then we list the hyper-parameters for training the latent dynamics model.Architecture of SCAE SCAE shares the same architecture as FLD. The architectures of the encoder enc and decoder dec are shown in Table 5 . BN denotes batch normalization and ELU represents the exponential linear unit. Hz. We parallize 4096 environments in Isaac Gym to collect samples. The summary of the policy training hyper-parameters can be found in Table 8 .The reward function used to train the robot policy consists of two categories r = r T + r R , where r T denotes the tracking rewards and r R represents the regularization rewards. The tracking reward calculates the weighted sum of individual rewards on each dimension bounded in [0, 1] with their weights in Table 9 ,EQUATIONThe reward of each dimension is generally formulated as, where i denotes the i th dimension. d i denotes the target value of this dimension while di represents the reconstructed value. The variable σ i denotes a temperature factor for each reward and can be found in Table 10 . The regularization reward is formulated as Equation 11, where the weights can be found in Table 11 and each term is detailed as follows:EQUATIONr R = w ar r ar + w qa r qa + w qT r qT ,with action rateEQUATIONwhere a ′ and a denote the previous and current actions, joint accelerationEQUATION)where q′ and q denote the previous and current joint velocity, ∆t represents the time step, and with joint torqueEQUATION)where T stands for torque.In the bi-level fine-tuning process, we retain most of the hyperparameters from the pre-training stage. Notable exceptions include the following: (i) We set a lower learning rate for the decoder update compared to the rate used in SCAE training. (ii) To align the magnitudes of the latent reconstruction loss and the motion reconstruction loss in Equation 8, we increase the coefficient β to 200. The key hyper-parameters used in BMI are summarized in Table 12 . These hyper-parameters may be further tuned for improved results. As this is an initial study of bi-level fine-tuning, we tested only a limited number of hyper-parameter configurations in our experiments.Learned Phase Timing 8x2 We visualize the motions learned by BMI. In addition to normal motions, such as stride in Figure 11a , which can be effectively learned by FLD, BMI successfully acquires two challenging motions kick and jump in which FLD fails. Figure 12a shows that BMI policy can naturally lift the kicking foot while maintaining the stability of the standing foot. Similarly, Figure 12b illustrates that the robot successfully jumps, with both feet leaving the ground.However, we note that our policy still struggles with the difficult spin-kick and cross-over motions which are highly dynamic and can significantly influence the robot balance. Consequently, the robot prioritizes maintaining balance over replicating these motion patterns. For example, the robot rarely lifts its kicking foot in spin-kick, and the legs do not fully cross in cross-over, as shown in Figure 13 .Project website: https://sites.google.com/view/bmi-corl2024. 8th Conference on Robot Learning (CoRL2024), Munich, Germany. 


Decision: This paper introduces an approach to tackle infeasible human motions in humanoid learning using a bi-level optimization-based imitation learning framework. The framework alternates between optimizing the robot policy and the latent dynamics.

Reviewers found the use of latent motion representation and bi-level optimization to be inspiring. However, before the rebuttal, there were concerns about the absence of real-world experiments and the lack of thorough discussions on the limitations and trade-offs of motion tracking based on latent space. Additionally, the paper lacks detailed explanations for some of the figures, equations, and videos.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns by including new experiments on sim-to-sim transfer and conducting additional analysis/ablations on the bi-level finetuning and the latent parameterization space. The authors also made substantial improvements to the presentation of the work. As a result, all reviewers now rate the paper as a Weak Accept, and the AC agrees that the paper is now much stronger, with the major limitation being the lack of real-world experiments. However, the AC acknowledges the challenges of conducting hardware experiments within the short rebuttal period and appreciates the inclusion of sim-to-sim transfer results.

To further strengthen the paper, the authors should consider Reviewer y7N1’s suggestion to scale to AMASS and transfer the policy to a higher-fidelity simulator like Mujoco or an actual real robot.

Reviews: review: title: title: Bi-Level Motion Imitation for Humanoid Robots
paper_summary: paper_summary: This paper introduces a way to handle infeasible human motions for humanoid learning by a bi-level optimization-based imitation learning framework which alternates between optimizing both the robot policy and the target mocap data.
main_review: main_review: Strengths:
- This paper solves the feasibility check of humanoid motion imitation by a bi-level optimziation by learning the latent motion representation and policy together. 
- The representation learning design is insightful and inspiring.

Weaknesses:
- Figure 1 needs more detailed caption to guide readers into the data flow and training pipelines.
- The state space and task objectives of motion tracking are not introduced. For example, what are the components of s_t? What rewards do you design for  the rewards in FIgure 2, and what is a_t, is a_t torque or PD setpoints?
- Motion tracking based on latent space is great. I agree with the advantages that the authors claim. But it also comes with several sacrifices. For example, what if you want the robot to walking left and waving the left hand, what latent space should you feed to the policy network?
- The biggest concern I hold for this paper is scalability since the authors only validate the motion representation learning on only 13 motions. What would be the effort and bottleneck of scaling SCAE on the entire AMASS dataset?
questions: questions: - I understand "physical feasible motions". But what is the definition of physical consistency? I was confused when reading the introduction. 
- "\Tau represents a known time window with H evenly spaced samples". How to get \Tau for each motion sequence?
- Introducing the loss of latent representation computed by feeding the reconstructed trajectory seems tricky to me. Because with a large coefficient \beta, the encoder will just simply converge give the same embedding z for all input motion. The ablation of the \alpha and \beta is needed.
- What is stopping us from seeing a sim2real demo on the MIT humanoid? Is it hardware or RL tuning?
- The metrics in Table is not sufficient enough. More detailed metrics on more diverse motions are needed.


I look forward to discussing with the authors during the rebuttal, and I am open to raise my score if my concerns could be addresed.
limitations: limitations: 2
review_summary: review_summary: The representation learning design of humanoid learning is novel. My concern is on the scalaibity of this method on 1000x larger motion datasets and sim2real side.
title: title: Review
paper_summary: paper_summary: The paper presents a framework for humanoid robot locomotion, to address the challenge of incorporating physically infeasible motion capture (MoCap) data into training datasets. The proposed method consists of three stages: training an auto-encoder, pre-training a policy, and fine-tuning the decoder and the policy. The proposed method named Bi-Level Motion Imitation (BMI) shows better physically feasible motions in simulation environments.
main_review: main_review: # Strengths

1. The paper introduces a good framework in humanoid robot learning/locomotion, to learn better locomotion skills from the reference motion datasets. The bi-level optimization stage seems to be new and novel.

2. The authors provide a detailed explanation of the SCAE and its role in learning sparse and structured motion representations. The mathematical formulation and the explanation of the bi-level optimization process are clear and well-articulated.


# Weaknesses

1. The proposed method is somewhat incremental to FLD, and the new stages such as bi-level optimization seem to be something common in the learning community, such as finetuing the networks. (however, I am not an expert in this field so I am not very sure.)
2. While the paper explains the methodology clearly, the complexity of the bi-level optimization process might be a barrier for some readers. A more simplified explanation or a high-level overview early in the paper could help in understanding.
3. Figure 2 might need to be largely improved or simplified for better illustrations. 
4. Lack of real robot experiments. This might be fine since humanoid robots are expensive, but maybe adding some experiments, such as Sim-to-Sim, to show its applicability, is necessary.
questions: questions: Please see weaknesses.
limitations: limitations: 3
review_summary: review_summary: I would recommend reject based on the current techniqual depth and presentations.
title: title: The quality of the paper, especially the experimental section, seems low.
paper_summary: paper_summary: This paper proposed a learning algorithm for self-consistent latent dynamics and an imitation learning method for bilevel fine tuning of policies and latent dynamics. Through an imitation task of 13 different motions of a humanoid robot in simulation, the authors successfully showed that sparse latent phases can be learned by the proposed method and that the latent manifolds are well-formed. The authors also claim that the proposed method improves imitation performance compared to a conventional imitation learning method.
main_review: main_review: Strengths:
* The paper, especially the proposed method section, is easy to understand. 

Weaknesses:
* The benefits of using latent dynamics are not shown experimentally. I would like to see a comparison between the policies of the proposed method and those learned directly from motion capture data without latent dynamics.
* Table 1 and Figure 6 show a comparison of the imitation performance of the conventional and proposed methods, but it is unclear how important the differences in values are.
* It is unclear why a change in the values in Table 1 and Figure 6 can be interpreted as an improvement in imitation performance.
* I would like to see a comparison of the changes in the reference motions from human motion capture data due to the bilevel fine tuning.
* The experiments were conducted only in simulation, and the applicability of the proposed method to real robots is not discussed.
* The accompanying video supposedly shows learned motions such as run, jog, step fast, etc., but it is not clear which motions are performed by the simulated humanoid robot.
questions: questions: Issues that need to be addressed by the authors are listed in Weaknesses in the Review section.
limitations: limitations: 1
review_summary: review_summary: The contribution of the paper is not sufficient: weakness in both quality and clarity of the paper.