OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

Authors: tairan  he
zhengyi  luo
xialin  he
wenli  xiao
chong  zhang
weinan  zhang
kris  kitani
changliu  liu
guanya  shi
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 15:56:55

Abstract: We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets. Videos at the anonymous website [https://anonymous-omni-h2o.github.io/](https://anonymous-omni-h2o.github.io/)

Main paper: 
 introduction: 
 How can we best unlock humanoid's potential as one of the most promising physical embodiments of general intelligence? Inspired by the recent success of pretrained vision and language models [1] , one potential answer is to collect large-scale human demonstration data in the real world and learn humanoid skills from it. The embodiment alignment between humanoids and humans not only makes the humanoid a potential generalist platform but also enables the seamless integration of human cognitive skills for scalable data collections [2, 3, 4, 5] .However, whole-body control of a full-sized humanoid robot is challenging [6] , with many existing works focusing only on the lower body [7, 8, 9, 10, 11, 12, 13] or decoupled lower and upper body control [14, 4, 15] . To simultaneously support stable dexterous manipulation and robust locomotion, the controller must coordinate the lower and upper bodies in unison. For the humanoid teleoperation interface [2] , the need for expensive setups such as motion captures and exoskeletons also hinders large-scale humanoid data collection. In short, we need a robust control policy that supports wholebody dexterous loco-manipulation, while seamlessly integrating with easy-to-use and accessible teleoperation interfaces (e.g., VR) to enable scalable demonstration data collection.In this work, we propose OmniH2O, a learning-based system for whole-body humanoid teleoperation and autonomy. We propose a pipeline to train a robust whole-body motion imitation policy via teach-student distillation and identify key factors in obtaining a stable control policy that supports dexterous manipulation. For instance, we find these elements to be essential: motion data distribution, reward designs, and state space design and history utilization. The distribution of the motion imitation dataset needs to be biased toward standing and squatting to help the policy learn to stabilize the lower body during manipulation. Regularization rewards are used to shape the desired motion but need to be applied with a curriculum. The input history could replace the global linear velocity, an essential input in previous work [3] that requires Motion Capture (MoCap) to obtain. We also carefully design our control interface and choose the kinematic pose as an intermediate representation to bridge between human instructions and humanoid actuation. This interface makes our control framework compatible with many real-world input sources, such as VR, RGB cameras, and autonomous agents (GPT-4o). Powered by our robust control policy, we demonstrate teleoperating humanoids to perform various daily tasks (racket swinging, flower watering, brush writing, squatting and picking, boxing, basket delivery, etc.), as shown in Figure 1 . Through teleoperation, we collect a dataset of our humanoid completing six tasks such as hammer catching, basket picking, etc., annotated with paired first-person RGBD camera views, control input, and whole-body motor actions. Based on the dataset, we showcase training autonomous policies via imitation learning.In conclusion, our contributions are as follows: (1) We propose a pipeline to train a robust humanoid control policy that supports whole-body dexterous loco-manipulation with a universal interface that enables versatile human control and autonomy. (2) Experiments of large-scale motion tracking in simulation and the real world validate the superior motion imitation capability of OmniH2O. (3) We contribute the first humanoid loco-manipulation dataset and evaluate imitation learning methods on it to demonstrate humanoid whole-body skill learning from teleoperated datasets. 
related works: 
 Learning-based Humanoid Control. Controlling humanoid robots is a long-standing robotic problem due to their high degree-of-freedom (DoF) and lack of self-stabilization [16, 17] . Recently, learning-based methods have shown promising results [7, 8, 9, 10, 11, 12, 13, 3, 18] . However, most studies [7, 8, 9, 10, 11, 12, 13] focus mainly on learning robust locomotion policies and do not fully unlock all the abilities of humanoids. For tasks that require whole-body loco-manipulation, the lower body must serve as the support for versatile and precise upper body movement [19] . Traditional goalreaching [20, 21] or velocity-tracking objectives [18] used in legged locomotion are incompatible with such requirements because these objectives require additional task-specific lower-body goals (from other policies) to indirectly account for upper-lower-body coordination. OmniH2O instead learns an end-to-end whole-body policy to coordinate upper and lower bodies.Humanoid Teleoperation. Teleoperating humanoids holds great potential in unlocking the full capabilities of the humanoid system. Prior efforts in humanoid teleoperation have used taskspace control [4, 22] , upper-body-retargeted teleoperation [23, 24] and whole-body teleoperation [25, 26, 27, 28, 29, 30, 3, 4] . Recently, H2O [3] presents an RL-based whole-body teleoperation framework that uses a third-person RGB camera to obtain full-body keypoints of the human teleoperator. However, due to the delay and inaccuracy of RGB-based pose estimation and the requirement for global linear velocity estimation, H2O [3] requires MoCap during test time, only supports simple mobility tasks, and lacks the precision for dexterous manipulation tasks. By contrast, OmniH2O enables high-precision dexterous loco-manipulation indoors and in the wild.Whole-body Humanoid Control Interfaces. To control a full-sized humanoid, many interfaces such as exoskeleton [31] , MoCap [32, 33] , and VR [34, 35] are proposed. Recently, VR-based humanoid control [36, 37, 38, 39] has been drawing attention in the graphics community due to its ability to create whole-body motion using sparse input. However, these VR-based works only focus on humanoid control for animation and do not support mobile manipulation. OmniH2O, on the other hand, can control a real humanoid robot to complete real-world manipulation tasks.Open-sourced Robotic Dataset and Imitation Learning. One major challenge within the robotics community is the limited number of publicly available datasets compared to those for language and vision tasks [40] . Recent efforts [40, 41, 42, 43, 44, 45, 46, 47] have focused on collecting robotic data using various embodiments for different tasks. However, most of these datasets are collected with fixed-base robotic arm platforms. Even one of the most comprehensive datasets to date, Open X-Embodiment [40] , does not include data for humanoids. To the best of our knowledge, we are the first to release a dataset for full-sized humanoid whole-body loco-manipulation. 
universal and dexterous human-to-humanoid whole-body control: 
 In this section, we describe our whole-body control system to support teleoperation, dexterous manipulation, and data collection. As simulation has access to inputs that are hard to obtain from real-world devices, we opt to use a teacher-student framework. We also provide details about key elements to obtain a stable and robust control policy: dataset balance, reward designs, etc.Problem Formulation. We formulate the learning problem as goal-conditioned RL for a Markov Decision Process (MDP) defined by the tuple M = ⟨S, A, T , R, γ⟩ of state S, action a t ∈ A, transition T , reward function R, and discount factor γ. The state s t contains the proprioception s p t and the goal state s g t . The goal state s g t includes the motion goals from the human teleoperator or autonomous agents. Based on proprioception s p t , goal state s g t , and action a t , we define the reward r t = R s p t , s g t , a t . The action a t specifies the target joint angles and a PD controller actuates the motors. We apply the Proximal Policy Optimization algorithm (PPO) [48] to maximize the cumulative discounted reward E T t=1 γ t-1 r t . In this work, we study the motion imitation task where our policy π OmniH2O is trained to track real-time motion input as shown in Figure 3 . This task provides a universal interface for humanoid control as the kinematic pose can be provided by many different sources. We define kinematic pose as q t ≜ (θ t , p t ), consisting of 3D joint rotations θ t and positions p t of all joints on the humanoid. To define velocities q1:T , we have qt ≜ (ω t , v t ) as angular ω t and linear velocities v t . As a notation convention, we use • to represent kinematic quantities from VR headset or pose generators, • to denote ground truth quantities from MoCap datasets, and normal symbols without accents for values from the physics simulation or real robot. Human Motion Retargeting. We train our motion imitation policy using retargeted motions from the AMASS [49] dataset, using a similar retargeting process as H2O [3] . One major drawback of H2O is that the humanoid tends to take small adjustment steps instead of standing still. In order to enhance the ability of stable standing and squatting, we bias our training data by adding sequences that contain fixed lower body motion. Specifically, for each motion sequence q1:T from our (b) Our sim-to-real policy is distilled through supervised learning from an RL-trained teacher policy using privileged information. (c) The universal design of OmniH2O supports versatile human control interfaces including VR headset, RGB camera, language, etc. Our system also supports to be controlled by autonomous agents like GPT-4o or imitation learning policy trained using our dataset collected via teleoperation.dataset, we create a "stable" version qstable 1:T by fixing the root position and the lower body to a standing or squatting position as shown in Fig. 2 . We provide ablation of this strategy in Appendix H.Reward and Domain Randomization. To train π privileged that is suitable as a teacher for a real-world deployable student policy, we employ both imitation rewards and regularization rewards. Previous work [18, 3] often uses regularization rewards like feet air time or feet height to shape the lowerbody motions. However, these rewards result in the humanoid stomping to keep balanced instead of standing still. To encourage standing still and taking large steps during locomotion, we propose a key reward function max feet height for each step. The max feet height for each step reward is designed to encourage the robot to take higher steps by rewarding it based on the maximum height achieved by the feet during the air phase of each step. We find that this reward, when applied with a carefully designed curriculum, effectively helps RL decide when to stand or walk. We provide a detailed overview of rewards, curriculum design, and domain randomization in Appendices E and F.Teacher: Privileged Imitation Policy. During real-world teleoperation of a humanoid robot, much information that is accessible in simulation (e.g., the global linear/angular velocity of every body link) is not available. Moreover, the input to a teleoperation system could be sparse (e.g., for VRbased teleoperation, only the hands and head's poses are known), which makes the RL optimization challenging. To tackle this issue, We first train a teacher policy that uses privileged state information and then distill it to a student policy with limited state space. Having access to the privileged state can help RL find more optimal solutions, as shown in prior works [50] and our experiments ( Section 4). Formally, we train a privileged motion imitator π privileged (a t |s p-privileged t , s g-privileged t ), as described in Figure 3 . The proprioception is defined as s p-privileged t ≜ [p t , θ t , qt , ω t , a t-1 ], which contains the humanoid rigidbody position p t , orientation θ t , linear velocity qt , angular velocity ω t , and the previous action a t-1 . The goal state is defined ass g-privileged t ≜ [ θt+1 ⊖ θ t , pt+1 - p t , vt+1 -v t , ωt+1 -ω t , θt+1 , pt+1 ],which contains the reference pose ( θt , pt ) and one-frame difference between the reference and current state for all rigid bodies of the humanoid.Student: Sim-to-Real Imitation Policy with History. We design our control policy to be compatible with many input sources by using the kinematic reference motion as the intermediate representation. As estimating full-body motion qt (both rotation and translation) is difficult (especially from VR headsets), we opt to control our humanoid with position pt only for teleoperation. Specifically, for real-world teleoperation, the goal state is s g-real t ≜ ( preal t -p real t , ṽreal t -v real t , preal t ). The superscript real indicates using the 3-points available (head and hands) from the VR headset. For other control interfaces (e.g., RGB, language), we use the same input 3-point input to maintain consistency, though can be easily extended to more keypoints to alleviate ambiguity. For proprioception, the student policy s p-real t ≜ (d t-25:t , ḋt-25:t , ω root t-25:t , g t-25:t , a t-25-1:t-1 ) uses values easily accessible in the real-world, which includes 25-step history of joint (DoF) position d t-25:t , joint velocity ḋt-25:t , root angular velocity ω root t-25:t , root gravity g t-25:t , and previous actions a t-25-1:t-1 . The inclusion of history data helps improve the robustness of the policy with our teacher-student supervised learning. Note that no global linear velocity v t information is included in our observations and the policy implicitly learns velocity using history information. This removes the need for MoCap as in H2O [3] and further enhances the feasibility of in-the-wild deployment.We train our deployable teleoperation policy π OmniH2O following the DAgger [51] framework: for each episode, we roll out the student policy π OmniH2O (a t |s p-real t , s g-real t) in simulation to obtain trajectories of (s p-real 1:T , s g-real 1:T ). Using the reference pose q1:T and simulated humanoid states s p 1:T , we can compute the privileged states s g-privileged t , s p-privileged t ← (s p t , qt+1 ). Then, using the pair (s p-privileged t , s g-privileged t), we query the teacher π privileged (a t privileged |s p-privileged t , s g-privileged t) to calculate the reference action a t privileged . To update π OmniH2O , the loss is: L = ∥a t privilegeda t ∥ 2 2 . Dexterous Hands Control. As shown in Figure 3 (c), we use the hand poses estimated by VR [52, 53] , and directly compute joint targets based on inverse kinematics for an off-the-shelf low-level hand controller. We use VR for the dexterous hand control in this work, but the hand pose estimation could be replaced by other interfaces (e.g., MoCap gloves [54] or RGB cameras [55] ) as well. 
experimental results: 
 In our experiments, we aim to answer the following questions. Q1. (Section 4.1) Can OmniH2O accurately track motion in simulation and real world? Q2. (Section 4.2) Does OmniH2O support versatile control interfaces in the real world and unlock new capabilities of loco-manipulation? Q3. (Section 4.3) Can we use OmniH2O to collect data and learn autonomous agents from teleoperated demonstrations? As motion is best seen in videos, we provide visual evaluations in our website.Experiment Setup. To answer Q1, we evaluate OmniH2O on motion tracking in simulation (Section 4.1.1) and the real world (Section 4.1.1). In simulation, we evaluate on the retargeted AMASS dataset with augmented motions Q (14k sequences); in real-world, we test on 20 standing sequences due to the limited physical lab space and the difficulty of evaluating on large-scale datasets in the real world. Detailed state-space composition (Appendix C), ablation setup (Appendix B), hyperparameters (Appendix K), and hardware configuration (Appendix A) are summarized in the Appendix. Metrics. We evaluate the motion tracking performance using both pose and physics-based metrics. We report Success rate (Succ) as in PHC [56] , where imitation is unsuccessful if the average deviation from reference is farther than 0.5m at any point in time. Succ measures whether the humanoid can track the reference motion without losing balance or lagging behind. The global MPJPE E g-mpjpe and the root-relative mean per-joint position error (MPJPE) E mpjpe (in mm) measures our policy's ability to imitate the reference motion globally and locally (root-relative). To show physical realism, we report average joint acceleration E acc (mm/frame 2 ) and velocity E vel (mm/frame) error.In Table 1 's first three rows, we can see that our deployable student policy significantly improves upon prior art [3] on motion imitation and achieves a similar success rate as the teacher policy.Ablation on DAgger/RL. We test the performance of OmniH2O without DAgger (i.e., directly using RL to train the student policy). In Table 1 (a) we can see that DAgger improves performance overall, especially for policy with history input. Without DAgger the policy struggles to learn a coherent policy when provided with a long history. This is due to RL being unable to handle the exponential growth in input complexity. However, the history information is necessary for learning a deployable policy in the real-world, providing robustness and implicit global velocity information (see Section 4.1.2). Supervised learning via DAgger is able to effectively leverage the history input and is able to achieve better performance.Ablation on History Steps/Architecture. In Table 1 (b), we experiment with varying history steps (0, 5, 25, 50) and find that 25 steps achieve the best balance between performance and learning efficiency. Additionally, we evaluate different neural network architectures for history utilization: MLP, LSTM, GRU and determine that MLP-based OmniH2O performs the best.Ablation on Sparse Input. To support VR-based teleoperation, π OmniH2O only tracks 3-points (head and hands) to produce whole-body motion. The impact of the number of tracking points is examined in Table 1 (c). We test configurations ranging from minimal (3) to full-body motion goal (22) and found that 3-point tracking can achieve comparable performance with more input keypoints. As expected, 3-point policy sacrifices some whole-body motion tracking accuracy but gains greater applicability to commercially available devices. Ablation on Global Linear Velocity. Given the challenges associated with global velocity estimation in real-world applications, we compare policies trained with and without explicit velocity information. In Table 1 (d), we find that linear velocity information does not boost performance in simulation, but it introduces significant challenges in real-world deployment (details illustrated in Section 4.1.2), prompting us to develop a policy with state spaces that do not depend on linear velocity as proprioception to avoid these issues. Appendix G), which suggests that the policy with history can effectively track motions without explicit linear velocity as input.History Steps and Architecture. Real-world evaluation in Table 2 (b) also shows that our choice of 25 steps of history achieves the best performance. The tracking performance of LSTM shows that MLP-based policy performs better in the real-world.To answer Q2, we demonstrate real-world capabilities of OmniH2O with versatile human control interfaces. All the capabilities discussed below utilize the same motion-tracking policy π OmniH2O .Teleoperation. We teleoperate the humanoid using π OmniH2O with both VR and RGB camera as interfaces. The results are shown in Figure 1 (a) and Appendix I, where the robot is able to finish dexterous loco-manipulation tasks with high precision and robustness. Language Instruction Control. By linking π OmniH2O with a pretrained text to motion generative model (MDM) [57] , it enables controlling the humanoid via verbal instructions. As shown in Figure 4 , with humans describing desired motions, such as "raise your right hand". MDM generates the corresponding motion goals that are tracked by the OmniH2O. Robustness Test. As shown in Figure 5 , we test the robustness of our control policy. We use the same policy π OmniH2O across all tests, whether with fixed standing motion goals or motion goals controlled by joysticks, either moving forward or backward. With human punching and kicking from various angles, the robot, without external assistance, is able to maintain stability on its own. We also test OmniH2O on various outdoor terrains, including grass, slopes, gravel, etc. OmniH2O demonstrates great robustness under disturbances and unstructured terrains.To answer Q3, we need to bridge the whole-body tracking policy (physical intelligence), with automated generation of kinematic motion goals through visual input (semantic intelligence). We explore two ways of automating humanoid control with OmniH2O: (1) using multi-modal frontier models to generate motion goals and (2) learning autonomous policies from the teleoperated dataset.GPT-4o Autonomous Control. We integrate our system, OmniH2O, with GPT-4o, utilizing a headmounted camera on the humanoid to capture images for GPT-4o (Figure 6 ). The prompt (details in Appendix M) provided to GPT-4o offers several motion primitives for it to choose from, based on the current visual context. We opt for motion primitives rather than directly generating motion goals because of GPT-4o's relatively long response time. As shown in Figure 6 , the robot manages to give the correct punch based on the color of the target and successfully greets a human based on the intention indicated by human poses. OmniH2O-6 Dataset. We collect demonstration data via VR-based teleoperation. We consider six tasks: Catch-Release, Squat, Rope-Paper-Scissors, Hammer-Catch, Boxing, and Pasket-Pick-Place. Our dataset includes paired RGBD images from the head-mounted camera, the motion goals Humanoid Learning from Demonstrations. We design our learning from demonstration policy to be π LfD ( pSparse-lfd t:t+ϕ |I t ), where π LfD outputs ϕ frames of motion goals given the image input I t . Here, we also include dexterous hand commands in pSparse-lfd t:t+ϕ . Then, our π OmniH2O (a t |s p-real t , s g-real t) serves as the low-level policy to compute joint actuations for humanoid whole-body control. The training hyperparameters are in Appendix L. Compared to directly using the π LfD to output joint actuation, we leverage the trained motor skills in π OmniH2O , which drastically reduces the number of demonstrations needed. We benchmark a variety of imitation learning algorithms on four tasks in our collected dataset (shown in Figure 7 ), including Diffusion Policy [58] with Denoising Diffusion Probabilistic Model [59] (DP-DDPM) and Denoising Diffusion Implicit Model [60] (DP-DDIM) and vanilla Behavior Cloning with a ResNet architecture (BC). Detailed descriptions of these methods are provided in Appendix D. To evaluate π LfD , we report the average MSE loss and the success rate in Table 3 , where we average the metrics across all tasks. More details for each task evaluation can be found in Appendix J. We draw two key conclusions: (1) The Diffusion Policy significantly outperforms vanilla BC with ResNet; (2) In our LfD training, predicting a sequence of actions is crucial, as it enables the robot to effectively learn and replicate the trajectory. 
limitations and future work: 
 Summary. OmniH2O enables dexterous whole-body humanoid loco-manipulation via teleoperation, designs universal control interfaces, facilitates scalable demonstration collection, and empowers humanoid autonomy via frontier models or humanoid learning from demonstrations.Limitations. One limitation of our system is the requirement of robot root odometry to transfer pose estimation from teleoperation interfaces to motion goals in the robot frame. Results from VIO can be noisy or even discontinuous, causing the motion goals to deviate from desired control. Another limitation is safety; although the OmniH2O policy has shown great robustness, we do not have guarantees or safety checks for extreme disturbances or out-of-distribution motion goals (e.g., large discontinuity in motion goals). Future work could also focus on the design of the teleoperation system to allow the humanoid to traverse stairs with only sparse upper-body motion goals. Another interesting direction is improving humanoid learning from demonstrations by incorporating more sensors (e.g., LiDAR, wrist cameras, tactile sensors) and better learning algorithms. We hope that our work spurs further efforts toward robust and scalable humanoid teleoperation and learning. substantial computational resources. By dedicating the first Orin NX to the ZED camera, and the second to policy inference, we ensure that each component operates with optimal performance. (2)In the second setup, a laptop (13th Gen i9-13900HX and NVIDIA RTX4090, 32GB RAM) serves as the computing and communication device. All devices, including the ZED camera, control policy, and Vision Pro, communicate through this laptop on its ROS system, facilitating centralized data handling and command dispatch. These two setups yield similar performance, and we use them interchangeably in our experiments.In this section, we provide an explanation of each ablation method. Main results in Table 1 • Privileged policy: This teacher policy π privileged incorporates all privileged environment information, along with complete motion goal and proprioception data in the observations. State space composition details in Table 4 .• H2O: A policy trained using RL without DAgger and historical data, utilizing 8 keypoints of motion goal in observations. State space composition details in Table 5 .• OmniH2O: Our deployment policy π OmniH2O that includes historical information and uses 3 keypoints of motion goal in observations, trained with DAgger. State space composition details in Table 6 .• OmniH2O-w/o-DAgger-History0: This variant of OmniH2O is trained solely using RL and does not incorporate historical information within observations. State space composition details in Table 7 .• OmniH2O-w/o-DAgger: This model is trained using RL, excludes DAgger, but includes historical information from the last 25 steps in observations. State space composition details in Table 8 .• OmniH2O-History0: This model is trained with DAgger, but excludes historical information from the last 25 steps in observations. State space composition details in Table 9 .• OmniH2O: This model is trained with DAgger and incorporates 25-step historical information within observations. State space composition details in Table 6 .Ablation on History steps/Architecture in Table 1 (b)• OmniH2O-History50/25/5/0: This variant of the OmniH2O with 50, 25, or 0 steps of historical information in the observations. State space composition details in Table 10 .• OmniH2O-GRU/LSTM: This version replaces the MLP in the policy network with either GRU or LSTM, inherently incorporating historical observations. State space composition details in Table 11 .• OmniH2O-22/8/3points: This variant of the OmniH2O policy includes 22, 8, or 3 keypoints of motion goal in the observations, with the 3 keypoints setting corresponding to the standard OmniH2O policy. State space composition details in Tables 6, 12 and 13.• OmniH2O-w-linvel: This variant is almost the same as OmniH2O but with root linear velocity in observations and past linear velocity in history information. State space composition details in Table 14 .In this section, we introduce the detailed state space composition of baselines in the experiments.Privileged Policy. This policy π privileged is the teacher policy that has access to all the available states for motion imitation, trained using RL. H2O. This policy has 8 keypoints input (shoulder, elbow, hand, leg) and with global linear velocity, trained using RL. OmniH2O. This is our deployment policy π OmniH2O with 25 history steps and without global linear velocity, trained using DAgger. OmniH2O-w/o-DAgger-History0. This policy has a history of 0 steps, trained using RL. OmniH2O-w/o-DAgger. This policy has the same architecture as OmniH2O but trained with RL. OmniH2O-History0. This policy has a history of 0 steps, trained using DAgger. OmniH2O-Historyx. This policy has a history of x steps, trained using DAgger. OmniH2O-GRU/LSTM. This policy uses GRU/LSTM-based architecture, trained using DAgger. OmniH2O-8points. This policy has 8 keypoints input (shoulder, elbow, hand, leg), trained using DAgger. OmniH2O-w-linvel. This policy has 25 history steps and global linear velocity, trained using DAgger. We conduct numerous ablation studies on LfD, aiming to benchmark the impact of various aspects on LfD tasks. The details of each ablation are as follows:Ablation on Dataset size.• 25/50/100% data: In this task, we use 25/50/100% of the dataset as the training set. The algorithm is DDPM which takes a single-step image as input and outputs 8 steps of actions.Ablation on Single/Sequence observation/action input/output.• Si-O-Si-A: Single-step observation and single-step action mean that we take 1 step of image data as input and predict 1 step of action as output.• Se-O-Se-A: Sequence-steps observation and sequence-steps actions mean that we take 4 steps of image data as input and predict 8 steps of action as output.• Si-O-Se-A: Single-step observation and sequence-steps actions mean that we take 1 step of image data as input and predict 8 steps of action as output.Ablation on Training Architecture..Behavior cloning which means we use resnet+MLP to predict the next 8 steps action from the current step's image.• DP-DDIM: We use DDIM as the algorithm which takes a single-step image as input and outputs 8 steps of actions.• DP-DDPM: We use DDPM as the algorithm which takes a single-step image as input and outputs 8 steps of actions.Reward Components. Detailed reward components are summarized in Table 15 . Reward Curriculum. We have modified the cumulative discounted reward expression to handle multiple small rewards at each time step differently, depending on their sign. The revised formula is given by: E T t=1 γ t-1 i s t,i r t,i where r t,i represents different reward functions at time t, and s t,i is the scaling factor for each reward, defined as:s t,i = s current if r t,i < 0 1 if r t,i ≥ 0 where s currentis the scaling factor. This scaling factor is adjusted dynamically: it is multiplied by 0.9999 when the average episode length is less than 40, and multiplied by 1.0001 when it exceeds 120. The init s current is set to 0.5, then the upper bound of this scaling factor is set to 1. This modification allows our policy to progressively learn from simpler to more complex scenarios with higher penalties, thereby reducing the difficulty for RL in exploring the optimal policy.Detailed domain randomization setups are summarized in Table 16 . The illustration of using the ZED camera VIO module and the comparison of VIO with neural state estimators are shown in Figure 8 . We train our neural velocity estimators using a supervised learning approach. The process involves repeatedly deploying our policy in simulation with different motion goals. In every environment step, we use the root linear velocity to supervise our velocity estimator. The ablation study on motion data distribution is shown in Figure 9 . The policy trained without motion data augmentation is hard to stand still and make upper-body moves. Additional VR-based and RGB-based teleoperation demo are shown in Figure 10 . As shown in Figure 11 , we collected 6 LfD tasks' dataset to enable the robot to autonomously perform certain functions. Catch-Release: Catch a red box and release it into a trash bin. This task has 13234 frames in total. Squat: Squat when the robot sees a horizontal bar approaching that is lower than its head height. This task has 8535 frames in total. Hammer-Catch: Use right hand to catch a hammer in a box. This task has 12759 frames in total. Rock-Paper-Scissors: When the robot sees the person opposite it makes one of the rock-paperscissors gestures, it should respond with the corresponding gesture that wins. This task has 9380 frames in total. Here is the example prompt we use for Autonomous Boxing task:You're a humanoid robot equipped with a camera slightly tilted downward on your head, providing a first-person perspective. I am assigning you a task: when a blue target appears in front of you, extend and then retract your left fist. When a red target appears, do the same with your right fist.If there is no target in front, remain stationary. I will provide you with three options each time: move your left hand forward, move your right hand forward, or stay motionless. You should directly respond with the corresponding options A, B, or C based on the current image. Note that, yourself is also wearing blue left boxing glove and right red boxing glove, please do not recognize them as the boxing target. Now, based on the current image, please provide me with the A, B, C answers.For Autonomous Greetings with Human Task, our prompt is:You are a humanoid robot equipped with a camera slightly tilted downward on your head, providing a first-person perspective. I am assigning you a new task to respond to human gestures in front of you. Remember, the person is standing facing you, so be mindful of their gestures.  
 
 Appendix:  


Decision: The reviewers talk positively about the paper. The reviewers highlight that this is one of the first papers doing full-body mimicry for humanoids. The paper does many different qualitative experiments, including tennis, drawing, watering plants, and picking objects in indoor and outdoor environments. The main weaknesses are

* that the paper makes unsubstantiated claims about improving over previous work and the experiment complexity. The statements mainly exaggerate the contributions and shy away from limitations.

* the clarity of the paper can be improved. Especially the paper is partially overloaded the additional GPT4o and imitation learning experiments provide little value / insights. The paper would have been better without these experiments. The freed space should be used to elaborate on the limitations. 

All in all, I agree with the reviewers and recommend accepting the paper. For the final submission, the authors should tone down the oversold statements and add a discussions about the limitations. There are still a lot of problems with the presented whole-body controller. The learned policies still have many undesirable properties such as tap dancing, the current controller kind of fails on dynamic tasks and all performed tasks do not require precise tracking. The paper does not need to solve all of these problems but it would be great to describe them and tell the reader where the performance would need to be improved and not list strawmen limitations in the limitation section.

Reviews: review: title: title: Review
paper_summary: paper_summary: The paper presents a humanoid whole-body control RL policy, OmniH2O, and integrates it with various teleoperation methods. The RL training reward, observation, and architecture are modified to remove the body velocity observation while improving tracking performance compared to prior work. The release includes a teleoperation dataset recorded using the controller.
main_review: main_review: Strengths:
- The universal interface supporting various input modalities (mocap, RGB, language) is a successful integration of many diverse components.
- The OmniH2O-6 dataset will provide robot data to the community for task/embodiment combinations that are not currently accessible.
- Compared to H2O, removing the requirement for mocap tracking during deployment allows the policy to be deployed outside of the lab.

Weaknesses:
- While improvements over H2O are claimed, the extent and source of these improvements are not clearly articulated.
- Robustness claims are not supported by comparisons or data.
- The role of history length and velocity estimation from history leave several open questions, see below.
- The demonstrated novelty from integrating various teleoperation interfaces with this controller is limited. The key capability of the controller is to imitate diverse whole-body motions, but the demonstrated behaviors might also be obtained using a decoupled controller for the legs/body.
questions: questions: A key claim of the paper is that OmniH2O "significantly improves upon prior art" H2O. Right now, it's hard to interpret which of the differences account for most of the performance. Is it possible to enumerate the exact difference between OmniH2O and H2O in a table with an ablation for each difference? As far as I can tell, the ablations (Table 1) already include the differences: [(1) number of keypoints, (2) lin vel observation], but there are also [(3) modified reward function, (4) modified domain randomization, (5) standing trajs added to the motion dataset, (maybe others?)]. Ablations 1 and 2 don't seem to explain much of the performance gap; can the improvement can be attributed to a specific design choice like a reward term? 

Relatedly in Table 1, H2O and OmniH2O are evaluated on dataset $\hat{Q}$. Is this a fair comparison since OmniH2O was trained on $\hat{Q}$ but H2O was trained on a different dataset $Q$ and therefore being evalauted out-of-distribution?

The difference between the student with no history (OmniH2O-History0) vs. training with history (OmniH2O) is only 0.3%. This difference seems small; is it statistically significant across multiple seeds of training? If not, is it valid to assert that history utilization is an essential element? As it stands, I would be inclined to draw the opposite conclusion from this table and say that history length seems to have no trend / irrelevant for the OmniH2O policy. Therefore, the role of linear velocity observation would be mainly in easing the learning dynamics, and it does not need to be reconstructed from the history for humanoid locomotion. This is somewhat surprising since other works on bipedal (https://arxiv.org/pdf/2401.16889) and even quadruped locomotion (e.g. https://arxiv.org/abs/2301.10602, https://arxiv.org/abs/2202.05481) found a history module important for velocity estimation. Can you report the statistical significance in Table 1 and comment on this interpretation of the results?

Following on this point, Table 2 suggests that omniH2O outperforms omniH2O-History0 by a wide margin in the real world despite nearly identical simulation performance. Does this suggest that using a history of observations helps mitigate the sim2real gap? What would explain this since there does not seem to be an obvious mechanism for the history length to influence the sim2real quality while both are enjoying high performance on the training data?

“OmniH2O demonstrates great robustness under disturbances and unstructured terrains.” “OmniH2O shows superior robustness against human strikes and different outdoor terrains.” Superior to what? I am not sure how the reader is supposed to evaluate these claims. Is the robustness better than Unitree’s default controller, better than H2O, or other or cited works like https://arxiv.org/abs/2402.16796 or https://arxiv.org/pdf/2303.03381 ? The disturbance rejection looks somewhat similar to me. The locomotion in all videos takes place on flat surfaces and slight inclines which I do not believe should be referred to as unstructured terrains. What would you consider a structured terrain?

Does the teleoperation from verbal instruction w/ gpt-4 have significant differences from SayTap (https://arxiv.org/pdf/2306.07580) that prompts a VLM to output motion parameters for a quadruped and Code as Policies https://arxiv.org/abs/2209.07753 that prompts a VLM to output code?

Is there a fundamental advantage of whole-body control over a decoupled approach? It seems like all behaviors could potentially be teleoperated with an interface like https://arxiv.org/abs/2402.16796 which separates control of the arms and legs while also allowing the user to specify a base height command for squatting. It would strongly motivate the coupled training if OmniH2O is capable of some more dynamic movements involving more evident coordination between the upper and lower body. The "embodiment feasible" motion dataset appears to contain several motions like kicking and jumping that involve this type of coordination. Was sim-to-real a limitation for these motions? If so, maybe one could add videos of the robot doing tracking these moves in simulation?
limitations: limitations: 2
review_summary: review_summary: The paper is a nice demonstration of humanoid teleoperation and the claims/experiments open many interesting questions. The paper's clarity and novelty would be enhanced by more details on the difference between OmniH2O and H2O, evaluating the statistical significance of the history related claims, clarifying robustness claims, and showing whole-body motions that are not possible via decoupled control.
title: title: Interesting and novel humanoid robotics sim2real demo with weaknesses in presentation and evaluation.
paper_summary: paper_summary: The paper describes an approach for training a humanoid controller that can mimic human motions. Simulated and real-world experiments are provided to demonstrate the controller's performance relative to a number of ablations. The controller is used to produce imitation learning data for several tasks and a policy is learned on top of the controller for each task.
main_review: main_review: Overall this is one of the first (among a couple others) works on sim2real learning of full-body humanoid mimicry. Thus, from an application perspective the novelty is high. 

While I enjoyed reading the paper, I found some of the technical descriptions to be difficult to understand. In addition, there are certain weaknesses in the empirical investigation that can be improved. Below I mention some of the key issues I identified. 

How did you select the squatting and standing pose used in the training? This could bias the system to less stable poses, which should ultimately depend on the task. More discussion of this choice would be useful. 

The main paper should provide at least some description of the reward term "max feet height for each step." since that is claimed to be crucial and is not self-explanatory. I tried to parse Appendix E that the authors pointed to, but it mostly provided factual details without motivation or definition of the symbols. 

I was not able to understand the rationale for the dynamic reward function described in the appendix. Is there a fundamental principle here that can be formalized more soundly? 

Why didn't you try to include the hand control in the overall learning framework? The videos showing hand movement are a bit misleading and this should be made clear early on in the paper. 

It would have been very interested to have at least a couple of non-standing real-world sequences. The additional dynamics of such sequences are non-trivial to deal with and it is important to know if the current approach has any fundamental limitations in that respect. Even a negative result would be interesting. The excuse given for not including such sequences (not enough lab space) is not very reasonable given videos already shown. 

Many of the values in Table 1 are quite similar, including those that are marked in bold as somehow significantly better. I'm not sure I can agree with conclusions regarding the relative performance of some of these choices. These are also hard to evaluate since they are apparently averaged over all 14,000 sequences.  Was there any randomization in the experiments (e.g. multiple randomizations for each sequence)? If so, the confidence intervals could be given? 

The OmniH20 NN architecture was not mentioned until the experimental section. This should be mentioned earlier. 

I did not understand the "ablation to sparse input" section. Does your OmniH20 architecture allow inputs with different types of sparsity or do you need to train a student for each sparsity you consider. I tried to figure this out from the paper text, but couldn't piece things together. 

How were the real-world experiments conducted? Was there just one trial of each motion? Why isn't there a success rate here?

How does Figure 5 show "superior robustness"? Compared to what? There is nothing quantitative here upon which to draw a conclusion.

Post Rebuttal:

I read the rebuttal to my review an the other reviews. I can only say that while I still appreciate the work, because there is very little that has been in this area so far, I now appreciate the limitations more. 

I feel that the submission is overselling the demonstrated capabilities of this approach in a number of ways consistent with my questions. The submitted video of simulated dynamic motions to one of the other reviewers was not very convincing --- it clearly would not transfer to hardware. The video submitted of the 20 real-world tests was a bit of a mystery to me. Maybe that was just one motion. 

I would say that the authors need to be much more upfront from the start about the lack of actual dynamic motion demonstrations, which are a major challenge. Otherwise, it is very easy for people to get a false sense of the state-of-the-art and actually not recognize future innovations when they occur---think this to be a solved problem. 

Accordingly, I downgraded my ranking to a "weak accept".
questions: questions: Is there a fundamental principle regarding the non-stationary reward formulation described in the appendix?

Why didn't you try to include the hand control in the overall learning framework? 

What can you say about the statistical significance of results in Table 1 (see comment in review)? 

How does your approach deal with different levels of sparsity in the input ("ablation to sparse input")? 

How were the real-world experiments conducted? Why isn't there a success rate here?
limitations: limitations: 2
review_summary: review_summary: Overall the paper has a good degree of novelty in the sense that we are at the leading edge of  sim2real learning for humanoids. The clarity of the paper can be improved as well as aspects of the evaluation. However, the novelty of the demonstrations makes up for these weaknesses.
title: title: Interesting application of general humanoid teleoperation policy, but lack of advancement compared with previous works.
paper_summary: paper_summary: This paper introduces a bi-level autonomous humanoid policy that can be commanded by vocal instructions. The low-level policy can be operated by human operator by teleoperation. The high-level policy can be deployed using GPT-4o, MGM, or a trained diffusion policy. The author demonstrates the training framework of the low-level policy and releases the dataset OmniH2O-6 dataset to train the high-level diffusion policy.
main_review: main_review: This paper describes the bi-level framework of humanoid policy that can be commanded by vocal instructions. The writing is clear and the technical details are well-organized. The author provides a clear motivation. However, compared with previous works, the advancement of this work is not addressed clearly. The author should provide a clear comparison with previous works or address the advancement of this work with more detailed information.

**strength**

* The author provides a clear motivation and well-organized technical details.

* The author provides a clear description of the training framework of the teleoperation policy.

* Clear illustration on using GPT-4o for the full-autonomous tasks.

**weakness**

* The research advancement is not clear compared with previous works.

* Details of learning from demonstrations are not sufficient in the manuscript.

* Lack of thorough investigation on the performance of different methods in learning from demonstrations.

* Lack of details on the OmniH2O-6 dataset. For example, how is the lower-limb motion captured when collecting the dataset using VR? In the training section of the teleoperation policy, the policy is designed to follow the lower-limbs motion.
questions: questions: 1. What is the reference trajectory when performing outdoor locomotion tests?

2. Why distillation is needed for teleoperation policy? 

3. Is it possible to use a more accurate odometry system to provide the linear velocity of the robot to achieve a better performance? For example, using an in-door tracking system for a proof of concept.
limitations: limitations: 2
review_summary: review_summary: The advancement of this work is not clear compared with previous works. I recommend a weak reject on this paper.