OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
Authors: Tairan He
Zhengyi Luo
Xialin He
Wenli Xiao
Chong Zhang
Weinan Zhang
Kris M. Kitani
Changliu Liu
Guanya Shi
Venue: CoRL 2024 (2024)
Keywords: Humanoid Teleoperation
Humanoid Loco-Manipulation
RL
TL;DR: We propose OmniH2O, a learning-based system for whole-body humanoid teleoperation and autonomy.

Abstract: We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets. Videos at the anonymous website [https://anonymous-omni-h2o.github.io/](https://anonymous-omni-h2o.github.io/)

OmniH20: Universal and Dexterous Human-to-
Humanoid Whole-Body Teleoperation and Learning
Tairan Het!, Zhengyi Luo"l, Xialin H Wenli Xiao', Chong Zhang',
Weinan Zhang", Kris Kitani', Changliu Liu', Guanya Shi'
Carnegie Mellon University SShnggaii Jiao Tong University 'Equal Contributions
Page: https: oomi hwwannhumaneeiiiommcmco id. com
Code: https:// www.coo////////o. LeCAR Lab /human2humanoid
(a) Teleoperation
(b) Agent
OmniH20 enables telcoperating a full-size humanoid robot (Unitree HI) to complete tasks that require both high-precision manipulation and locomotion. (b) OmniH20 also enables full autonomy through visual input, controlled by GPT-4o or a policy learned from teleoperated demonstrations. Videos: see our website: http://omnihummanhmmaaoi.... humanhhumamoid.ccmm
Figure 1:
We present OmniH20 (Omni Hluman-to-Humannqe a learning-base based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as universal control interface, OmniH2O enables various ways for a hu- man control full-sized humanoid with dexterous hands, including using real- time teleoperation through VR headset, verbal instruction, and RGB camera. Om- niH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such GPT-4o. OmniH20 demonstrates ver- satility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans, as shown in Figure We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning real-world deployable policy with sparse input by imitating a privileged teacher policy, and reward designs enhance bustness and stability. We release the first humanoid whole-body control dataset, OmmiH20-6, containing six everyday tasks, and demonstrate humanoid whole- body skill learning from teleoperated datasets.
Abstract:
Keywords: Humanoid Telcoperation, Humanoid Maniplaaipll,iion, RL
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
How can we best unlock humanoid's potential one of the most promising physical embodiments of general intelligence? Inspired by the recent success of pretrained vision and language models [1], one potential answer is to collect large-scale human demonstration data in the real world and learn humanoid skills from it. The embodiment alignment between humanoids and humans not only makes the humanoid a potential generalist platform but also enables the seamless integration of human cognitive skills for scalable data collections [2. 3, 4, 5].
However, whole-body control of full-sized humanoid robot is challenging [6], with many existing works focusing only on the lower body [7, 8, 9, 10, 11, 12, 13] or decoupled lower and upper body control [14, 4, 15J. To simultaneously support stable dexterous manipulation and robust locomotion, the controller must coordinate the lower and upper bodies unison. For the humanoid teleoperation interface [2], the need for expensive setups such as motion captures and exoskeletons also hinders large-scale humanoid data collection. In short, we need robust control policy that supports whole- body dexterous loco-manipulation, while seamlessly integrating with easy-to-use and accessible teleoperation interfaces (e.g., VR) to enable scalable demonstration data collection.
In this work, we propose OmniH20, a learning-basee system for whole-body humanoid teleoper- ation and autonomy. We propose a pipeline train robust whole-body motion imitation policy via teach-student distillation and identify key factors in obtaining stable control policy that sup- ports dexterous manipulation. For instance, distribution, reward designs, and state space design and history utilization. The distribution of the motion imitation dataset needs to be biased toward standing and squatting to help the policy learn to stabilize the lower body during manipulation. Regularization rewards are used to shape the desired motion but need to be applied with a curriculum. The input history could replace the global linear velocity, an essential input previous work [3] that requires Motion Capture (MoCap) to obtain. We also carefully design our control interface and choose the kinematic pose an intermediate representation to bridge between human instructions and humanoid actuation. This interface makes our control framework compatible with many real-world input sources, such as VR, RGB cameras, and autonomous agents (GPT-4o). Powered by our robust control policy, we demonstrate telcop- crating humanoids to perform various daily tasks (racket swinging, flower watering, brush writing, squatting and picking, boxing. basket delivery, etc.), as shown in Figure 1. Through teleoperation, we collect a dataset our humanoid completing six tasks such as hammer catching, basket picking, etc., annotated with paired first-person RGBD camera views, control input, and whole-body motor actions. Based the dataset, we showcase training autonomous policies via imitation learning.
find these elements to be essential: motion data
In conclusion, our contributions are as follows: (1) We propose pipeline to train robust humanoid control policy that supports whole-body dexterous loco- mic-mmnnpullttion with a universal interface that enables versatile human control and autonomy. (2) Experiments of large-scale motion tracking in simulation and the real world validate the superior motion imitation capability of OmniH20. (3) We contribute the first humanoid docommanipulation dataset and evaluate imitation learning methods on it to demonstrate humanoid whole-body skill learning from teleoperated datasets.
2 Related Works
Learning-based Humanoid Control. Controlling humanoid robots a long-standing robotic prob- lem due their high degree-of freedom
offree-of-freedom (DoF) and lack of self stlf-stabilizztion [16, 17]. Recently, leasning-based methods have shown promising results [7, 8, 9, 10, 11, 12, 13, 18]. However, most studies [7, 8, 9, 10, 11, 12, 13] focus mainly on learning robust locomotion policies and do not fully unlock all the abilities of humanoids. For tasks that require whole-body
coo-mmnipplltion, the lower body must serve as the support for versatile and precise upper body movement [19]. Traditional goal- reaching [20, 21] or selocity-tracking objectives [18] used in legged locomotion are incompatible with such requirements because these objectives require additional task-specific lower-body goals (from other policies) to indirectly account for upper-lower-book body coordination. OmniH20 instead learns end-to-end whole-body policy to coordinate upper and lower bodies.

---

Humanoid Teleoperation. Teleoperating humanoids holds great potential in unlocking the full capabilities of the humanoid system. Prior efforts in humanoid teleoperation have used task- teleoperation [23, 24] and whole-body teleopera- tion [25, 26, 27, 28, 29, 30, 3,4) Recently, H2O [3] presents an RL-based whole-body teleoperation framework that uses a third- third-person RGB camera to obtain full-body keypoints of the human telcop- erator. However, due to the delay and inaccuracy of RGB-based pose estimation and the requirement for global linear velocity estimation, H2O [3] requires MoCap during test time, only supports sim- ple mobility tasks, and lacks the precision for dexterous manipulation tasks. By contrast, OmniH20 enables high-precision dexterous loco-manipulation indoors and in the wild.
space control [4, 22], hnper-body-retargeed
Whole-body Humanoid Control Interfaces. To control a full-sized humanoid, many interfaces such as exoskeleton [31], MoCap [32, 33), and VR [34, 35] are proposed. Recently, VR-based humanoid control [36, 37, 38, 39] has been drawing attention in the graphics community due to its ability to create whole-body motion using sparse input. However, these VR-based works only focus on humanoid control for animation and do not support mobile manipulation. OmniH20, on the other hand, can control a real humanoid robot to complete real-world manipulation tasks.
Open-sourced Robotic Dataset and Imitation Learning. One major challenge within the robotics community is the limited number of publicly available datasets compared to those for language and vision tasks [40]. Recent efforts [40, 41, 42, 43, 44, 45, 46, 47] have focused on collecting robotic data using various embodiments for different tasks. However, most of these datasets are collected with fixed-base robotic arm platforms. Even one of the most comprehensive datasets to date, Open X-Embodiment [40], does not include data for humanoids. To the best of our knowledge, we are the first to release a dataset for full-sized humanoid whole-body doco-manipulation.
Universal and Dexterous Human-to-Huumnooid Whole-Body Control
In this section, we describe our whole-body control system to support teleoperation, dexterous ma- nipulation, and data collection. As simulation has access inputs that are hard obtain from real-world devices, opt elements to obtain stable and robust control policy: dataset balance, reward designs, etc.
teacher-student framework. We also provide details about key
Problem Formulation. We formulate the learning problem as goal-conditioned RL for a Markov Decision Process (MDP) defined by the tuple M = (S, A, 7, R, 7) of state S, action a, A, transition T, reward function R, and discount factor 7. The state 8, contains the proprioception sf and the goal state sy. The goal state sf includes the motion goals from the human teleoperator or autonomous agents. Based proprioception si, goal state sf. and action a, define the reward = R (s), sf, a.). The action a, specifies the target joint angles and PD controller actuates the motors. We apply the Proximal Policy Optimization algorithm (PPO) [48] to maximize the cumulative discounted reward E ET=1" (rr In this work, we study the motion imitation task where policy T0mmiH20 is trained to track real-time motion input shown in Figure 3. This task provides a universal interface for humanoid control as the kinematic pose can be provided by many different sources. We define kinematic pose as Q. (0:. consisting of 3D joint rotations 0 and positions P, of all joints on the humanoid. To define velocities J1:7, we have d A (w,, v.) as angular W; and linear velocities v;. As a notation convention, we use to represent kinematic quantities from VR headset or pose generators, T to denote ground truth quantities from MoCap datasets, and normal symbols without accents for values from the physics simulation or real robot.
Human Motion Retargeting. We train our motion imitation policy using retargeted motions from the AMASS [49] dataset, using a similar retargeting process as H2O [3]. One major drawback of H2O is that the humanoid tends to take small ad- justment steps instead of standing still. In order to enhance the ability of stable standing and squatting, we bias our train- ing data by adding sequences that contain fixed lower body motion. Specifically, for each motion sequence श1.r from our
Figure 2: (a) Source motion; (b) Retar- geted motion; (c) Standing variant; (d) Squatting variant.
3

---

(a) Human Motion Retargeting
(b) Sim-to-Real Training
Controller (anodi))
Largescale Motion Dalasel (Raw)
Simulation
Privileyed Proprioception
Privillegdd
Imitation Policy
Privileged Motion
Syntiurt
Phase
552 dan
Reinforeement
Learning
Retargeting Feasibiliity Filter Motion Datnoet (Kmboodiment Feasible)
Phose
Imitatiin Goal
Sim-to-Real Proprioception
Superuised
Sim-to-Real Imitaaiin Pollcyy
Learning
4,
Sim Real Goal
Foratio galta
(c) Universal Real-world Teleoperation and Learning
Versatile Human Control Interface
Sim Real Humanoid Deployment
Physical Teleogeration
Motion Goal
Proprioception
Body Pose
PD Coutroller
Right
Sim Roal Imitation Policy
VR
BUB
Joysticks
Emwdieletor
MiKcpp
Instruction
Human Morion Generative Model
"Uft your left hand above your shoulder."
Last Hined
Inverse Kinematics
Hand Controller
(MDM)
Human
Exteroception
Autonomous Agent
Diffusion Policy R
Telesperated Datasel Collection
Model
Prompt Example
"You are humonois robot cauioped with a commin iilted downoard on your head... Rirpand to humor gestures descriptons 99 numbers to desired loft/ight and 3D pasitio..."
Propriocaption
Omnil1z0-6
Exteroceptoon
front
Motion Goals
Figure 3: (a) OmniH20 retargets large-scale human motions and filters out infeasible motions for humanoids. (b) Our sim-to-real policy distilled through supervised learning from an RL-trained teacher policy using privileged information. (c) The universal design of OmniH20 supports versatile human control interfaces including VR headset, RGB camera, language, etc. Our system also supports to be controlled by autonomous agents like GPT-4o or imitation learning policy trained using our dataset collected via teleoperation.
dataset, we create a "stable" version gstable by fixing the root position and the lower body to a stand- ing or squatting position as shown in Fig. 2. We provide ablation of this strategy in Appendix H.
Reward and Domain Randomization. To train Triivileged that is suitable teacher for real-world deployable student policy, we employ both imitation rewards and regularization rewards. Previous work [18, 3] often uses regularization rewards like feet air time or feet height to shape the lower- body motions. However, these rewards result in the humanoid stomping to keep balanced instead of standing still. To encourage standing still and taking large steps during locomotion, we propose a key reward function max feet height for each step. The max feet height for each step reward is designed to encourage the robot to take higher steps by rewarding it based on the maximum height achieved by the feet during the air phase of each step. We find that this reward, when applied with a carefully designed curriculum, effectively helps RL decide when to stand or walk. We provide a detailed overview of rewards, curriculum design, and domain randomization in Appendices E and F.
Teacher: Privileged Imitation Policy. During rorldworld teleoperation of a humanoid robot, much information that is accessible in simulation (e.g., the global linear/angular velocity of every body link) is not available. Moreover, the input to a teleoperation system could be sparse (e.g.. for VR- based teleoperation, only the hands and head's poses are known), which makes the RL optimization challenging. To tackle this issue, We first train a teacher policy that uses privileged state information and then distill it to a student policy with limited state space. Having access to the privileged state can help RL find more optimal solutions, as shown in prior works [50] and our experiments ( Section 4). Formally, we train a privileged motion imitator Tprivileged (a, |s. peppiiileegedd , sd-privllged) as
described in Figure 3. The proprioception is defined as s--privileged & [p,, 0,, qr, wi, ax-1), which contains the humanoid rigidbody position Pt. orientation 0r, linear velocity dt. angular velocity wi, and the previous action a;-1- The goal state is defined as sy privileged A 10.41 @ Pe.1 P,, 0.+1 U. 0 - W,, difference between the reference and current state for all rigid bodies of the humanoid.
which contains the reference pose (0,,) p:) and one-frame
Student: Sim-to-Real Imitation Policy with History. We design our control policy to be compat- ible with many input sources by using the kinematic reference motion the intermediate represen- tation. As estimating full-body motion qe (both rotation and translation) is difficult (especially from

---

VR headsets), we opt to control our humanoid with position p. only for teleoperation. Specifically, for real-world telepppation,,,,,, the goal state is sf
sg-real - (f);aa -pt - ). The superscript real indicates using the 3-points available (head and hands) from the VR headset. For other control in- terfaces (e.g., RGB, language), we use the same input 3-point input to maintain consistency, though can be easily extended to more keypoints alleviate ambiguity. For proprioception. the student policy sp-ical (d4-25::, d 25:2; (t : 9t-25:t, (t-25-1:t-1) uses values easily accessible in the real-world, which includes 25-step history of joint (DoF) position dz-25:t- joint velocity d2-25st root angular velocity w/-23:1, root gravity 9t-25:ts and previous actions 0t-25-1:t-1 The inclu- sion of history data helps improve the robustness of the policy with our teacher-student supervised learning. Note that global linear velocity 82 information is included in our observations and the policy implicitly learns velocity using history information. This removes the need for MoCap in H2O [3] and further enhances the feasibility of in-the-wild deployment.
We train our deployable teleoperation policy TOmmiH20 follow- ing the DAgger [51] framework: for each episode, we roll out the student policy TOmmizzo (f ,8 )-real) in simulation to obtain trajectories of (sprral $1T s--eal). Using the ref- erence pose Q1:T and simulated humanoid states s we can compute the privileged states sf-prvillged of. ef-priviloged + (sf. Q(+1). Then, using the pair (sf- of )f-privllege), we query the teacher Tprivileged (a privilege |s. privileee St p-prrivilegdd to calculate the reference action a, privileged To update 70mil1120 the loss is: -= ||a, privileged - arl3.
Policy Distillation.
Dexterous Hands Control. As shown in Figure 3(c), we use the hand poses estimated by VR [52, 53], and directly compute joint targets based inverse kinematics for off-the-shelf low-level hand controller. We use VR for the dexterous hand control in this work, but the hand pose estimation could be replaced by other interfaces (e.g., MoCap gloves [54] or RGB cameras [55]) as well.
4 Experimental Results
In our experiments, we aim to answer the following questions. Q1. (Section 4.1) Can OmniH20 accurately track motion in simulation and real world? Q2. (Section 4.2) Does OmniH20 support versatile control interfaces in the real world and unlock new capabilities of loco-manipulation? Q3. (Section 4.3) Can we use OmniH20 to collect data and learn autonomous agents from teleoperated demonstrations? As motion is best seen in videos, we provide visual evaluations in our webs ite.
4.1 Whole-body Motion Tracking
Experiment Setup. To answer Q1, evaluate OmniH20 on motion tracking in simulation (Sec- tion 4.1.1) and the real world (Section 4.1.1). In simulation, evaluate the retargeted AMASS dataset with augmented motions Q (14k sequences); in real-world, we test 20 standing sequences due to the limited physical lab space and the difficulty of evaluating on aarge-scale datasets in the real world. Detailed state-space composition (Appendix C), ablation setup (Appendix B), hyperpa- rameters (Appendix K), and hardware configuration (Appendix A) are summarized in the Appendix. Metrics. We evaluate the motion tracking performance using both pose and thysics-based met- rics. We report Success rate (Succ) as in PHC [56], where imitation is unsuccessful if the average deviation from reference farther than 0.5m at any point in time. Suce measures whether the hu- manoid can track the reference motion without losing balance or lagging behind. The global MPJPE Eg-mpipe and the root-relative mean per-joint position error (MPJPE) Empipe (in mm) measures our policy's 's ability imitate the reference motion globally and locally (root-relative). To show physical realism, we report average joint acceleration Ence (mm/frame") and velocity Evel (mm/frame) error.
4.1.1 Simulation Motion-Tracking Tracking Results
In Table I's first three rows, we can see that our deployable student policy significantly improves upon prior art [3] on motion imitation and achieves a similar success rate as the teacher policy.
Ablation DAgger/RL. We test the performance of OmniH20 without DAgger (i.e., directly using RL to train the student policy). In Table ((a) we can see that DAgger improves performance
5

---

Table 1: Simulation motion imitation evaluation of OmniH20 and baselines on dataset Q
Note that all the
variants are trained with exact same rewards, domain randomizations and motion dataset Q
All sequences
Successful sequences
Method
State Dimension Sim2Real
Suce 1 Exuerre
Emgis
Eas
Eval
Eggapee
Emprr
Euc
E,g
Privileged policy
S R
94.77%
126.51
70.68
3.57
6.20
122.71
69:06
H20 131
8 C. R
87.52%
148.13
81.06
5.12
7.89
133.25
75,99
2.40
5.75
OmniH20
8 1906
94.10%
141.11
77.82
3.70
6.54
135.49
75.75
2.30
5.47
(a) Ablation on DAgger/RL
OmmiH22-wwo-Dggeee www.---Agee--sstoryy
S R200
90.62%
163.44
91.29
5.12
8.80
153.31
87.59
3.15
7.27
SomiH20-wwo-DAgee
R1005
11%
223.27
128.90
16.29
182.15 13
119.54
9.10
0mnil120 HistoryO
8 R
93.80%
141.21
78.52
3.74
6.62
134,90
76. 11
2.25
5.48
Ommil120
8 R1066
141.11
77.82
3.70
6.54
(b) Ablation on History stepa/Architecttre
OmmiHH2-Hissoryyt
S C R-220
93.56%
141.51
78.51
4.01
6.79
135.
76.07
2.36
5.55
0mmil120 History5
S C R-00
93.60%
139.23
77.82
3.91
6.66
132.67
75.33
2.24
5.41
S R
93.80%
141.21
78.52
3.74
6.62
134.90
76. 11
2.25
5.48
OmmilH20-GBB
Riu
92.85%
147.67
80.84
6.93
10
0mmil120 LSTM
8 R250
91.03%
147.36
80.34
4.12
7.04
142. 64
78.59
2.37
5.72
Ommiit20-Hissory255 (Ours)
S R106
141.11
77.82
3.70
(c) Ablation on Tracking Points
OmmiH20. 220onnss
8 R
94.72%
127.71
70.39
3.62
6.25
123.87
68.92
2.22
5.34
0mmil120- Spoints
S R47L0
94.31%
129.30
71.70
3.78
6.39
125.14
70.07
2.22
5.26
OmmiHH20-3points (Ours)
S R
94.10%
141.11
77.82
3.70
6.54
135,49
75.75
2.30
5.47
Ablation Linear Velocity
OmniH220-w-livel
8 c R
93. 80%
138.18
78.
3.94
6.61
132.44
75.98
2.29
5.40
OmmiH20
S R100 1905
94.10%
141.11
77.82
3.70
6.54
135.49
75.75
2.30
5.47
overall, especially for policy with history input. Without DAgger the policy struggles to learn coherent policy when provided with long history. This is due to RL being unable to handle the exponential growth in input complexity. However, the history information is necessary for learning a deployable policy in the real-world, providing robustness and implicit global velocity information (see Section 4.1.2). Supervised learning via DAgger is able to effectively leverage the history input and is able to achieve better performance.
Ablation on History Steps/Arccitecture. In Table 1(b), we experiment with varying history steps (0, 5, 25, 50) and find that 25 steps achieve the best balance between performance and learning efficiency. Additionally, we evaluate different neural network architectures for history utilization: MLP, LSTM, GRU and determine that MLP-based OmniH20 performs the best.
A Ablation on Sparse Input. To support VR-based teleoperation, T0mmiH20 only tracks 3-points (head and hands) produce whole-body motion. The impact of the number of tracking points is examined in Table Icc). We test configurations ranging from minimal (3) to full-body motion goal (22) and found that 3-point tracking can achieve comparable performance with more input keypoints. As expected, 3-point policy sacrifices some whole-body motion tracking accuracy but gains greater applicability to commercially available devices.
Ablation on Global Linear Velocity. Given the challenges associated with global velocity esti- mation in real-world applications, we compare policies trained with and without explicit velocity information. In Table 1(d), we find that linear velocity information does not boost performance simulation, but introduces significant challenges in real-world deployment (details illustrated in Section 4. 4.2,,, prompting us develop policy with state spaces that do not depend on linear velocity as proprioception to avoid these issues.
Table 2: Real-world motion tracking evaluation on 20 standing motions in Q
4.1.2 Real-world Motion-Trackinn Results
Tesea sequeeooss
Ablation on Real-world Linear Velocity Esti- mation. We exclude linear velocity in state space design global linear velocity obtained by algorithms such as visual inertial odome- try (VIO) can be rather noisy, as shown in Ap- pendix G. Our ablation study (Table 2(a)) also shows that policies without velocity input has better performance compared with policies us-
Method
Stose Dimensints
Ageepo
Auggs L. Eice 4 Esd
H20 [3]
Sc.. 32
87.33
53.32
6,113
5.87
OmmiH20
C 32 MM
47.94
41,87
1.84
2.20
Ablution Real- world
Velucity Estimation
Von 28 12774
V/A
Omnif w-immelGRU)
RiTes
123745
50.93
2.36
Poe
Riee
47.94
(b) Ablation on History depp/Aveehitectture
Omail120 HissoryO 120 Hissoyy
5 c g30
$3.25
46.00
4.86
4.45
62.18
46.50
2.65
2.90
OmilH3O-Hanoayh LSTM
325000
50.24
41.11
2.37
2.71
C. 3200
87.00
46.06
3.89
3.88
ing velocities estimated by VIO MLP/GRU neural estimators (implementation details in
OmsiH20
C 32100
47.94
41.87
1.84
2.20
Use ZED SDK to estimate the lincer velocity.
Unable to tinish the roal- world test due to fulling the ground

---

Figure OmniH20 policy tracks motion goals from language-based human motion generative model [57].
(a) Disturbances
(b) Outdoor Terrains
Figure 5: Omnil120 shows superior robustness against human strikes and different outdoor terrains.
Appendix G), which suggests that the policy with history can effectively track motions without explicit linear velocity imput.
History Steps and Architecture. Real-world evaluation in Table 2(b) also shows that our choice of 25 steps of history achieves the best performance. The tracking performance of LSTM shows that MLP-based policy performs better in the real-world.
4.2 Human Control via Universal Interfaces
To answer Q2, we demonstrate real-world capabilities of OmniH20 with versatile human control interfaces. All the capabilities discussed below utilize the same srtiiinntaacing policy T0miil12- Teleoperation. We teleoperate the humanoid using #0mmi220 with both VR and RGB camera as interfaces. The results are shown in Figure ((a) and Appendix where the robot is able to finish dexterous loco-manipulation tasks with high precision and robustness.
Language Instruction Control. By linking T0mmiH20 with a pretrained text to motion generative model (MDM) [57], enables controlling the humanoid via verbal instructions. As shown in Fig- ure 4, with humans describing desired motions, such as "raise your right hand". MDM generates the corresponding motion goals that tracked by the OmniH20.
Robustness Test. As shown in Figure 5, we test the robustness of our control policy. We use the same policy TommHHO across all tests, whether with fixed standing motion goals or motion goals controlled by joysticks, either moving forward backward. With human punching and kicking from various angles, the robot, without external assistance, is able maintain stability its own. We also test OmniH20 on various outdoor terrains, including grass, slopes, gravel, etc. OmniH20 demonstrates great robustness under disturbances and unstructured terrains.
4.3 Autonomy via Frontier Models or Imitation Learning
To answer Q3, we need to bridge the whole-body tracking policy (physical intelligence), with au- tomated generation of kinematic motion goals through visual input (semantic intelligence). We explore two ways of automating humanoid control with OmniH20: (1) using multi-mode frontier models generate motion goals and (2) learning autonomous policies from the teleoperated dataset.
GPT--o Autonomous Control. We integrate our system, OmniH20, with GPT-4o, utilizing a head- mounted camera on the humanoid to capture images for GPT-4o (Figure 6). The prompt (details in Appendix M) provided to GPT-4o offers several motion primitives for it to choose from, based on the current visual context. We opt for motion primitives rather than directly generating motion goals because of GPT-4o's relatively long response time. As shown in Figure 6, the robot manages to give the correct punch based the color of the target and successfully greets human based on the intention indicated by human poses.
Omnil20066 Dataset. We collect demonstration data via VR-based telcoperation. We consider six tasks: Catch-Ree Release Squat, Rope-Paper- Scisssos,, Hammer-Cate Boxing, and Pasket-Pick Place. Our dataset includes paired RGBD images from the head-mooe mounted camera, the motion goals

---

(a) Autonomous Boxing
(b) Autonomoux Greetings with Human
Figure OmniH20 sends egocentric RGB views to GPT-4o executes the selected motion primitives.
Catch Release
Squat
(c)
(c) Paper Scissor
Figure 7: Omnil120 autonomously conducts four tasks using LID models trained with our collected data.
of HI's head and hands with respect the root, and joint targets for motor actuation, recorded 30Hz. For simple tasks such minutes of data are recorded, and for tasks like Hammer-Cater and approximately 10 minutes, leading to 40-min real-world humanoid teleoperated demonstrations in total. Detailed task descriptions of the six open-sourced datasets are in Appendix J.
Squat, and
approximately 5 we collect
Humanoid Learning from Demonstrations. We design our learning from demonstration policy be (p fid {
Table 3: Quantitative LID average per- formance tasks over 10 runs.
where LLID outputs & frames of motion goals given the im-
Metrics
All Tasks
age input I. Here, we also include dexterous hand commands in P p Then, our T0miiH20 (a s s g.-rell) serves as
(a) Ablation Data size
p-real
d5%
50%
100%dat
the low-level policy to compute joint actuations for humanoid whole-body control. The training hyperparameters are in Appendix I.. Compared to directly using the TLD to out- put joint actuation, we leverage the trained motor skills in T0mmiH20. which drastically reduces the number of demonstra- tions needed. We benchmark variety of imitation learning algorithms on four tasks in our collected dataset (shown in Figure 7), including Diffusion Policy [58] with Denoising Dif- fusion Probabilistic Model [59] (DP-DDPM) and Denoising Diffusion Implicit Model [60] (DP- DDIM) and vanilla Behavior Cloning with a ResNet architecture (BC). Detailed deseriptions of these methods are provided in Appendix To evaluate ALDD we report the average MSE loss and the success rate in Table 3, where we average the metrics across all tasks. More details for each task evaluation can be found in Appendix J. We draw two key conclusions: (1) The Diffusion Policy significantly outperforms vanilla BC with ResNet; (2) In our LfD training, predicting sequence of actions is crucial, as it enables the robot to effectively learn and replicate the trajectory.
MSE Loss
1.30E-2 4/10
7,48-33
5,25E-4
Succ rale
6.5/10
8/00
(b) Ablation on Sequence observationsaaction
Si-O-Si-A
Se-O-Se-A
Se-O.Se-A
MSE
4.89E- 6,5/10
9.91E-4
5.25E-4
Succ rale
8.75/10
8/10
(c) A Ablation on RCDDIMDDDMM
BC
DF-DDIM
DP-DDPM
MSE Loss
3.63
1,90.3
5.25E-4
Suce zate
1/10
7.75/10
Limitations and Future Work
Summary. OmniH20 enables dexterous whole-body humanoid recommanipulation via teleopera- tion, designs universal control interfaces, facilitates scalable demonstration collection, and empow- ers humanoid autonomy via frontier models or humanoid learning from demonstrations.
Limitations. One limitation of our system the requirement of robot root odometry to transfer pose estimation from teleoperation interfaces to motion goals in the robot frame. Results from VIO can be noisy or even discontinuous, causing the motion goals to deviate from desired control. Another limitation safety; although the OmniH20 policy has shown great robustness, we do not have guarantees or safety checks for extreme disturbances out-of-distribune large discontinuity in motion goals). Future work could also focus the design of the teleoperation system to allow the humanoid to traverse stairs with only sparse upper-body motion goals. Another interesting direction is improving humanoid learning from demonstrations by incorporating more sensors (e.g., LiDAR, wrist cameras, tactile sensors) and better learning algorithms. We hope that our work spurs further efforts toward robust and scalable humanoid teleoperation and learning.
motion goals (e.g.,
8

---

Acknowledgments
The authors express their gratitude Ziwen Zhuang, Xuxin Cheng, Jiahang Cao, Wentao Dong, Toru Lin, Ziqiao Ma, Aoran Chen, Chunyun Wen, Unitree Robotics, Inspire Roboties, Damiao Technology for valuable help the experiments and graphies design.
References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] K. Darvish, L. Penco, J. Ramos, R. Cisneros, J. Pratt, E. Yoshida, S. Ivaldi, and D. Pucci. Teleoperation of humanoid robots: A survey. IEEE Transactions on Robotics, 39(3):1706 1727, 2023.
[3] He, Z. Luo, W. Xiao, Zhang, K. Kitani, C. Liu, and Shi. Learning human-to-humannen real-time whole-body teleoperation. arXiv preprint arXiv:2403.04458 2024.
[4] M. Sco, S. Han, K. Sim, S. H. Bang, C. Gonzalez, L. Sentis, and Y. Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8. IEEE, 2023.
[5] Z. Fu, Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.021/ 2024.
[6] F. L. Moro and L. Sentis. Whole-body control of humanoid robots. Humanoid robotics: reference, pages 1161-1183, 2019.
[7] J. Siekmann, Y. Godse, A. Fern, and J. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7309-7315. IEEE, 2021.
[8] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2811-288 1-2817. IEEE, 2021.
[9] H. Duan, B. Pandit, M. S. Gadde, B. J. van Marum, J. Dao, C. Kim, and A. Fern. Learning vision-based bipedal locomotion for challenging terrain. arXiv preprint vrXiv:2309.1459: 2023.
[10] J. Dao, K. Green, H. Duan, A. Fern, and J. Hurst. Sim-to-real learning for bipedal locomotion under unsensed dynamic loads. In 2022 International Conference Robotics and Automation (ICRA), pages 10449-10455. IEEE, 2022.
[11] I. Radosavovic, B. Zhang, B. Shi, J. Rajasegaran, S. Kamat, Darrell, K. Sreenath, and J. Ma- lik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402.19460 2024.
[12] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(9))cadd9959, 2024.
[13] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. arXiv preprint arXiv:2401, 16889, 2024.
[14] K. Harada, S. Kajita, H. Saito, M. Morisawa, F. Kanehiro, Fujiwara, K. Kaneko, and H. Hirukawa. A humanoid robot carrying heavy object. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pages 1712-1717. IEEE, 2005.

---

[15] M. Murooka, I. Kumagai, M. Morisawa, F. Kanehiro, and A. Kheddar. Humanoid loco- manipulation planning based on graph search and reachability maps. IEEE Robotics and Au- tomation Letters, (0Q:1880-1844) 2021.
[16] J. W. Grizzle, J. Hurst, B. Morris, H.-W. Park, and K. Sreenath. Mabel, a new robotic bipedal walker and runner. In 2009 American Control Conference, pages 2030-2036. IEEE, 2009.
[17] K. Hirai, M. Hirose, Haikawa, and Takenaka. The development of honda humanoid robot. In Proceedings. 1998 IEEE international conference on robotics and automation (Cat. No. 98CH136166), volume 2, pages 1321-1326. IEEE, 1998.
(18) X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 2024.
[19] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149 PMLR, 2023.
[20] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision free high-speed legged locomotion. arXiv preprint arXiv:2401. 17583, 2024.
[21] Y. Yang, G. Shi, X. Meng, W. Yu, T. Zhang, Tan, and B. Boots. Cajun: Continuous adaptive jumping using learned centroidal controller. Conference on Robot Learning, pages 2791- 2806. PMLR, 2023.
[22] S. Dafarra, U. Pattacini, G. Romualdi, Rapetti, R. Grieco, K. Darvish, G. Milani, E. Valli, I. Sorrentino, P. M. Viceconte, et al. icub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots. Science Roboties, 9(86):eadh3834, 2024.
[23] J. Chagas Vaz, D. Wallace, and P. Y. Oh. Humanoid lccommanipulation of pushed carts utilizing virtual reality teleoperation. In ASME International Mechanical Engineering Congress and Exposition, volume 85628, page V07BT07A027. American Society of Mechanical Engineers, 2021.
[24] M. Elobaid, Y. Hu, G. Romualdi, S. Dafarra, J. Babic, and D. Pucci. Telexistence and tele- operation for walking humanoid robots. In Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2, pages 1106-1121. Springer, 2020.
[25] F.J. Montecillo- Pueente,ll,,e M. Sreenivasa, and J.-P. Laumond. On real-time whole-body human to humanoid motion transfer. 2010.
[26] K. Otani and K. Bouyarmane. Adaptive whole-body manipulation in multi-contact motion retargeting, In 2017 IEEE-RAS 17th International Conference Hu- manoid Robotics (Humanoids), pages 446 456-453. IEEE, 2017.
[27] K. Hu, C. Ott, and D. Lee. Online human walking imitation in task and joint space based on quadratic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3458-3464. IEEE, 2014.
[28] Tachi, Y. Inoue, and F. Kato. Telesar vi: Telexistence surrogate anthropomorphic robot vi. International Journal of Humanoid Robotics, 17(05):205001,, 2020.
[29] Y. Ishiguro, K. Kojima, F. Sugai, S. Nozawa, Y. Kakiuchi, K. Okada, and M. Inaba. High speed whole body dynamic motion experiment with real time master-slave humanoid robot system. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 5835-5841, IEEE, 2018
10

---

[30] O. Porges, M. Connan, B. Henze, A. Gigli, C. Castellini, and M. A. Roa Garzon. A wear- able, ultralight interface for bimanual teleoperation of a compliant, whole-bndy-controllee hu- manoid robot. In 2019 International Conference on Robotics and Automation, ICRA 2019.
IEEE, 2019.
[31] Y. Ishiguro, T. Makabe, Y. Nagamatsu, Y. Kojio, K. Kojima, F. Sugai, Y. Kakiuchi, K. Okada, and M. Inaba. Bilateral humanoid telcoperation system using whole-body exoskeleton cockpit tablis. IEEE Robotics and Automation Letters, 3(4):6419-642)) 2020.
[32] C. Stanton, A. Bogdanovych, and Ratanasena. Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning. In Proc. Australasian Conference on Robotics and Automation, volume 8, page 51, 2012.
[33] D. Dajles, F. Siles, et al. Teleoperation of a humanoid robot using optical motion capture system. In 2018 IEEE International Work Conference Bioinspired Intelligence (IWOBI), pages 1-8. IEEE, 2018.
[34] L. Fritsche, F. Unverzag. J. Peters, and R. Calandra. First-person tele-operation of a humanoid robot. In 2015 TEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 997-1002. IEEE, 2015.
[35] M. Hirschmanner, C. Tsiourti, T. Patten, and M. Vincze. Virtual reality teleoperation of humanoid robot using markerless human upper body pose imitation. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 259-265. IEEE, 2019.
[36] A. Winkler, Won, and Ye. Questsim: Human motion tracking from sparse sensors with simulated avatars. In SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
[37] S. Lee, S. Starke, Y. Ye, J. Won, and A. Winkler. Questenvsim: Environment-awan simulated motion tracking from sparse sensors. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[38] Y. Ye, L. Liu, Hu, and S. Xia. Neural3points: Learning to generate physically realistic full-body motion for virtual reality users. In Computer Graphics Forum, volume 41, pages 183-194. Wiley Online Library, 2022.
[39] Z. Luo, J. Cao, J. Merel, A. Winkler, J. Huang, K. Kitani, and W. Xu. Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582. 2023.
[40] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- 2023.
els. arXiv preprint arXiv:2310.0888
[41] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
[42] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, Hansen-Estruch, A. W. He, V. My- ers, M. J. Kim, M. Du, et al. Bridgedata v2: A dataset for robot learning scale. In Conference on Robot Learning, pages 1723-1733 1736. PMLR, 2023.
[43] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at seale, arXiv preprint arXiv.2212.06817, 2022.
[44] A. Brohan, N. Brown, Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: lasiua-la models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023,

---

[45] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /2213, 2024.
[46] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
[47] T. Lin, Y. Zhang, Q. Li, H. Qi, two multifingered hands. arXiv preprint arXiv:2404.16823, 2024.
B. Yi, Levine, and J. Malik. Learning visuotactile skills with
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017. URL 47.
[49] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture as surface shapes. Proceedings of the IEEE International Conference on Computer Vision, 4019-0ctob:5441- 55400, 2019. ISSN 1550-5499.
[50] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47):eabc5986, 2020.
[51] Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. arXiv preprint 10Xiv:1011.0688, 2010.
[52] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-tclevision: open-source tele-operation with vision, 2024.
[53] Y. Park and P. Agrawal. Using apple vision pro to train and control robots, 2024. URL https: //ggt////ttttt com/ improbable-AA/Viss wwwonrooeeeeeeee
[54] K. Shaw, A. Agarwal, and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning, arXiv preprint (30.:23390004440 2023.
[55] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based telcoperation dexterous robotic hand-arm system. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9176 IEEE, 2020.
[56] 7. Luo, J. Cao, AlexanderWinkler, K. Kitani, and W. Xu. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10895-10904, October 2023.
[57] G. Tevet, Raab, B. Gordon, Shafir, D. Cohen-Or, and H. Bermano. Human motion diffusion model, 2022.
[58] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 23X 2023.
[59] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural 2020. information processing systems, 33:6840-6851, 2020.
[60] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint
arXiv:2010.02502, 2020.
[61] Unitree. Unitree's first universal humanoid robot, 2018. URL http::::/ww.wnittttttttt com/h1 hll
[62] Damiao. Static, dynamic, fictitious, strong, 2018. URL https: http://wwwwwwwwcc//tt//tttttt com/.
12

---

[63] Inspire-robots. Smaller and pecher-precisoo motion control experts. 2018. URL https: //inspired www.rorbo..toooooooo....
[64] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning, 2022.
[65] I.. Campanaro, S. Gangapurwala, W. Merkt, and Havoutis. Learning and deploying robust locomotion policies with minimal dynamics randomization, 2023.
13

---

Appendix
A Real Robot System Setup
14
Simulation Baseline and Ablations
15
State Space Compositions
16
D LfD Baselines
20
E Reward Functions
20
F Domain Randomizations
21
G Linear Velocity Estimation
22
H Ablation on Dataset Motion Distribution
22
I Additional Physical Teleoperation Results
22
J Dataset and Imitation Learning
23
K Sim2real Training Hyperparameters
24
L LID Hyperparameters
24
M GPT-4o Prompt Example
25
Appendix
More real-world experrmmmnt videos are at the website https: : omn1 www.hhahmmmoooo.dccom com.
A
Real Robot System Setup
Our real robot employs Unitree H1 platform [61], outfitted with Damiao DM-J4310-2EC mo- tors [62] and Inspire hands [63] for its manipulative capabilities. We have two versions of real robot computing setup. (1) The first one has two 16GB Orin NX computers mounted on the back of the HI robot. The first Orin NX connected to a ZED camera mounted on the waist of H1, which performs computations to determine Hl's own location for positioning. The camera operates 60 Hz FPS. Additionally, this Orin NX connects via Wi-Fi to our Vision Pro device continuously receive mo- tion goal information from human operator. The second Orin NX serves as main control hub. It receives the motion goal information, which it uses input for our control policy. This policy then outputs torque information for each of the robot's motors and sends these commands to the robot. As control for the robot's 's fingers and wrists does not require inference, it is directly mapped from the Vision Pro data to the corresponding joints on the robot. The policy's 's computation frequency set at 50 Hz. The two Orin NX units are connected via Ethernet, sharing information through common ROS (Robot Operating System) network. The final commands to HI are consolidated and dispatched by the second Orin NX. Our entire system has low latency of only 20 milliseconds.
It's worth noting that designed the system in this way partly because the ZED camera requires
14

---

substantial computational resources. By dedicating the first Orin NX to the ZED camera, and the second to policy inference, we ensure that each component operates with optimal performance. (2) In the second setup, a laptop (13th Gen i9- 19903X0 and NVIDIA RTX4090, 32GB RAM) serves as computing and communication device. All devices, including the ZED camera, control policy, and Vision Pro, communicate through this laptop on its ROS system, facilitating centralized data handling and command dispatch. These two setups yield similar performance, and them interchangeably in our experiments.
B Simulation Baseline and Ablations
In this section, we provide an explanation of each ablation method. Main results in Table
Privileged policy: This teacher policy priiileged incorporates all privileged environment
information, along with complete motion goal and proprioception data the observations. State space composition details in Table
H2O:
policy trained using RL without DAgger and historical data, utilizing 8 keypoints motion goal in observations. State space composition details Table
OmniH20: Our deployment policy T0mmiHo0 that includes historical information and uses 3 keypoints of motion goal in observations, trained with DAgger. State space composition details Table
Ablation on DAgger/RL in Table ((a)
OmniH220wwooDAAeeee Sttp:/o-DAgge--Hssttrr This variant of OmniH20 is trained solely using RL and does not incorporate historical information within observations. State space composi- tion details in Table
OmniH20-w/ 11000-wwo-DAggeee This model is trained using RL, excludes DAgger, but includes historical information from the last 25 steps in observations. State space composition de- tails Table
OmniH20-Hissorry.. This model is trained with DAgger, but excludes historical informa- tion from the last 25 steps in observations. State space composition details in Table 9.
OmniH20: This model is trained with DAgger and incorporates 25-step historical infor- mation within observations. State space composition details in Table 6.
Ablation on History steps/Architeecture in Table 1(b)
OmmiHH20-History$$25555.. This variant of the OmniH20 with 50. 25, or 0 steps of
historical information in the observations. State space composition details in Table 10.
OmmiH20-GRU/LSTM: This version replaces the MLP in the policy network with either GRU or LSTM, inherently incorporating historical observations. State space composition details Table 11.
Ablation on Tracking Points in Table 1(c)
0mmiH22022/8/ppointt This variant of the OmniH20 policy includes 22, 8, key- points of motion goal the observations, with the 3 keypoints setting corresponding the standard OmniH20 policy. State space composition details in Tables 12 and 13.
Ablation Linear Velocity in Table 1(d)
OmniH220-w-liveet This variant is almost the same as OmniH20 but with root linear velocity in observations and past linear velocity in history information. State space com- position details in Table 14.
15

---

C State Space Compositions
In this section, we introduce the detailed state space composition of baselines in the experiments. Privileged Policy. This policy "privileged is the teacher policy that has access all the available states for motion imitation, trained using RL.
Table 4: State space information in Privileged Policy setting
State term
Dimensions
(Proprioception) Rigid Body position
66
(Proprioception) Rigid Body rotation
138
(Proprioception) Rigid Body velocity
69
(Proprioception) Rigid Body angular velocity
69
(Motion goal) Rigid Body position difference
69
(Motion goal) Rigid Body rotation difference
138
(Motion goal) Rigid Body velocity difference
69
(Motion goal) Rigid Body angular velocity difference
69
(Motion goal) Local Rigid Body position
69
(Motion goal) Local Rigid Body rotation
138
Actions
19
Total dim
913
H2O. This policy has 8 keypoints input (shoulder, elbow, hand, leg) and with global linear velocity, trained using RL.
Table 5: State space information in H2O setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base velocity
3
Base angular velocity
3
Base gravity
3
Motion goal
72
Actions
19
Total dim
138
OmniH20. This is deployment policy TomaiHzo with 25 history steps and without global linear velocity, trained using DAgger.
16

---

Table 6: State space information in Omnil120 setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
Base gravity
3
Motion goal
27
Actions
19
Single step total dim
90
History state term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
Actions
19
History Single step total dim
63
Total dim
1665(63#25 90)
OmilHHOOwwooDAgeeeHHHsttrr.. This policy has history of 0 steps, trained using RL,
Table 7: State space information in Omnill00-woo-Agee--Hiioo setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Motion goal
27
Actions
19
Total dim
90
Ommil20-w6o-Agger: This policy has the same architecture as OmniH20 but trained with RL.
Table 8: State space information in OmniH220-//o -nniiH2O-wo-DAgger setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
Motion goal
27
Actions
19
Single step total dim
90
History state term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Actions
19
History Single step total dim
63
Total dim
1665(63 90)
Omnit20-HHisoryd., This policy has history of steps, trained using DAgger.
17

---

Table 9: State space information in OmniH120-HHisory0 setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Motion goal
27
Actions
19
Total dim
90
OmniH220-Histrrxx. This policy has a history of steps, trained using DAgger.
Table 10: State space information in OmniH20 OmniH22--Hssoryxx setting
State term
Dimensions
DoFF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Motion goal
27
Actions
19
Single step total dim
90
History state term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
Actions
19
History Single step total dim
63
Total dim
63*x + 90
OmiiH20-GRU/LSTT.. This policy GRU/LSTT--base. architecture, trained using DAgger.
Table 11: State space information in OmniH20-GRU/LSTE setting
State
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Motion goal
27
Actions
19
Total dim
90
OmniH20-22points.
This policy has 22 keypoints input (every joint the humanoid), trained
using DAgger.
18

---

Table 12: State space information in OmanlH20-222oiin setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
Base gravity
3
Motion goal
198
Actions
19
Single step total dim
261
History state term
Dimensions
DoF' position
19
DoF velocity
19
Base angular velocity
3
Base gravity
Actions
19
History Single step total dim
63
Total dim
1836(63325+261
OmniH20-8points.
This policy has keypoints input (shoulder, elbow, hand, leg), trained using
DAgger.
Table 13: State space information in Omnil12 20-Soot2--spoints setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Motion goal
72
Actions
19
Single step total dim
135
History state term
Dimensions
DoF position
19
DoF velocity
19
Base angular velocity
3
Base gravity
3
Actions
19
History Single step total dim
63
Total dim
17 1710063*25+133
OmniH20-w-linvel. This policy has 25 history steps and global linear velocity, trained using DAg-
ger.
19

---

Table 14: State space information in Omnil120-w-linve setting
State term
Dimensions
DoF position
19
DoF velocity
19
Base velocity
Base angular velocity
3
Base gravity
Motion goal
27
Actions
19
Single step total dim
93
History state term
Dimensions
DoF position
19
DoF velocity
19
Base velocity
Base angular velocity
3
Base gravity
3
Actions
19
History Single step total dim
66
Total dim
1743(66*25 93)
D LfD Baselines
We conduct numerous ablation studies on LID, aiming benchmark the impact of various aspects on LfD tasks. The details of each ablation are as follows:
Ablation on Dataset size.
25/50/100% data: In this task, we use 25/50/100% of the dataset as the training set. The algorithm is DDPM which takes a single-step image as input and outputs steps actions.
Ablation on Single/Sequence observation/action input/output.
Si-O-Si-A: Single-step observation and single-step action mean that take 1 step of image data input and predict 1 step of action output.
Se-O-Se-At Sequence-steps observation and sequence-steps actions mean that take steps of image data as input and predict steps of action as output.
Si-O-Se-A: Single-step observation and sequence-steps actions mean that we take step of image data as input and predict 8 steps of action as output.
Ablation on Training Architecture..
BC: Behavior cloning which means resnet+MLP predict the next steps action from the current step's image.
DP-DDIM: We use DDIM as the algorithm which takes a single-step image as input and outputs 8 steps of actions.
DP-DDPM: We use DDPM as the algorithm which takes single-step image as input and outputs 8 steps of actions.
Reward Functions
Reward Components. Detailed reward components are summarized in Table 15.
20

---

Table 15: Reward components and weights: penalty rewards for preventing undesired behaviors for sim-to-real
transfer, regularization to refine motion, and task reward achieve successful whole-body tracking real-time.
Term
Expression
Weight
Penalty
Torque limits
II T
Tmuiny Tmax
-2
DoF position limits
11 d1 &f Amin: Imax
-125
DoF velocity limits Termination
11 (d. & [umin: Imax
-50
1
termination
-250
Regularization
DoF acceleration
[del
-0.000011
DoF velocity
|
-0.004
Lower-body action rate Upper-body action rate
lower
lower
-3
a
apper
a
upper
-0.625
Torque
|
-0.0001
Feet air time
Tair 0.25 [64]
1000
Max feet height for each step
max max feet height for each
0.25, 03
1000
Feet contact force
Preet
-0.75
Stumble
1
Fary
5 X F
-0.00125
feet
Slippage
|| 1
feet
1 (Frect 2 1)
-37.5
Feet orientation
feet
-6.55
In the air Orientation
(Flet
Frighht
<1)
200
feet
9
200
Task Reward
DoF position
exp(-0.22 d de
32
exp(-0.25|| /2, d. (
DoF velocity
16
Body position
exp(-0.5|p - -D:13)
30
Body position VRpoints
exp(- exp(-0.5||pp,"
50
Body rotation
exp(-0.1|(0,
0,1))
20
Body velocity
exp( 10.00
6.11)
8
Body angular velocity
exp 0.01 Wr 2
8
Reward Curriculum. We have modified the cumulative discounted reward expression to handle multiple small rewards at each time step differently, depending on their sign. The revised formula is given by: E
St,Te, where Tt, represents different reward functions time
and st is the scaling factor for each reward, defined as: $1, Seurrent if, < where Scurrent rf, 2
the scaling factor. This scaling factor is adjusted dynamically: it is multiplied by 0.9999 when the average episode length is less than 40, and multiplied by 1.0001 when exceeds 120. The init Scurrent is 0.5, then the upper bound of this sealing factor is set to This modification allows policy progressively learn from simpler to more complex scenarios with higher penalties, thereby reducing the difficulty for RL in exploring the optimal policy.
F Domain Randomizations
Detailed domain randomization setups summarized in Table 16.
21

---

Table 16: Here describe the range of dynamics randomization for simulated dynamics randomization,
ternal perturbation, terrain, which are important for sim-to-real transfer, robustness, and generalizability.
Term
Value
Dynamics Randomization
Friction
((002, 1.1)
Base CoM offset
U(-0.1, 0.1)m
Link mass
(4077, 1.3) default kg
P Gain
U(0.75, 1.25) x default
D Gain
(40775 1.25) x default
Torque RFI [65]
0.1 x torque limit N m (20, 60)ms
Control delay
Motion reference offset
U[[-0.02, --002,, 0.02). -0.1, 0.
External Perturbation
Push robot
interval Vay 1m/s
Randomized Terrain
Terrain type
flat, rough, low obstacles [20]
G Linear Velocity Estimation
The illustration using the ZED camera VIO module and comparison VIO with neural state estimators are shown in Figure 8. We train our neural velocity estimators using a supervised learning approach. The process involves repeatedly deploying policy in simulation with different motion goals. In every environment step, we use the root linear velocity to supervise our velocity estimator.
of Coanlyy
(a) X-axis velocity
(b) Y-uxis velocity
(c) Zraxis velocity
(d) VIO setup
Figure 8: The illustration using ZED camera VIO module, and the comparison of velocity estimation VIO with neural state estimators.
H Ablation on Dataset Motion Distribution
The ablation study motion data distribution is shown in Figure 9, The policy trained without motion data augmentation is hard to stand still and make apper-body moves.
(a) w/ motion data augmentation
(b) w/o motion data augmentation
Figure 9; The ablation data augmentation.
I Additional Physical Teleoperation Results
Additional VR-based and RGB-based telcoperation demo shown in Figure 10.
22

---

VR-based Teleoperation
(a) RGB-based Teleoperation
Figure 10: More physical teleoperation showcases.
Catch-Release
(b) Squat
Hammer- Catch
(d) Sack-Papee-Seissorr
(e) Boxing
(0 Pick-Place
Figure 11: OmmiH20-6 dataset.
Dataset and Imitation Learning
As shown in Figure 11, collected 6 LfD tasks' dataset to enable the robot autonomously perform certain functions.
Catch-Release: Catch a red box and release it into trash bin. This task has 13234 frames in total. Squat: Squat when the robot sees a horizontal bar approaching that lower than its head height. This task has 8535 frames in total.
Hammer-Catch: Use right hand to catch a hammer a box. This task has 12759 frames total. Bock-Paper-Seissork When the robot sees the person opposite makes one of the rock-paper- scissors gestures, it should respond with the corresponding gesture that wins. This task has 9380 frames in total.
23

---

When you see blue boxing target, throw a left punch; when you see red one, throw right punch. This task has 11118 frames in total.
Boxing:
Paskkae-iikk-Paae:: Use your right hand to pick up the box and place the middle when the box on the right side, and use your left hand if box is on the left side. If you pick up the box with your right hand, place on the left side using your left hand; if picked up with your left hand, place it the right side using your right hand. This task has 18436 frames in total.
The detailed performance of tasks is documented in Table 17
Table 17: Quantitative LfD autonomous agents performance for 4 tasks.
Merics
Squat
Hammer
Paper Stissen
Ablation
25
50%iaaaa
100%daa
25%dats
50%date
100F
25%
30%idat
100%
259a
50%adaa
MSE Liss
3016-3
D4F-4
9.89F-5
1,256-4
1.10F4
7,07E-5
2.185-2
1.58F-2
3.2064 W10
2.728-2 3/10
1.396-2 910
Sucz ric
1/10
3/10
6/10
9/10
10/00
10/10
3/10
6/10
10/10
un Seguenee alsrevationnsction
SHO-SI-A
Se-O-Se-A
N--O-SeA
Se-O-Se-A
MSE Loss
2528
1,722
9,890
5,18
9,00
5,00
2.228
3,60
3,290 4
1430
5,300
1,06-3
10/10
10/10
5/10
610
1011
9/10
Ablation on MCDDDM/DDEM
BC
DP DDIM
DP DEPM
BC
DP DDMM
DP DDMM
BC
DP DDIM
DP DDPM
BC
DP DDID
DP DIPM
1390-3
6.346-4
6,42E-5
7,075-5
4.50
3-1E-4
3.29.44
1466-2
10/10
0/10
K Sim2real Training Hyperparameters
The hyperparameters for our RI/DAgger policy training are detailed in Table below.
Table 18: Hyperparameters
Hyperparameters
Values
Batch size
64
Discount factor (7)
0.99
Learning rate
0.001
Clip param
0.2
Entropy coef
0.005
Max grad norm
0.2
Value loss coef
1
Entropy coef
0.005
Init noise std (RL)
1.0
Init noise std (DAgger)
0.001
Num learning epochs
5
MLP size
[512, 256, 128)
L LID Hyperparameters
In order to make the robot autonomous, we have developed Learning from Demonstration (LID)
approach utilizing diffusion policy that learns from dataset collected. The default training hyperparameters are shown below in Table 19.
24

---

Table 19: Training Hyperparameters for the Lid Training
Hyperparameter
Default Value
Batch Size
32
Observation Horizon
1
Action Horizon
8
Prediction Horizon
16
Policy Dropout Rate
0.0
Dropout Rate (State Encoder)
0.0
Image Dropout Rate
0.0
Weight Decay
IE-5
Image Output Size
32
State Noise
0.0
Image Gaussian Noise
0.0
Image Masking Probability
0.0
Image Patch Size
16
Number of Diffusion Iterations
100
M GPT-40 Prompt Example
Here the example prompt we use for Autonomous Boxing task:
You'te 'e humanoid robot equipped with camera slightly tilted downward on your head, providing a first-person perspective. am assigning you task: when blue target appears in front of you, extend and then retract your left fist. When a red larget appears, do the same with your right fist. If there is no target in front, remain stationary. will provide you with three options each time: move your left hand forward, move your right hand forward, or stay motionless. You should directly respond with the corresponding options A, B, C based on the current image. Note that, yourself is also wearing blue left boxing glove and right red boxing glove, please do not recognize them as the boxing target. Now, based on the current image, please provide me with the A, B, C aasswer.
For Autonomous Greetings with Human Task, our prompt is:
You are humanoid robot equipped with a camera slightly tilted downward on your head, providing first-person perspective. I am assigning you a new task respond to human gestures in front of you. Remember, the person is standing facing you, so be mindful of their gestures. If the person extends their right hand to shake hands with you, use your right hand to shake their right hand (Option A). If the person opens both arms wide for a hug, open your arms wide to reciprocate the hug (Option B). If you see the person waving his hand as a gesture to say goodbye, respond by waving buck (Option C). If no significunt gestures are made, remain stationary (Option D). Respond directly with the corresponding options A, B. C, or D based on the current image and observed gestures. Directly reply with A, B, C, or D only, without any additional characters.
It is worth mentioning that we can use GPT-4 not only to choose motion primitive but also to directly generate the motion goal. The following prompt exemplifies this process:
You are humanoid robot equipped with a camera slightly tilted downward on your head, providing first- person perspective. I am assigning you a task respond to human gestures in front of you. If the person extends his left hand for handshake, extend your left hand to reciprocate. If they extend their right hand, respond by extending your right hand. If the person opens both arms wide for a hug, open your arms wide to reciprocate the hug. If no significant gestures are made, remain stationary. Respond numbers to represent the desired left and right hand 3D position with respect to your root position. For example: [0.25, 0.2, 0.3, 0. 0.15, -0.19, 0.27) means the desired position of the left hand 0.25m 0.25mforward, forward, 0.2m left, and 0. 3m high compared to pelvis position, and the desired position of the right hand is O. 1m forward, 0. 0.19m right and 0.27m high compared to pelvis position. The default stationary position should be (0.2. 0.2, 0.2, 0.2, -0.2, 0.2), Now please respond the 6d array based on the image to respond to the right hand shaking. left hand shaking, and hugging.
25


Reviews:
[{"review_id": "QsYEKJvNrX", "reviewer": "Reviewer_2Dtf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a humanoid whole-body control RL policy, OmniH2O, and integrates it with various teleoperation methods. The RL training reward, observation, and architecture are modified to remove the body velocity observation while improving tracking performance compared to prior work. The release includes a teleoperation dataset recorded using the controller.", "summary_of_recommendation": "The paper is a nice demonstration of humanoid teleoperation and the claims/experiments open many interesting questions. The paper's clarity and novelty would be enhanced by more details on the difference between OmniH2O and H2O, evaluating the statistical significance of the history related claims, clarifying robustness claims, and showing whole-body motions that are not possible via decoupled control.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "A key claim of the paper is that OmniH2O \"significantly improves upon prior art\" H2O. Right now, it's hard to interpret which of the differences account for most of the performance. Is it possible to enumerate the exact difference between OmniH2O and H2O in a table with an ablation for each difference? As far as I can tell, the ablations (Table 1) already include the differences: [(1) number of keypoints, (2) lin vel observation], but there are also [(3) modified reward function, (4) modified domain randomization, (5) standing trajs added to the motion dataset, (maybe others?)]. Ablations 1 and 2 don't seem to explain much of the performance gap; can the improvement can be attributed to a specific design choice like a reward term? \n\nRelatedly in Table 1, H2O and OmniH2O are evaluated on dataset $\\hat{Q}$. Is this a fair comparison since OmniH2O was trained on $\\hat{Q}$ but H2O was trained on a different dataset $Q$ and therefore being evalauted out-of-distribution?\n\nThe difference between the student with no history (OmniH2O-History0) vs. training with history (OmniH2O) is only 0.3%. This difference seems small; is it statistically significant across multiple seeds of training? If not, is it valid to assert that history utilization is an essential element? As it stands, I would be inclined to draw the opposite conclusion from this table and say that history length seems to have no trend / irrelevant for the OmniH2O policy. Therefore, the role of linear velocity observation would be mainly in easing the learning dynamics, and it does not need to be reconstructed from the history for humanoid locomotion. This is somewhat surprising since other works on bipedal (https://arxiv.org/pdf/2401.16889) and even quadruped locomotion (e.g. https://arxiv.org/abs/2301.10602, https://arxiv.org/abs/2202.05481) found a history module important for velocity estimation. Can you report the statistical significance in Table 1 and comment on this interpretation of the results?\n\nFollowing on this point, Table 2 suggests that omniH2O outperforms omniH2O-History0 by a wide margin in the real world despite nearly identical simulation performance. Does this suggest that using a history of observations helps mitigate the sim2real gap? What would explain this since there does not seem to be an obvious mechanism for the history length to influence the sim2real quality while both are enjoying high performance on the training data?\n\n“OmniH2O demonstrates great robustness under disturbances and unstructured terrains.” “OmniH2O shows superior robustness against human strikes and different outdoor terrains.” Superior to what? I am not sure how the reader is supposed to evaluate these claims. Is the robustness better than Unitree’s default controller, better than H2O, or other or cited works like https://arxiv.org/abs/2402.16796 or https://arxiv.org/pdf/2303.03381 ? The disturbance rejection looks somewhat similar to me. The locomotion in all videos takes place on flat surfaces and slight inclines which I do not believe should be referred to as unstructured terrains. What would you consider a structured terrain?\n\nDoes the teleoperation from verbal instruction w/ gpt-4 have significant differences from SayTap (https://arxiv.org/pdf/2306.07580) that prompts a VLM to output motion parameters for a quadruped and Code as Policies https://arxiv.org/abs/2209.07753 that prompts a VLM to output code?\n\nIs there a fundamental advantage of whole-body control over a decoupled approach? It seems like all behaviors could potentially be teleoperated with an interface like https://arxiv.org/abs/2402.16796 which separates control of the arms and legs while also allowing the user to specify a base height command for squatting. It would strongly motivate the coupled training if OmniH2O is capable of some more dynamic movements involving more evident coordination between the upper and lower body. The \"embodiment feasible\" motion dataset appears to contain several motions like kicking and jumping that involve this type of coordination. Was sim-to-real a limitation for these motions? If so, maybe one could add videos of the robot doing tracking these moves in simulation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrAXPXtSFT", "reviewer": "Reviewer_ziut", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes an approach for training a humanoid controller that can mimic human motions. Simulated and real-world experiments are provided to demonstrate the controller's performance relative to a number of ablations. The controller is used to produce imitation learning data for several tasks and a policy is learned on top of the controller for each task.", "summary_of_recommendation": "Overall the paper has a good degree of novelty in the sense that we are at the leading edge of  sim2real learning for humanoids. The clarity of the paper can be improved as well as aspects of the evaluation. However, the novelty of the demonstrations makes up for these weaknesses.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Is there a fundamental principle regarding the non-stationary reward formulation described in the appendix?\n\nWhy didn't you try to include the hand control in the overall learning framework? \n\nWhat can you say about the statistical significance of results in Table 1 (see comment in review)? \n\nHow does your approach deal with different levels of sparsity in the input (\"ablation to sparse input\")? \n\nHow were the real-world experiments conducted? Why isn't there a success rate here?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3xtG8a4MNA", "reviewer": "Reviewer_huqu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a bi-level autonomous humanoid policy that can be commanded by vocal instructions. The low-level policy can be operated by human operator by teleoperation. The high-level policy can be deployed using GPT-4o, MGM, or a trained diffusion policy. The author demonstrates the training framework of the low-level policy and releases the dataset OmniH2O-6 dataset to train the high-level diffusion policy.", "summary_of_recommendation": "The advancement of this work is not clear compared with previous works. I recommend a weak reject on this paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. What is the reference trajectory when performing outdoor locomotion tests?\n\n2. Why distillation is needed for teleoperation policy? \n\n3. Is it possible to use a more accurate odometry system to provide the linear velocity of the robot to achieve a better performance? For example, using an in-door tracking system for a proof of concept.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers talk positively about the paper. The reviewers highlight that this is one of the first papers doing full-body mimicry for humanoids. The paper does many different qualitative experiments, including tennis, drawing, watering plants, and picking objects in indoor and outdoor environments. The main weaknesses are

* that the paper makes unsubstantiated claims about improving over previous work and the experiment complexity. The statements mainly exaggerate the contributions and shy away from limitations.

* the clarity of the paper can be improved. Especially the paper is partially overloaded the additional GPT4o and imitation learning experiments provide little value / insights. The paper would have been better without these experiments. The freed space should be used to elaborate on the limitations. 

All in all, I agree with the reviewers and recommend accepting the paper. For the final submission, the authors should tone down the oversold statements and add a discussions about the limitations. There are still a lot of problems with the presented whole-body controller. The learned policies still have many undesirable properties such as tap dancing, the current controller kind of fails on dynamic tasks and all performed tasks do not require precise tracking. The paper does not need to solve all of these problems but it would be great to describe them and tell the reader where the performance would need to be improved and not list strawmen limitations in the limitation section.
Decision: Accept