HumanPlus: Humanoid Shadowing and Imitation from Humans
Authors: Zipeng Fu
Qingqing Zhao
Qi Wu
Gordon Wetzstein
Chelsea Finn
Venue: CoRL 2024 (2024)
Keywords: Humanoids
Learning from Human Data
Whole-Body Control
TL;DR: a full-stack humanoid system for learning motion and autonomous skills from human data

Abstract: One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training.Yet, doing so has remained challenging in practice due to the complexities in humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack of a data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, we introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy transfers to the real world and allows humanoid robots to follow human body and hand motion in real time using only a RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids to collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing a shoe to stand up and walk, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations.

HumanPlus: Humanoid Shadowing and Imitation from Humans
Zipeng Fu' Qinqqing Zhao' Qi Wa Gordon Wetzstein Chelsea Finn
Stanford University projecct co-leads https: nttps://human htwpa////amaniiaaaiiiaaaaa ai..aittt..iiioo
Shadowing
Autonomous Skills
Figure 1: We present full-stack system for humanoid robots to learn motion and autonomous skills from human data. Our system enables robots shadow fast, diverse motions from human operator, including boxing and playing table tennis, and to learn autonomous skills like wearing shoe, folding clothes, and jumping high.
One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training. Yet, doing so has remained challenging practice due to the complexities humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train low-level policy in simulation via reinforcement learning using existing 40- hour human motion datasets. This policy transfers to the real world and allows humanoid robots follow human body and hand motion in real time using only RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning train skill policies using egocentric vision, allowing humanoids complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations.
Abstract:
Keywords: Humanoids, Whole- Whole-Body Control, Learning from Human Data
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

1 Introduction
Humanoid robots have long been of interest in the robotics community due to their human-liks form factors. Since our surrounding environments, tasks and tools are structured and designed based on the human morphology, human-sized humanoids are the nature hardware platforms of general-purpose robots for potentially solving all tasks that people can complete. The human-like morphology of humanoids also presents unique opportunity to leverage the vast amounts of human motion and skill data available for training, bypassing the scarcity of robot data. By mimicking humans, humanoids can potentially tap into the rich repertoire of skills and motion exhibited by humans, offering promising avenue towards achieving general robot intelligence.
However, it has remained challenging in practice for humanoids to learn from human data. The complex dynamics and high-dimensional state and action spaces of humanoids pose difficulties in both perception and control. Traditional approaches, such as decoupling the problem into perception, planning and tracking. and separate modularization of control for arms and legs [1, 2, 3, 31, can be time- consuminn to be designed and limited in scope, making them difficult to scale to the diverse range of tasks and environments that humanoids are expected to operate in. Moreover, although humanoids closely resemble humans compared other forms of robots, physical differences between humanoids and humans in morphology and actuation still exist, including number of degrees of freedom, link length, height, weight, vision parameters and mechanisms, and actuation strength and responsiveness, presenting barriers for humanoids to effectively use and learn from human data. This problem is further exacerbated by the lack of off off-the-shelff and integrated hardware platforms. Additionally, we lack an accessible data pipeline for whole-body teleoperation of humanoids, preventing researchers from leveraging imitation learning as a tool to teach humanoids arbitrary skills. Humanoids developed by multiple companies have demonstrated the potential of this data pipeline and subsequent imitation learning from the data collected, but details aren't publicly available, and autonomous demonstrations of their systems are limited to couple tasks. Prior works use motion capture systems, first-person- view (FPV) virtual reality (VR) headsets and exoskeletons to teleoperate humanoids [4, 5, 6, 7], which are expensive and restricted in operation locations.
In this paper, we present a full-stack system for humanoids to learn motion and autonomous skills from human data. To tackle the control complexity of humanoids, we follow the recent success in legged robotics using large-scale reinforcement learning in simulation and sim-to-real transfer [8, 9] to train a low-level policy for whole-body control. Typically, beasenng-based low-level policies are designed to be task-specific due to cone-conu reward engineering [10, 1 11], enabling the humanoid hardware to demonstrate only one skill at a time, such walking. This limitation restricts the diverse range of tasks that the humanoid platform is capable of performing. At the same time, we have a 40-hour human motion dataset, AMASS [12), that covers wide range of skills. We leverage this dataset by first retargeting human poses to humanoid poses and then training a task-agnostic low-level policy called Humanoid Shadowing Transformer conditioning on the retargeted humanoid poses. Our pose-conditioned low-level policy transfers to the real world zero-shot.
After deploying the low-level policy that controls the humanoid given target poses, we can shadow human motion to our customized 33-DoF 180cm humanoid in real time using single RGB camera. Using state-of-the-ar human body and hand pose estimation algorithms [13, 14], we can estimate real-time human motion and retarget it to humanoid motion, which is passed as input to the low-level policy. This process is traditionally done by using motion capture systems, which are expensive and restricted in operation locations. Using line of sight, human operators standing nearby can teleoperate humanoids to collect whole-body data for various tasks in the real world, like boxing, playing the piano, playing table tennis and opening cabinets to store heavy pot. While being teleoperated, the humamoid collects egocentric vision data through binocular RGB cameras. Shadowing provides an efficient data collection pipeline for diverse world tasks, bypassing the sim-to-real gap of RGB perception.
Using the data collected through shadowing, we perform supervised behavior cloning to train vision-based skill policies. A skill policy takes in humanoid binocular egocentric RGB vision as inputs and predicts the desired humanoid body and hand poses. We build upon the recent success of imitation learning from human- -pooidde demonstrations [15, 16), and introduce a transformer- based architecture that blends action prediction and forward dynamics prediction. Using forward dynamics prediction on image features, our method shows improved performance by regularizing on image feature spaces and preventing the vision-based skill policy from ignoring image features and overfitting to proprioception. Using up to 40 demonstrations, our humanoid can autonomously complete tasks such as wearing a shoe to stand up and walk, unloading objects from warehouse
2

---

RGB Cameras
Humanoid DoFs
33
goom
Hand + Wrist Dofs
Jeach)
Mass
60kg
Sine [om)
55W150:
Z23cm
6-Dof Hand
Arm Payload
7.5kg
Hand Payload
1kg
Repestability
1-DoF Writt
Battery Life
-h (stending)
Max Pulling Force
32N
Figure 2: Hardware Details. Our HumanPlus robot has two egocentric RGB cameras mounted on the head, two 6-DoFF dexterous hands, and 33 degrees of freedom total.
racks, folding a sweatshirt, rearranging objects, typing, and greeting with another robot with 60-100%
success rates.
The main contribution of this paper is a full-stack humanoid system for learning complex autonomous skills from human data, named HumanPlus. Core this system is both (1) a real-time shadowing system that allows human operators to whole-body control humanoids using single RGB camera and Humanoid Shadowing Transformer, low-level policy that is trained on massive human motion data in simulation, and (2) Humanoid Imitation Transformer, an imitation learning algorithm that enables efficient learning from 40 demonstrations for binocular perception and high-DoF control. The synergy between our shadowing system and imitation learning algorithm allows learning of whole-body manipulation and locomotion skills directly in the real-world, such as wearing a shoe to stand up and walk, using only up to 40 demonstrations with 60-6600-100000 100% success.
2 Related Work
Reinforcement Learning for inantly focusing locomotion. While model-based control [17, 18, 19, 20, 21, 1, 22] has made tremendous progress on a wide variety of humanoid robots [23, 24, 25, 26, 27, 28], learning-bat based methods can achieve robust locomotion performance for humanoids [29, 30, 31, 32, 33, 10, 34] and biped robots [35, 36, 37, 38, 39, 40, 41, 42) due to their training on highly randomized environments in simulation and their ability adapt. Although mncommanipulation and mobile manipulation using humanoids are mostly approached via model-predictive control [43, 44, 45, 46, 47, 48), there has been some recent success on applying reinforcement learning and sim-to-real to humanoids for box relocation by explicitly modeling the scene and task in simulation [11], and for generating diverse upper-body motion [49]. In contrast, we use reinforcement leaning to train a low-level policy for task-agnostic whole-body control requiring no explicit modeling of real-world scenes and tasks in simulation.
learning for humanoids has been predom-
Teleoperation of Humanoids. Prior works develop humanoid and dexterous teleoperation by us- ing human motion capture suits [4, 5, 50, 51], exoskeletons [52, 53, 54, 55, 53), haptic feedback devices [56, 57, 58], and VR devices for visual feedbacks [6, 59, 60] and for end-effector con- trol [61, 62, 63, 64]. For example, Purushottam al. develop whole-body teleoperation of wheeled humanoid using an cxoskeleton suit attached to force plate for human motion recording. In terms of control space, prior works have done teleoperation in operation spaces [65, 5], upper-body telcopera- tion [66, 60). and whole-body teleoperation [67, 68, 69, 70, 59, 71, 72J. For example, He et al. use an RGB camera to capture human motion to whole-body teleoperate a humanoid. Seo et al. use VR controllers to teleoperate bimanual end-effectors and perform imitation learning the collected data to learn static manipulation skills. In contrast, our work provides a full-stack system that consists of a low-cost whole-body teleoperation system using single RGB camera for controlling every joint of a humanoid, enabling manipulation, squatting and walking, and an efficient imitation pipeline for learning autonomous manipulation and locomotion skills, enabling complex skills like wearing shoe to stand up and walk.
Robot Learning from Human Data.Human data has been used extensively for robot learning. including for pre-training visual or intermediate representations or tasks [73, 74, 75, 76, 77, 78, 64, 79] leveraging Internet-scale data [80, 81, 82], and for imitation learning on in-domain human data [83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97]. For example, Qin et al. use in-domain human hand data for dexterous robotic hands to imitate. Recently, human data has also been used for training humanoids [49, 72]. Cheng et al. use offline human data for training humanoids to generate diverse uoper-body motion, and He et al. use offline human data for training whole-body teleoperation interface. In contrast, we use both offline human data for learning low-level whole-
3

---

RGB camera (60Hz)
Body Hand Pose
Humanoid Shadowing Transfomer
Body & Hand Pose Rotargeting
Estimation (25Hg))
(50Hz)
Figure 3: Shadowing and Retargeting. Our system uses one RGB camera for body and hand pose estimation.
body policy for real-time shadowing, and online human data through shadowing for humanoids to imitate human skills, enabling autonomous humanoid skills.
3 Human Hardware
Our humanoid features 33 degrees of freedom, including two 6-DoF hands, two 1-DoF wrists, and a 19-DoF body (two 4-DoF arms, two 5-DoF legs, and I-DoF waist), shown on the left of Figure 2. The system is built upon the Unitree HI robot. Each arm is integrated with an Inspire- Robots RH56DFX hand, connected via customized wrist. Each wrist has Dynamixel servo and two thrust bearings. Both the hands and wrists are controlled via serial communication. Our robot has two RGB wehcams (Razer Kiyo Pro) mounted on its head, angled 50 degrees downward, with pupillary distance of 160mm. The fingers can exert forces up to 1ON, while the arms can hold items up to 7.5kg. The motors on legs can generate instant torques of up 360Nm during operation. Additional technical specifications of our robot are provided on the right of Figure 2.
4 Human Body and Hand Data
Offiine Human Data. We use public optical marker-based human motion dataset, AMASS [12] to train our lowelevel Humanoid Shadowing Transformer. The AMASS dataset aggregates data from several human motion datasets, containing 40 hours of human motion data on diverse ranges of tasks, and consisting of over 11,000 unique motion sequences. To ensure the quality of the motion data, we apply filtering process based motions are parameterized using the SMPL-X [99] model, which includes 22 body and 30 hand 3-DoF spherical joints, 3 3-dimensional global translational transformation, and 3-dimensional global rotational transformation.
approach outlined in [98]. Human body and hand
Retargeting Our humanoid body has subset of the degrees of freedom of SMPL-X body, consisting of only 19 revolute joints. To retarget the body poses, we copy the corresponding Euler angles from SMPL-X to our humanoid model, namely for hips, knees, ankles, torso, shoulders and elbows. Each of the humanoid hip and shoulder joints consists of 3 orthogonal revolute joints, so can be viewed as one spherical joints. Our humanoid hand has 6 degrees of freedom: DoF for each of the index, middle, ring, and little fingers, and 2 DoFs for the thumb. To retarget hand poses, we map the corresponding Fuler angle of each finger using the rotation of the middle joint. To compute the 1-DoF wrist angle, we use the relative rotation between the forearm and hand global orientations.
Real-Time Body Pose Estimation and Retargeting. To estimate human motion in the real world for shadowing, use mroddGGrodndd Humans with Accurate Motion (WHAM) [13] to jointly estimate the human poses and global transformations in real time using a single RGB camera. WHAM uses SMPL-X for human pose parameterization. Shown in 3, perform real cal-timme human-to- -oman-to-humanoid body retargeting using the approach described above. The body pose estimation and retargeting runs 25 fps on an NVIDIA RTX4090 GPU.
humanoid
Real-Time Hand Pose Estimation and Retargeting, We use HaMeR [14], transformer- hand pose estimator using a single RGB camera, for real-time hand pose estimation. HaMeR predicts hand poses, camera parameters, and shape parameters using the MANO [100] hand model. We perform real-time suman-to-humaan ho-munddo hand retargeting using the approach described above. Our hand pose estimation and retargeting runs at 10 fps on an NVIDIA RTX4090 GPU.
5 Shadowing of Human Motion
We formulate our levelevel policy, Humanoid Shadowing Transformer, as a decoder-only transformer, shown on the left side of Figure 4. At each time step, the input to the policy is humanoid proprioception and a humanoid target pose. The humanoid proprioception contains root state (row, pitch, and base angular velocities), joint positions. joint velocities and last action. The humanoid target pose consists of target forward and lateral velocities, target roll and pitch, target yaw velocity and target joint angles, and is retargeted from human pose sampled from the processed AMASS dataset mentioned

---

Joint Pos Setports
Humanaid Shackwing Transformer
Toqque
(10c0H
Targe: Pose
Humanoid Shadowing
Featur Loss
Transformer
Inputs
(50Hz)
Rabot Plopriocsption
Humanoid Imitation Transformar (25H-z)
Tirger
Peopoo
Pige Care divent
Pasitional Emboddiings Figure 4: Model 1s Architectures. Our system consists of a decoder-only transformer for low-level control, Humanoid Shadowing Transformer, and a decoder-only transformer for imitation learning, Humanoid Imitation Transformer.
in Section 4. The output of the policy is 19-dimensional joint position setpoints for humanoid body joints, which are subsequently converted to torques using 1000Hz PD controller. The target hand joint angles are directly passed to the PD controller. Our low-level policy operates at 50Hz and has a context length of 8, it can adapt to different environments given the observation history [31]. We use PPO [101] to train our Humanoid Shadowing Transformer in simulation by maximizing
discounted expected return El- y're). where r; the reward at time step f, T is the maximum
episode length, and y is the discount factor. The reward is the sum of terms encouraging matching target poses while saving energy and avoiding foot slipping. We list all the reward terms in the Table 4. We randomize the physical parameters of the simulated environment and humanoids with details in Table 5.
After training Humanoid Shadowing Transformer in simulation, deploy it zero-shot humanoid in the real world for real-time shadowing. The proprioceptive observations are measured using only onboard sensors including IMU and joint encoders. Following Section 4 and shown in Figure 3, we estimate the human body and hand poses in real time using a single RGB camera, and retarget the human poses to humanoid target poses. Illustrated in Figure 1, human operators stand near the humanoid to shadow their real-time whole-body motion to our humanoid, and use line of sight to observe the environment and behaviors of humanoid, ensuring a responsive teleoperation system. When the humanoid sits, we directly send the target poses to the PD controller, since we don't 't need the policy to compensate gravity, and simulating sitting with rich contacts is challenging. While being teleoperated, the humanoid collects egocentric vision data through binocular RGB cameras. Through shadowing, we provide an efficient data collection pipeline for various real-world tasks, circumventing the challenges of having realist RGB rendering, accurate soft object simulation, and diverse task specifications in simulation.
6 Imitation of Human Skills
Imitation learning has shown great success on learning autonomous robot skills given demonstrations on wide range of tasks [102, 103, 15, 16, 104]. Given the real-world data collected through shad- owing, we apply the same recipe to humanoids to train skill policies. We make several modifications to enable faster inference using limited onboard compute, and efficient imitation learning given binocular perception and high-DoF control.
In this work, we modify the Action Chunking Transformer [15] by removing its encoder-decoder architecture develop decoder-only Humanoid Imitation Transformer (HIT) for skill policies. as depicted on the right side of Figure 4. HIT processes the current image features from two egocentric RGB cameras, proprioception, and fixed positional embeddings inputs. These image features are encoded using a pretrained ResNet encoder. Due to its decoder-only design, HIT operates by predicting chunk of 50 target poses based on the fixed positional embeddings at the input, and it can predict tokens corresponding to the image features at their respective input positions. We incorporate an L2 feature loss on these predicted image features, compelling the transformer to predict corresponding image feature tokens for future states after execution of ground truth target pose sequences. This approach allows HIT to blend target pose prediction with forward dynamics prediction effectively. By using forward dynamics prediction on image features, our method enhances performance by regularizing image feature spaces, preventing the vision-based skill policies from ignoring image features and overfitting to proprioception. During deployment, HIT operates at 25Hz onboard, sending predicted target positions the low-level Humanoid Shadowing Transformer asynchronously, while discarding the predicted future image feature tokens.
5

---

Rearrange Objects
Reurrange
Cost # of Operators Whole-Body
Appracch Object (8)
Pick Object (8)
Place Object
Whole Stand (%)
Lower Objects (s)
Task (x)
Kinesthetic $50
3
X
2.10
1.38
3.12
6.60
90.5
ALOHA
$7050
2-3
X
2.70
1.30
3.15
7.15
100
Meta Quest
$250
2
X 3.57
1.63
3.67
8.87
95.3
Ours
$50
1
1.76
0.95
2.59
5,20
100
15.34
Table 1: Teleop Comparisons & User Studies. We report averaged completion time for 6 participants on 2 tasks.
7 Tasks
We select six imitation tasks and five shadowing tasks that need bimanual dexterity and whole-body control. Shown in Figure 7, these tasks span diverse array of capabilities and objects relevant to practical applications.
In the Wear a Shoe and Walk task, the robot (1) flips a shoe, (2) picks it up, (3) puts on, (4) presses it down to secure the fit on the left foot, (5) tangles the shoelaces with both hands, (6) grasps the right one, (7) grasps left one, (8) ties them, (9) stands up and (10) walks forward. This task demonstrates the robot's 's ability in complex bimanual manipulation with dexterous hands and capability in agile locomotion like standing up and walk while wearing shoes. The shoe placed on the table uniformly randomly on 2cm line along of robot's 's forward facing. Each demonstration has 1250 steps or 50 seconds.
In the Warehouse task, the robot (1) approaches the paint spray on warehouse shelves with its right hand, (2) grasps the sprayer, (3) retracts the right hand, (4) squats, (5) approaches the cart on the back of the quadruped, (6) release the spray, and (7) stands up. This tasks tests the whole-body manipulation and coordination the robot. The standing location of the robot randomized along a 10cm line. Each demonstration has 500 steps or 20 seconds.
In the Fold Clothes task, while maintaining balance, the robot (1) folds left sleeve, (2) folds right sleeve, (3) folds the bottom of the sweatshirt, requiring both dexterity to manipulate fabric with complex dynamics and maintaining an upright pose. The robot starts in a standing position, with uniformaly randomly sampled root yaw deviation from +10 to -10 degrees. The sweatsheet is uniformly randomly placed with a deviation of 10cm x 10cm on the table and -30 degrees to 30 degrees in rotation. Each demonstration has 500 steps or 20 seconds.
In the Rearrange Objects task, while maintaining balance, the robot (1) approaches the object, (2) picks up the object, and (3) places the object into a basket. The complexity arises from the diverse shapes, colors, and orientations of the objects, requiring the robot to choose the appropriate hand based on the object's location and to plan its actions accordingly. In total, we uniformly sample from 4 soft objects, including stuffed toys and a ice bag, where the object is placed uniformly randomly along a 10cm line on either left or right of the basket. Each demonstration has 250 steps or 10s. In the Type "Al" task, the robot (1) types the letter 'A', (2) releases the key, (3) types the letter T,, and (4) releases the key. Despite being seated, the robot needs high precision in manipulation. Each demonstration has 200 steps or 8 seconds.
In the Two-Robot Greeting task, the robot (1) approaches the other robot with the correct hand after observing the other bimanual robot starts extend one hand/arm, (2) touches the hand with the other robot, and (3) releases the hand. The other robot uniformly samples which hand to extend and stops in an end-effector region of 5cm 5em 5cm. The robot needs to quickly and accurately recognize which hand to use and approaches the other robot with the correct hand while maintaining balance. Each demonstration has 125 steps or 5 seconds.
For Shadowing Tasks, we demonstrate five tasks: boxing, opening two-door cabinet to store pot, tossing, playing the piano, playing table tennis, and typing "Hello World", showcasing the mobility and stability in shadowing fast, diverse motions and manipulating heavy objects. Videos of qualitative shadowing results can be found on the project website: ftatp::/hhuanoidd-a gi th
8 Experiments on Shadowing
8.1 Comparisons with Other Teleoperation Methods
We compare our telcoperation system with three baselines: Kinesthetic Teaching, ALOHA [15]. and Meta Quest, shown in Figure 5. For Kines- thetic Teaching, both arms are in passive mode and manually positioned. For ALOHA, we build a pair of bimanual arms for pupputeering from two WidowX 250 robots with similar kinematic structure as our humanoid arms. For Meta
Kinesthetic Teaching
ALOHA
Meta Quest
Figure 5: Baseline Teleoperation Systems.
Quest, we use positions of the controllers for operational space control through inverse kinematics

---

Maximum Force Thresholds
Whole-Body Skills
forward backward (1) leftward (1) rightward (1) Recovery Time (1) Squai(() Jump(") Stand
Ours
32N
44N
70N
100N
1.2s
0.44m
0.35m
H1 Default
24N
36N
40N
40N
15s
0.85m
0m
X
Table 2: Robustness Evaluation. Our low-level policy (Ours) can withstand large disturbance forces, has a shorter recovery time, and enables more whole-body skills than the manufacturer controller (H1 Default).
with gravity compensation. Shown in Table 1, all baselines do not support whole-body control and require at least two human operators for hand pose estimation. In contrast, our shadowing system simultancously control humanoid body and hands, requiring only one human operator. Also, both ALOHA and Meta Quest are more expensive. In contrast, our system and Kinesthetic Teaching only require a single RGB camera.
Shown in Table 1, we conduct user studies on 6 participants to compare our shadowing system with three baselines in terms of teleoperation efficiency. Two participants have prior teleoperation experience, while the remaining four have varying levels of expertise. None of the participants has prior experience with our shadowing system. The participants are tasked perform the Rearrange Objects task and its variant, Rearrange Lower Objects, where an object is placed on a lower table of height 0.55m, requiring the robot to squat and thus necessitating whole-body control. We record the average task completion time over six participants, with three trials each and three unrecorded practice rounds. We also record the average success rates of stable standing during teleop- eration using our lewelevel policy. While ALOHA enables precise control of robot joint angles, its fixed hardware setup makes it harder to adapt to people with different heights and body shapes, and does not support whole-body control of humanoids by default. Meta Quest often results in singularities and mismatches between target and ac- tual poses in Cartesian space due to the limited 5 degrees of freedom of each humanoid arm plus a wrist, resulting to the longest com- pletion time and destabilized standing arm singularities. While Kinesthetic Teaching is intuitive and has a low time-to-compll time-t--completion, it requires multiple operators, and sometimes external forces on arms during teaching causes the humanoid to stumble. In contrast, our system has the lowest time-to- completion, has the highest success rate of stable standing, and is the only method that can be used for whole-body teleoperation, solving the Rearrange Lower Objects task. 8.2 Robustness Evaluation
32N
24N
y
HI Default
36N
44N
Figure 6: Maximum Force Thresholds. Our low-level policy can withstand larger forces com- pared to H1 Default controller.
Shown in Table 2. We evaluate our low-level policy by comparing it to the manufacturer default controller (HI Default). The robot must maintain balance while manipulating with objects, so we assess robustness by applying forces to the pelvis and record the minimum forces causing instability. Shown in Figure 6, our policy withstands significantly larger forces and has a shorter recovery time. When robot is unbalanced, the manufacturer default controller takes several steps and up to 20 seconds to stabilize the robot, while ours typically recovers within one or two steps and below seconds. More recovery steps result in jittery behavior and compromise manipulation performance. We also show that our policy enables more whole-body skills that are not possible by the default controller like squatting, high jumping, standing up from sitting on a chair.
Experiments on Imitation
Shown in Table 3, we compare our imitation learning method Humanoid Imitation Transformer with three baseline methods: HIT policies with monocular inputs (Monocular). ACT [15], and Open-loop trajectory replay, across all tasks: Fold Clothes, Rearrange Objects, Type "Ar", Two-Robot Greeting, Warehouse, and Wear Shoes and Walk, detailed in Section and Figure 7. Although each skill policy solves its task continuously autonomously without stopping, we document the success rates of consecutive sub-tasks within each task for better analysis. We conduct 10 trials per task. We calculate the success rate for sub-task by dividing the number of successful attempts by the number of total attempts. For example in the case of Put on Shoe sub-task, the number of total attempts equals the number of success from the previous sub-task Pick up Shoe, as the robot could fail and stop at any sub-task.
Our HIT achieves higher success rates than other baselines across all tasks. Specifically, our method only method solves the Wear a Shoe and Walk task, achieving success rates of 60% given 40 demonstrations, where all other methods fail. This is because our method uses binocular perception,
7

---

Fold Clothes (40 demos)
Rearrange Objects (30 demos)
Fold Left
Fold Right
Fold
Stand
Whole
Approach Object
Pick Object
Place Object
Stand
Whole
Sleeve
Sleeve
Botton
Task
Tark
HIT (Ours)
100
100
100
100
100
100
90
100
100
90
Monocular
80
50
100
100
40
80
88
100
100
70
ACT
100
100
100
100
100
70
86
83
100
50
Open-loop
20
50
0
0
Type "Al" (30 demos)
Two-Robot Greeting (30 demos)
Type "A"
Leave "A"
Typc "I"
Leave
Whole
Appreach Hand
Touch
Release Hand
Whole
Stand
"T"
Tark
Hand
Tark
HIT (Ours)
90
100
89
100
100
90
100
100
Monocular
90
100
44
100
40
100
80
100
100
80
ACT
30
20
0
100
90
100
100
90
Open-loop
82
100
79
100
60
50
0
Warehouse (25 demos)
Appresch
Grasp
Retract
Squat
Approach
Release
Stand
Whole
Up
Task
HIT (Ours)
100
90
100
100
100
100
100
90
Monocular
100
80
100
100
100
100
100
ACT
100
90
100
100
100
100
100
Open-loop
60
0
0
Wear Shoe and Walk (40 demos)
Flip
Pick Up
Put on
Press
Tangle
Grasp Right
Grasp Left Shoelace
Tie Shoelace
Stand
Walk
Whole
Shoe
Shoe
Shoe
Shoe
Shoclaces
Shoelaces
Up
Forward
Task
HIT (Ours)
100
100
80
100
100
100
75
100
100
100
60
Monocular
0
ACT
50
60
Open-loop
Table 3: Comparisons Imitation. We show success rates of Humanoid Imitation Transformer (Ours), HIT with monocular input, ACT and open-loop trajectory replay across all tasks. Overall HIT (Ours) outperforms others.
and avoids overfitting to proprioception. ACT' fails in Wear a Shoe and Walk and Typing "Al" tasks where it overfits to proprioception, where the robot repeatedly attempts and stucks at Pick up Shoe and Leave "A" respectively after successful completing them, avoiding uses visual feedback. Monocular shows lower success rates due to its lack of depth information from a single RGB camera, yielding rough interaction with the table in Fold Clothes. It fails the Wear a Shoe and Walk task completely, where depth perception is crucial. However, due to its narrower field of view, it completes some sub-tasks more successfully than other methods in the Typing "Al" task. Open-loop only works in Typing "A!" with no randomization, and fails in all other tasks that require reactive control.
10 Conclusion, Limitations and Future Directions
In this work, present HumanPlus. full-stack system for humanoids to learn motions and autonomous skills from human data. Throughout the development of our system, we encountered several limitations. Firstly, our hardware platform offers fewer degrees of freedom compared to human anatomy. For instance, it uses feet with 1-DoF ankles, which restricts the humanoid's ' ability to perform agile movements such as lifting and shaking one leg while the other remains stationary. Each arm has only 5 DoFs, including a wrist, which limits the application of 6-DoF operational space control and may result in unreachable regions during shadowing. Furthermore, the egocentric cameras are fixed on the humanoid's 's head and are not active, leading to a constant risk of the hands and interactions falling out of view. In addition, we currently use fixed retargeting mapping from human poses to humanoid poses, omitting many human joints that do not exist on our humanoid hardware. This may limit the humanoids to learn from a small subset of diverse human motions. Currently, pose estimation methods do no work well given large areas of occlusion, limiting the operating regions of the human operators. Lastly, we focus on manipulation tasks with some locomotion tasks like squatting, standing up and walking in this work, as dealing with long- long-horizon navigation requires a much larger size of human demonstrations and accurate velocity tracking in the real world. We hope to address these limitations in future, and to enable more autonomous and robust humanoid skills that can be applied in various real-world tasks.
8

---

Acknowledgements
We thank Steve Cousins and Oussama Khatib at Stanford Robotics Center for providing facility support for our experiments. We also thank Inspire-Robots and Unitree Robotics for providing extensive supports on hardware and low-level firmware. We thank Huy Ha, Yihuai Gao, Chong Zhang. Ziwen Zhuang, Jiaman Li, Yifeng Jiang, Yuxiang Zhang, Xingxing Wang. Tony Yang. Walter Wen, Yunguo Cui, Rosy Wang, Zhiqiang Ma, Wei Yu, Xi Chen, Mengda Xu, Peizhuo Li, Tony Z. Zhao, Lucy X. Shi and Bartie for helps on experiments, valuable discussions and supports. This project is supported by The AI Institute, ONR grant N00014-21-1-225 and Okawa Foundation. Zipeng Fu supported by Pierre and Christine Lamond Fellowship.
References
[1] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake. Optimization-bon locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 2016.
[2] S. Feng, E. Whitman, X. Xinjilefu, and C. G. Atkeson. Optimization based full body control for the atlas robot. In International Conference on Humanoid Robots, 2014.
[3] J. Chestnutt, M. Lau, G. Cheung, J. Kuffner, J. Hodgins, and T. Kanade. Footstep planning for the honda asimo humanoid. In ICRA, 2005.
[4] K. Darvish, Y. Tirupachuri, G. Romualdi, L. Rapetti, Ferigo, F. J. A. Chavez, and D. Pucci. Whole-body geometric retargeting for humanoid robots. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), 2019.
[5] S. Dafarra, K. Darvish, R. Grieco, G. Milani, U. Pattacini, L. Rapetti, G. Romualdi, M. Salvi, A. Scalzo, I. Sorrentino, et al. icub3 avatar system, arXiv preprint arXiv:2203.069: 2022.
[6] L. Penco, N. Scianca, V. Modugno, I.. Lanari, G. Oriolo, and S. Ivaldi. A multimode teleoperation framework for humanoid ieco-manipulation: An application for the icub robot. IEEE Robotics & Automation Magazine, 2019.
[7] D. Kim, B.-J. You, and S.-R. Oh. Whole body motion control framework for arbitrarily and simultaneously assigned upper-body, tasks and walking motion. Modeling, Simulation and Optimization of Bipedal Walking, 2013.
[8] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04000 2021.
[9] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022.
[10] I. Radosavovic, B. Zhang, B. Shi, J. Rajasegaran, S. Kamat, T. Darrell, K. Sreenath, and
J. Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402.19469, 2024.
[11] J. Dao, H. Duan, and A. Fern. Sim-to-real learning for humanoid box loco-manipulation. arXiv preprint arXiv.2310.03191, 2023.
[12] Mahmood, N. Ghorbani, N. Troje, G. Pons-Moll, and M. Black. AMASS: Archive of motion capture surface shapes. In International Conference on Computer Vision, 2019.
[13] S. Shin, J. Kim, E. Halilaj. and M. J. Black. Wham: Reconstructing world-grounder humans with accurate 3d motion. In Computer Vision and Pattern Recognition (CVPR), 2024.
[14] G. Pavlakos, D. Shan, Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024.
[15] T. Z. Zhao, V. Kumma, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. RSS, 2023.
[16] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. Proceedings of Robotics: Science and Systems (RSS), 2023.

---

[17] M. H. Raibert. Legged robots that balance. MIT press, 1986.
[18] S. Kajita, F. Kanehiro, K. Kaneko, K. Yokoi, and H. Hirukawa. The 3d linear inverted pendulum mode: A simple modeling for a biped walking pattern generation. In Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robolics in the the Next Millennium (Cat. No. O1CH37 180), 2001.
[19] E. R. Westervelt, J. W. Grizzle, and D. E. Koditschek. Hybrid zero dynamics of planar biped walkers. IEEE transactions on automatic control, 2003.
[20] S. Collins, A. Ruina, R. Tedrake, and M. Wisse. Efficient bipedal robots based on passive- dynamic walkers. Science, 2005.
[21] Y. Tassa, T. Erez, and E. Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
[22] Q. Nguyen, A. Agrawal, X. Da, W. C. Martin, H. Geyer, J. W. Grizzle, and K. Sreenath. Dynamic walking on madomll-varyinng discrete terrain with one-step preview. In Robotics: Science and Systems, 2017.
[23] I. Kato. Development of wabot 1. Biomechanism, 1973.
[24] K. Hirai, M. Hirose, Y. Haikawa, and T. Takenaka. The development of honda humanoid robot. In Proceedings. 1998 IEEE international conference on robotics and automation, 1998.
[25] M. Chignoli, D. Kim, F. Stanger-Jones, and S. Kim. The mit humanoid robot: Design, motion planning. and control for acrobatic behaviors. In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), 2021.
[26] G. Nelson, A. Saunders, N. Neville, B. Swilling, J. Bondaryk, D. Billings, Lee, R. Playter, and M. Raibert. Petman: A humanoid robot for testing chemical protective clothing. Journal of the Robotics Society of Japan, 2012.
[27] Stasse, T. Flayols, R. Budhiraja, K. Giraud-Fsclasse J. Carpentier, J. Mirabel, A. Del Prete, P. Souères, N. Mansard, F. Lamiraux, et al. Talos: A new humanoid research platform targeted for industrial applications. In 2017 JEEE-RAR 17th International Conference on Humanoid Robotics (luumnoidds),00
[28] N. A. Radford, P. Strawser, K. Hambuchen, S. Mchling, W. K. Verdeyen, A. S. Donnan, J. Holley, J. Sanchez, V. Nguyen, L. Bridgwater, et al. Valkyrie: Nasa's first bipedal humanoid robot. Journal of Field Robotics, 2015.
[29] L. Krishna, G. A. Castillo, U. A. Mishra, A. Hereid, and S. Kolathaya. Linear policies are sufficient to realize robust bipedal walking challenging terrains. IEEE Robotics and Automation Letters, 2022.
[30] R. P. Singh, Z. Xie, P. Gergondet, and F. Kanehiro. Learning bipedal walking for humanoids with current feedback. IEEE Access, 2023.
[31] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 2024.
[32] B. van Marum, A. Shrestha, H. Duan, P. Dugar, J. Dao, and A. Fern. Revisiting reward design and evaluation for robust humanoid standing and walking, arXiv preprint 1IX1v32404.19173, 2024.
[33] Q. Zhang, P. Cui, D. Yan, J. Sun, Y. Duan, A. Zhang, and R. Xu. Whole-body humanoid robot locomotion with human reference. arXiv preprint 2xX.v:2402. 2024.
[34] A. Tang. T. Hiraoka, N. Hiraoka, F. Shi, K. Kawaharazuka, K. Kojima, K. Okada, and M. Inaba. Humanmimic: Learning natural locomotion and transitions for humanoid robot via wasserstein adversarial imitation. arXiv preprint 230.::2200.1122255 2023.
10

---

[35] H. Benbrahim and J. A. Franklin. Biped dynamic walking using reinforcement learning. Robotics and Autonomous systems, 1997.
[36] Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. Van de Panne. Feedback control for cassie with deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
[37] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. arXiv preprint arXiv:2 105.08328, 2021.
[38] J. Siekmann, Y. Godse, A. Fern, and J. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[39] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[40] A. Kumar, Z. Li, Zeng, D. Pathak, Sreenath, and J. Malik. Adapting rapid motor adaptation for bipedal robots. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2022.
[41] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learmiin for versatile, dynamic, and robust bipedal locomotion control, arXiv preprint arXiv:2401.16888, 2024.
[42] R. Tedrake, T. W. Zhang, and H. S. Seung. Stochastic policy gradient reinforcement learning on a simple 3d biped. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)IEEE Cat. No. 04CH37566), 2004.
[43] K. Harada, S. Kajita, H. Saito, M. Morisawa, Kanehiro, K. Fujiwara, K. Kaneko, and H. Hirukawa. A humanoid robot carrying a heavy object. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, 2005.
[44] H. Arisumi, J.-R. Chardonnet, A. Kheddar, and K. Yokoi. Dynamic lifting motion of humanoid robots. In Proceedings 2007 IEEE International Conference on Robotics and Automation, 2007.
[45] A. Settimi, D. Caporale, P. Kryczka, M. Ferrati, and L. Pallottino. Motion primitive based ran- dom planning for loco-manipulation tasks. In 2016 IEEE-RAS 16th International Conference Humanoid Robots (Humanoids), 2016.
[46] P. Ferrari, M. Cognetti, and G. Oriolo. Humanoid whole-body planning for moc-maaipulttionn tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
[47] K. Harada, S. Kajita, F. Kanehiro, K. Fujiwara, K. Kaneko, K. Yokoi, and H. Hirukawa. Real-time planning of humanoid robot's gait for force-controlled manipulation. IEEFASME Transactions on Mechatronics, 2007.
[48] S. Sato, Y. Kojio, K. Kojima, F. Sugai, Y. Kakiuchi, K. Okada, and M. Inaba. Drop preven- tion control for humanoid robots carrying stacked boxes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
[49] X. Cheng, Y. Ji. J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16990 2024.
[50] R. Cisneros, M. Benallegue, K. Kaneko, H. Kaminaga, G. Caron, A. Tanguy, R. Singh, L. Sun, A. Dallard, C. Fournier, et al. Team janus humanoid avatar: A cybernetic avatar embody human telepresence. In Toward Robot Avatars: Perspectives on the ANA Avatar XPRIZE Competition, RSS Workshop, 2022.
[51] Dragan, K. C. Lee, and Srinivasa. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference Human Robonn-Robot Interaction (HRI), 2013.
11

---

[52] A. Purushottam, C. Xu, Y. Jung, and J. Ramos. Dynamic mobile manipulation via whole-body bilateral teleoperation of wheeled humanoid. IEEE Robotics and Automation Letters, 2023.
[53] Y. Ishiguro, T. Makabe, Y. Nagamatsu, Y. Kojio, K. Kojima, F. Sugai, Y. Kakiuchi, K. Okada, and M. Inaba. Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit tablis. IEEE Robotics and Automation Letters, 2020.
[54] J. Ramos and S. Kim. Humanoid dynamic synchronization through whole-body bilateral feedback teleoperation. IEEE Transactions on Robotics, 2018.
[55] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke. Nimbro avatar: Interactive immersive telepresence with force-feedhack telemanipulation. In 2021 TEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5312-5319,203
[56] L. Peternel and J. Babič. Learning of compliant human-row roboot interaction using full-body haptic interface. Advanced Roboties, 2013.
[57] A. Brygo, I. Sarakoglou, N. tion with vibrotactile based balancing feedback. In Haptics: Neuroscience, Devices, Modeling, and Applications: 9th International Conference, EuroHaptics 2014, Versailles, France, June 24-26, 2014, Proceedings, Part II 9, 2014.
and N. Tsagarakis. Humanoid robot teleopera-
[58] J. Ramos and S. Kim. Dynamic locomotion synchronization of bipedal robot and human operator via bilateral feedback teleoperation. Science Robotics, 2019.
[59] S. Tachi, Y. Inoue, and F. Kato. Telesar vi: Telexistence surrogate anthropomorphic robot vi. International Journal of Humanoid Robotics.
[60] J. Chagas Vaz, D. Wallace, and Y. Oh. Humanoid meceppanppllttion of pushed carts utilizing virtual reality teleoperation. In ASME International Mechanical Engineering Congress and Exposition, 2021.
[61] T. Lin, Y. Zhang, Q. Li. H. Qi, B. Yi, S. Levine, and Malik. Learning visuotactile skills with two multifingered hands. arXiv:2404. 16823, 2024.
[62] S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969. IEEE, 2023.
[63] T. Portela, G. B. Margolis, Ji, and P. Agrawal. Learning force control for legged manipulation. arXiv preprint arXiv:2405.01444 2024.
[64] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube. arXiv preprint arXiv:2202. 10448, 2022.
[65] M. Seo, S. Han, K. Sim, S. H. Bang, C. Gonzalez, L. Sentis, and Y. Zhu. Deep imitation learning for humanoid loco- manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), 2023.
[66] M. Elobaid, Y. Hu, G. Romualdi, S. Dafarra, J. Babic, and D. Pucci. Telexistence and teleoper- ation for walking humanoid robots. In Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2, pages 1106-1122 1121. Springer, 2020.
[67] F.-J Cinteciill--ueenee. M. Sreenivasa, and J.-P. Laumond. On real-time whole-body human to humanoid motion transfer. 2010.
[68] Ishiguro, K. Kojima, F. Sugai, S. Nozawa, Y. Kakiuchi, K. Okada, and M. Inaba. High speed whole body dynamic motion experiment with real time master-slave humanoid robot system. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5835-5841. IEEE, 2018
[69] K. Otani and K. Bouyarmane. Adaptive whole-body manipulation in multi-contact motion retargeting. 2017 JEEE--AS 17th International Conference on Humanoid Robotics (Humanoids), pages 446-453. IEEE, 2017.
12

---

[70] K. Hu, C. Ott, and D. Lee. Online human walking imitation in task and joint space based on quadratic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3458-3464. IEEE, 2014.
[71] L. Penco, B. Clément, V. Modugno, E. M. Hoffman, G. Nava, D. Pucci, N. G. Tsagarakis. J.-B. Mouret, and S. Ivaldi. Robust real-time whole-body motion retargeting from human to hu- manoid. In 2018 IEEE-RAS /8th International Conference Humanoid Robots (Humunoids), pages 425-432, 432. IEEE, 2018.
[72] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to- humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2003.0443 243.044464, 2024.
[73] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representa- tion for robot manipulation. arXiv preprint arXiv:2203.1260) 2022.
[74] T. Xiao, I. Radosavovic, T. Darell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint 2211:22033.01773, 2022.
[75] I. Radosavovic, T. Xiao, S. James, Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training In Conference on Robot Learning, 2023.
[76] A. Majumdar, K. Yadav, S. Arnadd, J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, T. Wu, J. Vakil, et al. Where are we the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 2024.
[77] N. Hansen, Z. Yuan, Y. Ze, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre- training for visuo-motor control: Revisiting a ceeming-from-scraaee baseline. arXiv preprint arXiv:2212.05744 2022.
[78] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint 1rXiv:2103.1663 2021.
[79] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipu- lation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):1419-44444 2021.
[80] K. Grauman, A. Westbury, E. Byme, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[81] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, Yianilos, M. Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[82] D. Damen, H. Doughty, G. M. Farinella, Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), 2018.
[83] C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint 243 2024.
[84] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play, arXiv preprint arXi::2302.12422 2023.
[85] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj. S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 202/ IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), 2021.
[86] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: behaviors from raw video via context translation. In 2018 IEEE international conference on robotics and automation (ICRA), 2018
Learning to imitate
13

---

[87] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi- stage tasks via pixel-level translation of human videos. arXiv preprint 4rXhv:1912.04443,222 2019.
[88] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. In Conference Robot Learning, 2023.
[89] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 2019.
[90] M. Sicb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imitation. In Conference on Robot Learning, 2020.
[91] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. arXiv preprini SKNi:2207.004550 2022.
[92] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
[93] N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home. arXiv preprint arXiv:231 1.109,,,1,,,,,,. 2023.
[94] J. Wang. S. Dasari, M. K. Srirama, S. Tulsiani, and A. Gupta. Manipulate by seeing: Creating manipulation controllers from pre-trained representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3859-3868, 2023.
[95] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. In Conference on Robot Learning. PMLR, 2021.
[96] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 2020.
[97] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 2022.
[98] Z. Luo, J. Cao. A. W. Winkler, K. Kitani, and W. Xu. Perpetual humanoid control for real-time simulated avatars. International Conference on Computer Vision (ICCV), 2023.
[99] G. Pavlakos, V. Choutas, N. Ghorbani, Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from single image. In Proceedings IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), pages 10975-10985, 2019.
[100] J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 2017.
[101] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 170.0077777.647777 2017.
[102] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Ri-1: Robotics transformer for real-worke control at scale. In arXiv preprint a726072212.06817, 2022.
[103] O. X.-E. Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerice B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu, C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, Schiavi, H. Su, H.-S. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, I. Mordatch,
14

---

I. Radosavovic, I. Leal, J. Liang. J. Kim, J. Schneider, J. Hsu, J. Bobg, J. Bingham, J. Wu, J. Wu, J. Luo, J. Gu, Tan, Oh, J. Malik. J. Tompson, Yang,
Lim, Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, Oslund, K. Kawaharazuka, K. Zhang. K. Majd, K. Rana, K. Srinivasan, I.. Y. Chen, I.. Pinto, L. Tan, L. Ott, L. Lee, M. Tomizuka, M. Du, M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. Vuong. R. Rafailov, R. Tian,
Doshi, R. Martin-Martín, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Moore, S. Bahl, S. Dass, S. Song, S. Xu, Haldar, S. Adebola, S. Guist, S. Nasiriany. S. Schaal, S. Welker, S. Tian, S. Dasari, S. Belkhale, T. Osa, T. Harada, T. Matsushima, T. Xiao, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong. T. Darrell, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang. X. Zhu, X. Li, Y. Lu, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, hua Wu, Y. Tang, Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, and Z. J. Cui. Open X- X-Embodiment: Robotic learning datasets and RT-X models. https: arxiv. org/ abs 2310. 2310.08864, 2023.
[104] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.0211 2024.
15

---

Warehouse
#2
#4
45
Fold Clorhes
41,
42.
#3.
Rearrange Objects
$1.
+2.
Type "AP
R3.
Two Robot Greeting
#1
#3.
Shadowing Tasks
Figure Task Definitions. illustrate autonomous tasks through imitation learning, 5 shadowing tasks. Details are in Section 7.
16

---

Reward Teams
Expressions
target xy velocities
exp(-
[,, Viyl
102
target yaw velocities target joint positions target roll & pitch
exp(
Yyaw
9
11,
p)
1
energy feet contact feet slipping alive
|| 2
Vfoct 1 [Fret 11/2
Table 4: Rewards in Simulation. We denote linear x. velocity, linear y velocity, U/ae angular yaw velocity, 9 as joint positions, as joint velocities, roll, p as pitch, Vie feet velocities, e feet contact indicator, Fine as forees on feet, and 1 as targets.
Environment Params
Ranges
base payload
[-3.0, 3.0]kg
end-effector payload
[0, 0.5]kg
center of base mass
[-0.1, 0.1
motor strength friction control delay
[0.8, 1.1]
[0.3, 0.9]
[0.02, 0.04]s
Table Randomization Simulation. We uniformly sample from these randomization ranges during training simulation.
17


Reviews:
[{"review_id": "fnEicmB499", "reviewer": "Reviewer_ycZ6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, Human Shadowing Transformer and Human Imitation Transformer are introduced. Human Shadowing Transformer is trained in simulation to match 40 hours of human demonstrations from the AMASS dataset, including hand motions, proprioceptive only, to match reference poses. Using the shadowing transformer, task-specific data is collected (up to 40 demonstrations), and the imitation transformer is trained. This transformer is similar to action chunking with a modification to encourage image feature learning. The results on a 33 DoF humanoid, which includes 2 dexterous anthropomorphic hands, show 60-100% success rate for challenging whole-body manipulation tasks.", "summary_of_recommendation": "A stand out work that is beneficial for the community to expand, should be accepted.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "As above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBMXScW340", "reviewer": "Reviewer_mL5C", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a framework to mirror human motion without the need for expensive hardware and then uses this framework to collect demonstrations and train an autonomous control policy. The paper demonstrates the robustness and generalizability of the approach through a series impressive real robot experiments on H1 humanoid robot.", "summary_of_recommendation": "The paper demonstrates state-of-the-art results in challenging tasks for humanoid robots. The approach is straightforward and offers a promising alternative to expensive teleoperation hardware.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. How long are the transformers trained for? what is the simulator used? any domain randomization?\n\n2. What are the failure cases for the tasks shown? \n\n3. How are the forces applied for the robustness testing?\n\n4. What is the reason for the asymmetry in Fig. 7?\n\n5. Thoughts on improving the jittery motion?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5RrLnXBr5l", "reviewer": "Reviewer_uR2D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes an approach for training a humanoid controller that can mimic retargeted human motions. This controller in combination with vision-based human-motion tracking is used to generate real-world ego-centric demonstrations of tasks. A transformer based architecture is then used to learning to imitate the tasks. Primarily full-body manipulation tasks are demonstrated.", "summary_of_recommendation": "Overall there are some impressive demonstrations here and it is perhaps the first paper to close the loop on sim2real full-body demonstration learning using a learned full-body trajectory encoder. The clarity of the paper is the biggest weakness and would even be grounds for rejection if the application novelty were not so high. The depth of evaluation could also be improved.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "What was the curriculum for robot learning? Was it just randomly sample a motion from the 11,000+ dataset or was there a more structured approach?\n\nDo you have any evaluation results in simulation and real for the AMAAS training motions? \n\nHow are training episodes defined (e.g. cutoff) and are there force disturbances used during training?\n\nWhat is visible in the typing task to the HIT controller? Is the robot able to see letters or is it generalizing based on relative position? \n\nWhat device is used to apply forces in 8.2 and what are the details of the force application, e.g. duration and frequency?\n\nWhat does the shadowing demonstration process look like for the human for the shoe task, which involves, sitting, standing, and fine manipulation? How difficult was it to generate those full demonstrations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Pre-rebuttal**

This paper receives three strongly positive reviews, all acknowledging the impressive and state-of-the-art demonstration of the learned humanoid controller for full-body manipulation tasks.

The main mentioned weaknesses include:
- Missing details about training and evaluation (**uR2D**, **mL5C**).
- Additional evaluation.
  - Various design decisions (**ycZ6**).
  - AMASS trained controller (**uR2D**).
- Clarification on the generalizing capability.
  - OOD extrapolation (**mL5C**).
  - Taking in only the current step information in HIT (**uR2D**).
- Unclear difficulty level of the shadowing demonstration process (**uR2D**).


---
**Post-rebuttal**

All the reviewers acknowledge the merit of the rebuttal and continue to stand by their original position of strong accept. Overall the paper has delivered solid technical contributions and impressive robot demonstration on the popular frontier of humanoid learning, which would be of wide interest to the broad CoRL audience.

Additional feedback:
- The extra details from the rebuttal on the nuance of motion shadowing of "shoelaces" and "standing up and walking" provide an important insight to the data collection process (*"To ease the human efforts during data collection ... due to hard-to-model collisions between robot hips and chairs."*). AC encourages the authors to add them into the paper.
Decision: Accept