OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Authors: Jinhan Li
Yifeng Zhu
Yuqi Xie
Zhenyu Jiang
Mingyo Seo
Georgios Pavlakos
Yuke Zhu
Venue: CoRL 2024 (2024)
Keywords: Humanoid Manipulation
Imitation From Videos
Motion Retargeting

Abstract: We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately.  Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of $79.2\%$ without the need for labor-intensive teleoperation. More videos can be found on our 
website https://ut-austin-rpl.github.io/OKAMI/.

OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Jinhan Lift
Yifeng Zhu
Yuqi Xie
Zhenyu Jiang!
Mingyo Seo!
Georgios Pavlakos'
Yuke Zhu!?
UT Austin'
NVIDIA Research²
Abstract: We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAМТ uses open- open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, out- performing the state-of of-the-art baseline on open-wort imitation from observa- tion. Furthermore, OKAМI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve average success rate of 79.2% without the need for labor-intensive telcoperation. More videos can be found on our website https: //tu/uutuuus/// in- in-rpl github. io 1o/OKAMI/. /.
Keywords: Humanoid Manipulation, Imitation From Videos, Motion Retargeting
Single Video
Mornan
Imitation
Human Demo
Humas
Denso
Dema
Figure 1: OKAMI enables human teach the humanoid robot how perform task by providing single video demonstration.
Introduction
Deploying generalist robots to assist with everyday tasks requires them to operate autonomously in natural environments. With recent advances hardware designs and increased commercial avail- ability, humanoid robots emerge as promising platform deploy in living and working spaces. Despite their great potential, they still struggle to operate autonomously and deploy robustly in the
This work done while Jinhan Li visiting rescarcher UT Austin.
Equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

unstructured world. A burgeoning line of work has resorted to deep imitation learning methods for humanoid manipulation [1-3]. However, they rely on large amounts of demonstrations through whole-body teleoperation, requiring domain expertise and strenuous efforts. In contrast, humans have the innate ability to watch their peers do a task once and mimic the behaviors. Equipping robots with the abilitt imitate from visual observations will move us closer to the goal of training robotic foundation models from Intemet-scale human activity videos.
We explore teaching humanoid robots to manipulate objects by watching humans. We consider a problem setting recently formulated as "open-world imitation from observation," where a robot im- itates a manipulation skill from a single video of human demonstration [4-6]. This setting would facilitate users in effortlessly demonstrating tasks and enable humanoid robot to acquire new skills quickly. Enabling humanoids to imitate from single videos presents a significant challenge the video does not have action labels, but yet the robot has learn perform tasks in situations beyond what's demonstrated in the video. Prior works on one-shot video learning have attempted to optimize robot actions to reconstruct the future object motion trajectories [4. 5]. However, they have been applied to single-arm manipulators and are computationally prohibitive for humanoid robots due to their high degrees of freedom and joint redundancy [7]. Meanwhile, the similar kinematic structure shared by humans and humanoids makes directly retargeting human motions to robots fea- sible [8. 9]. Nonetheless, existing retargeting techniques focus on free-space body motions [10-14], lacking the contextual awareness of objects and interactions needed for manipulation. To address this shortcoming, we introduce the concept of "object-aware retargeting". By incorporating object contextual information into the retargeting process, the resulting humanoid motions can be effi ciently adapted to the locations of objects in open-ended environments.
To this end, we introduce OKAMI (Object-aware Kinematic retArgeting for huManoid Imitation), an object-aware retargeting method that enables a bimanual humanoid with two dexterous hands to imitate manipulation behaviors from a single RGB-D video demonstration. OКAМI uses two- stage process to retarget the human motions to the humanoid robot to accomplish the task across varying initial conditions. The first stage processes the video to generate a reference manipulation plan. The second stage uses this plan to synthesize the humanoid motions through motion retargeting that adapts to the object locations target environments.
OKAMI consists of two key designs. The first design is open-world vision pipeline that iden- tifies task-relevant objects, reconstructs human motions from the video, and localizes task-relevant objects during evaluation. Localizing objects at test time also enables motion retargeting to adapt to different backgrounds or new object instances of the same categories. The second design is the factorized process for retargeting, where we retarget the body motions and hand poses separately. We first retarget the body motions from the reference plan the task space, and then warp the retar- geted trajectory given the location of task-relevant objects. The trajectory of body joints obtained through inverse kinematics. The joint angles of fingers are mapped from the plan onto the dexterous hands, reprodueing hand-object interaction. With object-aware retargeting, OKAMI policies sys- tematically generalize across various spatial layouts of objects and scene clutters. Finally, we train visuomotor policies on the rollout trajectories from OKAMI through behavioral cloning to obtain vision-based manipulation skills.
We evaluate OKAMI on human video demonstrations of diverse tasks that cover rich object inter- actions, such picking, placing, pushing, and pouring. We show that its object-aware retargeting achieves 7 71.7% task success rates averaged across all tasks and outperforms the ORION [4] baseline by 58.3%. We then train closed-loop visuomotor policies on the trajectories generated by OKAMI, achieving an average success rate of 79.2%. Our contributions of AKAMI are three-fold:
1. OKAMI enables humanoid robot to mimic human behaviors from single video for dexterous manipulation. Its object-aware retargeting process generates feasible motions of the humanoid robot while adapting the motions to target object locations test time;
2. OKAМI uses vision foundation models [15, 16] to identify task-relevant objects with- out additional human inputs. Their common-sense reasoning ability helps recognize task-

---

relevant objects even if they are not directly in contact with other objects or the robot hands, allowing our method to imitate more diverse tasks than prior work;
3. We validate OKAMI's strong spatial and visual generalization abilities humanoid hard- ware. OKAMI enables real-robot deployment in natural environments with unseen object layouts, varying visual backgrounds, and new object instances.
2 Related Work
Humanoid Robot Control. Methods like motion planning and optimal control have been devel- oped for humanoid locomotion and manipulation [10, 12, 17]. These model-based approaches rely on precise physical modeling and expensive computation [11, 12, 18]. To mitigate the stringent requirements, researchers have explored policy training in simulation and sim-to-real transfer [10. 19J. However, these methods still require significant amount of labor and expertise in designing simulation tasks and reward functions, limiting their successes locomotion domains. In parallel to automated methods, variety of human control mechanisms and devices have been developed for humanoid teleoperation using motion capture suits [9, 12, 20-24], telexistence cockpits [25-29], VR devices [1, 30, 31J, or videos that track human bodies [17, 32). While these systems can control the robots to generate diverse behaviors, they require real-time human input that poses significant cognitive and physical burdens. contrast, OKAMI only requires single RGB-D human videos teach the humanoid robot new skills, significantly reducing the human cost.
Imitation Learning for Robot Manipulation. Imitation Learning has significantly advanced vision-based robot manipulation with high sample efficiency [33-44]. Prior works have shown that robots can learn visuomotor policies to complete various tasks with just dozens of demonstrations, ranging from long-horizon manipulation [34-36] to dexterous manipulation [37-39]. However, collecting demonstrations often requires domain expertise and high costs, creating challenges to scale. Another line of work focuses on one-shot imitation learning [40-44], yet they demand excessive data collection for meta-training tasks. Recently, researchers have looked into new problem setting of imitating from single video demonstration [4-6], referred to "open-world imitation from observation" [4]. Unlike prior works that abstract away embodiment motions due to kinematic differences between the robot and the human, we exploit embodiment motion information owing to the kinematic similarity between humans and humanoids. Specifically, we introduce object-aware retargeting that adapts human motions to humanoid robots.
Motion Retargeting. Motion retargeting has wide applications in computer graphics and 3D vision [8], where extensive literature studies how to adapt human motions to digital avatars [45-47]. This technique has been adopted in robotics for recreating human-like motions on humanoid or anthropomorphic robots through various retargeting methods, including optimization- based ap- proaches [11, 12, 20, 48], geometric-basee methods [49], and learning- techniques [10, 13, 17]. However, in manipulation tasks, these retargeting methods have been used within teleoperation systems, lacking vision pipeline for automatic adaptation object locations. OKA integrates the retargeting process with open-world vision, endowing it with object awareness so that the robot can mimic human motions from video demonstrations and adapt object locations at test time.
3 OKAМI
In this work, we introduce OKAMI, two-staged method that tackles open- world imitation from observation for humanoid robots. OK AMI first generates a reference plan using the object locations and reconstructed human motions from a given RGB-D video. Then, it retargets the human motion trajectories to the humanoid robot while adapting the trajectories based on new locations of the objects. Figure illustrates the whole pipeline.
We formulate humanoid manipulation task as discrete-time Markov Decision Process defined by tuple: M (S, P. Y, 1)), where S is the state space, A is the action space, P(-|s, a) is the transition probability, R(s) s) is the reward function, 7 E [0. 1) is the
Problem Formulation
3

---

'bortee
idennity Keyframes Through Changepoim Datactions
Hatur 1st of task: relevant ooject names
Track onjects across the video
Rercrence
RGB-D Video
Human Reconstruction Medel
lo
Reference Plan Generation
SMPL
loer fily targel, rel otojects and genorate reference
Robbr Chsarvation
Losa izo ralevant objocts lest time
Estimate between point elouds
Kinematlio
Reference Plan
Hand Finger Mapping
lo
IN
Target objects
Send Commands
Object-Aware Retargeting
SMPL- Trajectory
sagment
Using
Waraed motions
Hoost Execution
Figure 2: Overview of OKAМI. OKAMI is a two-staged method that enables humanoid robot to imitate a manipulation task from single human video. In the first stage, OKA generates reference plan using GPT- 4V and large vision models for subsequent manipulation. In the second stage, OKAMI follows the reference plan, where retargets human motions onto humanoid with object awareness. The retargeted motions are converted into sequence of robot joint commands for the robot to follow.
discount factor, and j is the initial state distribution. In context, S is the space of raw RGB-D observations that capture both the robot and object states, A is the space of the motion commands for the humanoid robot, R is the sparse reward function that returns 1 when task is complete. The objective of solving task is to find a policy that maximizes the expected task success rates from a wide range of initial configurations drawn from at test time.
We consider the setting of "open-world imitation from observation" [4], where the robot system takes recorded RGB-D human video, V as input, and returns humanoid manipulation policy that completes the task demonstrated in V. This setting is "open-world" as the robot does not have prior knowledge ground -ruth access to the categories or physical states of objects involved in the task, and it is "from observation" in the sense that video V does not come with any ground-truth robot actions. policy exccution is considered successful if the state matches the state the final frame from V. The success conditions of all tested tasks are described in Appendix B. Notably, two assumptions are made about in this paper: all the image frames in capture the human bodies, and the camera view of shooting V is static throughout the recording.
3.1 Reference Plan Generation
To enable object-aware retargeting, OKAMI first generates a reference plan for the humanoid robot to follow. Plan generation involves understanding what task-relevant objects are and how humans manipulate them.
Identifying and Localizing Task-Relevant Objects. To imitate manipulation tasks from videos V. O.MI must identify the task-relevant objects interact with. While prior methods rely on un- supervised approaches with simple backgrounds or require additional human annotations [50-53], OKAMI uses an off-the-shelf Vision-Language Models (VLMs), GPT-4, to identify task-relevant objects in V by leveraging the commonsense knowledge internalized in the model. Coneretely, OKAМI obtains the names of task-relevant objects by sampling RGB frames from the video demonstration V and prompting GPT-4V with the concatenation of these images (details in Appendix A.2). Using these object names, OKAMI employs Grounded SAM [16] to segment the objects in the first frame and track their locations throughout the video using Vidoe Object

---

Segmentation model, Cutie [54]. This process enables OKAMI to localize task-relevant objects in V, forming the basis for subsequent steps.
Reconstructing Human Motions. To retarget human motions to the humanoid robot, OKAMI reconstructs human motions from V to obtain motion trajectories. We adopt an improved version of SLAHMR [55], an iterative optimization algorithm that reconstructs human motion sequences. While SLAHMR assumes flat hands, our extension optimizes the hand poses of the SMPL-H model [56], which are initialized using estimated hand poses from HaMeR [57] (More details in Appendix A.1). This modification allows us to jointly optimize body and hand poses from monocular video. The output is a sequence of SMPL-H models capturing full-body and hand poses, enabling OKAMI to retarget human motions to humanoids (See Section 3.2). Additionally, the SMPL--H model can represent human poses across demographic differences, allowing easy mapping of motions from human demonstrators to the humanoid.
Generating a Plan from Video. Having identified task-relevant objects and reconstructed human motions, OKAMI generates reference plan from V for robots to complete each subgoal. OKAMI identifies subgoals by performing temporal segmentation on with the following procedure: We first track keypoints using CoTracker [58] and detect velocity changes of keypoints to determine keyframes, which correspond subgoal states. For each subgoal, we identify target object (in motion due to manipulation) and a reference object (serving as spatial reference for the target object's s movements through either contact or non-contact relations). The target object is determined based on the averaged keypoint velocities per object, while the reference object is identified through geometric heuristics or semantic relations predicted by GPT-4V (More implementation details of plan generation in Appendix A.4).
With subgoals and associated objects determined, generate reference plan ln- 4..... .v. where each step 4 corresponds to a keyframe and includes the point clouds of the target object Ouarget. the reference object Oreference. and the SMPL-H trajectory segment SMPL If no reference object is 4- required (e.g., grasping an object), Oreference is null. Point clouds are obtained by back-projecting segmented objects from RGB images using depth images [59].
3.2 Object-Aware Retargeting
Given a reference plan from the video demonstration, OKAMI enables the humanoid robot to imi- tate the task in V. The robot follows each step (, the plan by localizing task-relevant objects and retargeting the SMPL-H trajectory segment onto the humanoid. The retargeted trajectories are then converted into joint commands through inverse kinematics. This process repeats until all the steps are executed, and success is evaluated based on task-specific conditions (see Appendix B.
Localizing Objects at Test Time. To execute the plan in the test-time environment, OKAMI must localize the task-relevant objects in the robot's observations, extracting 3D point clouds track object locations. By attending to task-relevant objects, OKAMI policies generalize across various visual conditions, including different backgrounds the presence of novel instances of task-relevant objects.
Retargeting Human Motions to the Humanoid. The key aspect of object-awareness is adapting motions to new object locations. After localizing the objects, we employ a factorized retargeting process that synthesizes arm and hand motions separately. OKAMI first adapts the arm motions to the object locations so that the fingers of the hands are placed within the object-centric coordinate frame. Then OKAMI only needs to retarget fingers in the joint configuration to mimic how the demonstrator interacts with objects with their hands.
Concretely, we first map human body motions to the task space of the humanoid, scaling and adjust- ing trajectories to account for differences in size and proportion. OK AM then warps the retargeted trajectory so that the robot's arm reaches the new object locations (More details in Appendix A.5). We consider two cases in trajectory warping when the relational state between target and refer- ence objects is unchanged and when it changes, adjusting the warping accordingly. In the first case,
5

---

Sprinkle-salt
Plush-toy-in-basse
Close-the-laptop
Close- the tte-thheaeeeeeee drawer
Bagging
Figure 3: Visualization of initial and final frames of both human demonstrations and robot rollouts for all tasks.
we only warp the trajectory based on the target object locations; in the second case, the trajectory is warped based on the reference object location.
After warping, we inverse kinematics compute sequence of joint configurations for the arms while balancing the weights of position and rotation targets in inverse kinematics computa- tion to maintain natural postures. Simultaneously, we retarget the human hand poses to the robot's s finger joints, allowing the robot to perform fine-grained manipulations (Implementation details in Appendix A.3). In the end, we obtain a full-body joint configuration trajectory for execution. Since arm motion retargeting affine, our process naturally scales and adjusts motions from demonstrators with varied demographic characteristics. By adapting arm trajectories object locations and retar- geting hand poses independently, OKAMI achieves generalization across various spatial layouts.
Experiments
Our experiments designed to answer the following research question: 1) Is OKAМI effective for humanoid robot to imitate diverse manipulation tasks from single videos of human demonstra- tion? 2) Is it critical in OKAMI to retarget the body motions of demonstrators to the humanoid robot instead of only retargeting based on object locations? 3) Can OKAMI retain its performances con- sistently on videos demonstrated by humans of diverse demographies? 4) Can the rollouts generated by OKAM AMI used for training closed-loop visuomotor policies?
4.1 Experimental Setup
Task Designs. We describe the six tasks use in the experiments: 1) PlPPsshttoy basket:: placing plush toy in the basket; 2) Sprinkle-salt: sprinkling bit of salt into the bowl; 3) Close-the-drawes
pushing the drawer in to close it; 4) c1 ose-the-laptopp: closing the lid of the laptop: 5) P Place-sssiie Mance-snae www.ssssccss--sos on-plate: placing a bag of snacks on the plate. 6) Bagging: placing a chip bag into a shopping bag. We select these six tasks that cover diverse range of manipulation behaviors: require pick-and-plac behaviors daily objects; Sprinkle-sal ing behavior; Close-the- drawer and close-ther laptop require the humanoid interact with articulated objects, prevalent type of interaction in daily environments; Bagging involves dexterous, bimanual manipulation and includes multiple subgoals. While we mainly focus on real
and Place Place-snacks snteeee-snccks----pltete
the task that covers pour-

---

pesppig
Fibled Coumpaaioo
Demoroenet
83.0% 53
66.7%
68.7%
58.9%
68.9%
50.3%
Cortnkle-satt
Close-tte-
Clase-the-
Place-srarks- plase
Raggtng
Place-srarke snarks- plaso
Clase-the-
basket
laptop
drower
laptop
Figure 4: (a) Evaluation OKAMI all six tasks, including the failed trials, separated by failure mode. (b) Evaluation of OKAMI using videos from different demonstrations. Demonstrator is the main person recording videos for all evaluations in (a).
and the quantification of
and Close drawer simula- tion using RoboSuite [60] for easy reproducibility of OKAMI. See Appendix B.4.
robot experiments, we also implement
Hardware Setup. We use Fourier GRI robot our hardware platform, equipped with two 6-DoF Inspire dexterous hands and D435i Intel RealSense camera for video recording and test-time obser- vation. We implement joint position controller that operates at 400Hz. To avoid jerky movements, we compute joint position commands at 40Hz and interpolate the commands to 400Hz trajectories.
Evaluation Protocol. We 12 trials for each task. The locations of the objects are randomly initialized within the intersection of the robot camera's view and the humanoid arms' reachable range. The tasks are evaluated on tabletop workspace with multiple objects, including both task-relevant objects and various other objects. Further, we test new object generalization on P Place- snacks-on on-plateeeeaa wttpppeeeesssteeteeeppp -oush-toy-in--aseeeett -inkkeeeekee. and Sprinkle mrozinkle-salt saltt tasks, chang- ing the involved plate, snack bag, plush toy, and bowl to other instances of the same type.
Baselines. We compare result with baseline ORION [4]. Since ORION parallel-jaw grippers, it is not directly applicable in our experiments and we adopt it with minimal modifications: we estimate the palm trajectory using the SMPL-H trajectories, and warp the tra- jectory conditioning on the new object locations. The warped trajectory is used in the subsequent inverse kinematics for computing robot joint configurations.
proposed for
4.2 Quantitative Results
To answer question (1), we evaluate the policies of OKAMI across all the tasks, covering diverse behaviors such daily pick-place, pouring, and manipulation of articulated objects. The results presented in Figure 4(a). In our experiment, we randomly initialize the object locations so that the robot needs to adapt to the locations of the objects. This result shows the effectiveness of OKAMI in generalizing over different visual and spatial conditions.
To answer question (2), we compare OKAMI against ORION on representative tasks, and close-the-lapter OKAMI differs from ORION in that ORION does not condition the human body poses. OKAМI achieves 75.0% and 83.3% success rates, respectively, while ORION only achieves 0.0% and 41.2%, respectively. Additionally, we compare OKAMI against ORION on the two simulated and
Place-snacks-on-p22.
In the comparison experiment,
versions of Sprinkle-seae 82.0% and 84.0% success rates two tasks while ORION only achieves 0.0% and 10.0%. Most failures of ORION policies are due failing approach objects with reliable grasping poses (e.g., in Place- swwcks-ooooooooo the top-down grasp in human video), and failing to rotate the wrist fully achieve behaviors such as pouring. These behaviors originate from the fact that ORION ignores the embodiment information, thus falling short in performance compared to OK AMI. The superior performance of OK AMI sug- gests the importance of retargeting the body motion of the human demonstrators onto the humanoid when imitating from human videos.
tasks. In simulation, OKAMI achieves
ate task, ORION tries grasp the snack from the sides instead of
To answer question (3), conduct controlled experiment of recording vidcos of different demonstrators and test if OKAMI policies maintain strong performance across the video inputs.

---

Same as the previous experiment, we evaluate OKAMI on the PLace-snacks-oopllate and close-the-laptoo tasks. The results are presented in Figure 4(b). We show that for the task close-the-lapton, there is no statistical significance in performance change. As for task place-snacks-on-
while the evaluation maintains above 50%, the worst policy per- formance 16.7% worse than the best policy performance. After looking into the video recording, find that the motion of demonstrator 2 is relatively faster than the other two demonstrators, and faster motions create a noisy estimation of motion when doing human model reconstruction. Over- all, OK AMI can maintain reasonably good performance given videos from different demonstrators, but there is room for improvements on our vision pipeline to handle such variety.
4.3 Learning Visuomotor Policy With OKAМI Rollout Data
We address question (4) by training neural visuomotor policies on OKAMI rollouts. We first run OKAMI over randomly initialized object layouts to generate multiple rollouts and collect dataset of successful trajectories while discarding the failed ones. We train neural network policies on this dataset through a behavioral cloning algorithm. Since smooth execution is critical for humanoid manipu- lation, we implement the behavioral cloning with ACT [61], which predicts smooth actions via its temporal ensemble design, a trajectory smoothing component (more implementation details in Appendix B.5). We train visuomotor policies for Sprinkle-sa Figure 5 illustrates the success rates of these policies, demonstrating that OKAMI rollouts are effective data sources for training. We also show that the learned policies improve as more rollouts are collected. These results hold the promise of scaling up data collection for learning humanoid manipulation skills without laborious teleoperation.
50 Trajectories
100 Trajectories
83.3%
75.0%
58.3%
Sprinkle-salt
Bagging
Figure 5: Success rates of learned visuomotor policies on Sprinkle-s salt and bagging using 50 and 100 trajectories, respectively.
and Ragging
5 Conclusion
This paper introduces OKAMI that enables a humanoid robot to imitate a single RGB-D human video demonstration. At the core of OKAMI is object-aware retargeting, which retargets the human motions onto the humanoid robot and adapts the motions to the target object locations. OKAMI consists of two stages to realize object-aware retargeting. The first stage generating a reference plan for manipulation from the video. The second stage used for retargeting, where OKAMI retargets the arm motions in the task space and the finger motions in the joint configuration space. Our experiments validate the design of OKAMI, showing the systematic generalization of OKAМI policies. OKAMI enables efficient collection of trajectory data based on single human video demonstration. OKAMI-based data collection significantly reduces the human cost for policy train- ing compared to that required by teleoperation.
Limitations and Future Work. The current focus of OKAMI is the upper body motion retargeting of humanoid robots, particularly for manipulation tasks within tabletop workspaces. A promising future direction is to include lower body retargeting that enables locomotion behaviors during video imitation. To enable full-body loco-manipulation a whole-body motion controller needs to be implemented as opposed to the joint position controller used in OKAMI. Additionally, we rely on RGB-D videos in OK AMI, which limits us from using in-the-wild Internet videos recorded in RGB. Extending OKAMI to use web videos will be another promising direction for future works. At last, the current implementation of retargeting has limited robustness against large variations in object shapes. A future improvement would be integrating more powerful foundation models that endow the robot with general understanding of how to interact with a class of objects in spite of their large shape changes.
8

---

Acknowledgments
We would like thank William Yue for providing the initial implementation of the behavioral cloning policies, Peter Stone for his valuable support with task designs and demo shooting, Yuzhe Qin for sharing the dex-retargeting codebase, and Zhenjia Xu for his advice on developing the humanoid robot infrastructure.
References
[1] M. Seo, S. Han, K. Sim, S. H. Bang, C. Gonzalez, L. Sentis, and Y. Zhu. Deep imitation learn- ing for humanoid loco-manipulation through human teleoperation. In JEEE-RAS International Conference on Humanoid Robots (Humanoids), 2023.
[2] Y. Matsuura, K. Kawaharazuka, N. Hiraoka, K. Kojima, K. Okada, and M. Inaba. Development of whole-body work imitation learning system by biped and bi-armed humanoid. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 10374- 10381. IEEE, 2023.
[3] Asfour, P. Azad, F. Gyarfas, and R. Dillmann. Imitation learning of dual-arm arm manipulation tasks in humanoid robots. International journal of humanoid robotics, 5(02):183 10002)283-222 2008.
[4] Y. Zhu, A. Lim, P. Stone, and Y. Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint arXiv:2405.20321 2024.
[5] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada. Ditto: Demonstration imitation by trajectory transformation. arXiv preprint 2xXi::2403.12003 2024.
[6] D. Guo. Learning multi-step manipulation tasks from a single human demonstration. arXiv preprint arXiv:2312.153455 2023.
[7] T. Asfour and R. Dillmann. Human-like motion of a humanoid robot arm based on a closed-
form solution of the inverse kinematics problem. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) Cat. No. 03CH37453), volume 2, pages 1407-1412. IEEE, 2003.
[8] M. Gleicher. Retargetting motion to new characters. Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998.
[9] K. Darvish, Y. Tirupachuri, G. Romualdi, L. Rapetti, D. Ferigo, F. A. Chavez, and D. Pucci. Whole-body geometric retargeting for humanoid robots. In 2019 JEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 679-686. IEEE, 2019.
[10] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-hody control for humanoid robots. arXiv preprint arXiv:4402.16796 2024.
[11] S. Nakaoka, A. Nakazawa, Kanehiro, Kaneko, M. Morisawa, and K. Ikeuchi. Task model of lower body motion for a biped humanoid robot to imitate human dances. In 2005 JEEE/RS. International Conference on Intelligent Robots and Systems, pages 3157-3162. IEEE, 2005.
[12] K. Hu, C. Ott, and D. Lee. Online human walking imitation in task and joint space based on quadratic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3458-3464. IEEE, 2014.
[13] S. Choi, M. K. Pan, and J. Kim. Nonparametric motion retargeting for humanoid robots on shared latent space. In Robotics: Science and Systems, 2020.
[14] E. Demircan, Besier, S. Menon, and O. Khatib. Human motion reconstruction and synthesis of human skills. In Advances in Robot Kinematics: Motion in Man and Machine: Motion in Man and Machine, pages 283-292. Springer, 2010.

---

[15] OpenAl. Gpt-4 technical report, 2023.
[16] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 30.09999 2023.
[17] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning real-time whole-body teleoperation. In arXiv, 2024.
[18] A. Escande, N. Mansard, and P.-B. Wieber. Hierarchical quadratic programming: Fast online humanoid-robot motion generation. The International Journal of Robotics Research, 33(7): 1006-1028, 2014.
[19] Q. Liao, B. Zhang, X. Huang, X. Huang, Li, and Sreenath. Berkeley humanoid: A research platform for learning-based control. arXiv preprint arXiv:2407.21781, 2024.
[20] I.. Penco, N. Scianca, V. Modugno, L. Lanari, G. Oriolo, and S. Ivaldi. A multimode teleop- eration framework for humanoid mocommanipulation: An application for the icub robot. IEEE Robotics & Automation Magazine, 26(4):73-82, 2019.
[21] D. Kim, B.-J. You, and S.-R. Oh. Whole body motion control framework for arbitrarily and simultaneously assigned upper-body tasks and walking motion. Modeling. Simulation and Optimization of Bipedal Walking, pages 87-98, 2013.
[22] A. Di Fava, K. Bouyarmane, K. Chappellet, E. Ruffaldi, and A. Kheddar. Multi-contact motion retargeting from human to humanoid robot. In 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids), pages 1081-1086. IEEE, 2016.
[23] M. Arduengo, A. Arduengo, A. Colomé, J. Lobo-Prat, and C. Torras. Human to robot whole- body motion transfer. 2020 JEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), pages 299-305. IEEE, 2021.
[24] R. Cisneros, M. Benallegue, K. Kaneko, H. Kaminaga, G. Caron, A. Tanguy, R. Singh, L. Sun, A. Dallard, C. Fournier, et al. Team janus humanoid avatar: A cybernetic avatar to embody human telepresence. In Toward Robot Avatars: Perspectives on the ANA Avatar XPRIZE Competition, RSS Workshop, volume 3, 2022.
[25] Tachi, K. Komoriya, K. Sawada, T. Nishiyama, Itoko, M. Kobayashi, and K. Inoue. Telexistence cockpit for humanoid robot control. Advanced Robotics, 17(3):199-217, 2003.
[26] J. Ramos and S. Kim. Humanoid dynamic synchronization through whole-body bilateral feed- back teleoperation. JEEE Transactions on Robotics, 34(4):953 965, 2018.
[27] Y. Ishiguro, T. Makabe, Y. Nagamatsu, Y. Kojio, K. Kojima, F. Sugai, Y. Kakiuchi, K. Okada, and M. Inaba. Bilateral humanoid teleoperation system using ahhol-bodyy exoskeleton cockpit tablis. IEEE Robotics and Automation Letters, 5(4):6419-6426 2020.
(28) F. Abi-Farrajl, B. Henze, A. Werner, M. Panzirsch, C. Ott, and M. A. Roa. Humanoid teleoper- ation using task-relevant haptic feedback. In 2018 JEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pages 5010-5017, 2018. doi: 100.110//RROS.00 2018.8593521.
[29] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke. Nimbro avatar: Interactive immersive telepresence with foedba-c telemanipulation. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5312-5319. IEEE, 2021.
[30] M. Hirschmanner, C. Tsiourti, T. Patten, and M. Vineze. Virtual reality teleoperation of humanoid robot using markerless human upper body pose imitation. in 2019 iece-ras 19th international conference on humanoid robots (humanoids), 2019.
10

---

[31] D. Lim, D. Kim, and J. Park. Online telemanipulation framework on humanoid for both manip-
ulation and imitation. 2022 19th International Conference on Ubiquitous Robots (UR), pages 8-13, 2022. URL https::///// / /ttps:///ppi.. semant www.eetiiannooo..e.oe........oee wwcchllr........oo perg/CorpusIDD 250577582.
[32] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn, Humanplus: Humanoid shadowing and imitation from humans. arXiv preprini urXiv:2406. 2024.
[33] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany. C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108, 03298, 2021.
[34] A. Mandlekar, D. Xu, R. Martin-Martín, Savarese, and L. Fei-Fei. Learning to general- ize across long-horizon tasks from human demonstrations. arXiv preprini urXiv:2003.06085,
2020.
[35] C. Wang. L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.124222 2023.
[36] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210. 1339, 2022.
[37] C. Wang. Shi, W. Wang, R. Zhang, Fei-Fei, and C. K. Liu. Dexeap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788 2024.
[38] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, Levine, and J. Malik. Learning visuotactile skills with two multifingered hands. arXiv preprint arXiv:2404. 16823, 2024.
[39] Y. Ze, G. Zhang, Zhang, Hu, M. Wang, and Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954, 2024.
[40] M. Chang and S. Gupta. One-shot visual imitation via attributed waypoints and demonstration augmentation. arXiv preprint arXiv:2302.0488. 2023.
[41] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot composition of vision-based skills from demonstration. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2643-2650. IEEE, 2019.
[42] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns. Demonstrate once, imitate imme- diately (dome): Learning visual servoing for one-shot imitation learning. In 2022 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 8614-8622 IEEE, 2022.
[43] E. Johns. Coarse-to imitation learning: Robot manipulation from a single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pages 4613-4619. IEEE, 2021.
[44] N. Di Palo and E. Johns. Learning multi-stage tasks with one demonstration via self-replay. In Conference Robot Learning, pages 1180-1189. PMLR, 2022.
[45] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895-10904, 2023.
[46] X. B. Peng, Z. Ma. P. Abbeel, Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (ToG), 40(4): 1-20, 2021.

---

[47] B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[48] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake. Optimization-baser locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 40:429-455 2016.
[49] Y. Liang, W. Li, Y. Wang, R. Xiong, Y. Mao, and J. Zhang. Dynamic movement primitive based motion retargeting for dual-arm sign language motions. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 8195-8201. IEEE, 2021.
[50] Caelles, J. Pont-Tuset, F. Perazzi, A. Montes, K.-K. Maninis, and L. Van Gool. The 2019 davis challenge on vos: Unsupervised multi-object segmentation. arXiv preprint arXiv:1905.0073: 2019.
[51] Y. Huang. J. Yuan, C. Kim, Pradhan, B. Chen, L. Fuxin, and Hermans. Out of sight, still mind: Reasoning and planning about unobserved objects with video tracking enabled memory models. arXiv preprint arXiv:2309. 15278, 2023.
[52] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In 7th Annual Conference on Robot Learning, 2023.
[53] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishhan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xia, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint Sttv:2303300000. 2023.
[54] H. K. Cheng, W. Oh, Price, J.-Y. Lee, and A. Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151-3161, 2024.
[55] V. Ye, G. Pavlakos, Malik, and A. Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR, 2023.
[56] J. Romero, D. Tzionas, and M. J. Black. Embodied hands. ACM Transactions on Graphics, 36 (6):1-17,2) 2017.
[57] G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024.
[58] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: is better to track together. arXiv preprint arXiv:2307.0763 2023.
[59] Q--. Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint
2018
[60] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Marte A. Joshi, S. Nasiriany, and Y. Zhu. bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint 2009 12293, 2020.
[61] T. 7. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 13305 2023.
[62] S. Goel, G. Pavlakos, Rajasegaran, A. Kanazawa, and Malik. Humans in 4D: Reconstruct- ing and tracking humans with transformers. In ICCV, 2023.
[63] Y. Xu, J. Zhang, Q. Zhang, and D. Tao. ViTPose++: Vision transformer for generic body pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] S. Caron, Y. De Mont-Marin, R. Budhiraja, and S. H. Bang. Pink: Python inverse kinematics based on Pinocchio, 2024. URL https: htt/t/gttuu///t// coom/stepaane-eeee cateeee-crooooppi..i/
12

---

[65] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W. Chao, and D. Fox. Anyteleop: general vision-based dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, 2023.
[66] R. Killick, P. Fearnhead, and I. A. Eckley. Optimal detection of changepoints with linear computational cost. Journal of the American Statistical Association, (115000:1590-1598 2012.
[67] X. Cheng, J. Li, S. Yang. G. Yang, and X. Wang. Open-television: Teleoperation with immer- sive active visual feedback. arXiv preprint drXhv:2407.01512 2024.
[68] Darcet, M. Oquab, Mairal, and Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16566 2023.
[69] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, Sharma, S.-W. Li, W. Galuba, M. Rabbat, Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, Mairal, Labatut, A. Joulin, and Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
13

---

A Implementation Details
A.1 Human Reconstruction From Videos
Method. For the 3D human reconstruction, we start by tracking the person in the video and getting an initial estimate of their 3D body pose using 4D Humans [62]. This body reconstruction cannot capture the hand pose details (i.e., the hands are flat). Therefore, for each detection of the person in the video, we detect the two hands using ViTPose [63], and for each hand, we apply HaMeR [57] to get an estimate of the 3D hand pose. However, the hands reconstructed by HaMeR can be incon- sistent with the arms from the body reconstruction (e.g.. different wrist orientation and location). To address this, we apply an optimization refinement to make the body and the hands consistent in each frame, and encourage that the holistic body and hands motion is smooth over time. This op- timization similar SLAHMR [55], with the difference that besides the body pose and location of the SMPL+H model [56], we also optimize the hand poses. We initialize the procedure using the 3D body pose estimate from 4D Humans and the 3D hand poses from HaMeR. Moreover, we use the 2D projection of the 3D hands predicted by HaMeR to constrain the projection of the 3D hand keypoints of the holistic model using a reprojection loss. Finally, we can jointly optimize all the parameters (body location, body pose, hand poses) over the duration of the video, as described in SLAHMR [55].
Our modified SLAHMR incorporates the SMPL-H model [56] to include hand poses in the hu- man motion reconstruction. We initialize hand poses in each frame using 3D hand estimates from HaMeR [57]. The optimization process then jointly refines body locations, body poses, and hand poses over the video sequence. This joint optimization allows for accurate modeling of how hands interact with objects, which is crucial for manipulation tasks.
The optimization minimizes the error between the 2D projections of the 3D joints from the SMPL-H model and the detected 2D joint locations from the video. We use standard parameters and settings as described in SLAHMR [55], adapting them to accommodate the SMPL-H model.
Inference Requirements. The model of human reconstruction we use is large and needs to run on a computer with sufficiently good computation speed. Here we provide details about the runtime performance of the human reconstruction model. We a desktop that comes with GPU RTX3090 that has the size of the memory 24 GB. For 10 seconds video with fps 30, it processes 10 minutes.
A.2 Prompts of Using GPT4V
In order to use GPT4V in OKAMI, need GPT4V's output be in typed format that the rest of the programs can parse the result. Moreover, in order for the prompts to be general across diverse set of tasks, our prompt does not leak any task information to the model. Here we describe the three different prompts in OKAMI for using GPT4V.
Identify Task-relevant Objects. OKAMI uses the following prompt invoke GPT4V that identify the task-relevant objects from provided human video:
Prompt: You need analyze what the human is doing in the images, then tell 1. All the objects in front scene (mostly on the table). You should ignore the background objects. 2. The objects of interest. They should be subset of your answer to the first question. They are likely the objects manipulated by human or near human. Note that there are irrelevant objects in the scene, such as objects that does not move at all. You should ignore the irelevant objects.
Your output format is:
The human is xxx.
All objects are xxx.
The objects of interest are: "ijson
14

---

"objects": ["OBJECT1", "OBJECT2", ...l,
Ensure the response can be parsed by Python 'json.loads', e.g.: no trailing commas, no single quotes, etc. You should output the names of objects of interest a list ["OBJECTI",, "OB- JECT2", ...] that can be easily parsed by Python. The name string, egg. "apple", "pen", "keyboard", etc.
Identify Target Objects. OKAMI uses the following prompt to identify the target object of each step in the reference plan:
Prompt: The following images shows a manipulation motion, where the human is manipulating an object.
Your task is to determine which object being manipulated in the images below. You need choose from the following objects: {a list of task-relevant objects).
Tips: the manipulated object is the object that the human is interacting with, such as picking up, moving, or pressing, and it is in contact with the human's 's {the major moving arm in this step) hand.
Your output format is:
''json
"haniipulate__bbjett_rra
"MANTFULATE_LOBECTTLLAA
Ensure the response can be parsed by Python 'json.loads', e.g.: no trailing commas, no single quotes, etc.
Identify Reffrence Objects. Here is the prompt that asks GPT4V to identify the reference object of each step in the reference plan:
Prompt: The following images shows a manipulation motion, where the human is manipulating the object imanipulate_objcee .bjec...ammee... name].
Please identify the reference object in the image below, which could be object which to place caanipulaae_object_naee with. Note that there may not necessarily have an reference object, sometimes human may just playing with the object itself, like throwing it, or spinning it around. You need to first identify whether there a reference object. so, you need to output the reference object's name chosen from the following objects: (a list of relevatl objects).
an object that caanipulaae_objectnnaaa is interacting
Your output format is:
"**json
"reference_object_nnnee
TREFFRENCE_OBJJCTT_NN D
or "None",
Ensure the response can be parsed by Python 'json.loads' e.g.: no trailing commas, no single quotes, etc.
15

---

A.3 Details on Factorized Process for Retargeting
Body Motion Retarget. To retarget body motions from the SMPL-- representation to the hu- manoid, we extract the shoulder, clbow, and wrist poses from the SMPL-H models. We then use inverse kinematics to solve the body joints on the humanoid, ensuring they produce similar shoulder and elbow orientations and similar wrist poses. The inverse kinematics is implemented using an open-sourced library Pink [64]. The IK weights we use for shoulder orientation, elbow orientation, wrist orientation, and wrist position are 0.04, 0.04, 0.08, and 1.0, respectively.
Hand Pose Mapping. As we describe in the method section, we first retarget the hands from SMPI.-H models to the humanoid's dexterous hands using a hybrid implementation of inverse kine- matics and angle mapping. Here are the details of how this mapping is performed. Once we obtain the SMPL-H models from video demonstration, we can obtain the locations of 3D joints from the hand mesh models from SMPL-H. Subsequently, we can compute the rotating angles of each joint that correspond to certain hand poses. Then we apply the computed joint angles to the hand meshes of a canonical SMPL-H model, which is pre-defined to have the same size as the humanoid robot hardware. From this canonical SMPL--H model, we can get the 3D keypoints of hand joints and use an existing package, dex-retarget, an off-the-shelf optimization package to directly compute the hand joint angles of the robot [65].
Inverse Kinematics. After warping the arm trajectory, we use inverse kinematics to compute the robot's joint configurations. We assign weights of 1.0 to hand position and 0.08 to hand rotation, prioritizing accurate hand placement while allowing the arms to maintain natural postures.
For retargeting human hand poses to the robot, we map the human hand joint angles the corre- sponding joints in the robot's hand. This enables the robot replicate fine-grained manipulations demonstrated by the human, such grasping and object interaction. Our implementation ensures that the retargeted motions are physically feasible for the robot and that overall execution appears natural and effective for the task at hand.
A.4 Additional Details of Plan Generation
For temporal segmentation, we sample keypoints from the segmented objects in the first frame and track them across the video using CoTracker [58]. We compute the average velocity of these keypoints at each frame and apply an unsupervised changepoint detection algorithm [66] detect significant changes in motion, identifying keyframes that correspond to subgoal states.
To determine contact between objects, we compute the relative spatial locations and distances be- tween the point clouds objects. the distance between objects falls below a predefined threshold, we consider them to be in contact. For non-contact relations that are difficult to infer geometri- cally-such as a cup pouring task use GPT4V predict semantic relations based on the visual context. GPT4V can infer that the cup is the recipient in pouring action even if there is direct contact.
A.5 Trajectory Warping
Here, we mathematically describe the process of trajectory warping. We denote the trajectory for robot as rabat retargeted from SMPI. in the generated plan. Denote the starting point and end point Ti:fe41
of rbout as Paaans Pend, respectively. Note that all points along the trajectory are represented in SE(3) space.
Each point pt on the original retargetd trajectory can be described by the following function:
Pt = Pstant + robot (1) - Psmn)
where i E (ti...., (,,1,,+11,,,1 ). rooot robol((t) (t;) =
Pstart rrobol rrobol(t (f++1) = Pend-
When warping the trajectory, either only needs to adapt the trajectory to the target object location, adapt the trajectory to the new locations of both the target and the reference objects,
16

---

as described in Section 3.2. Without loss of generality, we denote the SE(3) transformation for the starting point sun, and the SE(3) transformation for the end point is Tod- Now the warped trajectory can be described by the following function:
Pt start Pstart + (f) Tstart Pstart)
(2)
where probot(1) (t) - - (t) Puin (Tend Pend Ttaan Poturi) + Tsturt Pstart Vt E ( (.+1). }. In this way,
Prod Paae:
we have probol (t,) = ssttnt Pstarts (robo((4441) = Tend Pendd- Note that this trajectory warping assumes the end point of trajectory not the same as the starting point, which is common assumption for most of the manipulation behaviors.
B
Additional Experimental Details
B.1 Success Conditions
We describe the success conditions we use to evaluate if a task rollout is successful or not.
Sprinkle-salt: The salt bottle reaches position where the salt is poured out into the bowl.
P Plush Plush-toy-in-te toy- The plush toy is put inside the container, with more than 50% of the toy inside the container.
clsse-the-laptop: The display is lowered towards the base until the two parts meet at the hinge (aka the laptop is closed).
The drawer is pushed back the containing region, either it's drawer or a layer of cabinet.
Srose--tee-zeee
The snack placed on top of the plate, with more than 50% of the snack package on the plate.
P paacessaccks-on-pLee
Bagging: The chip bag is put into the shopping bag which is initially closed.
B.2 Implementation Baseline
We implement the baseline ORION [4] with minimal modifications apply it our humanoid setting. First, estimate the palm trajectory from SMPL-H trajectories by using the center point of the reconstructed fingers the palm position at each time step. Next, we warp the palm trajectory based on the test-time objects' locations. Finally, we use inverse kinematics to solve for the robot's s body joints, with the warped trajectory serving as the target palm position.
B.3 Details on Different Demonstrators
Figure 6 shows the videos of three different human demonstrators performing Place- Place-snacks wwwsssaacssaee oaa--o--p22 on-plate and Close- -aa-ttheeelpggg laptop tasks. We calculate the success rates of imitating different videos. and the results are shown Figure 4(b).
B.4 Simulation Evaluation
For easy reproducibility, replicate two tasks, Sprinkle-sale and Close-the-dres in simulation (Figure 7). We implement these tasks using robosuite [60], which recently provided cross-embodiment support, including humanoid manipulation. We use "GRIFixeeLLowerBddy as the robot embodiment in these two tasks.
Note that for the policy of each task, we use the same human video as the ones used real robot experiments. We compare three methods in simulation: OKAMI (w/vision), OKAМI (wo vision), and ORION. OKAMI (w/vision) the same method (w/o vision) is the simplified version of OKAMI where we assume the model directly gets the
in our real robot experiments. OKAМI
17

---

Figure 6: The initial and end frames of videos performed by different human demonstrators. The first row is ceteo--nncks----p-hae task, and the sccond row is Close-the-lapte task.
ground-truth poses of objects. The evaluation results are shown in Table 1, where each reported number is the success rate averaged over 50 rollouts.
We notice that the simulation results are generally better than the real robot experiments. The per- formance difference comes from the easy physical interaction between dexterous hands and objects compared to the real robot hardware. Also, OKAMI without vision achieve much higher suc- cess rate than OKAMI with vision because the noise and uncertainty of perception abstracted away. Specifically, a large portion of uncertainties come from the partial observation of object point clouds, and the estimation of the object location is off the ground ground-truth truth locations of objects, while the success of OK AMI highly depends on the quality of trajectory warping, which dependent on the correct estimation of object locations. This simulation result also indicates that the performance of OKAMI is expected improve if more powerful vision models with higher accuracy are available.
Method
Sprinkle -altt Close the drawer
OKAMI (w/ vision)
82%
84%
OKАМТ (w/o vision)
100%
100%
ORION
0%
10%
Table The average success rates (%) across different methods in two tasks, Sprinkle- and Close-the-on the-drawer
B.5 Visuomotor Policy Details
We choose ACT [61] in our experiments for behavioral cloning, an algorithm that has been shown effective in learning humanoid manipulation policies [67]. Notably, we choose pretrained Di- no [68, 69] the visual backbone of policy. The policy takes single RGB image and 26- dimension joint positions as input and outputs the action of the 26-dimension absolute joint position for the robot to reach. In Table 2, we show the hyperparameters used for behavioral cloning.
18

---

KL weight
10
chunk size
60
hidden dimension
512
batch size feedforward dimension
45
3200
epochs
25000
learning rate
5e-5
temporal weighting
0.01
Table 2: The hyperparameters used in ACT.
Close-the-drawer
Sprinkle-salt
Figure The screenshots of starting and ending frames of two simulation tasks, Close-the-drawer and Spwrnkle-sssllll....
19


Reviews:
[{"review_id": "r9l8vff3Pm", "reviewer": "Reviewer_Qi9R", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the problem of teaching humanoid robot manipulation skills through single human video. The proposed method identifies task relevant objects with VLM such as GPT4V, and extracts human trajectories with SMPL-H. The human trajectories are retargeted to robot agent and transformed into the local coordinates of test objects. The joint configurations for controlling the robot are then derived through inverse kinematics. Experiments are performed on a humanoid robot in the real world.", "summary_of_recommendation": "My rating is based on the writing issues and insufficient experiment results.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Questions and suggestions: \n\nHave the authors attempted tasks that require fine-grained motions, in addition to those achievable with coarse motion? Comparing the proposed method with additional baselines, such as Behavior Cloning-based approaches, could also strengthen the work; \n\nI would suggest the authors include more implementation details on the identification of reference objects and target objects for each subgoal, the extraction of point clouds, and the adaptation and warping of human trajectories to robot agents and test object coordinates. Providing mathematical formulas would also be helpful.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MOstw67Zre", "reviewer": "Reviewer_XCSN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel framework called OKAMI, which allows humanoid robots to imitate manipulation tasks demonstrated by humans in single RGB-D video clips. The method proposed addresses the challenges inherent in \"open-world imitation from observation\" settings, where robots are required to understand and adapt to new environments without explicit annotations of actions or prior knowledge of object states.", "summary_of_recommendation": "This is a good paper, I suggest to accept.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Can you demonstrate how the potential of OKAMI on series tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7sBx5mLTd", "reviewer": "Reviewer_MHD2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "humanoid robotic imitation by one-shot human motion retargeting from demonstration videos", "summary_of_recommendation": "ok tech report with missing details", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Especially the section comparing the performance of OKAMI to ORION it was not clear how grasps were computed in this case. Large differences in success rates were reported, but it was difficult to attribute these to particular failure modes and thus get a good intuition of what problems OKAMI avoids that ORION is making. In addition to this, having more than two tasks would be helpful in better demonstrating the advantages of OKAMI.\n\nHow exactly is the motion re-targeting done, given that the rendered SMPL-H model contains several relevant fingers and contact points how is the combination of these weighted and mapped to the humanoid hand and for each point how are differences e.g. in rotation and position weighted. Is this re-targeting done as an optimization procedure or a one-step calculation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ce4Zc9fdYr", "reviewer": "Reviewer_MV4q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a method to solve humanoid robot manipulation tasks from a single human video. The approach consists of two stages. In the first stage, the provided human video is processed in order to extract a reference trajectory of the task. In the second stage, the reference trajectory is retargeted to take into account the given scene arrangement at test time, as well as the similar but different kinematics of the humanoid compared to the human demonstrator. The method is evaluated on 5 real world tasks against one baseline (ORION).", "summary_of_recommendation": "High quality paper, more a system architecture than an algorithmic contribution", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "During the rebuttal, the authors should address the points above. I would like to stress the fact that I believe that this work is already in the current form of high quality, and that the points I mentioned above are meant to make it exceptional.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- A promising approach for teaching humanoid robots new skills from a single RGBD video demonstration, with real-world experiments.
- The paper quality is great overall. The experiments conducted are robust, with user studies and comparisons to one baseline method.


Weaknesses:
- Across the reviews, a common concern is the lack of detailed explanation on key components like trajectory warping, object identification, and adaptation processes, which affects the reproducibility and understanding of the methods.
- Reviewers raise questions around the analysis of the presented results. Why does ORION fail while OKAMI succeeds in experiments? Perhaps a more thorough analysis can be performed in simulation, including an ablation experiment.

Post rebuttal discussion:
Authors provide more details missing in initial paper, as well as analysis. Overall a good paper.
Decision: Accept