Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Authors: Xuxin Cheng
Jialong Li
Shiqi Yang
Ge Yang
Xiaolong Wang
Venue: CoRL 2024 (2024)
Keywords: Teleoperation
VR/AR
Imitation Learning
TL;DR: An immersive teleoperation system with active visual feedback

Abstract: Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system $\textbf{Open-TeleVision}$ that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (can sorting, can insertion, folding, and unloading) for 2 different humanoid robots and deploy them in the real world. The entire system will be open-sourced.

Open-TeleVisionn Teleoperation with
Immersive Active Visual Feedback
Xuxin Cheng"!
Jialong Li
Shiqi Yang'
Ge Yang*
Xiaolong Wang'
UC San Diego'
MIT
https://robot-ttv
http:///oobott.....u........ io/
Autonomous
Teleoperation
Boston
San Diego
Figure 1: Autonomous and teleoperated sessions using our setup. a-c: robots executing long- tasks autonomously. f-j: robots executing fine-grained tasks with our horizon teleoperation system. unloading, in-hand passing; b: HI can-sorting; GR-1 can- immersive
sorting; d: can-insertion; e: towel folding; f: earplugs packing: g: drilling; l:pipetting; i: two operators teleoperate two robots interactively. The operator of H1 robot is approximately 3000 miles away from the GR-1 operator. j: interactions with humans.
Abstract: Teleoperation serves powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-qualigg diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open- Tele Vision that allows operators to actively perceive the robot's surroundings stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's ' robot embodiment. We validate the effectiveness of our
mind is transmitted system by collecting data and training imitation learning policies on four long- horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for two different humanoid robots and deploy them the real world.
"qqual contribution
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Teleoperation
Training and Deployment
Stereo Video
Neck Action
Demonstration
Action Chunking
Arm Action
Hand Action
Transformer Encoder Decoder
60Hz
Positions
Hand, Head, Wrist Pose
Deployment
Training
Figure 2: Teleoperated data collection and learning setup. Left: our teleoperation system. VR devices stream the hand, head, and wrist poses to the server. The server retargets the human poses to the robot and sends joint position targets the robot. Right: we train imitation policy for each task with action chunking transformer. The transformer encoder captures the relationship of image and proprioception tokens and the transformer decoder outputs action sequences of a certain chunk size.
Introduction
Learning-baa braniigg-basdddd robotic manipulation has advanced to new level in the past few years, thanks to large-scale real-robot data [1, 2]. Teleoperation has been playing an important role in data collection for imitation learning, where it not only offers accurate and precise manipulation demonstrations, but also provides natural and smooth trajectories that allow the learned policies to generalize to environment configurations and tasks. Various teleoperation approaches have been studied using VR devices [3, 2, 4, 5], RGB cameras [6, 7, 8), wearable gloves [9, 10, 11], and customized hardwares [12, 13].
There are two major components in most teleoperation systems: actuation and perception. For actuation, using joint copy to puppeteer the robot provides high control bandwidth and precision [12, 14, 15]. However, this requires the operators and the robot to be physically in the same location, not allowing for remote control. Each robot hardware needs to be coupled with specific teleoperation hardware. Importantly, these systems are not able to operate multi- multi-finger dexterous hands yet. For perception, the straightforward way is to observe the robot task space with the operator's 's own eyes in third-person view [7, 6, 3] or first- pest-person view [16, 17]. This inevitably will cause occlusion on the operator's 's sight during teleoperation (e.g., occluded by robot arms or torso), and the operator cannot ensure the collected demonstration has captured the visual observation needed for policy learning. Importantly, for fine-graing grained manipulation tasks, it hard for the teleoperator look closely and intuitively at the object during manipulation. Displaying third-person static camera view using passthrough in the VR headset [3, 2, 18] encounter similar challenges.
In this paper, we propose to revisit teleoperation systems with VR devices. We introduce Open- TeleVision general framework to perform telcoperation with high precision applicable to different VR devices on different robots and manipulators. We experiment with two humanoid robots including Unitree H1 [19] humanoid robot with multi-finge finger hands and Fourier GR1 [20] humanoid robot with grippers, bimanual manipulation tasks (Fig. 1). With the captured human operators' hand pose, perform re-targeting control multi-finger robot hands or parallel-jaw grippers. We rely inverse kinematics to convert the operator's hand root position to the robot arm end-effector position.
Our major contribution to allowing fine-grained manipulations comes from perception, which in- corporates VR systems with active visual feedback. We use a single active stereo RGB camera on the robot head equipped with two three DoFs actuation, mimicking human head movement to observe a large workspace. During teleoperation, the camera moves along with the operator's 's head, streaming real-time, ego-centric 3D observations to the VR device. The human operator what

---

(a) Head movements.
(b) Arm and hand movements.
Figure 3: Open-Tele Vele-so enables high-DOF control of head-mounted active camera and upper body movements. Left: the robot's intuitive spatial perception. Right: the robot's arm and hand movements are mapped from the operator with IK and motion retargeting.
head follows the movement of the operator, providing
the robot sees. This first-person active sensing brings benefits for both teleoperation and policy learning. For teleoperation, it provides more intuitive mechanism for the user to explore a broader view when moving the robot's 's head, and attend to the important regions for detailed interactions. For imitation learning, our policy will imitate how to move the robot head actively together with manipulation. Instead of taking further and static captured view as inputs, active camera provides a natural attention mechanism to focus on next-step manipulation related regions and reduce the pixels process, allowing smooth, real-time, and precise close-loop control. Another important innovation in perception is streaming stereoscopic video from the robot view to human eyes. The operator can have better spatial understanding which is shown to be crucial for completing tasks in our experiments. We also show training with stereo image frames improves the performance of the policy.
Our experiments follow the telcoperation and imitation learning paradigm. We experiment with multiple fine-grat fine-grained manipulation tasks across robots shown in Fig. 1. For teleoperation, qualitatively show the benefits of using active camera allows more intuitive and focused observation, and quantitatively show streaming stereoscopic video allows better success rate and shorter completion time across different users. For imitation learning, we quantitatively report active camera sensing enables faster inference for real-time smooth control, and using stereoscopic inputs achieves better manipulation performance. Our policy can conduct long-horizon tasks such as inserting multiple cans sequence. key benefit of our system is enabling remote control by an operator via the Internet.
2 Vision System
Our system overview is shown Fig. 2. We develop a web server based on Vuer [21]. The VR devices stream the operator's ' hand, head and wrist poses in SE(3) the server, which handles the human-to-robot motion retargeting. Fig. 3 shows how the robot's head, arm, hand follows the human operators movements. In turn, the robot streams stereo video at a resolution of 480x640 for each eye. The entire loop happens at 60 Hz. While our system is agnostic to VR device model, we choose Apple VisionPro as our VR device platform in this paper.
Hardwares: For robots we choose two humanoid robots as shown in Fig. 4: Unitree HI [19] and Fourier GR-1 [20]. We only consider their active sensing neck, two seven-DoF arms and end- effectors, while other DoFs are not used. H1 has six DoF's for each hand from [22], while GR-1 has one-DoF jaw gripper. For active sensing, we design a gimbal with two revolute DoFs (yaw and pitch) mounted on top of the torso of H1. The gimbal is assembled from 3D printed parts and powered by DYNAMIXEL XL330-M288-T motors [23]. For GR-1, we use the three-DoF neck (yaw, roll, and pitch) that comes with the manufacturer. A ZED Mini [24] stereo camera used with both robots provide stereo RGB streaming. We mostly feature humanoid robots in our setup because the teleoperation issue of occlusion and lack of intuitiveness of current system is the most prominent. While our system is specifically tailored for humanoid robots to maximize the capability
3

---

Stereo Camera
Dof Arm
Jaw Gripper
DoF Hand
Figure 4: Reference design of Open-Tele Vision on two types of hardware. Left: Unitree H1 [19] with six-DoF Inspire [22] hands. The head contains yaw and pitch motors. Right: Fourier GR-I [20] with jaw gripper. The active neck is from the manufacturer with yaw roll and pitch motors.
for immersive teleoperation experiences, it is versatile enough to be applied to any setup featuring two arms and one camera.
3 Experiments
In this section, we aim answer the following questions:
How do the key design choices of our system affect the performance of imitation learning results?
How effective is our teleoperation system in collecting data?
We choose ACT [12] our imitation learning algorithm with key modifications. First, replace the ResNet with more powerful visual backbone V2 [25, 26], pretrained vision transformer (ViT) by self-supervised learning. Second, we use two stereo images instead of four images from individually arranged RGB cameras as the input to the transformer encoder. The D2 backbone produces x 22 tokens for each image. The state token is projected from the current joint position of the robots. We use absolute joint positions as the action space. For H1 the action dimension is 28 (7 for each arm, 6 for each hand, and 2 for active neck). For GR-1 the action dimension is 19 (7 for each arm, for each gripper, 3 for active neck). The proprioception token is projected from the corresponding joint position readings.
We choose four tasks with an emphasis on precision, generalization and long horizon to show the effectiveness of our proposed teleoperation system Fig. 10. These tasks require actively looking to the left or right of the workspace using single active camera, otherwise multi-camera setup on the table. They also involve significant object location randomization and different manipulation strategies. We further show the data collection intuitiveness and speed by conducting user studies all four tasks and comparing them with humans doing the same tasks.
3.1 Imitation Learning Results
We ablate our key design choices the system and show from real-world experiments their effective- ness. The two baselines are w. R8 which uses the original ACT visual backbone ImageNet Pre-trained ResNet18 8 [27], and w/o Stereo Input, which only takes the visual tokens from the left image instead of both. All models are trained using AdamW optimizer [28, 29] with learning rate single RTX 4090 GPU. We conduct Can Sorting task on both H1 and GR-1 robots. All other tasks are conducted with H1 robot only.
of 5e 5, batch size of 45 and for 25k iterations
From our experiment results shown in Tab. 1, we notice that the original ResNet backbone (ImageNet pre-trained) fails to adequately perform all four tasks. the original ACT implementation, four cameras used (two fixed, wrist-mounted) to alleviate the deficiency of explicit spatial information from using RGB images. Our setup involves only two stereo RGB cameras, potentially making spatial information retrieval more challenging for ResNet backbone.

---

HI Can Sorting
GR-1 Can Sorting
Can Insertion
Baselines
Pick
Place
Pick
Place
Pick
Insert
W. V2 (Ours)
92%
88%
87%
60%
90%
87%
w. ResNet18
74%
58%
83%
50%
53%
70%
w/o Stereo Input
46%
52%
73%
63%
47%
63%
Folding
Unloading
Basclines
Lift
Fold
Move
Fold
Extract
Pass
Place
Dino (Ours)
100%
100%
100%
100%
100%
100%
100%
w. 18
100%
100%
100%
100%
85%
100%
95%
w/o Stereo Input
100%
100%
60%
100%
70%
95%
100%
Table 1: Success rate of autonomous policy. We record five real-world episodes for each task. Each episode contains complete task cycle defined in Sec. 3. On GR-1, 1, each Can Sorting roll-out contains six pickings and six placings, accumulating 30 trials for each sub-task.
Can Sorting: We evaluate the success rate of picking up the can and the accuracy of placing it in the designated case separately. According to the results on HI in Tab. I, our model has the highest success rate both evaluation metrics. w. ResNer18 is distinctly inferior in both picking and sorting, likely due its backbone's 's limitations. Without implicit depth information from stereo inputs, w/o Stereo Input fails to properly pick up the can (23/50 success rate). Its low sorting accuracy (26/50 success rate) is highly correlated to its poor performance in the previous stage, as an experimenter must frequently help it to grasp the can, an action that interferes with visual inference.
The results of Can Sorting with GR-1 are also reported in Tab. In the picking sub-task, our model consistently outperforms the other baselines. However, in the placing sub-task, none of the models reach satisfactory accuracy. This phenomenon can be attributed to the different morphologies between the dexterous hand and the gripper: when a robot hand grasps the can, the camera is able to see the color of the can and make its actions based on what it sees; in contrast, when gripper grasps the can, the camera can barely make out the color due to significant occlusion 10b), complicating visual inference. The other limitation is that ACT's ability to make long-horizon inference dependent on its chunk size. our setup, using a chunk size of 60 with an inference frequency of 60 Hz effectively provides the robot with one second of memory: when the robot is supposed to drop the without direct visual confirmation, it has likely forgotten the color of the can which was visible at the time of pickup.
Can Insertion: We evaluate the success rate of picking up and insertion separately. The results (Tab. 1) suggest that our model surpasses the baseline models both metrics. Successfully pinching up the soda can with only two fingers and adjusting pose fit into the slot requires precise control underpinned by spatial reasoning. This capability is notably absent in w/o Stereo Input.
Folding: We evaluate the success rate based on the policy's s ability to perform two consecutive folds without dropping the towel. The results are reported in Tab. 1. Both ours and w. ResNet18 models achieve a 100% success rate in performing the folds. We speculate that the high success rate this task across all models is due to its high repetitiveness in actions. On the other hand, w/o Stereo Input occasionally fails (2/5 fails) the third stage, in which the robot should gently move the towel the edge of the table to ease the second fold. We observe that w/o Stereo Input tends press its hands too hard the table and prohibit the successful movement of the towel. This action is likely due to the lack of depth information from using single RGB image, as this step requires the robot to adjust the force exerted by its hands based on the distance to the table.
We evaluate the success rate on three consecutive stages individually: extracting the tube, in-hand passing, and placing. As shown in Tab. 1, our model reaches 100% success rate in all three stages. Both w. ResNet18 and w/o Stereo Input fail extracting the tube from certain slots (3/20 fails and 6/20 fails, respectively). The extraction is particularly hard for w/o Stereo Input, partly because it
Unloading:
5

---

Static Wide Angle
Active (Ours)
Cropped
Figure 6: Comparison between wide-angle lenses and cropped views. Left: HI Can Sorting. Right: Unloading. Wide angle images are of resolution 1280x720 and have a 102% x 57%(V) field-of-view. Cropped images are cropped from the bottom middle from the original wide-angle images, and are resolution 680x480. Red marks are the points of interests (Pols) for the tasks. For Can Sorting, the Pols are the bin with cans to be sorted, pick location, coke bin and sprite bin. For Unloading, the Pols are tube slot, in- in-hand passing, and the drop location.
cannot estimate the relative orientation between the tube and the hand correctly. In-hand passing and placing are relatively simpler because these two stages do not involve much randomization during data collection. Nonetheless, w. ResNer18 and w/o Stereo Input still fail at them occasionally (both 1/20 fails) while our model completes these two stages with no mistake.
3.2 Generalization
We evaluate the generalization capabilities of model under randomized conditions. In the
12cm
Can Sorting task conducted with HI, assess
12cm
-3
its success rate of picking from a 4x4 grid with each cell measuring 3 cm, as depicted in Fig.5 (left). The results are detailed in Fig.5 (right), which indicate that our policy generalizes well large areas covered in the dataset, maintain- ing 100% success rate. Even in the peripheral regions, which are rare absent in the demon- strations, our model still exhibits adaptability to complete the grasp at times.
2
5
Figure 5: Distribution of can placements. Left:
The cans are placed in f4 grid. Right: number of successful pickings heatmap with five trials at each location.
Why Use Active Sensing? We compare the view from a wide-angle camera and our active camera setup with cropped view in Fig. 6. A single static wide-angle camera still have trouble capturing all of the points-of-intes -f-iterrsttt (Pol). One has to mount multiple cameras [12, 14] tune the camera positions for each task. The static wide angle camera also captures non-relevant information that brings additional computation cost for both training and deployment as shown in Tab. 2. Our TeleVision system is 2x faster for training with the same batch size and can accommodate 4x data in one batch on a 4090 GPU. During inference, our system is also 2x faster, leaving sufficient time for IK and retargeting computation reach 60Hz deployment control frequency. When using wide angle images input, the inference speed lower, about 42 frames per second.
Furthermore, with static camera. the operator needs to gaze at Pol on the edge of the image, which brings additional discomfort and non-intuitiveness as humans use central (foveal) vision to focus [30, 31].
User Study. We validate design choice of streaming and displaying stereo images in the VR headset through user study. The use study is performed on four participants with varied levels of

---

Method
Average Training Batch Time (s)
Average Deploying Step Time (s)
Cropped Active (Ours) Cropped Active 10
0.41
0.012 (83Hz)
0.10
Wide Angle
0.32
0.024 (42Hz)
Computation cost comparisons between the active vs non-active vision setup. We sample 100 batches for training and 100 deployment steps and average their computation time. The number in the baseline name indicates batch size. Cropped Active is down- from 640x480 to 308x224, resulting in 352 image tokens for each camera. Wide Angle is down-sampled with the same scale to 588x336, resulting in 1008 image tokens for each camera. We use batch size 10 for Wide Angle due the memory limit of RTX 4090 GPU.
Table 2:
Stereo
Mono
User
Can Sorting
Can Insertion
Folding
Unloading
Can Sorting
Can Insertion
Folding
Unloading
#1
67
54
30
73
110
116
49
94
#2
59
55
44
87
91
92
67
119
#3
55
59
27
52
76
68
37
96
#4
82
62
36
55
88
62
47
78
Mean
66
58
34
67
85
50
97
(a) Completion time for individual participant in second.
Stereo
Mono
User
Can Sorting
Can Insertion
Folding
Unloading
Can Sorting
Can Insertion
Folding
Unloading
#1
100%
100%
100%
100%
100%
33%
100%
50%
100%
100%
100%
100%
100%
83%
100%
50%
#3
100%
100%
100%
100%
90%
83%
100%
50%
100%
100%
100%
100%
80%
83%
100%
50%
Mean
100%
100%
100%
100%
93%
71%
100%
50%
(h) Success rate for individual participant.
Table 3: User study results. Users see stereo video stream in Stereo and see single RGB video from the left camera Mono.
prior exposure to VR devices. The participants are graduate students aged 20-25. Each participant is asked to complete all four tasks under guidance and is given roughly five minutes to familiarize with the system and the tasks. Their performances both stereo and monocular (Mono) setup are detailed Tab. 3. Both metrics (task completion time and success rate) suggest that Stereo surpasses Mono by large margin. Additionally, based on qualitative results of user feedback. using stereo image remarkably better than using single RGB image. Typically, human eyes adaptive enough to infer depth and spatial relationships from single RGB image. However, in teleoperation cases where the operator must actively interact with different objects, such intuitions are often proven insufficient. Leveraging the spatial information available in stereo images can substantially alleviate the discomfort when teleoperating robots with only images.
4 Related Work
Data Collection. Learning based methods have achieved great success in locomotion control with massive simulation data and Sim2Real transfer [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]. On the other hand, collecting real-robot demonstrations for imitation learning has shown to be more

---

effective way for robotic manipulation [45, 46, 1, 48]. This is mainly due to the large Sim2Real gap with complex contacts between the manipulator and objects and surroundings. To collect real- world data, teleoperation has emerged as the mainstream approach using RGB cameras [6, 7, 8], mocap gloves [9, 10, 11], and VR devices [3. 2]. There are also more conventional frameworks exploiting exo-skeleton devices [13] or mirroring arms for joint copy [12. 14, 15]. For example, the recent ALOHA framework [12] provides precise control fine-grained manipulation tasks with exact joint mapping. In this paper, instead of adopting joint copy, we find VR-based teleoperation system with hand retargeting can also achieve precise control of fine-grained manipulation tasks. Tab. 7 in the supplementary material summarizes the difference between our system and previous teleoperation systems.
Bimanual Teleoperation. Access to intuitive and responsive teleoperation system is essential for collecting high-fidelit demonstrations. Most existing teleoperation systems restricted using grippers [13, 49, 15, 16] or single-hand setups [50, 51, 52], which tend to be either non-intuitive or limited in their capabilities. We believe enabling manipulation with multi--inger hands in 3D space allows more robust manipulation conducted on diverse tasks, providing significant advantages over existing teleoperation systems. For example, it will be very challenging for a parallel-jaw gripper to stably grasp a Pringles Chips tube. For the handful of bimanual- setups [3, 2), they require operators to control the robot by directly observing its hands during task exccution or seeing an RGB image from a fixed camera. In contrast, our system integrates active head rotations, offering intuitive interface as if the robot is an avatar of the operator itself. Our system allows the operator to control the robot remotely far away (e.g.. control the robot on the West Coast from the East Coast). This is also the major innovation that differentiates our work from concurrent works on humanoid teloperation [53, 54]. Our system allows smooth teleoperation on fine-gran ann-ggrined tasks, and precise control policy for long-horizon execution. Furthermore, our web-based system accessible from all kinds of portable devices such as Quest (~500 USD) or VisionProo~33000 USD), while [55, 56, 57] require dedicated teleoperation stations that can be very expensive. For example, The two Franka Emika Panda arms in [55] will cost approximately 20000 USD, excluding other components of the station and limit the mobility of the system (operators must come to the station to start operating).
stereo display with
Imitation learning. The topic of imitation learning has covered a wide range of literature. we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [16, 12, 14, 58, 59, 60, 61, 46, 62, 1, 63, 47, 64, 65], learning from play data [66, 67, 68), and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 52]. Besides learning for manipulation, motion imitation for physical characters and real robots have also been widely studied in [75, 76, 36, 77, 78, 79, 80, 81, 82, 80, 83, 84, 85, 86, 87, 88]. This paper falls into imitation learning for manipulation tasks using real-robot data collected by human teleoperation.
Conclusion and Limitations
In this paper, we propose system for immersive teleoperation with stereoscopic video streaming and active perception with actuated necks. We show that our system can enable precise and long-horizon manipulation tasks and the data collected with our system can be readily utilized by imitation learning algorithms. We also conduct user studies to show the importance of stereo perception for human operators. While the stereo perception is crucial for the operators' spatial understanding, there is still lack of other forms of feedback, such haptic feedback, which is typically the dominant feedback with first-person visual occlusion and in tactile-intensive tasks. A system that enables the relabeling of expert data could be very helpful for increasing success rate, which is also missing from our system now. The future work can be extended to mobile version which utilizes all the DoFs of the robot.
8

---

Acknowledgments
This work was supported, in part, by Amazon Research Award and the Intel Rising Star Faculty Award.
References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06887, 2022.
[2] A. Iyer, Z. Peng, Y. Dai, I. Guzey, Haldar, S. Chintala, and L. Pinto. Open teach: A versatile teleoperation system for robotic manipulation, 2024.
[3] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik. Learning visuotactile skills with two multifingered hands. arXiv:2404. 16823, 2024.
[4] S. Dafarra, U. Pattacini, G. Romualdi, L. Rapetti, R. Grieco, K. Darvish, G. Milani, E. Valli, I. Sorrentino, P. M. Viceconte, A. Scalzo, S. Traversaro, C. Sartore, M. Elobaid, N. Guedelha, C. Herron, A. Leonessa, F. Draicchio, G. Metta, M. Maggiali, and D. Pucci. icub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots. Science Robotics, 9(6)): cadh3834, 2024. doi:10. /126///crrbooiccaahh33333333 URL https : http://wwww.w....////t////p///et abs/10. htt////crobboticcooooooo adh3834.
science. org/doi/
[5] L. Fritsche, F. Unverzag, J. Peters, and R. Calandra. First-person tele-operation of humanoid robot. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 997-1002, 2015. doi: 10. (09/HUMANODSS220..77.666. 10
[6] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang. Y.-W. Chao, and D. Fox. Anyteleop: A general basend dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, 2023.
[7] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning robotic hand imitator by watching humans on youtube. arXiv preprint arXiv:2202. 2022.
[8] S. Li, J. Jiang, P. Ruppel, H. Liang, X. Ma, N. Hendrich, F. Sun, and J. Zhang. A mobile robot hand-arm teleoperation system by vision and imu. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10900-10906. IEEE, 2020.
[9] M. Caeiro-Rodriguse I. Otero-González, F. A. Mikic-Fonte, and M. Llamas-Nistal. A system- atic review of commercial smart gloves: Current status and applications. Sensors, 21(8):2667, 2021.
[10] H. Liu, Z. Zhang, X. Xie, Zhu, Y. Liu, Y. Wang, and S.-C. Zhu. High-fidelity grasping in virtual reality using glove-based system. In 2019 international conference on robotics and automation (icra), pages 5180-5186. IEEE, 2019.
[11] H. Liu, X. Xie, M. Millar, M. Edmonds, F. Gao, Y. Zhu, J. Santos, B. Rothrock, and S.-C. Zhu. A glove- based system for studying hand-object manipulation via joint pose and force sensing. In 2017 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6617-6624. IEEE, 2017.
[12] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 0rxiv:2304.13705, 2023.
[13] H. Fang, H.-S. Fang. Y. Wang, Ren, J. Chen, R. Zhang, W. Wang, and C. Lu. Low-cost exoskeletons for learning whole-arm manipulation in the wild. arXiv preprint ar arXiv:2309.144 149755 2023.

---

[14] L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arvi:: 2403. 129/0, 2024.
[15] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.1300: 13037, 2023.
[16] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024.
[17] C. Wang. Shi, Wang. R. Zhang, Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint vrXiv:2403.07788. 2024.
[18] Y. Park and P. Agrawal. Using apple vision pro to train and control robots, 2024. URL https: //gt thub. com/ http///ppooaalll-A///ssooooooPPt
[19] Unitree Robotics, H1, 2024, www. www. www.tre...omme.llmmmmmm [Online; accessed Feb. 2024].
[20] Fourier Intelligence, GR-1, www.fourierr.itllllieeecc../a./ [Online; accessed Jun. 2024].
[21] Vuer, 2024, docs. docs.vuer..... wwe.. aieee//t/nl, [Online; accessed Jun. 2024].
[22] Inspire Robots, Dexterous Hands, 2024, www iir.ppreerrootsssttrr/coooll the-dexterous hands, [Online; accessed Jun. 2024).
[23] Dynamixel-X, 2024, www www.robotis.uuk www..bbboc....a/ppmmaaaaaa-11131 [Online; accessed Jun. 2024].
[24] Zed Mini, Stereo Camera, 2024, www.robbiss.uss/taaaiiillll---- accessed Jun. 2024).
[Online,
[25] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafranice, Khalidov, P. Fernandez, D. Haziza, F. Massa, El-Nouby, R. Howes, P.-Y. Huang, Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[26] Darcet, M. Oquab, Mairal, and P. Bojanowski. Vision transformers need registers, 2023.
[27] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
1b0:10..109/CCVPR2011...
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 1rXiv:1412 1412.6980, 2014.
arXiv preprint
[29] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL /
[30] W. S. Tuten and W. M. Harmening. Foveal vision. Current Biology, 31(11):R701-R700, 2021.
[31] I. Levy, U. Hasson, G. Avidan, Hendler, and R. Malach. Center center-periphery organization of human object areas. Nature neuroscience, 45555333-339, 2001.
[32] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. arXiv preprint arXiv:2205.022 022244 2022.
[33] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0403: 2021.
10

---

[34] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emergence gaits in legged robots. Conference on Robot Learning (CoRL., 2021.
[35] J. Fu, Y. Song, Wu, F. Yu, and D. Scaramuzza. Learning deep sensorimotor policies for vision-based autonomous drone racing, 2022.
[36] A. Escontrela, X. B. Peng. W. Yu, Zhang, A. Iscen, K. Goldberg, and Abbeel. Adversarial motion priors make good substitutes for complex reward functions. 2022 ieee. In International Conference Intelligent Robots and Systems (IROS), volume 2, 2022.
[37] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2811-2817. IEEE, 2021.
[38] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning, arXiv preprint arXiv:2105.08326 2021.
[39] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. Conference on Robot Learning, pages 403-415. PMIR, 2023.
[40] X. Cheng, A. Kumar, and D. Pathak. Legs as manipulator: Pushing quadrupedal agility beyond locomotion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[41] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138- 189. PMLR, 2023.
[42] H. Duan, B. Pandit, M. S. Gadde, B. J. van Marum, J. Dao, C. Kim, and A. Fern. Learning vision-based bipedal locomotion for challenging terrain. arXiv preprint arXiv:2309. 14594, 2023.
[43] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL), 2023.
[44] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309.14344 14344, 2023.
[45] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 3 2018.
[46] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mereat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, F. Tran, A. F. Wang. Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. Lim, J. Malik, R. Martín-Martín S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip. 7hu, Kollar, Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[47] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-actin action models transfer web knowledge to robotic control. arXiv preprint arXiw:2307.15818 2023,

---

[48] N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home. arXiv preprint arXiv:23 1..1111160,, 2023.
[49] M. Seo, Han, K. Sim, S. H. Bang. C. Gonzalez, L.. Sentis, and Y. Zhu, Deep imitation learning for humanoid loco-manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8. IEEE, 2023.
[50] S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969. IEEE, 2023.
[51] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letlers, 7()): 10873-10881, 2022.
[52] J. Wang, Y. Qin, K. Kuang, Y. Korkmaz, A. Gurumoorthy, H. Su, and X. Wang. CyberDemo: Augmenting Simulated Human Demonstration for World Dexterous Manipulation. arXiv preprint arXiv: 2312.09237. 2024.
[53] T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi. Omnih2o: Universal and dexterous human-to-humandde whole-body teleoperation and learning. In arXiv, 2024.
[54] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn. Humanplus: Humanoid shadowing and imitation from humans. In arXiv, 2024.
[55] M. Schwarz, C. Lenz, R. Memmesheimer, B. Pätzold, A. Rochow, M. Schreiber, and S. Behnke. Robust immersive telepresence and mobile telemanipulation: Nimbro wins avatar xprize finals. In 2023 SEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8. 8. IEEE, 2023.
[56] J. Lee, R. Balachandran, K. Kondak, A. Coelho, M. De Stefano, M. Humt, J. Feng. T. Asfour, and R. Triebel. Virtual reality via object pose estimation and active learning: Realizing telepresence robots with acrial manipulation capabilities. arXiv preprint arXiv:2210.09675. 2022.
[57] J. Vogel, D. Leidner, A. Hagengruber, M. Panzirsch, B. Bauml, M. Denninger, U. Hillenbrand, L. Suchenwirth, P. Schmaus, M. Sewtz, et al. An ecosystem for heterogeneous robotic assistants in caregiving: Core functionalities and use cases. IEEE Robotics & Automation Magazine, 28
(3):12-28, 2020.
[58] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel. S. Feng, R. Tedrake, and Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics:
Science and Systems (RSS), 2024.
[59] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[60] Pari, N. M. Shafiullah, S. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:21 2021.
[61] A. Mandlekar, D. Xu. J. Wong, S. Nasiriany, C. Wang, R. Kulkamni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2 03299, 2021.
[62] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models, arXiv preprint arXiv:2310.08866 2023.
12

---

[63] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[64] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In Proceedings of Robotics: Science and Systems (RSS), 2024.
[65] C. Gardenas-Perez. G. Romualdi, M. Elobaid, S. Dafarra, G. L'Erario, S. Traversaro, P. Morerio, A. Del Bue, and D. Pucci. Xbg: End-to-end imitation learning for autonomous behaviour in human-robot interaction and collaboration. arXiv preprint arXiv:2406. 10n.:34406.1533,,. 2024.
[66] J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047 2022.
[67] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.11422. 2023.
[68] R. Mendonca, S. Bahl, and D. Pathak. Alan Autonomously exploring robotic agents in the real world. ICRA, 2023.
[69] H. Xiong, R. Mendonca, K. Shaw, and D. Pathak. Adaptive mobile manipulation for articulated objects the open world. arXiv preprint urXiv:2401.14403, 2024.
[70] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:200.004550 2022.
[71] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. CoRL, 2022.
[72] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[73] Y. Zhu, A. Lim, P. Stone, and Y. Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint wcXiv:2405.20322. 2024.
[74] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. RSS, 2023.
[75] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. Apr. 2020.
[76] Y. Wang, Z. Jiang, and J. Chen. Amp in the wild: Learning robust, agile, natural legged locomotion skills. arXiv preprint arXiv:2304. 204.:2304410888, 2023.
[77] Y. Fuchioka, Z. Xie, and M. Van de Panne. Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5092-5098. IEEE, 2023.
[78] R. Yang, Z. Chen, J. Ma, C. Zheng, Y. Chen, Q. Nguyen, and X. Wang. Generalized animal imitator: Agile locomotion with versatile motion prior. arXiv preprint arXiv:2310.01408, 2023.
[79] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics- based character control. ACM Transactions on Graphics (ToG), 40(4):1-20, 2021.
[80] X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), July 2022.
13

---

[81] C. Tessler, Y. Kasten, Y. Guo, S. Mannor, G. Chechik, and X. B. Peng. Calm: Con- ditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701597. do1:00011455358842235555553 URL https://ddoi.org/1//
1145/3588432.... 3591544.
[82] M. Hassan, Y. Guo, T. Wang, M. Black, S. Fidler, and X. B. Peng. Synthesizing physical URL org/
character-sceme interactions. 2023. 40110..1145/358452235555333
10. 01444//358832..3591555 3591525.
[83] J. Won, D. Gopinath, and J. Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. ACM Trans. Graph., 39(4), 2020. URL https://doi. org/ 10. 00111153338666... 3392381.
[84] T. Wang, Y. Guo. M. Shugrina, and S. Fidler. Unicon: Universal neural controller for physics- based character motion, 2020.
[85] H. Zhang, Y. Yuan, V. Makoviychuk, Y. Guo, Fidler, X. B. Peng, and K. Fatahalian. Learning physically simulated tennis skills from broadcast vidcos. ACM Trans. Graph., 42(4), jul 2023. ISSN 0730-0301. 1oi: 10.1144//359208. URL https: /ooi org/ 10 1145/3592408.
[86] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics- based character skills. ACM Trans. Graph., 37(4):143:1- 143:14, July 2018. ISSN 0730-0301. dot:100.144/3197517..333 org/10. 145 3197517. 32013311.
hRL acm.
[87] X. Cheng, Y. Ji, Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:240..16796. 2024.
[88] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-human to-humanoid real-time whole-body teleoperation. In arXiv, 2024.
[89] J. Carpentier, G. Saurel, G. Buondonno, J. Mirabel, F. Lamiraux, O. Stasse, and N. Mansard. The pinocchio c++ library fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. In IEEE International Symposium on System Integrations (SII), 2019.
[90] Carpentier, F. Valenza, N. Mansard, al. Pinocchio: fast forward and inverse dynamics for poly-articulated systems.
2015-20022 2015-2021.
[91] Pinocchio CLIK, 2024, paster/doxygen- seseen-hmml//e httlmmd_dddooooooomomedo
laas. oepettoocc...aa...rrro///aaaaaaa htt.:///aa..nttooooooooooaaeeaea wtw..r-ew.mmee..---eee-eeeeeeeee html, [Online, accessed Jun. 2024).
[92] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-.. Chao, Q. Wan, S. Birchfield, N. Ratlfff and D. Fox. Dexpilot: Vision-based teleoperation dexterous robotic hand-arm system. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164 9170. IEEE, 2020.
[93] D. Kraft. Algorithm 733: TOMP-fortran modules for optimal control calculations. ACM Transactions on Mathematical Software, 20:262-281, 1994, doi: 06::101115//221111111222
[94] S. G. Johnson. The NLopt nonlinear-optimizztiion package. stevengj/nlopt, 2007.
https: httpt/ggttttttttee/e
14

---

A Arm and Dexterous Hand Control
Arm Control: The human wrist poses are first converted into the robot's coordinate frame. Specifi- cally, the relative positions between the robot end-effectors and the robot head are expected to match those between the human wrists and head. The orientations of the robot wrists are aligned with the absolute orientations of the human wrists, estimated during the initialization of the Apple VisionPro hand tracking backend. This differentiated treatment of end-effectors' positions and orientations ensures the stability of the robot end-effectors when the robot's ss head moves along with human's 's head. we employ the Closed-loop Inverse Kinematics (CLIK) algorithm based on Pinocchio [89, 90, 91] to compute the joint angles of the robot's 's arm. The input end-effector poses are smoothed using an SE(3) group filter, implemented with Pinocchio's 's SE(3) interpolation, enhancing the stability the IK algorithm. To further mitigate the risk of IK failures, our implementation incorporates a joint angle offset when the arm's 's manipulability approaches its limits. This correction procedure has minimal impact end-effector tracking performance, as the offset is projected onto the nullspace of the robot arm's 's Jacobian matrix, thereby preserving tracking accuracy while addressing the constraints.
The human hand keypoints are translated into robot joint angle commands through dex-retargeting, highly versatile and fast-computing motion retargeting library [6]. Our approach utilizes vector optimizers on both dexterous hand and gripper morphologies. The vector optimizers formulate the retargeting problem optimization problem [6, 92] while the optimization is defined based on user-selected vectors:
Hand Control:
min
|ao - f((g)ll + 8119 - 9-111?
In the above formulation, Gr denotes the robot joint angles at time f, and of is the i-th keypoint vector on the human hand. The function f(():) computes i-th keypoint vector on the robot hand using forward kinematics from the joint angles 4:- The parameter a a scaling factor that accounts for the hand size difference between the human hand and the robot hand (we set as 1.1 for Inspire hands). The parameter 33 weights the penalty term that ensures temporal consistency between consecutive steps. The optimization is conducted in real-time using Sequential Least-Squares Quadratic Program- ming (SLSQP) algorithm [93] as implemented in NLopt library [94]. The computation of forward kinematics and its derivatives are conducted in Pinocchio [90].
For dexterous hands, we employ seven vectors to synchronize the human and robot hands: five vectors represent the relative positions between the wrist and each fingertip keypoint; two additional vectors, spanning from the thumb fingertip to the primary fingertips (index and middle), are incorporated to enhance the motion accuracy during fine-grained tasks. For grippers, optimization is achieved using single vector, defined between the human thumb and index fingertips. This vector is aligned with the relative position between the gripper's s upper and lower ends, enabling intuitive control over the grippers opening and closing motions by simply pinching the operator's 's index and thumb fingers.
B Discussion on Comparing with Prior Teleoperation Systems
We discuss from two critical perspectives of teleoperation: actuation and perception.
Actuation. Various approaches have been studied for teleoperating robots through human commands, including visual tracking, motion-capture devices, and joint copying through customized hardware. While using motion-capture gloves for teleoperation seems the most intuitive, the commercially available gloves are not only costly but also unable to provide wrist pose estimations. The joint copying method has drawn significant attention recently, following the success of ALOHA[12]. This method offers precise and dexterous control. Historically, this method was considered costly, requiring using an additional pair of identical robotic arms for teleoperation; nonetheless, this issue has been mitigated by the adoption of low-cost exoskeleton devices to transmit commands[13] Despite their simplicity, joint copying systems currently limited to using grippers and have not yet been extended to operate multi-finger hands. Conversely, visual tracking employs off- the-shelf
15

---

hand pose extractors to track finger movements, but relying solely on RGB or RGBD images can lead to noisy and imprecise data. The recent surge in VR technology has led to the development of teleoperation systems that utilize VR tracking. VR headset manufacturers often integrate built-in hand-tracking algorithms that fuse data from diverse types of sensors, including multiple cameras, depth sensors, and IMUs. Hand-tracking data collected through VR devices are generally considered more stable and accurate than self-developed vision-tracking systems, while the latter only utilize subset of the mentioned sensors (RGB (RGB+RGBD(6), Depth+IMU[8], etc.).
Perception. While being the other critical component of teleoperation, perception has been consider- ably less explored than actuation within this field. Most existing telcoperation systems require the operators directly observe the robot's 's hands using their own eyes. While direct viewing provides the operators with depth sensing, leveraging humans' inherent capability for stereoscopic vision, it restricts the system be non-remote, necessitating the physical presence of the operator. Some teleoperation systems circumvent this by streaming RGB images, enabling remote controll6, 7]. However, if the operator opts for remote controlling by watching an RGB stream, the benefits of depth sensing provided by the human eye are lost. Despite being capable to provide both remote controlling and depth sensing, these two features are mutually exclusive in these systems. OPEN TEACH[2] merges the two in a mixed-reality fashion, yet it still requires the operator to be proximity to the robot, otherwise the depth sensing is unavailable. Prior to Open-Tele Vision, no system offered both remote control and depth sensing simultaneously: the operator forced to choose between either direct viewing, which demands physical presence, RGB streaming, which abandons depth information. By utilizing stereo streaming, our system is the first to provide both functionalities within a single setup.
C Discussion of Visual Occlusion
To support our proposed assumption that the unsatisfactory performance observed GR-/ Can Sorting task stems from visual occlusion caused by GR-l's gripper end-effector, we perform controlled experiment. In the experiment,
add color labels to the to mitigate the occlusion factor, as depicted in Fig. 7 left. The other settings are identical to those described in the GR-/ Can Sorting task in the article. Results are recorded in Table. 4.
GR-1 Can Sorting
Baselines
Pick(new)
Pick(old)
Place(new)
Place(old)
W. V2 (Ours)
97%
87%
100%
60%
w. ResNet18
90%
83%
97%
50%
w/o Stereo Input
47%
73%
93%
63%
Table 4: Success rate for GR-I Can Sorting. The experiments are conducted under identical settings and number of trials as outlined in Tab. 1. Columns marked as (old) contain the original results using unlabeled cans, while the columns marked as (new) contain the results of the new experiment using labeled cans.
The results indicate substantial improvement in the success rate of the placing task across all three baselines, achieved by using labeled cans. Our model reach a 100% accuracy rate in the placement, compared to the previous 0.60; notable gains are also observed in the other baselines, with w. ResNet18 improving from 0.50 0.97, and w/o Stereo Inpul improving from 0.63 0.93. On the other hand, while success rates of picking also increase for model and ResNet18, w/o Stereo Input does not exhibit similar improvements. This disparity further validates our claim that successful can-picking requires spatial information from stereo images.
As with HI in Sec. 3.2, we perform an experiment to evaluate the model's gen- cralization capability with Can Sorting on GR-1 with labeled cans. Its results are similarly collected from 4x4 grid (the same as Fig. 5 left) with each cell mea- suring 3 em. Generalization results shown in the heatmap in Fig. right.
16

---

Figure 9: Our web-based cross-platform system enables access across different devices, Left: Apple Vision Pro. Middle: Meta Quest. Right: Macbook, iPad and iPhone. On VR devices, the users can enter an immersive session to start teleoperation with hand and wrist pose streaming. On other devices, hand and wrist streaming not available but the user can still see the streamed images and control the robot's active neck by dragging on the devices' screen.
The results suggest that our model can easily adapt most of the random locations covered in our experiment, reaching 100% grasping curacy in nearly all locations on the grid. The results as shown here for GR-Z Can Sorting are also notably better than the results as shown in Fig. for HIl Can Sorting. The difference may also be attributed to differences in end-effector morphologies. Grasping soda can, which re- quires less dexterity and more tolerance, is better suited to grippers than to robotic hands.
5
5
33
5
5 5
Figure Can labeling and generalization re- sults. Left: Figure depicting labeled cans. Right: number of successful pickings heatmap with five trials at each location.
For HI robot's setup, the anthropomorphic hands we use are provided by Inspire Robots [22]. A close-up of one of the Inspire Hands is shown in Fig. 8.
D Dexterous Hand
Each hand has five fingers and twelve DoFs, among which six are actuated DoFs: two actuated DoFs are on the thumb and one on each of the remaining fingers. Each non-thumb finger possesses a single actuated revolute joint at the metacarpoppallangel (MCP) joint, serving as the entire finger's actuating DoF. The proximal interphalangeal (PIP) joints of these four fingers are driven by the MCP joints through linkage mechanisms, adding four underactuated DoFs. The thumb is equipped with two actuated DoF' at the carpometacarpal (CMC) joint. The thumb's 's MCP and interphalangeal (IP) joints are also driven by linkage mechanisms, contributing to additional two underactuated Dolss
Figure 8: Inspire
E Teleoperation Interface
Hand [22].
Fig. 9 shows web-based cross- plass-platfomm interface that be accessed not only from VR devices but also laptops, tablets and phones.
F
Experimental Details and Hyperparameters
F1
Autonomous Task Description
Can Sorting: This task as shown in Fig. 10a involves sorting randomly placed Coke cans (red) and Sprite cans (green) table. The cans are placed the table one-by-one but with random positions and types (Coke or Sprite). The goal is pick up each on the table and toss it into the designated
17

---

#2
#
End
Can Sorting
(a) The robot picks up randomly placed (#1). places the red can in the left (#2), picks up the next can (#3), places the green can right case (#4). When the sorting box at the edge of table is empty (as
shown #1, 43) and there is no more can on the table, robot poses the ending gesture (End).
GR-1 Can Sorting
#1
#2
#3
End
(b) GR-1 robot doing the same Can Sorting task as Fig. 10a.
Can Insertion
#3
#4
End
(c) The robot picks up the (#I), inserts the into the slot (#2), the other hand to repeat picking and inserting(#3, #4). When slots have filled, the robot poses ending gesture (End).
Towel Folding
End
(d) robot picks up two corners of towel (#I), folds towel (#2), gently moves towel to edge of the table (#3), folds the towel twice (#4). When the towel has been folded into a satisfactory configuration, the robot poses the ending gesture (End).
#1
#2
#3
#4
End
Unloading
(e) robot extracts tube from random slot (#I, #4), passes from right to left (#2), places the tube the designated position (#3). When there not a tube anymore, the rohot poses the ending gesture (End).
Figure 10: Data collection using HI on four tasks. Each row represents one task. At the end of each task, the operator postures the same ending gesture signify the successful completion of one demonstration. Then our system will stop recording data. We do not crop the ending gesture from dataset. The learned policy successfully react ending conditions.
case: left for Coke and right for Sprite. Solving this task requires the robot to adaptively generalize upon the position and orientation each can for accurate grasping. It also demands that the policy can adjust its planned motions based on the color of the can it is currently holding. Each episode consists of sorting ten cans (five Sprite five Coke randomly) consecutively. Can Insertion: This task shown in Fig. 10c involves picking up soft drink cans from the table and carefully inserting them into slots within container in predefined sequence. While both involve manipulation of drink cans, this task demands more precise and fine-grained actions than the previous one as successful insertion necessitates high accuracy. In addition, a different grasping strategy is adopted this task. In the previous can-sorting task, the robot only needs to toss the can into designated area, hence we form grasp that involves the palm and all five fingers, which a tolerant but imprecise grasping strategy. In this one, insert the into slot that is only slightly larger than the (the diameter of the soda can is roughly 5.6 cm, and the diameter of the slot is roughly 7.6 cm), employed
18

---

more pinching like strategy that utilizes only the thumb and index fingers, enabling more granular adjustments in the placement of the cans. The two distinct grasping strategies demonstrate that our system able to accomplish tasks with complex hand gesture requirements. Each episode this task includes picking and placing all six cans into the right slots.
Folding: This task shown in Fig. 10d involves folding the towel twice. The distinction of task is that it manifests the system's 's capability to manipulate soft and compliant materials like a towel. The action sequence of this task unrolls as follows: pinching the two corners of the towel; lifting and folding; gently moving the towel to the edge of the table to prepare for a second fold; pinching, lifting, and folding again. Each episode of this task consists of one complete folding the towel.
Unloading: This task shown in Fig. 10e is composite operation that involves tube extraction followed by in-hand passing. In this task, chip tube is randomly placed into one the four slots within a sorting box. The goal is to identify the slot containing the tube, extract the tube using the right hand, pass it to the left hand and place the tube in a predefined location. To successfully execute this task, the robot needs both visual reasoning to discern the tube's location and accurate action coordination for extraction and in-hand passing. Each episode this task consists of picking up four tubes from four random slots, passing them another hand, and finally putting them on the table.
F.2 Experimental Details
Tasks
Average Episode Length (s)
Number of Episodes
HI Can Sorting
9315
10
GR-1 1 Can Sorting
6115
10
Can Insertion
84+7
20
Folding
44:5
20
Unloading
93:-
20
Table 5: Details about collected demonstration data for each task.
More experimental details are listed Tab. 5. All tasks, with the exception of Can Sorting (both H! Can Sorting and GR-I Can Sorting), use 20 human demonstrations for training. contrast, only 10 demonstrations are used for Can Sorting. This choice is primarily due to its repetitive nature: each episode consists of 10 (6 for GR-/ Can Sorting) individual can-sortings. Consequently, 10 demonstrations encompass 100 individual sorting rollouts, providing ample data for training.
F.3 Hyperparameters
The hyperparameters employed for training the ACT [12] models are detailed in Table. 6. While the majority of these hyperparameters are consistent across all baselines and all tasks, there are a few exceptions, including chunk size and temporal weighting. The detailed explanations are as follows.
KL weight chunk size
10
hidden dimension batch size feedforward dimension
512
45
3200
epochs
25000
learning rate
5e-5
temporal weighting
0.01
Table 6: Hyperparameters of ACT.
19

---

Boston
San Diego
Figure 11: Our system enables cross-country teleoperation over the internet. The operator at Boston, USA can operate a robot San Diego, USA (approximately 3000 miles away). The robot and operator pictures are flipped left to right for better illustration.
The definition of chunk size in the action chunking operation is outlined in the original ACT paper[ [2]. chunk
We chunk size of 60 for all tasks, with the exception of Can Insertion, in which size of 100. Using chunk size of 60 in our setup effectively provides the robot with approximately one second of memory, correlating with our inference and action frequency of 60Hz. Nonetheless, we notice that in Can Insertion task, using a larger chunk size, which corresponds to incorporating more historical actions, proves to be advantageous for the model to perform correct action sequences.
The definition of temporal weighting in the temporal aggregation operation is outlined in the original ACT paper[ 12], where an exponential weighting scheme w, ezp( -m i) is employed to assign weights to actions at different timesteps. We is the weight for the oldest action, adhering to ACT's setting. m is the temporal weighting hyperparameter mentioned in Table. 6. As m decreases, greater emphasis placed on more recent actions, rendering the model more reactive but less steady. We found that using a temporal weight m of 0.01 reaches satisfactory balance between responsiveness and stability for most tasks. However, for Unloading and Can Sorting tasks, adjust this parameter cater their specific needs. For unloading, m is set as 0.05, ensuring greater stability during in-hand passing; for Can Sorting, is set 0.005, providing quicker movements.
Teleoperated Performance. In Fig. 12, we include more teleoperation tasks that our system is capable of. The Wood-board Drilling task shows that our system can operate heavy-weight (1kg) tools that are designed for humans, thanks to its compatibility with dexterous hands, and can apply sufficient force to the wood board to drill it through. Such a task is virtually impossible for the grippers. The Earplugs Packing task demonstrates that our system is dexterous and responsive enough to perform agile bimanual arm- mandandd coordination. The Pipette task demonstrates that system is also capable of precise actions. This is also a task that is extremely hard or impossible for the grippers to achieve, as the usage of pipette is specialized for anthropomorphic hands. Even though the motors on HI humanoid robot are quasi-direct-drivg motors with planetary reducers, which are known to have gear clearance and far less accuracy and stiffness, our system can still achieve high-precision with human operators in the loop. Our system also achieves remote teleoperation as shown in Fig. 11. Please see our videos for visualization.
20

---

Wood -ooadd Drillngg
(a) The robot holds wood board of thickness 2cm with the left hand and uses an electric drill to drill holes on the board. This task requires precise control of the drill trigger using the index finger. Furthermore, our system enables fine control of the hand so that after drilling the first hole, the robot can let the board slide in hand to leave space for the second drilling.
Earplugs Packing
(b) The robot picks randomly placed earplugs on table places them into randomly placed latch boxes. The robot needs dexterous bimanual in hn-hand manipulation and adjustments to properly close the latch box.
Pipette
(c) robot utilizes its thumb DoF to control pipette to transfer liquid a petri to centrifuge tube. The diameter of the tube only 1.5cm so it requires high precision to complete the task.
Figure 12: More teleoperation experiments. These experiments aim to show our system's reliability and precision for wide variety of tasks.
Teleop System
Actuation
Hand
Bimanual
Perception
Remote
Depth
OPEN TEACH[2]
VR Controller
Direct View+RGB
НATO[3]
VR Tracking
Direct View
X
AnyTclcop161
RGB(D) Tracking
X
Direct View/RGB
V
Telekinesis( 7]
RGB Tracking
X
Direct View/RGB
/
7
Transtcleop(8)
IMU+Depth
Direct View
X
ALOHA[12]
Joint Copy
Direct View
X
AirExo[13]
Joint Copy
X
Direct View
X
GELLO[15]
Joint Copy
X
Direct View
X
Mobile ALOHA[16]
Joint Copy
X
Direct View
X
D1x
SLAM+Mocap
Direct View
X
Open-TeleVision
VR Tracking
Stereo
Table 7: Comparing Open-Tele Vision's capabilities with prior teleoperation systems. more detailed analysis to the contents in this table is in Appendix. B.
21


Reviews:
[{"review_id": "IMZwoxzRjd", "reviewer": "Reviewer_mH7J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In RSS 2023, a teleoperation system called ALOHA was presented in order to perform end-to-end imitation learning from teleopeartion for bnimanual manipulation. In the current submission, the authors follow this trend and present a system called Open-TeleVision, where the focus is on introducing stereoscopic vision. Several bi-manual manipulation tasks are demonstrated on a real system.", "summary_of_recommendation": "For me, the cons outweigh the pros, which I have outlined in the section above. The paper can improve by (a) stating the contribution and system level novelty in more clear way by paying tribute to the established field of telerobotics, (b) providing design decisions upon description of the system and (c) providing the promised open-source work during the review phase. --------------------------------------------------------------------- ( #post rebuttal comment: I have read the revised manuscript and the author rebuttal. I decided to increase my recommendation to weak accept, given that the novelty is more clear, e.g., the argumentation about the costs in the related work seems convincing and the provided system can serve the community. However, the design choices should be more thorough. I get that the paper can be a useful infrastructure to the community, but methodological lessons could be stronger)", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "In my understanding of the field, telerobotics have been one of the traditional topics, and teaching the robot through teleoperation data have been quite widely studied in early 2010s and also before that. Also, if we look at the DLR systems and also the recent AVATAR challenge, the teleoperation systems have reached impressive performance, which could also directly used for learning. I see ALOHA as an addition because that system is specialized in low cost hardware. In this regard, I wonder what is the contribution of this paper given the developments over last few decades? What makes the presented system specialized for modern imitation learning toolboxes?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3nFDvrMdB", "reviewer": "Reviewer_xknk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Open-TeleVision, which is an immersive teleoperation system for robots utilizing VR devices to mimic human movements and offer stereoscopic visual feedback. It showcases efficient data collection for imitation learning and illustrates enhanced performance on diverse tasks in comparison to alternative methods.", "summary_of_recommendation": "The paper introduces an efficient teleoperation system with a significant potential impact. Although the work is of high technical quality and well presented, further revisions are required to improve the discussion of results and limitations. Overall, it makes a valuable contribution to the field.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please make a note of the following: \n\n- The paper would benefit from a more thorough discussion of its limitations and the conditions under which the experiments were conducted.\n- Can you provide more details about the potential challenges and solutions for scaling the system to handle more complex and varied tasks?\n- The paper mentions that there are significant visual occlusions when using the GR-1 robot with grippers, particularly when the gripper grasps objects, which complicates visual inference. It would be helpful to include a discussion of possible methods to mitigate these challenges.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIgHLbTSNS", "reviewer": "Reviewer_QGk6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper describes a teleoperation system for dexterous robot arms. The system is used to collect human demonstrations for everyday tasks, and policies are learnt to imitate these demonstrations.", "summary_of_recommendation": "The paper needs important additional experiments to show the importance of the proposed teleoperation system.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Check the performance of policies without articulated head for the tasks chosen in the experiments.\n- Measure the accuracy of the teleoperation system by measuring the difference in the 3D location of various corresponding points on the human and robot arm.\n- Please clarify L114. What does \"projected from\" mean?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Pre-rebuttal**

This paper receives a mixed review from three reviewers.

On the positive side, reviewer **QGk6** recognizes the outstanding engineering effort put into the system and its potential contribution to the research community if open-sourced. Reviewer **xknk** acknowledges the empirical effectiveness of the system in improving teleoperation and imitation learning performance.

On the negative side, both reviewer **QGk6** and **mH7J** question about the claimed contribution on the active vision piece. Reviewer **QGk6** questions about the validation of head articulation in teleoperation and policy performance, as well as more quantitative evaluation of the teleoperation interface. Reviewer **mH7J** raises concerns about novelty, particularly on the active vision side and also the differentiation to the established telerobotics field. A lack of discussion on certain design choices is also brought up. Reviewer **xknk** mentions the lack of discussion on the limitation side.

---
**Post-rebuttal**

This paper initially received two weak reject and one weak accept. One reviewer acknowledged the merit of the rebuttal and raised the assessment from weak reject to weak accept.

After reading the paper and reviews, AC agrees with the main criticism, including unclear contribution, particularly with respects to the telerobotics field, as well as the missing empirical validation of the active vision component. However, onboard with the reviewer who overturned the rating, AC agrees that the rebuttal has sufficiently addressed the concerns. 

Considering the recent research landscape and the growing interest in the teleoperation of bimanual/humanoid robots, AC thereby recommends to accept the paper with the belief that the paper will be of sufficient interest to the CoRL audience.

AC also encourages the authors for a proper open source endeavor if the paper is accepted, since the contribution of the work is largely contingent upon the open source of the proposed system.

Typos:
- [Line 79] "we" -> "We"
Decision: Accept