OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation
Authors: Aadhithya Iyer
Zhuoran Peng
Yinlong Dai
Irmak Guzey
Siddhant Haldar
Soumith Chintala
Lerrel Pinto
Venue: CoRL 2024 (2024)
Keywords: Teleoperation
Robot Learning
Robotic Manipulation
TL;DR: Versatile Teleoperation System for Robotic Manipulation

Abstract: Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including multi-fingered hands, bimanual arms, and mobile manipulators, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feedback and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. A comprehensive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, Allegro, and Hello Stretch platforms, OPEN TEACH is fully open-sourced to promote broader adoption. Videos are available at https://anon-open-teach.github.io/.

OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation
Aadhithya Iyer* New York University
Zhuoran Peng New York University
Yinlong Dai New York University
Irmak Guzey New York University
Siddhant Haldar New York University
Soumith Chintala
Lerrel Pinto New York University
Meta Al
https://open-teach.tiiee htt.:/pppnnttaac..ihh.......bb
Open Teach
Figure We present OPEN TEACH, unified robot teleoperation framework that supports multiple arms and hands, allows mobile manipulation, is calibration-free and works across both simulation and real-world environments. OPEN TEACH uses a VR headset for telcoperation, offers low latency and high-frequency visual feedback. This high-freque fighffrequency operation allows human users correct for robot errors in real time, facilitating the exccution of intricate and long-horizon tasks. From making sandwich and ironing cloth to placing items basket and lifting it approaching a cabinet and opening OPEN TEACH delivers comprehensive, user-friendly teleoperation experience for a wide range of applications. OPEN TEACH is fully open-source.
*Correspondence to: audhihyyaiyer@@nyu.ddu
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Abstract: soen-soureed, user-friendly tools form the bedrock of scientific ad- vancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprictary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. Built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including mulle--ingeeed hands, bimanual arms, and mo- bile manipulators, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feed- back and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. comprehen- sive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, Allegro, and Hello Stretch plat- forms, OPEN TEACH fully open-sourced to promote broader adoption. Videos are available at https: http:/:://///eeeeeeeeeee teach. .iich.githi github. io/.
Keywords: Low-Cost Teleoperation, Imitation Learning, Robot Manipulation
1 Introduction
The integration of learning-basse based methods has sparked revolution in robotics, significantly enhanc- ing capabilities in manipulation [1, 2, 3, 4), locomotion [5, 6, 7, 8), and aerial robotics [9, 10, 11]. More recent work has been making advancements in complex single-task behavior learning [12. 13, 21, multitask scenarios [14, 15), multimodal behavior learning [16, 17, 18, 19), and efficient fine-tuning of pretrained behavior models [20, 21, 22]. A fundamental requirement across all these threads of research is the need to collect data in the form of task demonstrations.
Commonly used teleoperation systems include devices such as joysticks and 3D spacemouses [23, 24], commercial VR headsets [25, 26, 27, 13, 28, 29], kinesthetic teaching [30], and phone teleop- eration [31]. The aforementioned devices are cost-effective and easy set up. However, they often unintuitive to use and require extensive user-training to demonstrate intricate motions. Re- cently proposed exoskeleton-basee teleoperation frameworks like ALOHA [2], GELLO [32], and AirExo [33] attempt to alleviate this problem by having the human teleoperator directly control kinematically isomorphic version of the same robot arm. These frameworks directly impose the kinematic constraints of the robot arm during teleoperation making more compatible and intu- itive control the motion of the robot. Although highly effective, these systems can require an additional robot for each robot being controlled, have high initial setup costs, and are designed for specific robot morphologies.
The challenge of easy-to-use teleoperation devices is more apparent in dexterous manipulation prob- lems [34, 27, 13], owing to the high dimensional action space. Such frameworks typically involve the use of expensive gloves [36, 37, 38), extensive calibration processes [34, 27], or are susceptible monocular occlusions [27].
In this work, we present OPEN TEACH, open-source framework for robot teleoperation that sup- ports a variety of robots, including bimanual and multi-finger manipulation, all at price of $500. As shown in Figure 1, OPEN TEACH uses a VR headset (e.g. Quest 3) to put users / teachers in an immersive virtual world where they can view a robotic scene both through their eyes, via visual passthrough, as well as realtime streams from the robot's cameras. To control the robot, users can simply use hand gestures, which are detected using onboard hand-pose estimators 90Hz. As re- sult, even though OPEN TEACH is kinematics-unawee unaware,, the high frequency execution and improved hand pose detection accuracy enables users to collect high-quality real-time robot demonstrations.

---

We experimentally evaluate OPEN TEACH on 38 tasks across single arm, bimanual, multi-fire fingered, and mobile manipulation robot setups in both simulation and the real world. The tasks span from tabletop manipulation to contact-rich dexterous manipulation. Across different robot morphologies, we find that users can provide demonstrations at speeds on par with robot-specific teleoperation sys- tems and significantly faster than general-purpost systems like AnyTeleop [35]. Importandl, poli- cies trained on the data collected achieve an average success rate of 86% 10 tasks in simulation and the real world, validating the utility of policy learning using OPEN TEACHI. The contributions of this work is summarized as follows:
We present OPEN TEACH, an open-source system for plug-and- play teleoperation frame- work suitable for collecting demonstrations across different robot morphologies in both simulation and the real world.
2. We experimentally show that the demonstrations collected by OPEN TEACH can be used to train effective, general-purpose purpose manipulation behaviors.
3. Our user study on 15 users highlights the efficacy of OPEN TEACH for both experienced and new users.
OPEN TEACH is fully open-source with the mixed reality API, policy training code, and demonstra- tions collected using OPEN TEACH available at htttps://a //op:e:///peeeeee teach. github.io/ io/.
2 Related Work
2.1 Robot-Specific Teleoperation
Teleoperation, as a medium for human-robot interaction, has been crucial part of robotics. Recent strides in learning-based methods demand extensive data collection, giving rise to diverse teleop- eration systems - joysticks and spacemouses [23, 24), VR controllers [25, 26, 27, 13, 28, 29), RGB cameras [34, 39, 35, 40]. IMU sensors [41, 42, 43], kinesthetic teaching [30], phone teleoper- ation [31], gloves [36, 37, 38), marker-based motion capture systems [44, 45], and reacher-grabber sticks [46, 4]. However, several challenges remain in the robot-specific nature of these devices. For instance, devices like joysticks, spacemouses, and phones are limited to controlling robot arms due to their lack of fidelity for multi-fingered hands. There have been systems developed map the human hand pose to the robot pose [27, 13, 35, 47, 48, 491, but they are often restricted to only controlling robot hands. Further, all of these frameworks lack awareness of the robot's 's kinematic constraints, leading to challenges in intuitive control, especially in complex poses. As a solution, there are more conventional but expensive exo-skeleton based teleoperation systems [50, 51, that use second arm for controlling the manipulator arm. Recently proposed ALOHA [2, 53] affirms the effectiveness of this approach through impressive results in fine-grained bimanual manipulation. However, these systems require exact copy of the manipulator robot arm, rendering them costly and less practical for heavier robots. In addressing these issues, GELLO [32] and AirExo [33] in- troduce exo-skeleton teleoperation frameworks, utilizing kinematically isomorphic variant of the robot arm. This approach proves more affordable and lightweight, enhancing usability for humans. Despite their success in fine-grained manipulation tasks, these solutions are constrained to robot arms and face challenges in extending seamlessly to control robot hands. For multi-fingered robot hands, gloves, vision-based, and VR-based methods have been employed. These systems either as- sume fixed robot arm [27, 13] or are tied to specific robot setup [34, 54], making them difficult to transfer to new arm-hand systems and new environments.
2.2 Unified Teleoperation Frameworks
The robotics community has often sought to develop versatile systems that operate across di- verse environments and robots [55, 14, 56). Leveraging the success of learning-basee methods, achieving this requires telcoperation systems adaptable various robot variants, allowing for abun- dant data collection with minimal setup costs. While methods combining robot arms with multi- fingered hands exist [34, 39, 28, 57, 54], their applicability across robot variants remains unclear.
3

---

Human-to- Robot Retargeting
Network Server
Pose Detection
Hand Pose Detection
Wrist Pose Detection
Hardware
Visnal Fashbockk
Camera Streum
Oculus Passthrough
Figure 2: Overview the teleoperation module in OPEN TEACH. Provided a hand and wrist pose within the VR interface. the controller transmits keypoint data to the robot's server. The server then transforms and retargets these key points to align with the specific robot setup. Real-time visual feedback of the telcoperated scene promptly relayed back to the VR headset.
AnyTeleop [35] makes progress in this direction by proposing a robot-agnostic system compatible with multiple hands and arms. We build upon this idea in creating OPEN TEACH..
3 OPEN TEACH
In OPEN TEACH, a user wears Virtual Reality (VR) headset to provide demonstrations to a robot. This involves creating a virtual world for teaching, retargeting the teacher's hand and wrist pose to the robot joints, and finally controlling the robot. We compare OPEN TEACH with various other teleoperation systems across variety of robot types (Appendix ??) and observe that OPEN TEACH is the only framework that enables controlling multiple arms, hands, and mobile manipulators, is calibration-free, and is completely open-source. In this section, provide details about the VR- based teleoperation setup and the system design that enables data collection using this framework.
3.1 Placing an Operator in a Virtual World
We use the Meta Quest 3 VR headset to place the human teacher in virtual world. The headset surrounds the human in a virtual environment at resolution of 2064 x 2208 and a native refresh rate of 90Hz. The base version of this headset is affordable at $499 and is relatively light at 513g. Compared to the Meta Quest 2 VR headset used in prior work [13], the Quest provides a full-color passthrough allowing the human to get a direct view of the robot setup during teleoperation. These features, especially the full-color passthrough, allow for comfortable and intuitive operation by the Additionally, similar Arunachalam al. [13], the Quest API interface allows for creating custom mixed reality worlds that visualize the robotic system along with diagnostic panels in VR. Examples of virtual scenes have been shown in Fig. 2 and Fig. 3. It is important to highlight the exceptional clarity of the scene passthrough visible in Quest 3.
3.2 Pose Estimation with VR Headsets
Similar to Arunachalam et al. [13], we directly use the in-built hand pose estimator [58] of the Quest using 2 monochrome cameras. This is significantly more robust compared to single camera alternatives [59]. Further, since the cameras are internally calibrated, they do not require specialized calibration routines that are needed in prior multi-camera teleoperation frameworks [34, 35). Also, since the hand-pose estimator is integrated into the device, can stream real-time hand poses at 90Hz. This alleviates the challenge of obtaining hand poses at both high accuracy and high fre- quency, reported in prior work [34, 27].

---

Make Sandwich
Iron Clolb
Pack Basket
Figure 3: The demonstration collection process as viewed from within the VR application. Shown here is one task being performed for each real-world setup. High resolution images streamed at 90 Hz to the VR application allow for an immersive experience and enable reactive control the
3.3 Human to Robot Pose Retargeting
The inbuilt hand pose estimate from the VR headset provides us with the joint positions of all the fingers of the human hand and the wrist. With this information, we can design wrappers that use combinations of these joint positions to map the human hand poses to the robot poses for any given robot morphology. In this work, we variety of robot arms, each with either 2- 2-fingered gripper multi-fingered robot hand. Below, we provide the wrapper design for each robot morphology used in this work.
Robot Arm We establish 3D coordinate system using the wrist keypoint and knuckle points index and pinky fingers to define 2D plane along the palm and a perpendicular third axis. The wrist position maps the robot end effector position. Changes in the orientation of this hand coordinate system over time map to adjustments in end effector orientation.
Robot Hand We use the teacher's hand pose obtained from the VR to compute the individual joint angles in the teacher's hand. Given these joint angles, a straightforward method of retargeting is to directly command the robot's joints to the corresponding angles. In practice, this works well for all fingers except the thumb. To address this, we improve upon Arunachalam al. [13]. where the spatial coordinate of the teacher's thumb tip is mapped that of the robot hand and then an inverse kinematics solver is used to compute the joint angles of the thumb. More details about the improvement in thumb retargeting have been included in Appendix 5.1. Since the Allegro hand does not have a pinky finger, we ignore the teacher's pinky joints.
Two-fingered gripper detect the opening and closing of the two-fingered gripper, we utilize the pinch between the pinky finger and the thumb. The pinch is detected by computing the distance between the tips of the fingers and setting threshold on the pinch distance. We toggle mechanism for opening and closing the gripper where each pinch indicates toggling the alternate state of the gripper.
Mobile manipulator The same coordinate system established for controlling robot arms used for mapping the wrist's movements actions of the mobile robot. When the wrist moves forward, it extends the robot's arm, enabling it reach farther. Vertical wrist movements adjust the robot's height, while lateral wrist movements cause the robot move sideways by controlling its wheels. The 3D transformations across time are mapped to end effector orientation changes. The opening and closing of the gripper are controlled through the pinch between the index finger and the thumb.
These are just a few examples of how the hand pose data can be used to obtain a mapping between the human hand and a robot. The simplicity of the proposed framework has been summarized in Code Snippet ?? in the appendix. The primary idea behind OPEN TEACH is that given a VR headset, the end-user has the flexibility to design their own human-to-robot wrappers using the
5

---

human hand poses as imput. The framework has been designed for simple integration with any robot setup, allowing robot teleoperation with real-time streaming (up to 90Hz) and low-latency visual feedback. This significantly reduces the initial setup cost as compared to prior exoskeleton-ban teleoperation frameworks like GELLO [32] and AirExo [33].
3.4 Robot Control
Achieving minimal error and low latency is pivotal for OPEN TEACH to facilitate the intuitive tele- operation of the robot hand by the human teacher. In this study, we employ the Allegro Hand as our robotic hand, controlling it asynchronously through the ROS [60] communication framework. Using the computed robot joint positions from the retargeting procedure, PD controller outputs desired torques at frequency of 300Hz. To mitigate steady-state error, we include a gravity compensation module compute offset torques.
trim
We use three different robot arms for our evaluations - xArm, Franka Emika Panda, and Kinova Jaco. We use different controllers for each. The xArm is directly controlled through the official xArm API [61]. For the Franka Emika Panda, use the Deoxys controller
[62]. For the Kinova Jaco, we the controller open-sourced by Arunachalam et al. [27]. The streaming frequencies for all robots is provided in Table 1. Such high frequency teleoperation allows the human teacher to see the robot move in real time and immediately correct execution errors in the robot. The Hello Stretch controlled at 5Hz using the controller released by Shafiullah et al. [4]. Further, we acknowledge the fact that the human hand possesses fewer degrees of freedom than the robot. In response, we introduce pause functionality, allowing the teacher to momentarily halt teleoperation, reorient themselves, and resume the process. We also implement resolution adjustment feature, which provides performance boost for high-precision tasks such as delicately picking up tea sachet. Details about these implementations have been included in Appendix 5.1.
Experiments
Our experiments and tasks are designed to answer the following questions: (1) How versatile is OPEN TEACH across range of robotics setups? (2) How successful are policies trained with OPEN TEACH? (3) Can OPEN TEACH be used for performing complex, long-horizon tasks? (4) How intuitive is the system for new users?
4.1 Experimental Setup
We evaluate the versatility of OPEN TEACH by using to collect demonstrations on six different setups four in the real world and two in simulation. Each setup is combination of variant of robot arm with either an Allegro Hand or a 2-ngeee gripper. The real-world setups include:
1. Franka-Allegro: A Franka Arm with an Allegro Hand having the Xela tactile sensors.
2. Kinova-Allegro: Kinova Jaco Arm with an Allegro Hand with the Xela tactile sensors.
3. Bimanual: 2 xArm7 robot with 2-fingered grippers.
4. Stretch:
Hello Stretch mobile manipulator with 2-fingered gripper.
The Flaeka-Allegro and Kinova-Allegro comprise a single Intel Realsense camera for data collec- tion, whereas the Bimanual setup collects data from 5 different cameras. The Stretch has an iPhone attached the wrist for data collection [4]. The simulated environments include:
1. Allegro Sim: A floating Allegro Hand capable of performing static and dynamic tasks.
2. LIBERO Sim [23]: A Franka Arm with a 2-fingered gripper placed in varied scenes.
We demonstrate the usefulness of the collected data by training visual and visuotactile policies using behavior cloning [63] and inverse RL [64, 65].

---

Performance of policies learned on collected through OPEN TEACH. Franka-Allegro. Allogro Sim, and Libero Sim, TAVI [66]. FISH [21] and BC were spectively used train policies.
Table Teleoperation Frequency across all robots.
Number of Success
Robot Setup
Stream
Demos
Rate
Domain
Robot Selup
Frequency
Franka-Allego
Open Bix
9/10
Arm
End Eflector
Grasp Sponge
7/10
Keal
Franka Allegm
Pick Tea Sachet
7/10
Kinova-Allegro
60
60
Grasp Object and Twist
8/10
Bimanual
Allegm Sim
Flip Cahe
10/10
Stretch
5
Flip Sponge
10/10
Allegro
Pinch Grasp
7/10
LIBERO Sim
20
20
Libero Sim
Close Top Drawer of Cabinet
10
10/10
Turn Stove
10
9/10
Pick and Place Soup
into Rasket
50
9/10
Table 3: User study comparing OPEN TEACH with baselines when used by experts and users.
Median completion time
Tisk
Success Rate
successful demonstrations (in s)
Experi Open Teach
New
Experi
Holo-Dex
Any Teleop
Open Teach
Holo-Dex
Any Teleop
Open Teach
Open Teach
Flip cube
6.58
13.71
5.5
2.85
Pinch Grasp
0
0.2
0.8
17.49
18.94
18.72
3.71
Pour
N/A
N/A
0.8
N/A
40.97
14.83
Pick and Place
NIA
N/A
0.8
0.8
N/A
NIA
23.57
11.875
Open box
32.21
4.2 Imitation Learning with OPEN TEACH Data
Here, we describe the algorithms used for learning policies on data collected through OPEN TEACH.
1. Franka-Allegro: We record both visual and tactile data for this setup. The policies trained using TAVI [66], a demonstration-guude residual RL algorithm that collects a few expert demonstrations and learns a robot policy using both visual and tactile data.
2. Allegro Sim: We only record visual data for this setup and train policies using FISH [21].
3. LIBERO Sim [23]: We only record visual data for this setup. The policies are trained using transformer-based BC with GMM head [67] and action chunking [2].
4.3 How versatile is OPEN TEACH across robotic setups?
The primary idea behind OPEN TEACH is that given any robotic setup, user can purchase affordable off-the-shelf VR headset (in this case, Quest 3) and plug the headset and robot setup into the proposed framework to start teleoperating the robot without any additional hardware setup cost. To investigate its versatility, we use OPEN TEACH to teleoperate four different real world robotic setups, each having a different combination Kinova Allegro, Bimanual setup with 2 xArm7 robots, and Hello Stretch for mobile manipulation. We also exhibit the applicability of OPEN TEACH in simulation through evaluations on 2 simulated environment suites Allegro Sim and LIBERO Sim [23]. The frequency of teleoperation for each of the setups has been provided in Table 1. 'abble provides set of tasks performed on Franka- Allegro, Allegro Sim, and LIBERO Sim. A more comprehensive list of tasks, including those on the Kinova-Allegro, Bimanual, and Stretch setup have been provided in Fig. and Appendix 5.2.2.
robot arm end effector type - Franka Allegro,
4.4 How successful are policies trained with OPEN TEACH?
Table 7 provides the success rates of policies learned using imitation learning across both the real- world and simulated setups. We use TAVI [66] to learn visuotactile policies on Franka-Allegro, and FISH [21] to learn visual policies on Allegro Sim. Similar to prior work [66, 21], these policies were learned within 20 minutes and achieved an average success rate of 82%, validating the high
quality of the collected observation data. Behavior cloning policies LIBERO Sim achieve an average success rate of 93%, confirming the high quality of the collected action data. Overall, the
7

---

learned policies achieve an average success rate of 86% across all tasks and robot morphologies. This highlights the effectiveness of OPEN TEACH in collecting data for policy learning.
4.5 Can OPEN TEACH be used for performing complex, long-horizon tasks?
In this section, we emphasize the efficacy of OPEN TEACH in executing a diverse array of complex, long-horizon tasks across various robotic configurations. Appendix 5.2.2 includes real-world task rollouts for the Bimanual, Franka-Allegro, Kinova-Allegro and Stretch setups. OPEN TEACH allows the collection of demonstrations for intricate tasks, ranging from high-precision activities like USB insertion to delicate movements such as slicing cucumber. On the multi-fingered hand setup, we demonstrate a broad spectrum of tasks, encompassing extended activities like placing objects in basket and lifting it to contact-rich manipulation scenarios like opening a laptop and sliding a tea sachet off the table. detailed compilation of tasks performed both real-world and simulated setups, along with more task rollouts, have been included in Appendix 5.2. Videos showcasing these task rollouts can be found our project website.
4.6 How intuitive is the system for new users?
We assess the user-friendliness of OPEN TEACH through a comprehensive user study involving 15 new users. The study is conducted on the Franka-Allegro setup, chosen for its capacity to evaluate the system's performance on both the robot hand and the robot arm. Each participant allocated a 10-minute practice session to familiarize themselves with teleoperating the robot setup. Following this, they perform five trials for each of three distinct tasks using Holo-Dex [13], AnyTeleop [35], and OPEN TRACH. To mitigate potential biases, the order of tasks is randomized for each user.
In Table 3, we present a comparative analysis of success rates and median completion times for new users across Holo-Dex, AnyTeleop, and OPEN TEACH for the tasks of cube flipping and pinch grasp- ing. Given the relatively small user sample size, we chose to analyze the median rather than mean completion times mitigate potential biases from outliers. Since the Holo-Dex and Any TeTolo baselines lack open-source code for arm retargeting, volving movements. Thus, our comparison is limited to the cube flipping and pinch grasping that do not require arm manipulation. On these tasks, OPEN TEACH demonstrates a higher success rate along with significantly reduced median time to complete tasks compared to the other baselines.
were unable evaluate them on tasks in-
Table 3 also includes a comparison of success rates and median completion times between an expert and new users utilizing OPEN TEACH for all tasks. On average, new users exhibit a success rate that is 76% of the expert's and take 2.25x longer to complete a task. Details regarding individual user performances provided in Appendix 5.3. Intriguingly. some new users, despite their unfamiliarity with the framework, achieve comparable or superior performance to the experts in certain tasks. This observation highlights two factors: (1) the inherent variation in abilities among individuals, and (2) while our system is intuitive for new users, prolonged training leads to substantial improvement in their performance, with the potential for further enhancement with continued practice.
Limitations and Discussions
In this work, we introduce OPEN TEACH, an open-source unified framework designed to facilitate low-latency, high-frequency robot teleoperation. This versatile framework tailored accommo- date diverse tasks and is compatible with range of robot morphologies. However, we recognize few limitations in this work: (a) OPEN TEACH relies on the accuracy of the in-built hand pose detection in the VR headset. Inaccuracies, particularly when fingers are occluded from view, can diminish the quality of hand tracking, posing challenges to teleoperation. (b) In specific instances, the pose detector on the Oculus board may misconstrue finger positions, leading to difficulties in executing gestures like gripper closing, which relies on precise pinches between fingers. Addressing these challenges through future research on hand pose detection and tracking holds the potential to enhance the ease and intuitiveness of teleoperation using VR headsets.
8

---

Acknowledgments
We thank Mahi Shafiullah, Raunaq Bhirangi, Ben Evans, Yibin Wang, Venkatesh Pattabiraman, Haritheja Etukuru and Chenyu Wang for valuable feedback and discussions. This work was sup- ported by grants from Honda, Meta, Amazon, and ONR awards N00014-21-1-2758 and N00014- 22-1-2773. Lerrel Pinto is supported by the Packard Fellowship.
References
[]] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy: for goal- conditioned dynamic manipulation of deformable objects. arXiv preprini urXiv:2203.00665: 2022.
[2] Z. Zhao, Kumar, Levine, and Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2023.
[3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0668/ 22.0117, 2022.
[4] N. M. M. Shafiullah, A. Rai, H. Etukuru, Y. Liu, I. Misra, S. Chintala, and L. Pinto. On bringing robots home. arXiv preprint arXiv:2311.1609. 2023.
[5] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics, 2022.
[6] Y. Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter. Combining learning -bsed locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377-2382) 2022. doi: 10.1 109/LRA.2022.314355...
[7] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning walk in 20 minutes with model-free reinforcement learning, arXiv preprint arXiv:2208.07860, 2022.
[8] X. Cheng, Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309.143.. 2023.
[9] T. Zhang, G. Kahn, S. Levine, and P. Abbeel. Learning deep control policies for autonomous acrial vehicles with mpe-guided policy search. In 2016 IEEE international conference on robotics and automation (ICRA), pages 528 535. IEEE, 2016.
[10] D. Gandhi, L. Pinto, and A. Gupta. Learning to fly by crashing. In 2017 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 3948-304 3955. IEEE, 2017.
[11] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter. Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096-203: 2017. 1oi: 109/LRA. 2017.2720851.
[12] C. Wang. L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302. 12422, 2023.
[13] S. P. Arunachalam, I. Güzey, S. Chintala, and I.. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5962-5969. IEEE, 2023.
[14] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x mod- cls. arXiv preprint 2rX1v:2310.08864, 2023.

---

[15] H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Gener- alization and efficiency in rohot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.019/8, 2023.
[16] N. M. Shafiullah, Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955-22968. 2022.
[17] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10000 2022.
[18] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 2rXiv:2303.04117, 2023.
[19] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies, arXiv preprint arXiv:230400253. 2023.
[20] Haldar, V. Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. arXiv preprint arXiv:2206. 15469, 2022.
[21] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.0149p) 2023.
[22] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09:300 2020.
[23] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.
[24] N. E. Sian, K. Yokoi, Kajita, F. Kanehiro, and Tanie. Whole body teleoperation of humanoid robot development of simple master device using joysticks. Journal of the Roboties Society of Japan, 22(4):519-527, 2004.
[25] Z. Gharaybeh, H. Chizeck, and A. Stewart. Telerobotic control in virtual reality. In OCEANS 2019 MTS/IEEE SEATTLE, pages 1-8, 2019. 000:003319//0EAAN44099....1.9999
[26] T. Zhang, Z. McCarthy, Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
[27] P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.1322 2022.
[28] I. Radosavovic, T. Xiao, James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learn- ing with masked visual pre-training, 2022. URL ptttps://arxi..org//// wwwp//s//////.ooo..///oo 03109.
[29] A. George, A. Bartsch, and A. B. Farimani. Openvr: Teleoperation for manipulation. arXiv preprint arXiv:2305.09776, 2023.
[30] A. G. Billard, S. Calinon, and F. Guenter. Discriminative and adaptive imitation in uni-manual and bi-manual tasks. Robotics and Autonomous Systems, 54(5):370-384 2006.
[31] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, et al. Roboturk: crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[32] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint 230.:23039113777 2023.
10

---

[33] H. Fang, H.-S. Fang. Y. Wang, J. Ren, J. Chen, R. Zhang, W. Wang, and C. Lu. Low-cost ex- oskeletons for learning whole-arm manipulation in the wild. arXiv preprint arXiv:2309. 14975, 2023.
[34] A. Handa, K. Van Wyk, W. Yang. J. Liang, Y.W.. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation dexterous robotic hand-arm system. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9170, 2020. 001:10.1199/CRR4409445 202099999999022229999999911144
[35] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W. Chao, and D. Fox. Anytelcop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577, 2023.
[36] M. Laeiro-Rodriguez. I. verr-GGozzāeez F. A. Mikic-Fonte and M. Llamas-Nistal. A system- atic review of commercial smart gloves: Current status and applications. Sensors, 2021. ISSN 1424-8220. doi: 10.3390///22002266.
[37] V. Kumar and E. Todorov. Mujoco haptix: A virtual reality system for hand manipulation. 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 657-663, 2015. doi:10. LAL.LOHHMAAOODS000005555005
[38] Li, J. Jiang, Ruppel, H. Liang, X. Ma, N. Hendrich, F. Sun, and Zhang. A mobile robot hand-arm teleoperation system by vision and imu. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 19900-10906. IEEE, 2020.
[39] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning robotic hand imitator by watching humans youtube, 2022.
[40] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. RA-L, 2020.
[41] M. Laghi, M. Maimeri, M. Marchand, C. Leparoux. M. Catalano. A. Ajoudani, and A. Bicchi. Shared-autonomyy control for intuitive bimanual tele-manipulation. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Hummanoids) pages 1-9. IEEE, 2018
[42] S.-K. Kim, Hong, and D. Kim. walking motion imitation framework of humanoid robot by human walking recognition from imu motion data. 2009 9th IEEE-RAS International Conference on Humanoid Robots, pages 343-348. IEEE, 2009.
[43] Y. Wu, P. Balatti, M. Lorenzini, F. Zhao, W. Kim, and A. Ajoudani. teleoperation interface for loco- beco-manipulation control of mobile collaborative robotic assistant. IEEE Robotics and Automation Letters, 4(4):3593-3600, 2019.
[44] W. Zhao, J. Chai, and Y.-Q. Xu. Combining marker-based mocap and rgb-d camera for quiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPHH/enograppiis symposium on computer animation, pages 33-42, 2012.
[45] S. Liu, H. Jiang, J. Xu, S. Liu, and X. Wang. Semi-supervised 3d hand-object poses estimation with interactions time. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14687-14697, 2021.
[46] Pari, N. M. Shafiullah, S. Arunachalam, and Pinto. The surprising effectiveness of representation learning for visual imitation, 2021.
[47] S. Li. X. Ma. H. Liang, M. Gömer,, P. Ruppel, B. Fang, F. Sun, and Zhang. Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In 2019 Inter- national Conference on Robotics and Automation (ICRA), pages 416-422, IEEE, 2019.
11

---

[48] D. Antotsiou, G. Garcia Hereiidd and T.-K. Kim. Task-oriented hand motion retargeting for dexterous manipulation imitation. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[49] C. Meeker, M. Haas-Heger, and M. Ciocarlie. A continuous teleoperation subspace with empir- ical and algorithmic mapping algorithms for nonanthhoooomorpiic hands. IEEE Transactions on Automation Science and Engineering, 19(1):373-333 -366, 2020.
[50] Hulin, K. Hertkorn, P. Kremer, S. Schätzle, J. Artigas, M. Sagardia, Zacharias, and
C. Preusche. The dlr bimanual haptic device with optimized workspace. 2011 IEEE ternational Conference Robotics and Automation, pages 3441-3442. IEEE, 2011.
[51] B. G. Katz. A low cost modular actuator for dynamic robots. PhD thesis, Massachusetts Institute of Technology, 2018.
[52] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke. Nimbro avatar: Interactive im- mersive telepresence with force-feedback telemanippuattinn in 2021 ieee. In RSS International Conference Intelligent Robots and Systems (IROS), pages 5312-5319.
[53] Z. Fu, Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.0224 2024.
[54] S. Li, N. Hendrich, H. Liang, P. Ruppel, C. Zhang, and J. Zhang. A dexterous hand-arm teleoperation system based on hand pose estimation and active vision. IEEE Transactions Cybernetics, 2022.
[55] D. Shah, A. Sridhar, Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023. URL https::///aaat: org/ abs//b/b.../////////// 1..6..4855 14846.
[56] Octo Model Team, D. Ghosh. H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https http::///coooomml//e itteo-mmmeee...t....ii.....
github. io, 2023.
[57] M. Mosbach, K. Moraw, and S. Behnke. Accelerating interactive human-like manipulation learning with gpu-based simulation and high-quality demonstrations. In 2022 IEEE-RAS s International Conference on Humanoid Robots (Humanoids), pages 435-441. IEEE, 2022.
[58] Han, Liu, R. Cabezas, C. D. Twigg, Zhang, Petkau, T.-H. Yu, C.-J. Tai, Akbay,
Z. Wang, A. Nitzan, G. Dong, Y. Ye, L. Tao, C. Wan, and R. Wang. Megatrack: Monochrome egocentric articulated hand-tracking for virtual reality. 2020.
[59] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grund- mann. Mediapipe hands: On-device real-time hand tracking, 2020.
[60] Stanford Artificial Intelligence Laboratory al. Robotic operating system. URL https:
//www. ros. org.
[61] xAmm Developer. xArm Mam--Pthon-SSD SDK.
xarm python
sdk.
[62] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arxiv::210.11133) 2022. 40.:11044855/mxxi..22200 1339.
[63] D. A. Pomerleau. Alvinn: An autonomous land vehicle a neural network. Touretzky, editor, NewrIPS, volume KaufaanKnn 1988.
[64] A. Y. Ng. J. Russell, et al. Algorithms for inverse reinforcement learning. In ICML, 2000.
12

---

[65] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
[66] I. Guzey, Y. Dai, B. Evans, S. Chintala, and L. Pinto. See touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309.123000 2023.
[67] D. A. Reynolds et Gaussian mixture models. Encyclopedia of biometrics, (41(666-663, 2009.
13

---

Appendix
5.1
Framework details
5.1.1
Structure the framework
We use ZeroMQ for networking between nodes. The OPEN TEACH framework is divided into two parts teleoperation and data collection.
Teleoperation T'e teleoperator is divided into 5 components Detector, Keypoint Transformer, Op- erator, Controller, and Visualizer. A brief description of each has been provided below.
1. Detector: Receives the hand keypoints from the Meta Quest 3 and publishes them to ZMQ sockets.
2. Keypoint Transformer: Subscribes the keypoints published by the detector and maps them to the robot pose.
3. Operator: Receives the robot pose from the keypoint transformer and the current robot state from the controller. The operator computes the robot's ' actions which are published to a ZMQ socket.
4. Controller:
Subscribes an action from the operator and takes an action in the real or simu- lated environment. After taking the action, the controller publishes the current states of the environment for use by the operator.
5. Visualizer: Subscribes the RGB images from the camera process (or the environment in case of simulations) and puts it on the screen inside the VR app for visualization during teleoperation.
Data Collection data recorder process subscribes sensor information (RGB and Depth images, tactile readings, timestamps) and robot-specific information (joint states, gripper states, timestamps) from the corresponding sockets and logs them in corresponding files. The data is then compiled together by matching the timestamps between the sensor information and robot-specific data.
5.1.2 Thumb Retargeting for Robot Hand
Section 3.3 provides details about the design of the OPEN TEACH wrapper for the robot hand. To recap, given the individual joint angles in the teacher's hand from the VR headset, the joint angles for the robot hand can be computed by directly commanding the robot's 's joints to the corresponding angles. This works well for fingers except the thumb. Holo-Dex(13] 13] deals with this by mapping the spatial coordinate of the teacher's thumb tip that of the robot hand. Then an inverse kinematics solver is used compute the joint angles of the thumb. In this case, the retargeting of the thumb done in 2D space. These bounds, depicted in Fig. 4(a), define the thumb's reach limits. During retargeting, the thumb tip's zone on the 2D palm plane is detected, and a perspective transform from the human hand to the robot hand is applied, aligning the human thumb tip with the robot thumb tip on the 2D plane. However, using three separate bounds introduces jitters when the thumb tip transitions between zones and results stagnancy when outside the bounds. Further, Holo-Dex, the height the robot thumb tip is fixed, allowing it to only move along the 2D space.
To address these challenges, OPEN TEACH employs single, large zone spanning the entire thumb's workspace in 2D space(refer to Fig. 4(b)). When the thumb is within bounds, a perspective trans- formation retargets the human thumb tip to the robot thumb tip. In cases where the thumb goes out of bounds, the closest point within the bound is estimated and used for retargeting, avoiding stagna- tion. Additionally, instead of fixed height, the thumb allowed move perpendicular the 2D surface along the palm, mapping the height of the human thumb tip the robot thumb tip based on maximum and minimum height bounds. This approach ensures smoother thumb motion and enables the performance of more complex tasks compared to Holo-Dex [13].
14

---

Zonn
Porne
(o)
(b) Open
Figure 4: Thumb retargeting difference
Table 4: Time
Average time collect demo
Robot Setup
Task
(in s)
Fraeka-Allegro
Open box
45
Grasp sponge
60
Pick up satchet
60
Grasp object and twist
35
Kinova-Allegro
Unfold towel
40
Open pack of cream
10
Open ketchup bottle
40
Bimanual
Uncap marker
60
Sweep table
60
Pour sprinkles in bowl
40
Allegro Sim
Flip cube
3
Flip sponge
20
Pinch Grasp
15
LIBERO Sim
Close top drawer of cabinet
10
Turn on stove
25
Pick up and put soup
30
can the basket
5.2 Task Details
5.2.1
Demo Collections times
Table 4 provides the average times required collect demonstration for 16 tasks across real- world setups (Franka-Allegro, Kinova-Allegro, Bimanual) and simulated environments(Allegro sim, LIBERO sim).
5.2.2 Task Descriptions
Fig. 5, Fig. 6, Fig. 7. Fig. 8, Fig. 9, and Fig. 10 provide rollouts of all the tasks performed both in the real world and in simulated environments. Each task rollout is labeled with the name of the task and task description.
5.3 User Study
Following up from Section 4.6, provide the success rate and average completion times for all 15 users for each task performed in Table 5 and Table respectively. Each user roughly performed tasks average, with 5 trials for each task. As mentioned in Section 4.6, since the Holo-Des [13]
15

---

and AnyTeleop [35] baselines lack open source code for arm retargeting, we were unable to evaluate them on tasks involving arm movements. We observe a wide range of differences success rates and average completion times demonstrating the inherent variations across users.
Table Success rates the user study conducted across individuals. Each user roughly performs 3 tasks on average.
User
Method
Success Rate (in trials)
Flip Cube
Pinch Grasp
Pour
Pick and Place Open Box of Mints
User
Holo-Dex
0
AnyTellop
0.8
0.2
Open Teach
0.8
0.2
User 2
Holo-Dex
0.2
AnyTeleop
0.2
Open Teach
0.8
0.8
0.8
User 3
Holo-Dex
0
AnyTeleop
0.2
Open Teach
0.8
0.2
User 4
Holo-Dex
0
AnyTellop
0.2
Open Teach
0.8
0.6
0.4
User 5
Holo-Dex
0
AnyTeleop
0.6
Open Teach
0.2
0.4
User 6
Holo-Dex
AnyTeleop
0.6
Open Teach
0.8
0.2
User 7
Holo-Dex
0
AnyTelcop Teleop
Open Teach
0.6
0.8
0.8
0.4
User 8
Holo-Dex
AnyTeleop
Open Teach
User 9
Holo-Dex
0
AnyTTelop
0.4
Open Teach
0.8
0.6
User 10
Holo-Dex
0
AnyTeleop
0.2
Open Teach
0.6
0.4
User 11
Holo-Dex
Any Aeleop
Open Teach
0.8
0.4
User 12
Holo-Dex
AnyTeleop
Open Teach
User 13
Holo-Dex
Acleop
Open Teach
0.6
User 14
Holo-Dex
0
AnyTeleop
0.4
Open Teach
0.6
0.8
User 15
Holo-Dex
AnyTellopp
Open Teach
0.4
16

---

Table 6: Average completion times for successful trials for the study conducted individuals. Each user roughly performs 3 tasks on average. NS denotes cases where no successes were achieved.
User
Method
Average completion time for successful demonstrations (in s)
Flip Cube
Pinch Grasp
Pour
Pick and Place Open Box of Mints
User
Holo-Dex
4.6
NS
AnyTeleop
20.2
22.5
Open Teach
5.4
18.6
66
User 2
Holo-Dex
17.5
AnyTeleop
18.9
Open Teach
20.6
29.7
12.2
User 3
Holo-Dex
5.4
NS
AnyTeelop
18.3
7.8
Open Teach
5.1
12.6
11.3
User 4
Holo-Dex
11
NS
AnyTelcop
13.2
31.4
Open Teach
6.2
7.5
16.9
48.4
User 5
Holo-Dex
NS
AnyTellop
11.4
Open Teach
10.9
41.6
12.4
User 6
Holo-Dex
NS
aeTecp
12.7
Open Teach
10.5
23.57
User 7
Holo-Dex
NS
AnyTTelopp
NS
Open Teach
19.1
21.49
49
37.8
User 8
Holo-Dex
6.5
AnyTeleop
5.4
Open Teach
4.7
User 9
Holo-Dex
NS
AnyTeleop
49.9
Open Teach
65.3
NS
32.21
User 10
Holo-Dex
NS
AnyTTleop
48
Open Teach
30.8
40.3
48.7
21.3
User 11
Holo-Dex
6.7
AnyTTelepp
11.5
Open Teach
5.6
21.8
15.7
User 12
Holo-Dex
6.2
AnyTeleop
11
Open Teach
3.8
User 13
Holo-Dex
8.9
AnyTell
14.2
Open Teach
5.8
18.1
User 14
Holo-Dex
NS
AnyTeleop
49.9
Open Teach
65.3
132.5
User 15
Holo-Dex
13.2
AnTTeloop
14.6
Open Teach
6.3
53.1
17

---

Slice Cucumber: Subillze the cucumber with onz araa and use knile for cuting siice with the other amm
Make Sandwich:
by sequattially adling
Insert USIK: Insert USB chorging cablee into adapter.
Bimanuul
Toast Bread: Mace
bread in
Sweap Dirt OIT the Table: Sweap dist off the uble wsing a brush dusipan.
Unfoll Cloth: Unfold cloth and place the table.
Uncap Marker: E-ft marker with and with other.
Figure 5: Real world task rollouts demonstrating the ability of OPEN TEACH to perform intricate, long morghooi tasks.
18

---

Rimantal
Handover cam: Pick the with une am and hand over the other
Sprinkles Pour
Pack Lift llnsket: Moor
honey bottle and soda can inside basket, the handles and pick up basket
Pot Muffin the Oven: Place mufiin inside the and closs the duce.
Franka- Allegro
Undo latch and open box: Undo latch and open the box.
battery battory bolder.
Shell Game: Pla the shell game with cups and ball.
Figure 6: Real world task rollouts demonstrating the ability of OPEN TEACH to perform intricate, long morghooiizon tasks.
19

---

Open box: Lift the lid of the box.
Frunka- Allegro
Grasp dhjjcct Group
lural upwards.
Graep Sponge: Grapp sponge placed on table.
Piek Up Tea Sachei:: Slice the sachet and pick
Stack Blocks: Stack blocks of ench other.
Kinowa ie- Allegro
Open Ketehop Buttle: Open
Open Drawer: Slide the drawer and open
Figure 7: Real world task rollouts demonstrating the ability of OPEN TEACH to perform intricate, long horizon tasks.
20

---

Iron Towel: Move the hand to the iron and iron the towel.
Kinova-A
fitt
Laplop Opening: Open ofa leptop. The pasethrough upp heing sincemed the laprop's sereen.
placed
Open Box: open
Write Alphabet: Write the alphaicur "A" on poper placed on the table.
Allegro Sim
Flip Sponge: Flip sponge
Figure Real world rollouts demonstrating the ability of OPEN TEACH to perform intricate, mor-horizon tasks.
21

---

Allegro Slm
Pinch Grasp: Grasp the block with pinch and lift
Vlip cube ishand
Place white left plate: Pick the white and oion the white
LIBERO Sim
Pat pan and turn Pick
the Then the
Organize banket: Pirk up everything
Close tup drawer of cabinet: Approsch the top drower of the cabinet and pur close
Figure 9: Real world task rollouts demonstrating the ability of OPEN TEACH to perform intricate, monghooizoon tasks.
22

---

Open open
Streich
Open Doer: Move close to the cabinet door and open
plastic bag garbage plastic lag
parbage
Figure 10: Real world task rollouts demonstrating the ability of OPEN TEACH to perform intricate, tasks.
Table Performance policies learned on collected through OPEN TEACH. FISH and BC were used train policies for Allegro Sim and Libero Sim respectively, We report the mean and standard deviation for 25 evaluation trials across 3 seeds for each task.
Number of
Success Rate
Robot Setup
Task
Demos
(25 trials)
Allegro Sim
Flip Cube
6
0.97 + 0.03
Flip Sponge
6
0.79 + 0.05
Pinch Grasp
6
0.75 + 0.07
Libero Sim
Close Top Drawer of Cabinet
10
0.96 0.03
Turn on Stove
10
0.95 + 0.04
Pick and Place Soup
50
0.77 + 0.02
into Basket
Flip Cube
Pinch Grasp
0.8
Success Rate
Sucosesa Rale
0.8
0.4
0.4
02
0.0
0.0
AnyTeeop
Any Teleop
Figure 11: Success rates for the user study conducted across 15 individuals on 2 tasks Flip Cube and Pinch Grasp. We report the mean and standard deviation for 3 methods Helo-Dex, AnyTeleop. and Open Teach (Ours).
23

---

Flip Cube
Pinch Grasp
17.5
Timne for completion (in 6)
for completion 5)
20
10.0
20
2,5
Holo
AnyTeeopp
AnyThloop
OpenTeach
Figure 12: Average completion times for successful trials for user study conducted across individuals for 2 tasks - Flip Cube and Pinch Grasp. We report the mean and standard deviation for methods Holo-Dex, AnyTeleos Teleop, and Open Teach (Ours).
1.0
40
35
0.8
30
Success Rate
completoon
15
10
0.2
Open
Open Box
(a) Success Rate
(b) Time for completion (in seconds)
Figure 13: We compare the (a) success rate, and (b) average completion time (in seconds) for using OPEN TEACH between expert and 15 individuals participating in study. We report this comparison for Pour, Pick Place, and Open Box Mints.


Reviews:
[{"review_id": "W7m34PjZNw", "reviewer": "Reviewer_m2vB", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents the development of a teleoperation system that leverages the hand/fingers tracking available on the Quest 3 VR headset. The teleop system is designed to work with different robots and end effectors, single arm or bimanual tasks, and the library is open source.", "summary_of_recommendation": "The paper present a system for teleoperation that is adaptable to several robots and open source. As such, it would be useful to the community to have a report on the implementation of the system. However, from the research point of view for inclusion on the main program, it’s perhaps still a straightforward teleoperation system without significant new lessons. The paper is clear and easy to follow.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Questions above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7zWz9Yozl8", "reviewer": "Reviewer_rYe8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduce an affordable, open-source teleoperation system for robotic manipulation using a VR headset. It supports multiple robot types, offers high-frequency visual feedback, and works in both simulated and real environments. The system demonstrates versatility across various tasks, shows improved performance over existing frameworks, and produces data suitable for policy learning. OPEN TEACH aims to provide a user-friendly, comprehensive solution for robot teleoperation and data collection, addressing limitations of current systems and promoting broader adoption in robotics research.", "summary_of_recommendation": "The authors present a great system for robot teleoperation, the system contribution is enough to be accept. To further improve this paper, the author can consider explore the teleoperation system for downstream applications, e.g. how can this system impact imitation learning.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. The authors demonstrate OPEN TEACH's capability in data collection for robot learning. However, it would be valuable to discuss the unique characteristics of data collected via OPEN TEACH compared to data from exoskeleton-based systems like ALOHA. What are the potential advantages or limitations of OPEN TEACH-collected data in terms of quality, diversity, or other relevant factors?\n\n2. While the paper shows that policies can be trained using OPEN TEACH data, it would be beneficial to provide more insight into the data efficiency of this approach. Specifically, how much data collected through OPEN TEACH is typically required to train an imitation learning policy that exhibits moderate generalization capability? A quantitative analysis or comparison with other data collection methods would strengthen this aspect of the work.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cw1Lf6sZdX", "reviewer": "Reviewer_BA4K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a system called open teach, which is a vision-based teleoperation system for robotic manipulation. The paper claims that when compared to existing systems, within a single framework, open teach supports hands and manipulators, calibration-free and works across simulation and real world settings. Several manipulation tasks are demonstrated with the system. The user study confirms certain advancements over AnyTeleop, and the entire set up costs less than 500 dollars.", "summary_of_recommendation": "For me, the weakness outweighed the pros, thus recommending for a weak rejection. The paper is aiming to support the community through an open-source infrastructure. Thus, the open-source software and hardware should have been included for a review. Moreover, the paper should outline what is the main system level innovation from first principles, rather than focusing only on end-results. If these two points are addressed during the rebuttal, I am willing to change my recommendation since strenghts of the paper are also clear and have potential to serve the community.", "technical_quality": 2, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "If I think of vision-based telepresence systems, I always think of buggy hand gesture recognition. I agree that it is cheaper to build such systems, but my default choice would be something like ALOHA or more traditional DLR teleoperation systems like HUG or many of systems shown on recent AVATAR challenge. They are expensive, but more useful - with hardware, one can have all the advantages of open teach in supporting hands, manipulators, mobile manipulation and of course, real and simulation environments, and we do not have to suffer from vision-based softwares not being robust. \n\nHow accurate is the current hand gesture recognition in terms of usability? It was not satisfying that the paper mentions only at the end about such limitations. I think the tribute has to be paid to more traditional teleoperation systems throughout the paper other than ALOHA or recent learning-related papers, and comparison should be discussed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers found some strengths in this submission, but also clearly articulated some questions for the rebuttal phase.
Thank you for your detailed responses.
Decision: Accept