Quest2ROS: An App to Facilitate Teleoperating Robots

Authors: nils  ingelhag
michael c welle
martina  lippi
andrea  gasparri
danica  kragic
maciej  wozniak
Venue: humanrobotinteraction_org_HRI_2024_Workshop_VAM-HRI
Published: 2024-02-13 17:08:54

Abstract: Teleoperation is an integral part of robotics research. In this work, we present Quest2ROS, a stand-alone app available  %Metas AppLab 
for Oculus Quest 2 and 3 that facilitates the teleoperation of robots via ROS. Quest2ROS publishes the position and velocity of both hand-held controllers, the button pressed as well as enables haptic feedback via the controller vibration.
The Quest headset does not have to be worn at teleoperation time to not restrict the operator, furthermore, a simple way to align the coordinate frame of the controller with any given robot in the real world is provided.
We measure the tracking accuracy of a Quest 2 to be 0.46 mm on average, with mean latency between the Quest 2 and a ROS node being 82 ms and update frequency of relevant ROS topics being 71.96 Hz.

Main paper: 
 quest2ros: 
 Quest2ROS is a stand-alone app for Oculus 2 and 3 that lets the user easily retrieve the position, velocity, button inputs as well as controlling the haptic feedback of both hand controllers via ROS. It is based on the teleoperation framework presented in [19] . The Oculus app streams telemetry data via Wi-Fi to a TCP-endpoint running on a host specified by the user. The streamed controller data is then made available via ROS topics to other ROS nodes. An overview of the system is shown Figure 2 . To make the app more usable for controlling physical robots and actuators, the app is provided with built-in functionality to let the user set the reference frame in which the VR controller's pose and twist information are given.The app is implemented using the Unity game engine together with the Oculus Integration3 and ROS-TCP-endpoint 4 packages. The position and velocity of the controllers are continuously tracked by the headset using a combination of internal sensors and visual tracking. It is therefore important that there exists no visual occlusion between the headset and the controllers at all times during operation. The user interface is a configuration window accessible within the VR world of the app. It lets the user set the IP and port of the TCP to ROS gateway running on a host. This configuration has to match the configuration to the TCP-to-ROS endpoint. The configuration is by default locked to prevent accidental reconfiguration but can be unlocked by pressing A and B buttons, and both triggers on the right VR controller. The IP and port can then be set using a virtual keyboard. After both the IP and port are set, the user has to press Apply & Lock to set the configuration as well as to lock the screen again. An image of the user interface is shown in Figure 3 .The coordinate frame alignment lets the user set a new reference frame based on the position of one of the controllers. All pose and twist information is thereafter given in reference to the newly set coordinate system. The purpose of the coordinate frame alignment functionality is to make it easy for users to use the controller telemetry for robotic teleportation without complex coordinate transformations. Since the initial coordinate system is arbitrarily set to the momentary position of the VR headset at startup, coordinate frame alignment lets the user align the coordinate system to a physical stationary system within the room, e.g., with the robot coordinate system. The process of coordinate system alignment is shown in Figure 4 . Either controller is aligned to the desired reference frame and the buttons A and B for the right controller or X and Y for the left controller are pressed for 4 seconds to set the new coordinate system. All telemetry data from the controllers are now given in the newly set coordinate system (step 2).The position, rotation and velocities in and around each axes of the controllers are made available via ROS-topics. The coordinate axes of the controllers follow the right-hand rule, with x pointing forward, y to the left, and z pointing up , as shown in Figure 5 . The VR controller telemetry is given as separate pose and twist messages. The pose consists of two vectorsì 𝑝 =       𝑥 𝑦 𝑧       and ì 𝑞 =         𝑞 𝑤 𝑞 𝑥 𝑞 𝑦 𝑞 𝑧        , corresponding to the position and orientation, represented in quaternions, of the controller, respectively. The twist is given by the vectorsì 𝑣 =       𝑣 𝑥 𝑣 𝑦 𝑣 𝑧       and ì 𝜔 =       𝜔 𝑥 𝜔 𝑦 𝜔 𝑧      , which represent the linear and angular velocity in and around each axis, respectively. Via ROS publishers, the user is able to access realtime pose, twist, and controller inputs, i.e., buttons' state, of left and right controllers. A subscriber is available for each VR controller to facilitate haptic feedback to the user by specifying the vibration of the controller. The full list of topics is reported in Table 1 .To use Quest2ROS, the user needs to:(1) Install and start the TCP-to-ROS endpoint on a host.(2) Connect the VR headset to a Wi-Fi network from which the host is available. (3) Install and start the Quest2ROS app on an Occulus Quest 2 or 3. (4) Specify the TCP-to-ROS endpoint's IP address and port within the app and connect. (5) Remove the headset from the head and place it where it has direct sight of the controllers by its cameras. (6) (optional) Create a new reference frame within the room by aligning the right controller and pressing buttons A and B simultaneously for 4 seconds. More details on the setup and use of Quest2ROS system can be found on the project website.  
performance evaluation: 
 To evaluate the performance of the Quest2ROS system, the accuracy, latency, and update frequency of the controller telemetry were measured. In this section, the methods and results of these experiments are presented. Accuracy: The accuracy of the streamed controller data was evaluated by recording the measured positions while physically constraining a controller to follow a fixed circular path. By calculating how much the positional data deviates from the circle, we can assess how much the data fluctuates over time and space. The setup used to fix the controller to follow a circular path is shown in Figure 7 . The controller was manually rotated four turns while the headset was placed at a distance of approximately 1 meter while facing the controller unobstructed resulting in 2180 datapoints. Since the position of the circle in the VR reference system is not known beforehand, the position of the circle was optimized to minimize the mean distance from each recorded data point.The mean distance from the recorded data points to the optimized circle was 0.46 mm. A histogram showing the frequency of the recorded data points within different distance ranges is shown in Figure 8 . Latency: The latency of the streamed telemetry data was measured by recording the time from when a controller was physically moved till that the movement was registered in a subscriber ROS node. The time was measured using a high-speed camera with both the controller and the output of the ROS subscriber within its view. The high-speed camera used recorded 940 frames per second. The time was measured seven times using this method. The latency Frequency: The update frequency of the telemetry data was measured with the built-in ROS command 'rostopic hz' on all publishers to the ROS node. The average measured update frequency was 71.86 Hz. The individual measurements for each topic are reported in table 1 . Example usage: We showcase the usability of the App by connecting it to a Panda Franka Robot which has a bottle opener on the end effector and is teleoperated to open a bottle as shown in Fig 6 . We relay the twist message received on the topic q2r_right_hand_twist to a velocity cartesian controller which translates the received velocity to robot action as long as the left trigger button is held which is received via the q2r_right_hand_inputs topic. A full 30 min video of an operator performing the task is shown on the project website. 
conclusion: 
 Quest2ROS lowers the entry barrier for researchers and engineers to perform intuitive control of high degree of freedom robotic systems. The Meta Quest 2 and 3 are readily available consumer-grade electronics, and ROS is a common framework of choice within the research field of robotics. Joining these technologies in a seamless solution results in a tool that has the potential to be highly useful within the community. Videos showing the framework being used on different robotic platforms are available on the project website. 
: 
 movement of the robot in the Cartesian space. Other works address this by following a master-slave or puppeteer setup where the robotic system in question consists of a master robot that can be manually moved and the movements are mirrored to the slave robot or puppet. While this approach excels in achieving precise teleoperation and has proven its effectiveness in handling delicate tasks such as lid opening, juggling and cable tie zipping in [6] , its implementation is generally costly. Other teleoperation systems are highly specific and only applicable for the robot they were developed for such as [7] [8] [9] .In recent years, numerous Augmented Reality (AR) or Virtual Reality (VR) teleoperation projects for human-robot interaction have emerged, due to advancements in communication technologies and Virtual and Augmented Reality (VAM) hardware [10] . These projects have been developed for a variety of tasks, including cloth folding [11] , pick-and-place operations [12] , and diagnostics and debugging [13] . Additionally, VAM technologies have been used for demonstration learning [14] or enhancing action explainability and human-robot understanding [15, 16] . Additionally, many studies such as [17] and [18] have made significant efforts to enhance the visualization of robot sensory information within AR or VR environments. While these projects serve as impressive proof of concepts, their developers often had to begin from scratch due to the absence of required software and specific tools. Moreover, these projects tend to be specific to particular VAM and robot hardware.In this work, we present Quest2ROS, an app that facilitates intuitive robot-agnostic teleoperation on consumer-grade hardware using ROS middleware. In detail, the app is available for the Oculus Quest 2 and 3 and allows users to use the VR controller to teleoperate the robot, as shown in Fig. 1 . The VR headset needs to be placed with an unobstructed view of the VR controller for optimal tracking performances. The app is available in Metas Applab environment via an alpha version invitation link 1 . Further instructions, demonstration/evaluation videos as well as links to the ROS code repository can be found on the project's website2 . 
 
 Appendix: 7th Int. Work. on VAM for HRI at HRI 2024, 03/2024, Boulder, CO, USA Nils Ingelhag*, Michael C. Welle*, Martina Lippi, Maciej Wozniak, Andrea Gasparri, and Danica Kragichttps://www.meta.com/s/1stEAnW1uhttps://quest2ros.github.io/q2r-web/https://developer.oculus.com/downloads/package/unity-integration/https://github.com/Unity-Technologies/ROS-TCP-Connector 



Reviews: review: title: title: Accept
main_review: main_review: This paper presents Quest2ROS, a publicly available stand-alone app for the Quest 2 and Quest 3 virtual reality headsets that enables position tracking-based teleoperation of manipulator robots, through the movement of handheld controllers. The functionality and architecture of the Quest2ROS system is described, and an analysis of the system's tracking accuracy is conducted.

Strengths:
-  This paper introduces a publicly available instantiation of a component (live teleoperation) useful for a wide variety of VAM architectures, providing future researchers with the option to reduce their engineering efforts through the use of this off-the-shelf system.
- The position tracking evaluation is well-designed, and allows the accuracy of the system to be precisely measured under ideal conditions. These measurements are very useful for developers moving forward.

Weaknesses:
- The paper mentions that the headset does not need to be worn by the operator. This does seem helpful for certain task setups, where the operator is able to directly observe the robot performing its actions in real time from a close distance. However, the paper doesn't describe what (if anything) is visualized within the VR headset if it is worn while the teleoperation is underway. Does the system work effectively when the operator is not in the same room as the robot?
- The paper does not include any quantitative or qualitative evaluations about the human-factors side of the teleoperation (i.e., how easy is it to use), especially as compared with other teleoperation systems. This is good content to include in future work.
- Needing to place the headset so that it has direct line of sight to the controllers at all times seems like it might introduce some issues during actual use - this should also be explored in future human-subjects studies.

Minor notes:
- The paper could include some more details as to what the purpose of the reference frame is: does this match the current 6DOF pose of the controller to the current 6DOF pose of the end effector of the teleoperated robot? Of some other reference pose? What does resetting the reference frame do in terms of affecting the movement of the robot?
- A couple minor typos in Section 1: "pupped" should probably be "puppet," and the sentence "...the app is available for the Oculus Quest 2 and 3 and allows to use the VR controller to teleoperate the robot..." is worded strangely - "allows to use" should probably be "allows the use."

Summary:
This paper is clearly relevant to the VAM-HRI workshop and makes a easily recognizable contribution to the field at large. I recommend it for acceptance and presentation at this year's workshop.
title: title: Accept
main_review: main_review: The paper presents a stand-alone application compatible with Oculus Quest 2 and 3, designed to enhance the teleoperation of robots via the Robot Operating System (ROS). The app facilitates the publication of hand-held controller positions, velocities, button presses, and haptic feedback. It offers a simple method for aligning the coordinate frame of the controller with any robot in the real world, boasting an average tracking accuracy of 0.46 mm, a mean latency of 82 ms between the Quest 2 and a ROS node, and an update frequency of 71.96 Hz for relevant ROS topics.

## Strengths:
- **Innovative Integration:** Combines consumer-grade VR hardware with ROS to make robot teleoperation more intuitive and accessible.
- **High Precision and Responsiveness:** Demonstrates high tracking accuracy and reasonable latency, which are critical for delicate teleoperation tasks.
- **Flexibility and Robot Agnosticism:** The app's ability to easily align coordinate frames and its compatibility with different robots enhance its utility across various teleoperation scenarios.

## Weaknesses:
- **Limited Evaluation:** While the paper presents performance metrics, a broader evaluation including user studies could better assess usability and practical efficiency.
- **Hardware Dependency:** The reliance on specific VR hardware (Oculus Quest 2 and 3) might limit accessibility for users with different devices.

## Recommendations for Improvement:
- **Extend Hardware Support:** Develop versions for other VR/AR systems to broaden the application's accessibility and utility.
- **Comprehensive User Studies:** Conduct extensive user studies to evaluate the app's usability, effectiveness, and impact on teleoperation tasks across diverse user groups.
- **Enhance Documentation and Support:** Provide detailed documentation and tutorials to facilitate the adoption of Quest2ROS by researchers and practitioners in robotics.

In summary, I think this paper is a great fit for VAM-HRI, and I recommend acceptance.