## 2 Materials and Methods
### The Generation of the Virtual Scene  
An application has been developed for teleoperating a robot arm. This application enables users with little or no prior experience to control the robot arm easily and reduces the technical proficiency required for tasks. When the user starts the application, a virtual scene is generated in HoloLens2, as shown in Figure 4a. Initially, the scene includes a desktop, a robot arm, and a button. The desktop and robot arm are the digital twins of the real table and robot arm. They are used to help the user confirm the position of the robot arm and the objects. The button is used to confirm the user's actions and prevent accidental operations. After HoloLens2 receives the category and position of the objects, the digital twins of the objects at the corresponding position are constructed in the virtual scene, and the reference coordinate system is the base of the virtual robot arm. The digital twins of objects can be moved freely by the user, as shown in Figure 4b. The digital twins have physical properties and can simulate the operations of real senses, such as movement, collision, and falling. Users can observe the virtual scene from any angle to confirm the relative position of objects. When users observe the virtual scene in HoloLens2, users can also simultaneously observe the surrounding real environment, and the real environment includes the workspace of the robot arm. This allows users to quickly identify any unexpected events during the operation of the robot arm and press the emergency stop button to prevent risks. Specifically, a class ObjMsg is created to save the relevant information about the objects, and the information includes the object's name \((\text{name})\), the three-dimensional coordinate of the real objects \((\text{posx},\text{posy},\text{posz})\), and the quaternion of the real objects \((\text{rotx},\text{roty},\text{rotz},\text{rotw})\). In the ROS receive script, the \(\text{Subscriber}()\) function is used to receive ROS messages. When Unity receives the objects' information from ROS, the \(\text{ObjectPoseNow}.\text{Add}()\) function is used to save the objects' information to the dictionary \(\text{ObjectPoseNow}\). The dictionary then uses the object's name as the key and the three-dimensional coordinate and quaternion as the value. Then, the \(\text{ObjAdd}()\) function is used to sequentially add the prefabricated 3D models of the objects to the virtual scene. The core Algorithm 1 is as follows:  
```
Input : the dictionary of object ObjectPoseNow;
1:while1 do
2: obj = \(\text{Subscriber}()\)
3: ObjectPoseNow.Add(obj.name, ObjectMessage )
4: ObjAdd(obj.name, obj.posx, obj.posz, \(-\)obj.posy )
5:endwhile
```  
**Algorithm 1****ROS Subscriber**  
In the virtual scene, the user can move the digital twins of the objects to any position without affecting the real objects, as shown in Figure 5. When all digital twins have been moved to the target position, the user can press the confirmation button. After the user presses the confirmation button, HoloLens2 uses the script to select the moved digital twins. The script scans all objects in the virtual scene and calculates the difference between the current position of each object and its recorded position. Then, the script determines whether the object has been moved based on a set threshold. If the difference is less than the threshold, the object is considered not to be moved; otherwise, the current coordinates of the object are sent to ROS. Specifically, a script is created to send data to ROS. The script iterates through all the object's information \(\text{obj}_{\text{now}}\) in the virtual scene. The script uses the current object's name \(\text{obj}_{\text{name}}\) as the input and uses the \(\text{ObjectPoseNow}()\) function to obtain the object's information \(\text{obj}_{\text{saved}}\) which is stored in the ROS receive script. Then, Equation (8) is used to determine whether the current virtual object has been moved. If any conditions are met, the current virtual object is considered to be moved.