## 4 Mixed Reality for Human-Robot Interaction
### Mixed Reality-Enhanced Telemanipulation
#### Mixed Reality-Based Vision Mapping and Merging  
Mixed reality (MR) scenes feature immersive integration of multiple 3D/2D visual display modes to the users. At present, various teleoperation platforms have focused on MR-enhanced robot control [93]. It is not yet known whether using stereoscopic vision and point cloud within an MR environment can enhance users' stereoscopic perception and task performance in separated telerobotic operations. In [115], an MR control room for dynamic vision and movement mapping between the operator and dual-arm robot is introduced. The multiple monocular sensor displays and motion mapping approach via MR outperforms telerobotic platforms with direct camera feeds. The control room has objects and controls floating in space, which allows the user to perform movements relative to 3D markers to command the remote-controlled robot. However, the authors did not determine if the immersive experience led to performance improvements. In [31], the authors presented an MR teleoperation interface for mobile manipulation tasks with visual inputs from a monoscopic and stereoscopic camera setup for remote mobile manipulation tasks in hazardous production environments. This system is equipped with two monocular cameras and a stereoscopic camera at the robot's working site. Users acquire multi-view 2D images and stereo vision with depth cues in a Unity-generated MR control room. However, the intuitive control of the robotic platform as a whole was not assessed or presented. In addition, no comparative tests with typical telemanipulation systems were performed to verify the performance and efficiency of the proposed system. To reduce the training time for teleoperation, a multi-view merging method via MR was designed (Figure 7) in [121], and the platform provides users with intuitive control of the robot's motion by using commercial VR controllers. [PERSON] et al. [122] explored the use of stereoscopic view in an immersive manner for mobile robot teleoperation and navigation, and the results showed the stereoscopic view and immersive perception via virtual reality head-mounted displays (VR-HMDs) provided the user with depth cues and improved user performance and system usability. However, the effect of stereoscopic perception on telerobotic manipulation and the depth cues provided by other 3D visualization resources such as point cloud was not studied. The only exception is [118]; this work evaluates the impact of depth perception and immersion provided by integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in an MR environment. In particular, the MR-enhanced systems maintain spatial awareness and perceptual salience of the remote scene in 3D, facilitating intuitive mixed-reality human-robot interaction (MR-HRI). This study compared two MR-integrated 3D/2D vision and motion mapping schemes against a typical 2D baseline visual display method through pick-and-place, assembly, and dexterous manufacturing tasks. Incorporating 2D/3D vision mapping into mixed reality is just one facet of the growing adoption of XR-enhanced intuitive HRI; digital twins also play a pivotal role in facilitating intuitive teleoperation within the MR scene [116]. These digital twins, mirroring the physical robots, provide users with real-time information about the remote robot and seamlessly integrate into the human-robot interface [123; 124]. Within the mixed reality environment, users can interact with these digital twins naturally, enabling precise and intuitive control over robotic systems while obtaining real-time status information about the remote robot from the digital twins [125; 126]. Digital twins enhance the user's sense of presence and control in teleoperation by providing visual representations, spatial awareness, real-time monitoring, predictive capabilities, and interactivity [127; 128]. These capabilities empower operators, enabling them to teleoperate robots effectively and with a heightened sense of immersion and control.