# Visuo-Haptic Mixed Reality Simulation Using Unbound Handheld Tools
## 2 VHMR Simulation Setup  
An overview of the VHMR simulation setup is shown in Figure 2. The tool is tracked by a fixed optical tracker, fusionTrack 250, which has a root mean square (RMS) accuracy of 0.09 mm. The user wears a see-through head-mounted display (HMD), Magic Leap 1, and a Leap Motion Controller. The latter is for hand tracking to fix occlusions when the user's hand is between the user and the virtual objects. All three components operate up to 120 Hz.  
Figure 1: The proposed visuo-haptic mixed reality (VHMR) concept. ETHD, encountered-type haptic display; HMD, head-mounted display.  
Figure 2: Overview of the VHMR setup. PC-H, hard real-time industrial computer; PC-S, server computer.  
The ETHD consists of three components. A hexapod positioner, PI H-840.D2 (Physik Instrumente GmbH & Co. KG, Karlsruhe, BW, Germany), which has 10 kg payload, 50 mm/s maximum velocity, and 0.5 \(\upmu\)m repeatability. An ATI Gamma SI-130-10 force/torque transducer (ATI Industrial Automation, Apex, NC, USA) with a force capacity of 130 N and a resolution of 0.025 N. And a Schunk EGP 50 gripper (SCHUNK GmbH & Co. KG, Lauffen/Neckar, BW, Germany), which is currently used in validation experiments, but it is vital for our future work as discussed in Section 5. The haptic-tip is at the end of the gripper, and its reachable workspace is 160 \(\times\) 140 \(\times\) 50 mm. A hard real-time industrial computer (PC-H in Figure 2), Beckhoff C6920 (Beckhoff Automation GmbH & Co. KG, Verl, NRW, Germany), controls the ETHD over EtherCAT protocol. Control software was developed in MATLAB/Simulink (R2020a, MathWorks, Natick, MA, USA, 2020) and compiled to run with TwinCAT (Version 3.1, Beckhoff Automation GmbH & Co. KG, Verl, NRW, Germany, 2013) at 1 kHz cycle frequency.  
The range of stiffness that can be rendered by the ETHD is determined via a set of unilateral virtual wall experiments, where the wall is modeled as a linear compression spring (Figure 3a). The haptic-tip is positioned at the virtual wall boundary and constrained to move only in the vertical direction. Then, it is pushed inside the wall by an external force. The apparent stiffness is calculated from the measured force and displacement at steady-state. Figure 3b shows a plot of the percent error between the actual and commanded stiffnesses for the range from 0.03 N/mm to 200 N/mm. The error bars indicate the maximum and minimum values of three repeated experiments. Stiffnesses below 0.03 N/mm could not be rendered due to unstable behavior of the haptic-tip. High errors and low repeatability were observed up to 0.3 N/mm. After 0.3 N/mm, the error is within \(\pm\)2%.  
Server computer (PC-S in Figure 2) is a Windows PC for handling data transfer between components with a server application developed in Unity. It has a USB connection with the hand tracker and a 100 Hz user datagram protocol (UDP) connection with the tool tracker, PC-H, and HMD. PC-S also controls the sidescreens with individual cameras on either side of the setup to monitor simulations for development purposes, and also for an audience without HMDs.  
Coordinate frames associated with the system components are shown in Figure 4a. All components are referenced to a world frame {W}. It is chosen such that it coincides with the haptic-tip frame {H} when the hexapod is in zero-configuration. The transformation matrices \({}^{WT}_{O}T\), \({}^{WT}_{S}\), \({}^{WT}_{R}T\), and \({}^{WT}_{C}T\) are constant since the respective components are fixed. The HMD continuously references {G} relative to {S}. The hand tracker is fixed to the HMD, hence \({}^{GT}_{L}T\) is known. \({}^{RT}_{H}T\) is calculated based on joint variables of the hexapod, and \({}^{HT}_{L}T\) is constant.  
Figure 3: Stiffness fidelity: virtual wall model (**a**), percent error between the actual and commanded stiffnesses (**b**).