# A Mixed Reality Interface for a Digital Twin Based Crane
## 2 Materials and Methods
### Mixed Reality Application for the Overhead Crane
#### 2.3.1 Interaction Module  
The interaction module enabled crane control through interacting with virtual balls in the scene. There were two ways available for interaction, namely the movable target control and the fixed target control. The main difference between the two was whether the assigned crane target position was hooked to space or not.  
Workflow of Fixed Target Control  
In the fixed target control mode, there were five fixed targets, which were displayed as five yellow 3D sphere-shaped holograms of 20-cm-diameter size, with the text \"Target 1/2/3/4/5\" on top of them, spreading around at the lab space. Users could select any of them by using the head gaze combined with an air-tap gesture. Once a target was selected, its color turned from yellow to red as an indicator of the successful commit. If meanwhile the control condition was fulfilled, the crane should move to the selected target position.  
Workflow of Movable Target Control  
The movable target control mode worked similarly, with the only difference that, instead of choosing from a range of the fixed targets, the user could drag and drop the movable target to freely define the target location. The movable target was presented as a 20-cm-diameter green 3D sphere, with the text \"Movable Target\" on top of it. Once the target was firstly selected with a gaze and \"tap and hold\" gesture inputs, the sphere color turned to red and the spatial mapping mesh showed up in the scene for enhancing users' spatial awareness, then the user could navigate the sphere in the space while keeping the hold gesture. Once the user committed again with an air-tap gesture, the target was dropped and placed at a new location with its color turning blue and the spatial mapping mesh disappearing. If meanwhile the control condition was fulfilled, the crane would then start moving towards the target position. Figure 4 shows the HoloLens view during movable target control.  
Development of Interaction Module  
Both interaction scenes were developed with the MRTK and its input system in Unity, which support the gaze as well as the \"tap and hold\" gesture inputs. When the MR application ran, one script constantly computed the value differences between the crane current position and the selected target position of each subsystem, namely the hoist, trolley, and bridge. The crane trolley and bridge first moved forward/backward until the differences were under a pre-defined threshold. Considering the operational safety,  
Figure 4: HoloLens view while the user was navigating the movable target: the spatial mapping mesh of the AIIC space captured by the HoloLens camera was also displayed in the scene, to enhance users’ spatial awareness and reduce the risk of placing the movable target in inaccessible locations.  
only once the trolley and bridge completed their movements did the hoist start to move up/down until the crane finally arrived at the target location in all three dimensions.  
Regarding the parameters needed for the calculation above, the crane current position directly fetched through communication with the GraphQL server, while the selected target position was determined by the transformation of the position from Unity scene coordinates to the crane coordinates. The parameters in the transformation required a pre-calibration process as follows. We first manually controlled the crane to align its hook to each of the fixed targets, then read the crane positions from the crane GraphQL server (eventually from crane OPC UA) as well as the hologram position of the fixed targets from the Unity scene. Based on these, we calculated the linear relation between the positions under the crane system in the physical world and the ones in the Unity scene. The final scale and offset of this relation were determined by taking the average of the calculations over the five target sets. The calculated linear relation was then used to transform the target positions in Unity to determine the actual target positions that should be written into the crane system.