# Extended Reality Application Framework for a Digital-Twin-Based Smart Crane
## 1 Introduction
### Industrial XR Applications  
Currently, the manufacturing system is becoming more agile and flexible, more able to achieve small-scale production in an economically sustainable way [28]. Meanwhile, the integration of human operators into production, or HMI, is considered a key factor facing the increasing complexity of the manufacturing context [29]. Smart factory HMI functions as a wrapper for visualization, aggregation, and analysis to assure humans realize their full potential and adopt the role of strategic decision-maker and flexible problem-solver [30]. With modern production strategies, where systems and processes are self-optimized, operators are expected to perceive and engage with aggregate, prepared, and nicely presented HMI to monitor and intervene within processes (usually through physical interfaces).  
Mobile devices have been widely adopted in manufacturing and production processes, including overhead displays, tablets, and smartphones, with multi-modal (gaze, voice, gesture, tactile, and haptic) or multi-touch interaction capabilities. XR is acknowledged as having a vital role in enabling different HMI applications [4]. Representing AR, VR, and MR in the reality-virtuality continuum [30], XR technologies refer to the combined environment of all real and virtual assets where the human interacts with a machine, and are therefore expected to bring transformation to manufacturing. The development of XR technologies has brought many research directions, including industrial model preparation, information integration, data visualization, and XR application development. [PERSON] et al. [31] provided a structure for the geometry preparation automation process, which simplified the integration of XR applications into product development. [PERSON] [32] proposed a digital transformation of engineering processes using VR technology. [PERSON] et al. [33] presented a smart sensor dashboard of smart manufacturing, which can be used to design VR environments with interactive graphs. In addition, the implementation of XR technology can provide a convenient and natural human-machine interface. [PERSON] et al. [34] presented a method of programming robots using virtual reality and digital twins. [PERSON] et al. [35] proposed a mobile augmented reality remote monitoring system to help operators with low knowledge.  
When it comes to the crane, the industry has witnessed diverse XR applications for it. A conceptual speech-based HMI equipped with AR and interactive systems was proposed in [36] for controlling mobile cranes. An approach integrating 4D building information modeling and AR proposed in [37], provided users with real-time navigation for tower crane lifting operation with enhanced safety and efficiency. An assistance system for mobile cranes leveraging AR was developed in [38] to provide safety-related information for operators. An interactive virtual reality system integrated with a database was developed to evaluate mobile crane lift operations in [39]. However, these cases were developed according to the specific requirements, lacking a general framework for the development of XR applications. On the other hand, the data flow of those apps is limited to one direction, which is hardly used for a DT-based machine.