# WARM: Wearable AR and Tablet-Based Assistant Systems for Bus Maintenance
## 3 Solution Approach
### AR-Based Solution  
To develop the AR solution, an application based on AR with smart headset (such as Microsoft HoloLens, Figure 8) is proposed. These types of AR headset are a new type of hardware that integrates a video camera, a screen and an audio system. The appearance on the market of this type of headset provides a better way of integrating these technologies for non-experts. Figure 6: Maintenance report (1 st page). Handwritten report (**a**). Digital report automatically produced and transmitted to the back office by the WARM system (**b**). Figure 7: Maintenance report (2 nd page). Handwritten report (**a**). Digital report automatically produced and transmitted to the back office by the WARM system (**b**). AR specific algorithms have been implemented whose objective is to solve the problem of detection and tracking of objects through artificial vision. Due to the fact that objects in industrial environments are characterized by the absence of visual characteristics that can be used in the detection (the usual surfaces are metallic, without patterns or other visual characteristics), two different tracking methods were used to detect the bus and to locate the user respected to the work environment. The geometry and related coordinates that describe our problem are illustrated in Figure 9.  
A brief description of the coordinate systems will be given below:  
- W corresponds to the world coordinates system;  
- C corresponds to the camera coordinates system  
- B corresponds to the bus coordinates system with the origin in the license plate;  
- O corresponds to the object coordinates system, i.e., the virtual plane with checklist showed to the user through the headset. The tracking problem consists in finding a rigid transform function \(T_{A}^{B}\) which maps points from coordinates system A to coordinates system B. This transform is composed of a rotation and an offset. Rigid transforms are invertible by definition. Therefore, if a transform that goes from system A to B is known, the inverse transform that goes from B to A can be obtained directly by inverting the original matrix. In addition, if three coordinate systems are present (A, B and C) and the transforms from A to B and from B to C are known, it is possible to obtain the transform that goes from A to C by left-multiplying the transform matrices:  
\[T_{A}^{C}=T_{B}^{C}T_{A}^{B} \tag{1}\]  
The two tracking systems implemented in our approach are the following:  
1. Detection of the environment and positioning of the operator (spatial mapping): using the 4 infra-red cameras and inertial sensors of HoloLens device, the detection of the  
Figure 8: Microsoft AR HoloLens. Figure 9: All the proposed system coordinates and transforms. most relevant geometric characteristics of the objects of the environment is performed. Through the information obtained with this algorithm, the transformation matrix \(T_{W}^{C}\) of the device's camera is estimated and thus, it is able to track the operator in the environment. The MixedRealityToolkit-Unity toolkit (MRTK) has been used as an AR support for this spatial mapping and user interaction. It is a collection of scripts and components designed to accelerate the development of applications aimed at the HoloLens device and Windows Mixed Reality. 2. Vehicle detection: the detection of objects must be much more accurate and rigorous than the environment since it is the core of the activity and this depends on the exact positioning of all virtual objects for the operator's guidance. Due to the large size and poor homogeneity of bus surfaces, it is difficult to position them accurately through a direct detection of the vehicle or its textures. Therefore, it has been decided to develop an algorithm that analyze in real time the images captured by the HoloLens in search of certain fixed elements in all vehicles (license plates). Once the fixed element has been detected, its position can be computed respected to the camera (\(T_{C}^{B}\)). For the image processing, Unity 3D native Vufforia libraries have been used.