## 3 Architecture
### Augmented Reality Application  
The augmented reality app was developed for HoloLens using Unity3D ( [[https://unity.com/](https://unity.com/)]([https://unity.com/](https://unity.com/)), accessed on 20 December 2021) and the Mixed Reality Toolkit ( [[https://github.com/microsoft/MixedRealityToolkit-Unity](https://github.com/microsoft/MixedRealityToolkit-Unity)]([https://github.com/microsoft/MixedRealityToolkit-Unity](https://github.com/microsoft/MixedRealityToolkit-Unity)), accessed on 20 December 2021). The main functional modules of the application are as follows:  
* **Drone control commands**: This module is responsible for capturing the gesture/voice interactions of the user, translating them to specific control commands (i.e., pitch, roll, yaw, and throttle values), and sending them to the the Android app via the Kafka broker. * **Drone tracking**: The drone tracking module is responsible for monitoring the spatial relation between the real world elements (the drone and the app user) and its virtual representatives. Drone tracking is the backbone for many feedback cues (visual, textual, contextualized videos) that provide the user with enhanced spatial perception. Tracking is possible following an initial calibration process and an IMU-based approach. * **Video feedback**: The live video feed from the drone's camera is projected on a canvas inside the augmented environment. The user can simultaneously be aware of his/her surroundings in the real environment while also viewing the drone's video feed in his/her field of view (FoV). With minimal head movement and loss of focus, the user can perceive the surrounding environment from two different viewpoints. This video projection is modifiable and can either be placed in the center of the user's FoV as shown in Figure 2 (FPV mode) or on a smaller panel projected from the drone's virtual counterpart (Figure 3). The center placement is useful in tasks where higher detail of the drone's camera feed is required. This contextualized video feed mode is called \"egocentric\" view with the drone at the center and allows for easier perception of what the drone \"sees\" during the flight and of the spatial correspondence between the drone's and the user's point of view. In contrast, the FPV mode is useful in tasks where higher detail of the drone's camera feed is required. In addition, when using a drone equipped with an infrared camera, the user can choose between three different modalities, visible (RGB), infrared (IR), or fused (MSX). * AR feedback and visualization: GUI elements and various visualization features were added in the AR application to provide feedback and various controls. Emphasis was given to creating an easy-to-use application and increasing the SA of the user. In more detail:
* Visualization of the virtual hand joints overlaying on top of the user hands, making it possible for the user to directly check if his/her hands are correctly perceived by HoloLens (Figure 4). * Drone visualization based on the drone tracking module. A virtual drone is overlaid on top of the real drone so that the user is aware of the drone's relative
position even if it is not in a direct line of sight (e.g., behind a wall or building) or too far to be easily visible with a naked eye. * An extensive virtual menu that includes all high level commands and toggles for enabling the application features is present in the lower part of the application view. The menu can be be hidden in order to not obstruct the user's field of view. * A navigation feedback panel is placed at the top part of the application's window. In the feedback panel, the current perceived navigation commands are listed alongside with other drone- and user-related information such as the drones's flying altitude and horizontal distance from the user, the drone heading, and the user heading. Figure 3: The AR Egocentric View mode. The AR user can be shown in collinearity between the user and the drone’s camera in the right corner. Figure 2: First Person View (FPV) into the Augmented Reality environment.