# SafeSpect: Safety-First Augmented Reality Heads-up Display for Drone Inspections

Peisen Xu, Jérémie Garcia, Wei Tsang Ooi, Christophe Jouffrais

► To cite this version:

Peisen Xu, Jérémie Garcia, Wei Tsang Ooi, Christophe Jouffrais. SafeSpect: Safety-First Augmented Reality Heads-up Display for Drone Inspections. CHI 2025, Apr 2025, Yokohama, Japan. hal-04905310

HAL Id: hal-04905310

https://hal.science/hal-04905310v1

Submitted on 7 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# SafeSpect: Safety-First Augmented Reality Heads-up Display for
Drone Inspections

Peisen Xu

Department of Computer Science
National University of Singapore
Singapore, Singapore
IPAL
Singapore, Singapore
ps.xu@nus.edu.sg

Jérémie Garcia

ENAC, Université de Toulouse
Toulouse, France
jeremie.garcia@enac.fr

Wei Tsang Ooi

Computer Science
National University of Singapore
Singapore, Singapore
ooiwt@comp.nus.edu.sg

Christophe Jouffrais

IPAL
CNRS
Singapore, Singapore
IRIT
Univ of Toulouse 3
Toulouse, France
christophe.jouffrais@cnrs.fr

<figure><img src="image_2.png" /><figcaption>Figure 1: An adaptive AR drone control interface for building inspection task balances mission and safety information to reduce cognitive load. (A) shows a simplified interface for data collection. (B) helps track the drone's position and mission progress. (C) illustrates a safety view for attention switching between camera view and the line-of-sight. (D) displays the critical data such as heading, collision status and positional uncertainty near the drone for enhanced situational awareness.</figcaption></figure>

## Abstract

Current tablet-based interfaces for drone operations often impose a heavy cognitive load on pilots and reduce situational awareness by dividing attention between the video feed and the real world. To address these challenges, we designed a heads-up augmented reality (AR) interface that overlays in-situ information to support drone pilots in safety-critical tasks. Through participatory design

workshops with professional pilots, we identified key features and developed an adaptive AR interface that dynamically switches between task and safety views to prevent information overload. We evaluated our prototype by creating a realistic building inspection task and comparing three interfaces: a 2D tablet, a static AR, and our adaptive AR design. A user study with 15 participants showed that the AR interface improved access to safety information, while the adaptive AR interface reduced cognitive load and enhanced situational awareness without compromising task performance. We offer design insights for developing safety-first heads-up AR interfaces.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

**DOI** 10.1145/3706598.3714283

© 2025 Copyright held by the owner/authors.

ACM ISBN 979-8-4007-1394-1/25/04

https://doi.org/10.1145/3706598.3714283
# CCS Concepts

* Human-centered computing → Mixed / augmented reality; Participatory design.

## Keywords

Augmented Reality, Heads-up Display, Drones, UAV, Adaptive UI, Safety

## ACM Reference Format:

Peisen Xu, Jérémie Garcia, Wei Tsang Ooi, and Christophe Jouffrais. 2025. SafeSpect: Safety-First Augmented Reality Heads-up Display for Drone Inspections. In *CHI Conference on Human Factors in Computing Systems (CHI '25), April 26-May 01, 2025, Yokohama, Japan*. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3706598.3714283

# 1 Introduction

Drone inspections have become commonplace for maintaining hard-to-reach areas in urban environments. In addition to structure condition monitoring, drones are used for data collection for digital twins and tracking construction progress [3, 18]. However, the current reliance on 2D handheld devices for flight control and data monitoring presents significant challenges for pilots [29]. Pilots often experience "tunnel vision", focusing on the screen rather than the drone's physical surroundings, which increases the risk of collisions, especially in safety-critical scenarios [6]. Head-up Displays (HUDs) and augmented reality (AR) interfaces provide an opportunity to integrate drone data directly into the pilot's visual line of sight (VLOS), potentially reducing cognitive load [7, 40] and improving situational awareness [23].

A key challenge lies in designing an AR interface that effectively displays multiple critical elements such as remaining battery, GPS status, speed or orientation without overwhelming the pilot who is already facing cognitive burden [29]. Previous research has often focused on individual elements of the interface, such as camera feed [19], waypoints (for semi-autonomous flight) [52], altitude [55] or heading [47]. Rebensky et al. [41] suggest that directly adapting conventional 2D interfaces to heads-up displays might overwhelm new pilots, recommending AR HUDs to prioritize information based on task urgency and context.

Another challenge, as with every aeronautical activity, is to ensure safety. According to Rahmani et al. [38], in addition to GPS signal loss or battery failures, situational awareness, decision-based, and skill-based errors are the main issues leading to drone accidents. While previous research has improved the operational efficiency of drone flight with AR, such as camera alignment [9, 19], less attention has been given to critical safety scenarios, which are difficult to simulate in real-world experiments.

Our goal is to address these two gaps: (1) the need to manage information processing during an inspection task with AR interfaces, without overwhelming the pilot, and (2) the lack of research on AR interfaces in safety-critical situations. We investigate the following research questions:

* **RQ1.** How can 3D heads-up AR interfaces reduce the cognitive load of drone pilots performing facade inspection?

* **RQ2.** What design factors on the heads-up AR interface can ensure safety and situational awareness during critical scenarios of drone flight?

To answer these questions, we adopted a research-through-design approach [54], aiming at transforming drone interfaces "from their current state to a preferred state" that enhances both safety and efficiency during drone inspections. Through this process, we sought to create an AR prototype that could improve drone piloting under challenging conditions while minimizing cognitive overload.

In this paper, we first describe the participatory design workshops with five professional pilots. These workshops helped identify the potential benefits of using AR for monitoring mission progress and ensuring safety during facade inspection, as well as the need to balance the information presented to avoid overwhelming users. We then describe the iterative design process with three pilots of a VR-based drone inspection simulator. During this process, we explored how dynamic adaptation between safety and mission in AR could reduce cognitive load and enhance situational awareness. The Unity simulator allowed us to simulate the demanding inspection task and challenging scenarios such as GPS signal loss and obstacles without the risks associated with real-world testing. We then describe a user study with 15 experienced drone users comparing the adaptive AR interface (ADAPT-AR) with the conventional 2D interface (2D-ONLY) and a static, non-adaptive AR interface (FULL-AR) during a facade inspection task. The study was followed by semi-structured interviews. We conclude with design insights for developing safety-first heads-up AR interfaces.

# 2 Related Work

## 2.1 Risks and Challenges of Urban Drone Flight

Commercial drone operations, especially those conducted in densely populated urban areas, require rigorous safety assessment procedures. Through interviews with professional pilots, Ljungblad et al. [29] identified safety as the foremost concern of the pilots. Raketonarivo et al. [39] discovered three key factors that impact flight safety: (1) environmental factors like weather condition, geographical elements, airspace characteristics; (2) drone-specific parameters like automation, range, speed, and (3) mission-specific parameters such as payload, trajectories, timing of flight, take-off and landing zone. An online survey [38] highlighted situational awareness, decision-based errors, and skill-based errors as the primary human-factor-related causes of drone accidents. Poor communication, information display, and ambiguity in control modes were also identified as significant obstacles to effective human-drone collaboration. Moreover, Borowik et al. [6] showed with eye trackers that both beginner and advanced pilots tend to focus heavily on the camera feed during flights, leading to tunnel vision and near-collisions.

## 2.2 Heads-up AR Drone Control and Situational Awareness Support

Following the distinctions outlined by Meta of AR, MR, and VR [16], we focus on AR that overlays digital elements onto the physical world, enhancing the user's perception, rather than MR, which focuses on the interaction between real and virtual content.

Situational awareness (SA) was introduced by Endsley et al. [14] as the understanding of what is happening around us, the current situation and the one to come. AR has been shown to improve situational awareness by visualizing the 2D camera feed [19] and the
signal status [41] within the visual line-of-sight (VLOS) compared to a heads-down interface for drone operations. Addin et al. [2] compared a AR heads-up interface with a PC interface during critical drone flight operations. Their results suggest that augmented reality can provide significant situational awareness support for those involved in stressful situations.

In addition to showing flight parameters on a heads-up display, Garcia et al. [17] helps the safety pilots locate the drone with a localization ring. Previous work also showed that incorporating 3D AR visual elements, such as in-situ depictions of the drone's next movements [51, 52] or visualizing flight-path [9] enhances operational efficiency, accuracy and usability of the interface. Similarly, visualizing spatial relations between the drone and its surroundings through 3D projections and path connections aids depth perception [55]. Alternatively, Inonue et al. [21] visualize the height and the proximity of the drone via an augmented Third Person View (TPV) to enhance the perception of the drone's surroundings. Nevertheless, their setup requires a follower drone to observe and control the main drone.

Most of these studies were conducted either in indoor environments or outdoors in open fields and did not consider safety factors like GPS loss or wind turbulence. Additionally, many prior studies focused either on a limited set of information, such as the camera feed [19] and waypoints [9, 55] or on only one modality of information (i.e. 2D [41] or 3D [55]). Rebensky et al. [41] reported that presenting all flight data and the camera feed in 2D on a heads-up display can lead to visual clutter and increased cognitive load, which motivates further investigation into effective information presentation on AR interfaces.

## 2.3 Cognitive Load Management in AR

Ryo et al. [46] synthesized a taxonomy of AR and robotics, identifying real-time data visualization and explainable robotics via AR as key research avenues. Buchner et al. [7] reviewed AR applications, finding that AR generally reduces cognitive load by addressing the split-attention effect [4], which arises when information is spread across multiple sources. This aligns with the Cognitive Theory of Multimedia Learning [32], where spatial and temporal contiguity—such as real-time feedback within the visual line of sight (VLOS)—helps minimize extraneous cognitive load. Further, Li et al. [27] showed that 3D AR interfaces improve eye fixation patterns and reduce perceived workload compared to 2D AR. Despite these advances, Buchner et al. [7] highlighted a need for studies on value-added AR design principles. Our work addresses this gap by investigating how heads-up AR interfaces can reduce cognitive load in drone control tasks.

In addition, adaptive AR interfaces have been explored to mitigate information overload. Lindlbauer et al. [28] emphasized the importance of adapting AR interfaces to user context, while Tatzgern et al. [48] introduced a level-of-detail (LOD) technique using hierarchical clustering to organize AR elements. Lu et al. [30] demonstrated that semi-automated adaptive systems improve task performance. However, these efforts have not addressed safety-critical scenarios like drone control, where managing cognitive load and ensuring situational awareness are paramount. Our research extends this

work by applying adaptive AR design principles to enhance drone control interfaces in such contexts.

# 3 Participatory Design Workshops

We conducted a study with professional pilots to better understand the practical workflow and challenges of drone-based building inspections and identify the design requirements for a heads-up interface. We used participatory design techniques [44] to generate and validate design concepts.

## 3.1 Method

3.1.1 *Participants.* We recruited five expert pilots from the French Civil Aviation School, all of whom hold a professional certificate for conducting drone flights and have more than 3 years of relevant experience in the industry. The participants are currently working in the research field of drone development. The details of the pilots are listed in Table 1 (D1-D5).

We organized two separate sessions of the design workshop to accommodate all participants' schedules. The first session involved three pilots (D1-D3) and three researchers from the group as facilitators and design assistants. The second session involved two pilots (D4, D5) and two researchers.

3.1.2 *Procedure and Apparatus.* After a brief introduction regarding the goals of the workshop, we asked the pilots to brainstorm on how they would conduct the building inspection mission and what were the challenges during the mission. We provided a printed photo of a high-rise building in urban area, taken from a pedestrian point of view. We also printed illustrations of essential data for drone control extracted from the DJI GO 4 [13] and DJI GS Pro [12] interfaces that includes the typical flight data such as signal strength, flight mode, map actions, flight telemetry, way-points, obstacle detection, and avoidance. Before moving to the design phase, participants tried a specific HoloLens AR application to familiarize themselves with the technology. They could use gestures to control a virtual drone's take-off, landing, and autopilot. Each pilot was paired with a researcher who would assist in generating design ideas. They were asked to imagine conducting the mission with a AR headset and illustrate the interface that would be helpful for the mission and addressing the challenges. We used transparent film papers to draw the design ideas simulating AR views. After that, each pilot took turns sharing their ideas and drawings to trigger discussions.

## 3.2 Results

Participants actively engaged in the workshop, providing valuable insights into drone-based building inspections and contributing to the development of heads-up AR concepts for drone operations. Using low-fidelity prototypes, we explored AR solutions addressing three main areas: safety information, mission tracking, and adaptive visualizations to balance safety and mission-related information. Below, we detail our findings related to mission planning, operational challenges, and proposed AR design ideas.

3.2.1 *Mission Planning.* Participants discussed inspecting a high-rise facade in a densely populated area, emphasizing safety, battery optimization, and photo coverage. D1 highlighted the need for safe
<table><caption>Table 1: Professional Pilots Involved in the Participatory Design Workshop (D1-D5) and the Iterative Design (D5-D7)</caption><thead><tr><th>ID</th><th>Gender</th><th>Age</th><th>Experiences</th><th>Session</th></tr></thead><tbody><tr><td>D1</td><td>Male</td><td>30</td><td>6 years of experience in drone development, mainly in embedded system design and operating ground control station (GCS)</td><td>Workshop 1</td></tr><tr><td>D2</td><td>Male</td><td>56</td><td>15 years of drone development with test flight for fixed wings and rotorcrafts.</td><td>Workshop 1</td></tr><tr><td>D3</td><td>Male</td><td>25</td><td>1 year of drone pilot, 3 years of drone research. Worked with home-made drones and designed drone control algorithms</td><td>Workshop 1</td></tr><tr><td>D4</td><td>Male</td><td>42</td><td>18 years of drone development and experiment with fixed wings and rotorcraft</td><td>Workshop 2</td></tr><tr><td>D5</td><td>Male</td><td>42</td><td>15 years of drone development and research. Conducted flight for scientific research with prototypes and some commercial drones.</td><td>Workshop 2 & Iterative Design</td></tr><tr><td>D6</td><td>Male</td><td>28</td><td>3 years working experience in commercial aerial filming</td><td>Iterative Design</td></tr><tr><td>D7</td><td>Male</td><td>31</td><td>4 years of experience as drone instructor and 2 years experience in facade inspection</td><td>Iterative Design</td></tr></tbody></table>

landing zones and proposed starting inspections from the top to ensure safe battery management, though starting at the base could improve visibility for larger structures. D2 stressed the importance of photo overlap for photogrammetry, even at the cost of higher energy consumption. To address the building's curved geometry, participants recommended consistent waypoint distances, with collision detection aiding safe piloting. D5 emphasized defining flight zones to prevent drone drift.

### 3.2.2 Challenges.

**Split Attention.** All participants explained that for legal reasons, pilots must maintain a visual line of sight (VLOS) with the drone, but they also had to rely on the camera view for navigation and identifying defects. This becomes challenging when the drone is distant, making it hard to track its location and orientation (D1, D2). D1 stressed the need to frequently look up at the drone because the camera view alone is insufficient for assessing safety. D2 noted that VLOS piloting improves situational awareness since "with the camera view, you cannot move left or right quickly; you have to turn first." However, managing piloting with safety signal monitoring and data acquisition simultaneously causes information overload, often requiring to operators (D3).

D5 introduced the idea of "autonomy level," suggesting that with an unsupervised flight mode — where the drone follows a predefined path, maintains a set distance, and avoids obstacles — a single pilot could manage both flight and data acquisition. He, however, doubted that he could program it with the current 2D tablet interface.

**Battery Life.** Pilots were concerned with battery life, often proposing solutions such as battery replacement stations on balconies or landing to swap batteries. Pilots rely on low-battery alarms to interrupt missions (D1, D3). However, resuming missions to the latest observed point is difficult as they usually lack clear visual cues. D1 suggested recording the GPS location of flight interruptions. D1 proposed showing an estimate of remaining flight time to reduce stress and optimize mission planning.

**GPS loss and Drifting Issues.** GPS loss and drifting are critical safety challenges that require immediate manual control (D1, D3). D1 emphasized that "This is the situation when you have to look at the drone. If you don't know the heading, you don't know what command you should use."

Most commercial drones can handle moderate wind drifting without landing (D2). However, wind turbulence or GPS loss makes self-balancing unreliable, requiring manual altitude adjustment (D1, D2).

### 3.2.3 Heads-up Display Design Ideas.
Building on the discussed mission challenges, we used paper prototyping techniques to create and discuss AR visualizations related to the challenges identified in the scenario discussion phase. We detail our findings covering three main goals: enhancing safety, improving usability, and balancing information load.

**Use AR to Enhance Safety.** Most pilots mentioned the importance of showing the heading of the drone in the AR view, which helps the pilot know the orientation during manual piloting as presented in Figure 2.a.

Pilots also suggested various solutions for GPS and wind drifting issues. D1 proposed visualizing wind turbulence and GPS-denied areas as danger zones, but others found it counterproductive if it blocked the drone's view. D3 suggested displaying real-time wind direction and intensity above the drone, while D2 focused on using peripheral signals to indicate safety status as illustrated in Figure 2.d.

D4 proposed to visualize a ground projection from the current position of the drone, indicating the horizontal positioning and the ground area influenced by the fall damage. It can help the pilot estimate the ground risk from the current midair position. To mitigate the danger of midair collision, a spherical boundary can be visualized around the drone for an intuitive perception of the safety distance when piloting with VLOS. The color of the boundary is dependent on the current nearest obstacle distance as detected by the drone (D4, Figure 2.f). Alternatively, D1 proposed visualizing a projection casting from the drone to the surface of the building to indicate the distance (Figure 2.b).

D2 drew a path connecting the current drone location and the landing point. In addition, D1 and D5 brought up the idea of showing the needed battery remaining energy over this landing path, so that the pilot can glance at the visualization to know whether they have sufficient time to complete the mission (Figure 2.e). It helps the pilot project into a future state of the mission and prepares for landing, reducing the negative feeling of bewilderment when the battery alert occurs and improving situational awareness.
<figure><img src="image_4.png" /><figcaption>Figure 2: Examples of the safety and mission-related visual-ization illustrated by the pilots. (a) Display the heading of the drone, illustrated by D3. (b) Show the nearby waypoints and the photo-taking points over the trajectory, illustrated by D1. (c) Allow marking the point of interest from the camera view while monitoring, illustrated by D3. (d) Visualize safety signals at the peripheral vision, illustrated by D2. (e) Show the RTH path with battery remaining energy, illustrated by D1. (f) Display a boundary around the drone and the ground projection, illustrated by D4.</figcaption></figure>

*Use AR to Track Mission and Improve Usability.* All pilots agreed on the value of using AR to visualize waypoints and the flight trajec-tory over the building surface. D1 suggested displaying only nearby waypoints, while D3 recommended coloring completed segments of the trajectory differently. D1 and D5 added that marking the locations where photos were taken along the path would help track progress (Figure 2.b). D5 also highlighted that visualizing coverage across the building surface could assist in monitoring data quality and revisiting poorly captured points.

Several pilots stressed the benefit of integrating the camera feed within the VLOS view, allowing smoother attention shifts between tracking mission progress and focusing on the inspection task. D3 emphasized the need to mark points of interest (POIs) during the flight, such as areas with defects, low lighting, or locations where the drone may need to return home due to a low battery. He proposed using a simple tap gesture on the camera feed to mark defective regions for future reference (Figure 2.c).

*Balancing Mission and Safety Information to Reduce Complexity.* Pilots expressed concerns about the complexity of the AR visualiza-tion that could occlude some elements of the real world or create information overload (D2, D3). To address this, they proposed sepa-rate views for mission and safety to streamline information. The mission-focused view would include the camera feed, sensor data (eg. infrared sensors), waypoints, camera frustum, and coverage with marked points of interest, along with a status message for warning notifications. The safety-focused view would prioritize safety data monitoring, including GPS satellite count, connectivity,

battery levels, wind turbulence, and obstacle detection using high-contrast colors like red for urgent signals and green for normal ones as illustrated in Figure 2.d

Additionally, pilots suggested adapting the camera feed's placement depending on the context (D1, D4). During autopilot, the camera feed should be centered to focus on data collection. When manual piloting is required due to safety issues, the camera feed should be positioned peripherally, allowing pilots to use AR overlays (collision boundaries, headings, RTH paths) for better situational awareness while managing both the feed and VLOS.

## 4 Iterative Design of an Adaptive AR Interface and a Realistic Inspection Task

Using a technology probe approach [20] with three professional drone pilots (see table 1, D5-D7), we iteratively developed an in-teractive prototype and a challenging inspection task. We further explored how an AR interface could support pilots in safety-critical conditions and focus on the adaptation between safety and mis-sion tasks. To address technical and safety challenges, we created a simulator in Unity (Version 2022.3.12f1) and tested it in a VR environment with a Meta Quest 3 headset. This setup allowed us to focus on AR design without the risks of real-world testing. Pilots provided insights on AR views, system usability, and task realism, as well as on how to improve the interface and adapt it to various operational scenarios. We describe below the simulation environ-ment, inspection task, and how we integrated safety challenges, followed by the interactions and visualizations that support pilots in managing their missions and responding to events.

### 4.1 Inspection Task and Drone Simulation

To simulate the building inspection task, we built a 3D environment based on the city center of Singapore as illustrated in Figure 3. The dimensions of the inspected building were 34m (width) x 29.46m (length) x 62m (height). It is located at the corner of the street and the standing location of the pilot is on the opposite corner. We simulated the busy traffic with ambient noises of the street to add realism.

To create a realistic virtual drone, we adapted the velocity-based drone controller from Belkhale et al. [5]. Based on pilots' feedback, we finely tuned its behaviors to ensure its flight mechanics closely resemble those of a real one. We also created a drone simulation system that allows path planning, switching between manual and autopilot, battery, GPS, and wind simulation, and interactive visualization on 2D and AR interfaces.

The mission objective is to correctly spot all defects on the build-ing facade. We rendered six types of defects on the building facade, based on the common defects classifications shown in [8, 25]. These include paint peel, wall crack, wall dent, rotten surface, leakage, and delamination (Figure 3.b-g). To avoid repetition, we created at least three variations for each type of defect. We intentionally made some defects hard to spot by placing them under the shadowed area on the building surface (Figure 3.c). The defects are evenly laid out over the building facade, with 0-2 defects per layer. There are in total 11-12 defects on the entire facade of 12 layers. D7, who had experience with building inspection, found the defects realistic for the simulated task.
<figure><img src="image_5.png" /><figcaption>Figure 3: The simulated building inspection scenario and different types of defects used in our study. (a) Overall configuration of the scenario. (b) Paint peel. (c) Wall crack. (d) Wall dent. (e) Rotten surface. (f) Leakage. (g) Delamination.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: Interactive 2D Camera View. Left: Inputs from the Meta Quest Touch Pro VR Controllers (Only the Left Controller is shown in the picture). Right: Interactive buttons on the interface. The autopilot button (bottom right) will be highlighted if the autopilot is on.</figcaption></figure>

## 4.2 Implementation of the Camera View and Interactions

Following suggestions from our participatory design workshops, we created a floating midair camera view in AR. In addition to the drone video feed, usual flight information is displayed on the camera view as illustrated in Figure 4. We adopted the user interface design of DJI GO 4 [13], a popular flight control software developed by DJI, which features flight status messages, signal and system monitoring, a battery slider bar, telemetry data monitoring, and collision detection status. This implementation was based on suggestions by the pilots, saying that it would be a good option to rely on the familiar layout of the existing 2D interface, even in the new AR adaptive interface.

The user interacts with the simulated drone via the 2D interface and the joysticks of the controllers. There are three buttons to manage the flow of the drone flight: a take-off button, a return-to-home (RTH) button, and an autopilot toggle button. The autopilot function can only be used when near the trajectory, and it can be interrupted by any manual input from the joysticks. The user can trigger the defect marking by directly "clicking" on the camera view, which creates a red circle on the exact location of the marked point (Figure 4).

## 4.3 Simulating Safety Challenges

We simulated GPS loss, wind turbulence, and limited battery life as safety challenges that were identified during the workshops to replicate complex situations and design effective adaptation behaviors for such situations. Following the pilots' suggestions (D5-D7), we also added mid-air obstacles. Both GPS loss and obstacle detection cause the autopilot to stop, imposing pilots to manually operate the drone before enabling the autopilot again when safety issues are solved.

4.3.1 GPS Loss and Drifting. The GPS loss often occurs when the drone is close to the building. We manually defined several GPS-denied zones on the building facade. The GPS signal gradually decreases when entering the zone and gradually recovers when leaving the zone. We first implemented sudden changes but D5 commented that this did not feel realistic and D7 argued that "Normally the drone can still stabilise to a certain extent with visual sensors. Thus, the drift of global positioning should be a gradual process."

When the GPS is lost, the system cannot confidently determine the accurate global position of the drone, resulting in inaccuracies of drone-centered visualizations. This prevents the drone from stabilizing using positional data, which makes it start drifting due to the wind. In our simulation, the wind is represented as a force vector that gradually adjusts its direction and intensity to match a target force vector, which is updated at random intervals. The behavior was adjusted from the previous iteration, using constant direction based on D7's feedback that "one difficulty about flying in real world is that the wind isn't constant. It can change direction too.") The wind vector imposes force on the drone as long as it has taken off and affects the movement of the drone depending on the GPS status.

<figure><img src="image_7.png" /><figcaption>Figure 5: Mid-air obstacles. (a) A tall tree. (b) A gondola machine.</figcaption></figure>

4.3.2 Mid-air Obstacles. Commenting on the difficulty of our task, D7 explained that "there are way more things to consider in real world flight" such as poles extruding from the building, open windows, etc. The presence of unexpected mid-air obstacles, especially those blocking the flight path, imposes great challenges for drone navigation. We thus modelled two types of mid-air obstacles over
the facade: a tall tree and a gondola machine in operation, as illustrated in Figure 5. Typically, the drone cannot reliably detect small branches on a tree or thin wires with its sensor, forcing pilots to rely on direct sight to avoid obstacles.

## 4.4 Mission and Safety AR Elements

Mission and safety information are displayed as 3D in-situ visual cues, either centered on and moving with the drone or anchored in a fixed world position. The 2D panel presenting the video feed from the drone and safety information is anchored either to a head-fixed or a body-fixed location, depending on the mission context. We describe our AR designs following the main flight phases including pre-inspection, inspection, critical events, and returning to home.

### 4.4.1 Pre-inspection and Inspection Phases.
At the start of the mission, a safety buffer is visualized as a transparent green box surrounding the building, marking the safe area for operation (Figure 6.a). A 3D path plan is projected onto the building facade, with each waypoint represented by a holographic drone model, as shown in Figure 6.b.c.

Once the drone enters the mission boundary, the safety buffer visualization disappears, the autopilot can be activated and only the nearest waypoint to the drone's real-time position is shown, helping the pilot select the starting point for autopilot.

During autopilot, only the next waypoint is displayed, following a suggestion from D1 during the workshop. Also, the drone travels at a constant speed between waypoints, pausing briefly upon reaching each one. The pauses allow the pilot to mark defects without the risk of a camera shake, improving the quality of data collection. Coverage visualizations appear on the facade at each waypoint, confirming that data has been successfully captured for that region (Figure 6.c). The coverage visualization can also be displayed during the manual flight if the pilot briefly holds the drone's position at the waypoint.

### 4.4.2 Critical and Returning Phases.
When critical issues occur, safety and control information is displayed in situ through 3D visualizations, primarily as drone-centered visual cues. A horizontal locator ring appears at the drone's real-time position, helping the pilot locate the drone when it is far away, as shown in Figure 7.B. A red arrow on the ring indicates the drone's heading. Collision detection is visualized on the locator ring in a similar manner to the 2D interface, showing both the direction and distance of nearby obstacles, with colors (yellow or red) indicating proximity. We implemented dynamic scaling for the drone-centered visualization as the drone moves away from the pilot. The scale of the radius remains the same when the distance between the drone and the headset is within 25 meters, while it will grow proportionally when it moves beyond the range (Scale radius = max(1.0, distance×0.04)). This ensures that the pilot is able to see the visualization even if the drone is far away.

A blue vertical line is projected from the drone's position to the ground level, as presented in Figure 8.a. The altitude is shown near the center of the line. This not only visualizes the relative horizontal location of the drone to the surrounding structures, such as the building facade, but also indicates the projected fall area on the ground.

<figure><img src="image_8.png" /><figcaption>Figure 6: (a) Mission boundary (in green). (b) Generated path plan (in blue) based on building geometry and parameters. (c) Waypoints and coverage visualization during autopilot.</figcaption></figure>

As part of the GPS loss simulation, the accuracy of the drone-centered visualization is not guaranteed during GPS loss. To make the pilot aware of the horizontal positioning error, the positional uncertainty is visualized as a holographic disc beneath the drone, becoming more transparent as uncertainty increases to avoid blocking the view of the real drone (Figure 7.D). This allows pilots to notice GPS issues directly from the line of sight and make prompt decisions. In earlier iterations, GPS loss was indicated by a red cross above the drone (Figure 7.C), but D5 and D6 indicated that it was drawing too much attention away from the actual drone's position and argued for a more subtle, continuous indicator of positional error.

When the battery is running low, a Return-to-Home (RTH) path is visualized, connecting the drone's current position to the home point as illustrated in Figure 8. This path also displays battery sufficiency, similar to the battery slider on the 2D interface, based on suggestions from D1 and D5 during the design workshop. Normally, the path appears green. As the remaining battery approaches the low (25% of battery left) and RTH (dynamically adjusted based on the distance) threshold, a yellow bar begins to extend from the home point toward the drone's position along the path, indicating the need to trigger the RTH function. When the battery nears the critically low threshold (10% of battery left), a red bar gradually replaces the yellow, signaling that the drone should land immediately once the entire path turns red.

### 4.4.3 Discarded Visualizations.
Several visualizations initially discussed during the design workshops were discarded after pilot feedback. As illustrated in Figure 7.A, we experimented with displaying battery level on the drone's locator ring, drone inclination, a projected line to the building surface, and the camera frustum. However, pilots found these displays cluttered and unhelpful. D5 remarked that the inclination was "more disturbing than helpful,"
<figure><img src="image_9.png" /><figcaption>**Figure 7:** Iterative Design Decisions. A & B: Battery ring, surface projections, and inclination are removed and replaced with a white locator ring, red arrow heading, and circular collision detection. C & D: GPS error which blocks the real drone is replaced with a more subtle and transparent representation of positional uncertainty. E: Wind visualization was discarded in the final version.</figcaption></figure>

while D6 and D7 both found the battery ring unnecessary. D7 also suggested simplifying the camera frustum to an arrow to reduce visual clutter, leading us to revise these elements in the final version.

We also tested wind visualization in an early version. We visualized the wind as a cluster of animated 3D arrows pointing towards the detected wind direction. The strength of the wind was indicated by the speed of the animation and the color of the visualization (from white to red, Figure 7.E). However, D5 and D6 found the cues for wind direction and strength unclear and of little use. In practice, pilots preferred to rely on the camera view and VLOS when adjusting the drone's position. As a result, we removed the wind indicator to declutter the interface. Additionally, wind and GPS loss zones were not visualized, as they would require additional mission preparation that we did not focus on for our study.

## 4.5 Adaptation between Mission and Safety Views

Along with pilots, we refined the concept and behavior of the adaptation between two views, mission and safety. Figure 1 and Figure 9 provide an overview of the differences between the two views.

We mostly relied on the feedback provided by the participants to elicit which visualization would be integrated into which view and how to transition between them. Following principles from

Deuschel and Scully [11], we opted for preserving the spatial organization of the information and mostly focused on which information was required in each view to make it visible or not.

The first is **mission view**, which focuses on inspection during autopilot. Here, the 2D interface is centered in the pilot's vision (head-locked), with most interface elements hidden, showing only the 2D status messages, the 3D drone path, and coverage information.

The second is **safety view**, which focuses on manual piloting and the visibility of safety issues. The 2D interface is positioned in a body-locked view to assist with line-of-sight piloting, and all drone-centered visualizations—such as the position ring, heading, ground projection, and RTH path are displayed. Additionally, following the recommendation of Pfeuffer *et al.* [36] on using gaze to navigate multiple layers of information, we implemented dynamic transparency of the 2D interface based on the pilot's eye gaze approximated by the head-tracking feature (eye-tracking is not available on Meta Quest 3). The interface becomes faded when it is not close to the center of the view space, which is determined by a ray casting at the forward direction of the view camera.

The system automatically switches from the mission view to the safety view when abnormalities or emergency events are detected. These include near collisions, GPS loss, or low battery, as well as a manual override when the control mode is switched from autopilot to manual piloting (triggered by moving the joysticks). Upon switching to the safety view, AR safety visualizations are displayed instantly, and the 2D interface transitions smoothly from a head-locked to a body-locked position with a brief animation. The pilot can return to the mission view by clicking the autopilot button when no safety issues are present.

## 5 Comparative Study between 2D-Only, Full-AR and Adaptive-AR

We aimed to evaluate if the adaptive AR interface that adjusts the views based on the task context can reduce cognitive load and enhance situational awareness. We conducted a comparative user study of the three interface designs: a regular 2D-only interface (2D-ONLY), a static full-AR interface that displays all information simultaneously (FULL-AR), and the adaptive AR interface (ADAPT-AR).

### 5.1 Method

5.1.1 *Participants.* We recruited 15 participants (10 male, 5 female, mean age 24.93 years) through university platforms in Singapore and Toulouse. Participants were required to have prior experience with outdoor drone flights, including manual control or autopilot.

These amateur pilots are indexed as A1-A15. The study took approximately 1.5 hours per participant, and they were compensated $20 for their time. The experiment was approved by the IRB of our university.

In addition, due to limited availability, three professional drone pilots who had participated in the iterative design sessions (D5-D7) tested the three conditions of the prototype to provide qualitative feedback.
<figure><img src="image_10.png" /><figcaption>Figure 8: Battery remaining time on the 2D view and the 3D RTH path. (a) The white dot indicates the battery's low threshold (25% of battery remaining in our case). Ground projection is visualized in blue. (b) The yellow dot indicates the RTH threshold dynamically calculated based on the distance of the drone to the home point. (c) The red dot indicates the battery's critically low threshold (10% of battery remaining in our case).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 9: Adaptive behavior during the inspection and the critical phase of the drone flight. Left: Mission view that focuses on marking defects and tracking the mission progress. Right: Safety view that focuses on controlling the drone with both camera view and VLOS and being aware of the safety issues. Note that the colors of the simulated real-world elements have been desaturated to facilitate the visualization of AR elements.</figcaption></figure>

5.1.2 *Study Design.* The experiment follows a within-subject design, where each participant experienced the three conditions: 2D-ONLY, FULL-AR, and ADAPT-AR. The order of the conditions was counterbalanced using a Latin square to cancel any order effects.

5.1.3 *Task.* Participants performed a drone-based building inspection task similar to the one described in Section 4 in a virtual reality (VR) environment. The participants had to:

(1) Identify and mark defects on the building facade,

(2) Follow the predefined flight path and minimize deviations unless obstacles were encountered,

(3) Maintain safety by avoiding collisions and monitoring battery life for a safe landing.

Three experimental task configurations were created to prevent learning from one experimental condition to the other. Each configuration involved inspecting a different building facade (short or long) with different layouts of defects, mid-air obstacles, and GPS-denied zones. Battery life was set according to the facade size (4 minutes 20 seconds for short, 5 minutes 10 seconds for long).
Figure 10.a shows one of the configurations used in the experiment on the short facade. The three configurations were designed to be almost equivalent in terms of difficulty.

### 5.1.4 *Conditions*. We compared three Interface conditions:

**2D-ONLY.** This condition replicates a regular interface displayed on a tablet, similar to the DJI GO 4 app [13], and serves as a baseline. The interface is attached to the user's left hand in VR and displays a camera view and flight path plan simultaneously (Figure 10). The display replicates the vertical path plan visualizer and autonomous flight monitor of professional inspection software such as UgCS [15].

<figure><img src="image_12.png" /><figcaption>Figure 10: (a) One example of the configuration on the short facade. Yellow regions indicate GPS-denied areas. Red circles surround the defects. (b) 2D tablet-based Interface in the 2D-ONLY condition. Top: Corresponding path plan for the config. Bottom: Camera view (DJI GO 4 [13])</figcaption></figure>

**FULL-AR.** In this condition, all the information (camera view, flight path, safety indicators) is overlaid on the AR interface without adaptation. The interface remains static, showing all visualizations at once. The 2D display is head-fixed within the user's line of sight, eliminating the adaptive switching between head-fixed and body-fixed positions.

**ADAPT-AR.** The adaptive interface, as described in Section 4, selectively presents either mission-focused or safety-focused information based on the ongoing task context. It aims to simplify the interface by focusing the user's attention on the most relevant visual elements at any given moment.

### 5.1.5 *Procedure*
We used the Meta Quest 3 headset for the virtual reality simulation. The application was built as a standalone software pre-loaded on the headset. Participants wore the headset and used the controllers to interact. They were asked to stand within a safe space without obstacles during the experimental tasks.

The experiment began with a brief familiarization session, where participants used the 2D-only interface to take off, practice manual control, and experience simulated GPS loss, drifting, and mid-air obstacles. This helped them get comfortable with the VR environment and the controller.

After familiarization, participants completed a learning phase with their first interface condition where they practiced the inspection task and received performance scores (correctly marked defects, average deviation from the flight path). They could retry the learning phase if unsatisfied with their performance. Following this, participants completed the inspection task under the same interface conditions. During the actual test, the performance scores were hidden.

After each condition, participants completed a post-task questionnaire to evaluate cognitive load, situational awareness, task performance, safety information accessibility, and self-confidence. Finally, one researcher conducted a semi-structured interview to gather qualitative feedback and identify future design opportunities.

## 5.2 Data Collection and Analysis

### 5.2.1 *Performance*. To assess the impact of the Interface on the task performance, we logged several data and computed two key performance indicators:

* Percentage of marked defects: The number of correctly identified and marked defects.

* Deviation from flight path: The average shortest distance from each waypoint to the actual navigation path, indicating how well the pilot can follow the flight plan.

### 5.2.2 *Questionnaires*. After each condition, participants were asked to fill out three specific questionnaires (provided in Appendix A.) focusing on different aspects:

* Cognitive Load: We used the Bedford Workload Scale to assess cognitive load, a method used by NASA to evaluate cognitive workload [42]. The scale uses a hierarchical decision tree to quantify cognitive load.

* Situational Awareness: Since our task are characterized by its dynamic and undefined nature, we measured situational awareness using the SART (Situational Awareness Rating Technique) [14] following the guideline of [43]. It evaluates task demand (D), understanding (U), and supply (S) with a final situation awareness score calculated using the formula: $SA = U - (D - S)$.

* Participants rated their performance, the accessibility of safety information, and their confidence in using the interface for real-world inspections using a 7-point Likert scale.

For both performance and questionnaire data, we used a Repeated Measures (RM) ANOVA test to analyze the data if the normality assumption was met (Shapiro-Wilk test, *p* > 0.05), while for non-parametric data, we used the Friedman test to analyze the main effect and applied Conover's Post Hoc Comparisons. JASP [22] was used for data analysis.

### 5.2.3 *Semi-structured Interview*. Interviews were recorded and transcribed. One of the authors did a thematic analysis using QDA Miner software [37] to identify codes and group them into themes. These themes were consolidated with the other authors.

## 5.3 Results

### 5.3.1 *Quantitative Data*.
<figure><img src="image_13.png" /><figcaption>Figure 11: The three interface conditions in the user study. (1) 2D hand-held tablet interface with no AR visualization as the baseline. (2) AR interface that displays all information without adaptive features. (3) Adaptive AR interface that switches between mission and safety views.</figcaption></figure>

*Task Performances.* The mean percentage of marked defects for 2D-ONLY is 83.2%, while FULL-AR is 79.3%, and ADAPT-AR is 71.7%. The Shapiro-Wilk test rejects the normality of the data ($p_{adapt-ar} = 0.038$). The Friedman test shows no main effect of the interface on the marked percentage ($\chi_F^2 = 3.000, p = 0.223$, Kendall's W = 0.100).

The mean deviation from the flight path for 2D-ONLY is 2.84 meters, while FULL-AR is 2.84 meters, and ADAPT-AR is 2.62 meters. The Shapiro-Wilk test reveals that the normality is not met for 2D-ONLY and FULL-AR ($p_{2d-only} = 0.004, p_{full-ar} = 0.002$). The Friedman test shows no main effect of the Interface on the average deviation ($\chi_F^2 = 0.533, p = 0.766$, Kendall's W = 0.018).

<figure><img src="image_14.png" /><figcaption>Figure 12: Mean rating for the percentage of marked defects and average deviation from the flight path.</figcaption></figure>

*Cognitive Load.* The mean cognitive load ratings of ADAPT-AR is 4.13 (SD = 1.727), while 2D-ONLY is 5.40 (SD = 2.230) and FULL-AR is 4.47 (SD = 1.506).

The Shapiro-Wilk test of 2D-ONLY and FULL-AR rejected the normality of the data ($p_{2d-only} = 0.060, p_{full-ar} = 0.008$). The Friedman non-parametric test shows a significant main effect of the

Interface on cognitive load ($\chi_F^2 = 7.581, p = 0.023$, Kendall's W = 0.253). The Conover's post-hoc tests showed that there is a significant reduction of cognitive load between 2D-ONLY and ADAPT-AR ($p = 0.005, p_{bonf} = 0.016, p_{holm} = 0.016$). No significant difference was observed between other pair-wise comparisons. This indicates that the adaptive interface significantly reduces the cognitive load compared to the baseline (the current 2D-only tablet interface), while the full AR visualization has little improvement in terms of cognitive load.

*Situational Awareness.* The mean situational awareness is 17.47 (SD = 8.879) for 2D-ONLY, 21.40 (SD = 4.014) for FULL-AR, and 24.07 (SD = 7.126) for ADAPT-AR.

As the data passed the Shapiro-Wilk normality test, we did an RM ANOVA test, which shows a significant main effect of the Interface on the situational awareness ($F(1, 14) = 5.107, p = 0.013$, $\eta_p^2 = 0.267$). The Post hoc test shows a significant difference between 2D-ONLY and ADAPT-AR ($p_{bonf} = 0.021, p_{holm} = 0.021$). No significant difference is observed in the other pair-wise comparisons. These results suggest that the adaptive interface improves the pilot's situational awareness during the flight compared to the baseline.

*Subjective Performance.* The subjective performance is rated on a 1-7 scale. The mean subjective performance of 2D-ONLY is 4.20 (SD = 0.941) while FULL-AR is 5.40 (SD = 0.986) and ADAPT-AR is 5.73 (SD = 1.033).

The Shapiro-Wilk test shows that normality does not meet for both the data in 2D-ONLY and ADAPT-AR ($p_{2d-only} = 0.004, p_{adapt} = 0.005$). The Friedman test shows that there is a significant main effect of the Interface on the subjective performance ($\chi_F^2 = 12.360, p = 0.002$, Kendall's W = 0.412). The Conover's post hoc test shows significant differences for 2D-ONLY vs. ADAPT-AR ($p < .001$, $p_{bonf} < .001$, $p_{holm} < .001$), as well as 2D-ONLY vs. FULL-AR ($p = 0.002$, $p_{bonf} = 0.006$, $p_{holm} = 0.004$). This indicates that the AR, in general, improves the perceived performance.
<figure><img src="image_15.png" /><figcaption>Figure 13: Mean rating for cognitive load, situational awareness, subjective perception of the performance, accessibility of the safety information, and the confidence of using the current interface. The significant comparisons are highlighted with asterisk marks.</figcaption></figure>

*Safety Accessibility.* The accessibility of the safety information is rated on a 1-7 scale. The mean safety accessibility is 3.87 for 2D-ONLY (SD = 1.885), 5.87 for FULL-AR (SD = 0.915), and 5.07 for ADAPT-AR (SD = 1.486).

The Shapiro-Wilk test shows that the normality is met. The RM ANOVA test shows that there is a significant main effect of the Interface on the safety accessibility ($F(1, 14) = 7.222, p = 0.003, \eta_p^2 = 0.340$). The Conovor's post hoc test shows a significant contrast between 2D-ONLY and FULL-AR ($p_{bonf} = 0.014, p_{holm} = 0.014$). This indicates that only the full visualization improves the accessibility of the safety information.

*Confidence.* The confidence of using the interface is rated on a 1-7 scale. The mean confidence is 3.47 for 2D-ONLY (SD = 1.922), 4.87 for FULL-AR (SD = 1.457), and 4.87 for ADAPT-AR (SD = 1.727).

The Shapiro-Wilk test shows that the normality does not meet for the data in FULL-AR ($p = 0.041$). The Friedman test shows that there is a significant main effect of the Interface on the confidence ($\chi_F^2 = 9.733, p = 0.008, Kendall's W = 0.324$). The Conovor's post hoc test shows significant differences between 2D-ONLY and ADAPT-AR ($p = 0.006, p_{bonf} = 0.018, p_{holm} = 0.012$) and between 2D-ONLY and FULL-AR ($p = 0.002, p_{bonf} = 0.007, p_{holm} = 0.007$). This indicates that AR, in general, improves the confidence of using the interface.

### 5.3.2 Qualitative Data.
Participants provided rich feedback on the three interfaces. We present our findings below according to four major themes that emerged during the thematic analysis.

*AR Serves as an Assistive Feature for the Mission.* Participants found AR visualization highly beneficial for mission completion. Many favored the 3D path, with A12 noting, "I can follow the path instead of flying aimlessly," and D6 emphasizing the usefulness of the camera coverage indicators over the facade in tracking progress: "This one (AR) is definitely helpful... it's more mission related." A13 and D6 suggested adding coverage indicators to both the building surface and the camera view for better utility.

The Return To Home (RTH) path, showing remaining battery life, was praised by most participants (11/15) for improving safety and flight planning. A2 appreciated knowing when to return, saying "I know when I need to come back and I will plan (the rest of) the

flight accordingly". D6 commended the design as straightforward and informative: "It tells you how critical your battery life is... and the path is a straight line, which is very useful." A15 noted that the feature became increasingly helpful with familiarity, especially in locating power information.

*AR Supports Attention Switches, but Visibility Remains an Issue.* The 2D interface posed challenges in maintaining Visual Line of Sight (VLOS) and assessing drone orientation. AR visualizations, such as the drone locator, heading, and ground projection, aided most pilots (11/15) during attention switches. A3 noted, ""You don't see where the drone really is. Then you can use the drone locator," while A6 emphasized its utility when the drone is distant and barely visible. The ground projection also helped pilots assess the drone's position relative to obstacles, as A10 highlighted: ""It has height and position information... relative to the trees."" The AR collision indicator was useful for some (A1, A13, A14) but less effective for others, like A6 and D6, who preferred collision indicators in the video feed, as seen in commercial systems. D6 remarked, "The ring around the drone is not so noticeable... it's not really catching my attention," suggesting that AR displays could be improved for better visibility near obstacles, especially when the drone is far from the pilot.

*Adaptive Visualization Potentially Reduces Cognitive Load, but Requires Trust in the System.* The AR visualization can help the pilot with mission tracking and attention-switching, but the AR interface with full visualizations provides little improvement in cognitive load compared to the baseline 2D interface. However, both quantitative and qualitative feedback indicate that the adaptive interface significantly reduces the cognitive load over the baseline. D5 mentioned that "adaptive was better because it hides complexity and reduces the cognitive workload during automated flight phases." A7 said that it felt "less stressful" when using the adaptive mode.

Participants raised concerns about trusting the adaptive system. D6 preferred having all critical information visible at all times, stating, that "When I want it to appear, it doesn't appear and then it becomes a bit stressful." A10 and A15 shared similar sentiments with A10 noting, "if everything is shown, I can focus on what I need and ignore the rest... if you don't show it to me, I feel insecure." D5 had a more balanced perspective, initially feeling "surprised" by the
lack of information but later appreciating the reduced clutter for improved focus. However, he added, "You have to trust the system to actually detect problems." This lack of trust contributed to a lower safety information accessibility rating for the adaptive interface compared to the full AR interface.

*Adaptive Visualization Makes Warning more Salient, but Screen Placement and Behavior Need Improvement.* Participants found the adaptive interface effective in providing clear warning signals when switching to the safety view, helping them react quickly. D5 noted, "When I use autopilot, I need less concentration because I can use the alarms to react... it's even more important in adaptive as the camera view changes to make you realize there's a problem." A9 found the adaptive visualization helpful during manual piloting by removing irrelevant information, and A4 and A13 appreciated how it uncluttered the visual line of sight.

However, the inability to adjust the location and scale of the 2D interface in AR hindered usability, particularly for the adaptive interface. A7 noted discomfort with the 2D interface "following my head", while D6 and A11 preferred the flexibility of a hand-attached 2D interface. D5 also wished to remove the 2D interface from view, explaining, "The screen is always in the field of view. It forces you to look at it." Additionally, A13 mentioned a lag when the 2D interface repositioned, indicating that the head-tracking feature was not responsive enough, reducing overall usability.

# 6 Discussion

This section synthesizes insights from our design workshops, iterative process, and user study to address our research questions and highlight key design considerations for improving human-drone interaction with heads-up AR displays.

## 6.1 RQ1: Design Considerations for Reducing Cognitive Load in a Heads-up AR Drone Control Interface

Our results show that an adaptive AR interface can effectively decrease cognitive load. However, the rating (3.67 as visible in Figure 13) still slightly exceeds NASA's recommended threshold for nominal tasks [34] (a score of 3 or below). This may be due to participants' unfamiliarity with the interface compared to the more conventional 2D-only interface. With longer use and then higher familiarity, the workload rating could improve [50]. Additionally, allowing users to customize and control the adaptive interface could further optimize cognitive load and enhance user experience [45]. Below, we describe key design considerations based on our findings.

*Optimizing Attention Switches with Adaptive AR.* Drone control interfaces are inherently complex, requiring pilots to monitor multiple pieces of information simultaneously. Situational impairments such as vision tunneling, external distraction, or screen reflection can exacerbate these challenges and result in safety hazards [6, 38].

Heads-up AR devices, such as HoloLens 2, provide a wider display space that can alleviate information clutter observed in tablet-based interfaces. By appropriately presenting critical information and hiding non-essential data [32], AR designers can reduce cognitive load. For example, when GPS is lost, AR could prominently display the drone locator and heading information while hiding less

critical data such as the building coverage. This approach can also be applied during automated phases to focus on the main inspection task or any other critical events by identifying relevant data in each situation. In summary, **with adaptive AR interfaces, designers could reduce complexity and enhance attention-switching efficiency, particularly in critical scenarios.**

*Enhancing Visibility and Informativeness.* Visibility and informa- tiveness of AR cues are crucial for cognitive load and situational awareness. In our study, while we implemented dynamic scaling for the drone-centered visualization as it moved away from the pilot, some pilots found these cues most useful when the drone was nearby, with effectiveness decreasing as the drone moved fur- ther away from the building. The heading visualization also lost its usefulness with distance. Additionally, collision detection was not prominent enough and was often missed in AR due to visual cues being too far from the pilot. At the same time, AR visualiza- tions should avoid obstructing the real-world view or cluttering the space. Since the user's gaze is approximated by head-tracking, future designs could **improve clarity by increasing size, enhancing contrast, or using eye-tracking to emphasize key elements.**

Moreover, visual cues should guide the pilot's actions. For ex- ample, the RTH path with battery status provided clear guidance on when to land. While the AR inspection path and waypoints were useful, participants struggled to align with the path manually due to visual ambiguities. One participant suggested adding an offset indicator to aid alignment during manual piloting. Thus, **AR visual cues must be informative and adaptable to support decision-making without overwhelming users.**

*Streamlining Operations with Dual Control Mode.* Current com- mercial drones often interrupt autopilot when issues such as GPS loss or wind turbulence are detected. Our study suggests that AR can enhance the perception of mode interruptions and facilitate transitions between control modes. However, as indicated by sev- eral pilots and our observations, pilots often failed to resume to mission mode when safety issues were resolved. Adding clear in- dications about when to take manual control and how to switch back to autopilot if conditions are stable could help pilots transiting between modes. Also providing specific information on how to re-attend to the mission tasks after focusing on safety could be helpful. For instance, participants suggested adding visual cues on the future direction of the drone when the autopilot will be resumed. Future AR interfaces should **clearly delineate signals and actions required for transitioning between manual and autopilot modes to streamline the semi-autonomous drone operation.**

*Customizing and Controlling the Adaptive AR Interface.* Our work revealed diverse preferences for the placement and visualization of specific visual elements, both in AR and on the 2D interface. Unlike traditional tablets, heads-up displays offer flexibility in adjusting scale, transparency, and position. It also offers the possibility of multimodal interactions with gestures, voice, and gaze input capa- bilities [35] even if voice command could be better in piloting tasks compared to gesture-based interaction [26]. On the other hand, tablets can also be put away to rapidly remove interface clutter.
Users could benefit from interaction allowing them to tailor the interface according to their preferences, the specific flight conditions or their mission context both before the operation or with explicit control while operating. Future designs should **include customizable options for interface placement, content, and interaction modes, enhancing user control and satisfaction.**

## 6.2 RQ2: Safety Considerations for Dealing with Critical Scenarios Using a Heads-up AR Interface

AR visualization generally enhances perceived safety by improving situational awareness and depth perception [55]. In our study, participants gave better safety ratings to the non-adaptive AR visualization even if they rated the adaptive version with a better situation awareness score. According to Lavie et al. [24] high levels of adaptivity support performances in routine situations but could impair performances in unfamiliar situations. Based on our results, we discuss below design considerations for improving safety with an adaptive interface without compromising the existing advantages.

*Building Trust with the System.* Trust is crucial for adaptive systems that automatically adjust information visibility. Users may initially feel insecure when critical data is hidden. Trust is enabled by the "predictability of the behavior of other agents through understanding their motivations" [1]. The "transparency" of the autonomous agent system is also seen as an important factor for human trust [31]. To build trust, we thus suggest to **enhance transparency by explaining the rules and triggers behind adaptive behaviors, such as why certain information is displayed or hidden.** For instance, the interface can notify users when battery levels necessitate immediate RTH actions and explain the dynamic calculation of thresholds.

*Improving Warning Saliency.* The adaptive interface's ability to highlight warnings was found valuable, as removing the 2D screen forced pilots to look at the drone, but participants suggested stronger notifications than the ones we used. Participants made several suggestions such as visualizing the entire mission path with different colors to indicate the GPS loss or using audio alarms as in some commercial systems they were using. According to Woodward et al. [53], primary information must be in the field of view to capture attention. For secondary information, having world-anchored elements helps users by letting them decide when to look at the information without distracting them. Future AR applications for drone piloting should **enhance the saliency of safety warnings in the AR interface by using distinct visual or auditory cues and positioning critical information close to the drone or in their field of view, but without obstructing the direct view of the drone.** This approach ensures that pilots can promptly recognize and respond to safety issues.

## 6.3 Threats to Validity

Most professional pilots in our study are not committed in building inspections, with only one (D7) having experience. However, participatory design workshops and iterative design phases included pilots with over 15 years of experience, ensuring the validity of our

findings. For the comparative study, we recruited amateur pilots with drone expertise, such as extensive use for photography, test flights, or competitions, as professional pilots were difficult to recruit. Future research could benefit from a more diverse participant pool to better compare professional and recreational pilots.

While our study gave significant differences for subjective performance measures, the objective performance measures showed no significant difference. Using G*Power 3.1, we calculated the desired sample size for the marked percentage based on our existing data's effect size, which indicated a sample of 11. With 15 participants, the lack of significance is likely due to non-normal data distribution, limiting the sensitivity of the ANOVA test. Increasing the number of participants may yield significant differences. However, our primary goal was to examine the effect of our design on five factors (cognitive load, Situational Awareness, Subjective Performance, Safety Awareness, and Confidence) in a realistic building inspection scenario. Regarding Situational Awareness we chose the SART questionnaire as it is quick and easy to administer after the task but it has been found to correlate with performances and not only situational awareness [43]. Using freezing techniques such as SAGAT was not possible in our case since it would interrupt the task and lead to degraded pilot SA and additional efforts to restart. Future studies could focus on the effect of adaptive AR interfaces on piloting performance more specifically and leverage specific during-the-task questions to assess Situational awareness.

Our study was conducted in a simulated VR environment but real-world AR conditions pose additional challenges, such as adapting to varying lighting conditions that affect rendering fidelity. Optical see-through devices like HoloLens 2 struggle in bright light, and video see-through systems like Meta Quest 3 require sufficient ambient light. Furthermore, accurate positioning and tracking of virtual objects in our study relied on technologies like Real-Time Kinematics [10] and computer vision [33], though environmental changes can cause drift, impacting results. These challenges highlight the need for further research to address real-world discrepancies in AR applications.

## 7 Conclusion

This paper explores the potential of adaptive AR interfaces to support safety and efficiency in drone facade inspections. We addressed the challenges of managing pilot information overload and using AR in safety-critical situations. Through participatory design workshops and iterative testing with professional pilots, we developed an adaptive AR system that prioritizes information based on task urgency. This system was implemented and evaluated in a VR drone inspection simulator, comparing it against non-adaptive AR and traditional 2D interfaces.

The results reveal that the adaptive AR interface significantly reduces cognitive load and improves situational awareness compared to traditional 2D interfaces and non-adaptive AR interfaces. Key findings from our study suggest that incorporating adaptive design elements can optimize cognitive load by reducing attention switches, enhancing visibility and informativeness, and streamlining the semi-autonomous operation.
Our findings highlight key factors for improving the safety support of AR interfaces. Enhancing the transparency and explainability of the AR system's adaptive behaviors can increase pilot confidence. Additionally, making safety warnings and adaptive behaviors more salient through distinct visual cues ensures that pilots can quickly recognize and respond to potential hazards.

We believe that the principles of adaptive AR interface design described in this paper could be applied to other drone-related areas, such as search and rescue missions, traffic management, and human-swarm interaction, where managing information overload and enhancing safety are equally critical.

Future work will explore customization options and interactions that allow pilots to control adaptive behaviors, addressing both trust and performance requirements. Additionally, we aim to validate the effectiveness of adaptive AR interfaces in real-world scenarios.

## Acknowledgments

We are thankful for the support that this project has received from the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. We would also like to thank all the pilots that accepted to share their time and ideas about their work, in particular, Gautier, Murat, Fabien, Xavier, Florian, Ashley, Wan Yi and Zheng Hao. We extend our gratitude to the members of Synteraction Lab for their valuable feedback in different stages of our research.

## References

[1] Hussein A. Abbass, Jason Scholz, and Darryn J. Reid. 2018. *Foundations of Trusted Autonomy: An Introduction*. Springer International Publishing, Cham, 1–12. doi:10.1007/978-3-319-64816-3_1

[2] Dany Naser Addin and Benoit Ozell. 2021. Design and test of an adaptive augmented reality interface to manage systems to assist critical missions. *arXiv preprint arXiv:2103.14160* (2021).

[3] Naveed Anwar, Muhammad Amir Izhar, and Fawad Ahmed Najam. 2018. Construction monitoring and reporting using drones and unmanned aerial vehicles (UAVs). In *The Tenth International Conference on Construction in the 21st Century (CITC-10)* (Colombo, Sri Lanka). CITC Global, Colombo, Sri Lanka, 8 pages.

[4] Paul Ayres and Gabriele Cierniak. 2012. *Split-Attention Effect*. Springer US, Boston, MA, 3172–3175. doi:10.1007/978-1-4419-1428-6_19

[5] Suneel Belkhale, Alex Li, Gefen Kohavi, Murtaza Dalal, Daniel Ho, Franklin Rice, Allan Levy, Alex Chan, Kevin Yang, Valmik Prabhu, Asabella Maceda, Erin Song, Dilan Bhalla, Asad Abbasi, and Jason Kim. 2018. Unity Drone Simulator. https://github.com/UAVs-at-Berkeley/UnityDroneSim. A drone created for Unity with realistic drone physics, intended for Reinforcement Learning Simulation. Project AutoQuad, Spring 2018.

[6] Grzegorz Borowik, Monika Koźdón-Debecka, and Sebastian Strzelecki. 2022. Mutable Observation Used by Television Drone Pilots: Efficiency of Aerial Filming Regarding the Quality of Completed Shots. *Electronics* 11, 23 (Jan. 2022), 3881. doi:10.3390/electronics11233881 Number: 23 Publisher: Multidisciplinary Digital Publishing Institute.

[7] Josef Buchner, Katja Buntins, and Michael Kerres. 2022. The impact of augmented reality on cognitive load and performance: A systematic review. *Journal of Computer Assisted Learning* 38, 1 (2022), 285–303. doi:10.1111/jcal.12617

[8] Building and Construction Authority. 2022. Guidelines on Periodic Facade Inspection. https://www1.bca.gov.sg/docs/default-source/docs-corporate/building-control/periodic-facade-inspection/pfi-guidelines-for-competent-person.pdf

[9] Linfeng Chen, Kazuki Takashima, Kazuyuki Fujita, and Yoshifumi Kitamura. 2021. PinpointFly: An Egocentric Position-control Drone Interface using Mobile AR. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21)*. Association for Computing Machinery, New York, NY, USA, Article 150, 13 pages. doi:10.1145/3411764.3445110

[10] Francesco De Pace and Hannes Kaufmann. 2023. A systematic evaluation of an RTK-GPS device for wearable augmented reality. *Virtual Reality* 27, 4 (2023), 3165–3179.

[11] Tilman Deuschel and Ted Scully. 2016. On the Importance of Spatial Perception for the Design of Adaptive User Interfaces. In *2016 IEEE 10th International Conference on Self-Adaptive and Self-Organizing Systems (SASO) (Augsburg, Germany)*. 70–79. doi:10.1109/SASO.2016.13

[12] DJI. 2018. DJI GS Pro User Manual V2.0. https://dl.djicdn.com/downloads/groundstation_pro/20181102/GS_Pro_User_Manual_v2.0_EN_201811.pdf

[13] DJI. 2021. DJI GO 4. https://www.dji.com/sg/downloads/djiapp/dji-go-4

[14] Mica R Endsley, Randy Sollenberger, and Earl Stein. 2000. Situation awareness: A comparison of measures. In *Proceedings of the Human Performance, Situation Awareness and Automation: User-centered Design for the New Millennium*. SA Technologies Savannah, GA 3A, 15–19.

[15] SPH Engineering. 2022. Application: Vertical inspections with automatic vertical scan tool of UgCS. https://www.sphengineering.com/news/vertical-inspections-with-automatic-vertical-scan-tool-of-ugcs

[16] Meta for Work. 2024. What's the difference between AR, VR and MR. https://forwork.meta.com/sg/blog/difference-between-vr-ar-and-mr/

[17] Jérémie Garcia, Anke Brock, Nicolas Saporito, Gautier Hattenberger, Xavier Paris, Michel Gorraz, and Yannick Jestin. 2019. Designing human-drone interactions with the Paparazzi UAV System. In *1st International Workshop on Human-Drone Interaction* (Glasgow, United Kingdom). Ecole Nationale de l'Aviation Civile [ENAC]. https://hal.archives-ouvertes.fr/hal-02128390

[18] Srijeet Halder and Kereshmeh Afsari. 2023. Robots in inspection and monitoring of buildings and infrastructure: A systematic review. *Applied Sciences* 13, 4 (2023), 2304.

[19] Hooman Hedayati, Michael Walker, and Daniel Szafir. 2018. Improving collocated robot teleoperation with augmented reality. In *Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction* (Chicago, IL, USA). 78–86.

[20] Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, et al. 2003. Technology probes: inspiring design for and with families. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Fort Lauderdale, FL, USA). 17–24.

[21] Maakito Inoue, Kazuki Takashima, Kazuyuki Fujita, and Yoshifumi Kitamura. 2023. BirdViewAR: Surroundings-aware Remote Drone Piloting Using an Augmented Third-person Perspective. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems* (Hamburg, Germany). 1–19.

[22] JASP Team. 2024. JASP (Version 0.19.0)[Computer software]. https://jasp-stats.org/

[23] SeungJun Kim and Anind K Dey. 2009. Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Boston, Massachusetts). 133–142.

[24] Talia Lavie and Joachim Meyer. 2010. Benefits and costs of adaptive user interfaces. *International Journal of Human-Computer Studies* 68, 8 (2010), 508–524. doi:10.1016/j.ijhcs.2010.01.004 Measuring the Impact of Personalization and Recommendation on User Behaviour.

[25] Kisu Lee, Goopyo Hong, Lee Sael, Sanghyo Lee, and Ha Young Kim. 2020. MultiDefectNet: Multi-Class Defect Detection of Building Façade Based on Deep Convolutional Neural Network. *Sustainability* 12, 22 (Jan. 2020), 9785. doi:10.3390/su12229785 Number: 22 Publisher: Multidisciplinary Digital Publishing Institute.

[26] Wen-Chin Li, Jingyi Zhang, Samuel Court, Peter Kearney, and Graham Braithwaite. 2022. The influence of augmented reality interaction design on Pilot's perceived workload and situation awareness. *International Journal of Industrial Ergonomics* 92 (2022), 103382.

[27] Xiangdong Li, Cindy Zheng, Zhenghua Pan, Zhongnan Huang, Yuting Niu, Pengfei Wang, and Weidong Geng. 2023. Comparative Study on 2D and 3D User Interface for Eliminating Cognitive Loads in Augmented Reality Repetitive Tasks. *International Journal of Human-Computer Interaction* (2023), 1–17.

[28] David Lindbauer, Anna Maria Feit, and Otmar Hilliges. 2019. Context-Aware Online Adaptation of Mixed Reality Interfaces. In *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology* (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 147–160. doi:10.1145/3332165.3347945

[29] Sara Ljungblad, Yemao Man, Mehmet Aydin Baytaş, Mafalda Gamboa, Mohammad Obaid, and Morten Fjeld. 2021. What Matters in Professional Drone Pilots' Practice? An Interview Study to Understand the Complexity of Their Work and Inform Human-Drone Interaction Research. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21)*. Association for Computing Machinery, New York, NY, USA, Article 159, 16 pages. doi:10.1145/3411764.3445737

[30] Feiyu Lu and Yan Xu. 2022. Exploring Spatial UI Transition Mechanisms with Head-Worn Augmented Reality. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 550, 16 pages. doi:10.1145/3491102.3517723

[31] Joseph B Lyons. 2013. Being transparent about transparency: A model for human-robot interaction. In *2013 AAAI spring symposium series.*
[32] Richard E. Mayer. 2009. *Temporal Contiguity Principle*. Cambridge University Press, 153–170.

[33] Niluthpol Chowdhury Mithun, Kshitij S. Minhas, Han-Pang Chiu, Taragay Oskiper, Mikhail Sizintsev, Supun Samarasekera, and Rakesh Kumar. 2023. Cross-View Visual Geo-Localization for Outdoor Augmented Reality. In *2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)*. IEEE, 493–502. doi:10.1109/vr55154.2023.00064

[34] NASA. 2023. Cognitive Workload Assessment Methods, OCHMO-TB-032, Rev C. https://www.nasa.gov/wp-content/uploads/2023/12/ochmo-tb-032-cognitive-workload.pdf

[35] SS Muhammad Nizam, Rimaniza Zainal Abidin, Nurhazarifah Che Hashim, Meng Chun Lam, Haslina Arshad, and NAA Majid. 2018. A review of multimodal interaction technique in augmented reality environment. *Int. J. Adv. Sci. Eng. Inf. Technol* 8, 4-2 (2018), 1460.

[36] Ken Pfeuffer, Yasmeen Abdrabou, Augusto Esteves, Radiah Rivu, Yomna Abdelrahman, Stefanie Meitner, Amr Saadi, and Florian Alt. 2021. ARTention: A design space for gaze-adaptive user interfaces in augmented reality. *Computers & Graphics* 95 (2021), 1–12.

[37] Provalis Research. 2023. QDA Miner Lite. https://provalisresearch.com/products/qualitative-data-analysis-software/freeware/

[38] Hoda Rahmani and Gary R. Weckman. 2023. Working under the Shadow of Drones: Investigating Occupational Safety Hazards among Commercial Drone Pilots. *IISE Transactions on Occupational Ergonomics and Human Factors* 0, 0 (2023), 1–13. doi:10.1080/24725838.2023.2251009 Publisher: Taylor & Francis.

[39] Balita Heriniaina Rakotonarivo, Nicolas Drougard, Stéphane Conversy, and Jérémie Garcia. 2023. Cleared for Safe Take-off? Improving the Usability of Mission Preparation to Mitigate the Safety Risks of Drone Operations. In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems*. 1–17.

[40] Ashwin Ram and Shengdong Zhao. 2021. LSVP: Towards Effective On-the-go Video Learning Using Optical Head-Mounted Displays. *Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies* 5, 1 (2021), 1–27.

[41] Summer Rebensky, Meredith Carroll, Winston Bennett, and Xueyu Hu. 2022. Impact of Heads-Up Displays on Small Unmanned Aircraft System Operator Situation Awareness and Performance: A Simulated Study. *International Journal of Human–Computer Interaction* 38, 5 (March 2022), 419–431. doi:10.1080/10447318.2021.1948683 Publisher: Taylor & Francis.

[42] Alan H Roscoe and Georges A Ellis. 1990. *A subjective rating scale for assessing pilot workload in flight: A decade of practical use*. Royal Aerospace Establishment.

[43] Paul M Salmon, Neville A Stanton, Guy H Walker, Daniel Jenkins, Darshna Ladva, Laura Rafferty, and Mark Young. 2009. Measuring Situation Awareness in complex systems: Comparison of measures study. *International Journal of Industrial Ergonomics* 39, 3 (2009), 490–500.

[44] Douglas Schuler and Aki Namioka. 1993. *Participatory design: Principles and practices*. CRC press.

[45] Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips, and Nicolas Roussel. 2006. User interface façades: towards fully adaptable user interfaces. In *Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (Montreux, Switzerland) (UIST '06)*. Association for Computing Machinery, New York, NY, USA, 309–318. doi:10.1145/1166253.1166301

[46] Ryo Suzuki, Adnan Karim, Tian Xia, Hooman Hedayati, and Nicolai Marquardt. 2022. Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems* (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 553, 33 pages. doi:10.1145/3491102.3517719

[47] Daniel Szafir, Bilge Mutlu, and Terry Fong. 2015. Communicating directionality in flying robots. In *Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction*. 19–26.

[48] Markus Tatzgern, Valeria Orso, Denis Kalkofen, Giulio Jacucci, Luciano Gambieri, and Dieter Schmalstieg. 2016. Adaptive information density for augmented reality displays. In *2016 IEEE Virtual Reality (VR)*. 83–92. doi:10.1109/VR.2016.7504691

[49] Colin R Theodore, Carlos A Malpica, Christopher L Blanken, Mark B Tischler, Ben Lawrence, James E Lindsey, and Tom Berger. 2014. Effect of control system augmentation on handling qualities and task performance in good and degraded visual environments. In *AHS (American Helicopter Society) Annual Forum and Technology Display (Forum 70)*.

[50] Juhani E Tuovinen and John Sweller. 1999. A comparison of cognitive load associated with discovery learning and worked examples. *Journal of educational psychology* 91, 2 (1999), 334.

[51] Michael Walker, Hooman Hedayati, Jennifer Lee, and Daniel Szafir. 2018. Communicating robot motion intent with augmented reality. In *Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction*. 316–324.

[52] Michael E. Walker, Hooman Hedayati, and Daniel Szafir. 2019. Robot Teleoperation with Augmented Reality Virtual Surrogates. In *2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)* (Daegu, Korea). 202–210. doi:10.1109/HRI.2019.8673306 ISSN: 2167-2148.

[53] Julia Woodward, Jesse Smith, Isaac Wang, Sofia Cuenca, and Jaime Ruiz. 2023. Designing Critical and Secondary Information in Augmented Reality Headsets for Situational Awareness. *Journal of Engineering Research and Sciences* 2, 3 (2023).

[54] John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through Design as a Method for Interaction Design Research in HCI. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (San Jose, California, USA) (CHI '07). Association for Computing Machinery, New York, NY, USA, 493–502. doi:10.1145/1240624.1240704

[55] Stefanie Zollmann, Christof Hoppe, Tobias Langlotz, and Gerhard Reitmayr. 2014. FlyAR: Augmented Reality Supported Micro Aerial Vehicle Navigation. *IEEE Transactions on Visualization and Computer Graphics* 20, 4 (April 2014), 560–568. doi:10.1109/TVCG.2014.24

## A Post-Study Questionnaire

*Cognitive Load*. The Bedford Workload Rating scale used in our study is shown in Figure 14.

<figure><img src="image_16.png" /><figcaption>Figure 14: Bedford Workload Rating (adapted from [49]).</figcaption></figure>

*Situational Awareness*. The SART questionnaire consists of 3 different categories of situational awareness and 10 questions in total, each rated on basis of a 7 point Likert scale.

### (1) Demand

* Instability of Situation: How changeable is the situation? Is the situation highly unstable and likely to change suddenly (high) or is it very stable and straightforward (Low)?

* Complexity of Situation: How complicated is the situation? Is it complex with many interrelated components (high) or is it simple and straightforward (low)?

* Variability of Situation: How many variables are changing within the situation? Are there a large number of factors
varying (high) or are there very few variables changing (low)?

(2) Supply

* Arousal: How aroused are you in the situation? Are you alert and ready for activity (high) or do you have a low degree of alertness (low)?

* Concentration of Attention: How much are you concentrating on the situation? Are you concentrating on many aspects of the situation (high) or focused on only one (low)?

* Division of Attention: How much is your attention divided in the situation? Are you concentrating on many aspects of the situation (high) or focused on only one (low)?

* Spare Mental Capacity: How much mental capacity do you have to spare in the situation? Do you have sufficient to attend to many variables (high) or nothing to spare at all (low)?

(3) Understanding

* Information Quantity: How much information have you gained about the situation? Have you received and understood a great deal of knowledge (high) or very little (low)?

* Information Quality: How accessible and usable is the information you have in this situation? Is the knowledge communicated very useful (high) or is it not usable at all (low)?

* Familiarity with Situation: How familiar are you with the situation? Do you have a great deal of relevant experience (high) or is it a new situation (low)?

*Other Questions.* Similarly, these questions were rated on the basis of a 7-point Likert scale.

* How do you rate your performance with the current interface?

* How do you rate the current interface in terms of the accessibility of the safety information?

* How confident are you with using the current interface to conduct building inspection mission by yourself in the real world?