Dynamic InfoGraphics (DIG) Capstone: A Concept for Transparency of Intelligent Agent Systems Using Augmented Reality
Authors: Hartnett Gina 
Published: November 2024

Autonomous and artificial intelligence (AI) systems are beginning to inundate almost all industries, providing humans with enormous amounts of sensed information and aid or replacement with complex and rapid decision making. The use of these systems is growing in almost all military operations, providing aid to the Soldier or replacement with complex, rapid decision making and actions that become extremely difficult for humans to quickly understand (Ha et al., 2019; Holzinger et al., 2019; Le Roux, 2010) and, if needed, intervene. It becomes challenging for humans to decipher the plethora of sensed information (Ha et al., 2019; Holzinger et al., 2019). This is true for many Army operations in which decisions, actions, and mistakes happen quickly. The rapid and complex actions these systems take often make it difficult to understand the vast amount of sensed information or the transparency of system reasoning for a specific action. Currently, in Army rotary wing platforms, automated systems make some decisions and take various actions to alleviate workload (Crawford & Neal, 2006). Often, however, no explanations are provided to the pilot or crew. This lack of reasoning frequently leads to uncertainty in users (Bobek & Nalepa, 2021; Zhang & Hußmann, 2021; Jiang et al., 2022). This uncertainty will likely exacerbate the situation when AI is introduced into the cockpit (Sutthithatip et al., 2021). This information gap and uncertainty leaves operators with decreased situation awareness (SA) and user trust in these systems, especially if the system makes an obvious error (Pomranky et al., 2001; Pak et al., 2017; Schmitt et al., 2018). With the future battlefield becoming an exceedingly complex environment that is difficult for a Soldier to fully digest alone, Soldiers will need to understand their environment as quickly as possible when working with intelligent agents (IAs). The Dynamic InfoGraphics (DIG) concept (patent # US 11960710 B1) bridges that gap by providing the user transparency and rapid understanding of autonomous or IA system actions and decisions and increase user SA and user trust. This concept combines intuitive graphics with bits of textual information that are dynamic and quickly understood by the user. The goal is to present relevant informational DIGs in augmented reality (AR) to the user. This study presents a solution that increases SA and user trust in the system and improves user workload and overall mission performance as Soldiers are given a tool to facilitate a new capability in understanding. A total of 54 rotary wing pilots, medics and crew, and civilian aviation subject matter experts (SMEs) participated in this 2 (display presentation type: DIG [in AR], text [on an aircraft inside screen]) × 2 (level of transparency: less [L1 sense and display {what}]; more [L1/sense and display {what}, L2/transparency through XAI {why}, L3/trend forecasting through trend analysis {next}]) within-subjects design study. Results indicated that the DIG presentation type outperformed the text presentation type with increased accuracy, SA, user trust, and decreased decision time and workload. In addition, the more transparency the system shows, the higher the users’ trust.