# The effect of autonomy transparency in human-robot interactions : a preliminary study on operator cognitive workload and situation awareness in multiple heterogeneous UAV management

Ting Chen, Duncan Campbell, Felipe Gonzalez, Gilles Coppin

► To cite this version:

Ting Chen, Duncan Campbell, Felipe Gonzalez, Gilles Coppin. The effect of autonomy transparency in human-robot interactions : a preliminary study on operator cognitive workload and situation awareness in multiple heterogeneous UAV management. Australasian Conference on Robotics and Automation 2014, Dec 2014, Melbourne, Australia. pp.1-10. hal-01864490

HAL Id: hal-01864490

https://hal.science/hal-01864490

Submitted on 7 Jul 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The Effect of Autonomy Transparency in Human-Robot Interactions: A Preliminary Study on Operator Cognitive Workload and Situation Awareness in Multiple Heterogeneous UAV Management

*Ting (Brendan) Chen, *Duncan Campbell, *Felipe Gonzalez, and **Gilles Coppin

*Queensland University of Technology, Australia, **Lab-STICC, Télécom Bretagne, France
tb.chen@hdr.qut.edu.au

## Abstract

The autonomous capabilities in collaborative unmanned aircraft systems are growing rapidly. Without appropriate transparency, the effectiveness of the future multiple Unmanned Aerial Vehicle (UAV) management paradigm will be significantly limited by the human agent's cognitive abilities; where the operator's Cognitive Workload (CW) and Situation Awareness (SA) will present as disproportionate. This proposes a challenge in evaluating the impact of robot autonomous capability feedback, allowing the human agent greater transparency into the robot's autonomous status - in a supervisory role. This paper presents the motivation, aim, related works, experiment theory, methodology, results and discussions, and the future work succeeding this preliminary study. The results in this paper illustrates that, with a greater transparency of a UAV's autonomous capability, an overall improvement in the subjects' cognitive abilities was evident, that is, with a confidence of 95%, the test subjects' mean CW was demonstrated to have a statistically significant reduction, while their mean SA was demonstrated to have a significant increase.

## 1 Introduction

Interest in the area of collaborative unmanned aircraft systems in a Multi-Agent System (MAS), to complement the strengths and weaknesses of the human-machine relationship is continuously growing [Ososky et al., 2014]. The effects of automation on human-machine collaboration has been studied extensively to conclude that it is able to reduce a humans Cognitive Workload (CW) [Cummings et al., 2007; Parasuraman et al., 2000]. However, as automation capabilities continue to expand - particularly in the domain of Unmanned Aerial Vehicles (UAVs), there is an increased interest in deploying multiple UAVs in a close proximity to achieve a common goal [Baranzadeh et al., 2013; Patten et al., 2013;

Cummings and Mitchell, 2008; Jarvis and Tang, 2005; Miller et al., 2001].

The *multiple-to-one* ratio (where multiple human operators are required to manage one UAV) is the current UAV operational norm. This limits the effective management of teams of UAVs [Franke et al., 2005; Brown et al., 2005] in a Multi-Agent System (MAS). Currently, a typical UAV operation requires at least two operators; a pilot and a sensor operator to operate one UAV in a mission [Murphy and Burke, 2010]. It is also not uncommon that a third mission commander is involved to overlook the entire operation [Chaput, 2004]. Hence, to enable further growth in the area of effective management of multiple heterogeneous UAVs, the management ratio must be inverted to *one-to-multiple* (where one operator is managing multiple UAVs) [Chen et al., 2014; Brown et al., 2005]. However, this inversion of the human-machine management ratio introduces a number of cognitive challenges, such as an increase in operator CW and a reduction in Situation Awareness (SA) [Chen et al., 2014].

This study aimed to evaluate the impact of UAV autonomy visualisation in an MAS through autonomy transparency, on a human operator's cognitive CW and SA. This was achieved through the management of multiple UAV experiment. The results comparing a baseline experiment (the UAVs' navigation capability was inferred based on the UAVs' behaviour) to an evaluation experiment (the UAVs' navigation capability was graphically represented through flight path/trajectory visualisation) demonstrated that with a confidence of 95%, the human operator is able to experience a reduction in CW and a gain in SA with greater transparency of the autonomy. The rest of the paper is organised as follows; Section 2 focuses on the related works in this area, section 3 focuses on the methodology, section 4 focuses on the results and analysis, section 5 focuses on the discussion of the results, and finally, section 6 presents future work that succeeds this study.

## 2 Related Work

Research studies have identified the significance of system information transparency and its effect on the human cognitive performance in multiple heterogeneous UAV manage-
ment [Chen et al., 2014; Ososky et al., 2014; Miller, 2014; Fuchs et al., 2013; Chen et al., 2011]. The fore-mentioned system transparency can be considered as a two-fold problem; the transparency of the functional subsystems and capabilities, and the transparency of the system's autonomy capabilities to the human operator [Saget et al., 2008].

## 2.1 Multi-Agent System

An MAS refers to a system involving more than one human operator, robots, and an interface device. Multiple UAV operations and management by a single operator closely relates to the traditional MAS (a subfield of artificial intelligence), as they both involve dealing with a number of entities that require making some level of decisions [Legras and Coppin, 2007].

In an MAS setting, a human-centred approach to improve the human agent's cognitive performance in human-autonomy collaboration, involves the human agent having the knowledge of the robot agent's autonomy status [Chen et al., 2014; Saget et al., 2008].

## 2.2 Information Transparency

The concept of information transparency was established to enable the effective human-machine interaction in an authority sharing MAS. In order to achieve information transparency, both the human and the machine agent must have an awareness of each others internal status as illustrated through the dual-ended interaction [Saget et al., 2008]. Figure 1 illustrates this concept with a single robot-single operator interaction as a whole, which includes authority sharing, operating modes, control and interaction.

One element of information transparency is placed on the machine representation to the human agent [Miller, 2014; Chen et al., 2014; Fuchs et al., 2013; Chen et al., 2012]. Chen et al. initially proposed a functional capability framework to capture the functionalities of a typical UAV [Chen et al., 2012], the process of Information Abstraction used to produce the framework from an operational perspective was refined [Chen et al., 2013], hence producing an experimental framework and prototype to assess the effect of a layering approach to information visualisation on a human operators CW and SA [Chen et al., 2014]. The information representation focused on three levels: 1) Low, Level Of Detail (LOD), or highest amount of system raw data, 2) Hybrid/Adjustable LOD, or a mix of raw and aggregated UAV data as preferred by the operator (test subject) and 3) High LOD, or the most abstracted data/succinct information display. Their study suggested that there was a significant reduction in operator CW when the operators were able to manage their level of system information transparency, compared to maximum information output (low LOD) at no cost to the operator's SA [Chen et al., 2014]. Hence, during multiple UAV interaction interface designs, the adjustability of the system's functional capability information and transparency must be considered.

Fuchs et al. were also interested in investigating the effect of information representation on operator CW and SA [Fuchs et al., 2013]. Their study focused on the information layout rather than adjustability or availability, where existing UAV consoles were adapted and modified through usability iterative surveys from certified UAV operators. During each iteration, modifications to the consoles were carried out to improve the CW and SA (which was measured using SA Global Assessment Technique (SAGAT) [Endsley, 1988] and NASA-TLX [Hart and Staveland, 1988]). Their results demonstrated a positive impact to the CW and SA of the operator through information layout modification.

## 2.3 Autonomy Transparency

Recently, autonomy transparency was recognised as having a tremendous impact on operator SA, CW and trust [Miller, 2014; Chen et al., 2014; 2011]. Traditionally, in a Human-Robot Interaction (HRI) environment, the robot was considered a remote tool for the human agent to complete a mission or a task, however many, Ososky et al. [Ososky et al., 2014] viewed a robot or multiple robots as teammate(s) [Phillips et al., 2011]. To draw upon the strengths of each teammate (humans and robots), a bi-directional communication channel during task execution is required and is currently a limitation of modern-day HRI systems [Ososky et al., 2014].

In this work, we evaluated the effect of the visualisation of autonomy (increase in transparency) on a human operator's CW and SA in the context of handling multiple UAVs through a hypothetical disaster response search mission (figure 2). The visualisation directly communicates the LOA's implication on the UAVs' autonomous capability to the operator, hence enabling the operator to more effectively interact with the UAVs.

## 2.4 Functional Level Of Autonomy

The need to quantify autonomy was first identified by Sheridan [Sheridan and Verplanck et al., 1978], where the ten levels of Sheridan and Verplanck's (SV), LOA scale was proposed [Sheridan and Verplanck et al., 1978]. This is a uni-dimensional scale describing all aspects of HRI in a single dimension. Parasuraman et al. [Parasuraman et al., 2000] had identified that during HRI, the humans information process is very important, hence, a multidimensional LOA scale incorporating a simplified four-stage, human information process model was proposed, where the LOA of each stage was quantified using the SV scale.

Bruni et al. [Bruni et al., 2012] proposed a taxonomy to quantify autonomy through the perspective of human-automation collaboration to produce the Human-Automation Collaboration Taxonomy (HACT). HACT describes the human-automation relationship through the collaborative information-processing model, with the emphasis placed on the decision-making process through three basic roles [Bruni et al., 2012]; the moderator - the agent that main-
<figure><img src="image_2.png" /><figcaption>Figure 1: Authority sharing concepts in single robot-single operator interactions (Figure and caption adapted from [Saget et al., 2008]). Situation information is perceived by both the robot and the human to establish a representation of each agent. Interaction is improved by the communication of these representations with the other agent, hence, a collaborative understanding of the control operation modes is established.</figcaption></figure>

tains the forward-moving aspect of the decision-making process: the generator - the agent that generates feasible solutions based on the information and environmental data; the decider - the agent who determines which solution is to be selected and carried out. The agent mentioned, denotes either an autonomous agent, or a human agent, where five levels of collaboration between these agents are applied to each role.

## 2.5 Functional Capability Information

The additional experimental software and capabilities of the work developed by Chen et al. [Chen et al., 2014; 2013; 2012] were introduced. This included a functional capability framework using the information abstraction method, which enabled the subject to gain unrestricted access to the necessary UAV's system functions and status through specific gestures.

## 2.6 Autonomy Capability Information

The experimental software was augmented to measure the effect of the autonomy information transparency. Emphasis was placed on the autonomous navigation capability feedback.

Figure 4 illustrates the visualisation of a UAV's flight path in both the baseline and evaluation experiment. Figure 4a illustrated no representation of any form for the baseline experiment, as no transparency was intended. However, figure 4b

<figure><img src="image_3.png" /><figcaption>Figure 3: Representations of the three levels of searching autonomy capability; (a) UAV 1: Fully autonomous - auto identification and selection of the Items of Interest (IOIs), (b) UAV 3: Partially autonomous - auto identification, manual selection of the IOIs, and (c) UAV 2: No autonomy - Manual identification and selection of the IOIs.</figcaption></figure>

illustrated the visual representation of the immediate flight path of the UAVs for the evaluation experiment, where the UAVs encountered obstacles, and amendments to the flight path were required and carried out. The subject could register the change immediately, hence, awareness of each UAV's autonomous navigation capability could be achieved.

Autonomy was not only applied to the navigation capability, but also to autonomous searching. Each UAV had a certain level of searching autonomy where human intervention was required at three levels and depicted graphically as illustrated in figure 3; full autonomy, denoted by a solid circle in the centre of the UAV icon; part autonomy, denoted by a hollow circle in the centre of the UAV icon; and no autonomy, denoted by the absence of any circle/marking in the centre of the UAV icon.
<figure><img src="image_4.png" /><figcaption>Figure 2: A screen capture of the typical interface view of the experiment software prototype. A base map is illustrated with two UAVs (ID 1 and 2) following a pre-defined flight path. *UAV1* is displaying two levels of status and health information, while *UAV2* is displaying one level with a search instrument/scope opened.</figcaption></figure>

Figure 3a (UAV1) shows a UAV with no autonomous searching capabilities, which meant that all the identifying and selecting of Items Of Interest (IOIs) needs to be performed manually by the human operator. Figure 3b (UAV3) shows a UAV with partial autonomous searching capabilities, which meant that the identification of the items of interests were performed by the UAV autonomously, however, the human operator was still required to make an appropriate selection as false identification could have occurred. Finally, figure 3c (UAV2) shows a UAV with full autonomous searching capabilities, where the UAV has the capability to perform the identifying and selecting of the items of interests autonomously without human intervention.

# 3 Methodology

This section presents a general overview of the experiment, the apparatus and setup, the experiment mission scenario, the participants (test subjects) and the data collection instrument and process.

## 3.1 Overview

The aim of the experiment was to investigate the impact on a human agent's CW and SA associated through a continuous visual representation of each UAV's flight path when compared to no representation. The continuous visual representation

is hypothesised to provide a greater autonomy transparency of the UAV's navigation capability.

This experiment evaluates the aim through a simulated search operation during a disaster response mission in a game-like style set up on a touch sensitive tabletop device, where the operators (test subjects) were required to monitor UAVs and perform searches of the designated areas through tools provided in the experiment prototype.

As illustrated in figure 4, the key concept of the experiment investigated a comparison between visually representing the flight path of UAVs in an evaluation experiment (figure 4b), which resembled transparency in the autonomous flight path replanning capability - and no such representation in a baseline experiment (figure 4a).

This experiment required one operator (test subject) to concurrently monitor and manage four UAVs. The UAVs were heterogeneous in their functional LOAs, that is, each UAV contained a different level of functional autonomy. The amount of LOA transparency was presented in two levels with the focus placed on the UAV's autonomous path-planning capability, and the basic concept of LOA is explained to the test subject during the pre-experiment briefing, while the practical component, that is, what does the LOA look like and what does it mean to the test subject is explained in the equipment familiarisation phase after the briefing.

This experiment was carried out in two parts; part A re-
<figure><img src="image_5.png" /><figcaption>Figure 4: Illustration showing the conceptual difference between a configuration with the (a) The flight path and trajectory is hidden from the operator, hence autonomy is non-transparent; and (b) The intended and and newly-generated flight trajectory and path is visually represented, hence autonomy is transparent.</figcaption></figure>

required the operator to perform under no transparency, that is, no visual representation of the UAV's previous, present or projected flight path was displayed at any time during the experiment (figure 4); part B required the operator to perform with some level of automation transparency, that is, a graphical representation of the UAV's previous and current flight path was displayed to allow the operator to have an awareness of the UAV's path-planning capabilities.

The authors hypothesised a reduction in CW and an increase in SA to be exhibited in part B compared to part A. This is because the UAV's autonomous capabilities are more transparent, allowing the operator to gain a greater awareness of the UAV's autonomous status.

The experiment was conducted at the Australian Research Centre for Aerospace Automation (ARCAA), which is an off-site research facility of the Queensland University of Technology (QUT). The Interactive Systems Group (ISG) supplied the necessary apparatus to conduct the experiment. The experiment was designed to take approximately 90 minutes to complete. The experiment required each participant to perform two similar missions and answer the SA questionnaires and CW rating forms.

During the course of the 90 minutes, the subject was presented with a sequence of events as described below, and their session was video and audio recorded for further analysis:

* Experiment briefing - 10 min

* Equipment familiarisation - 20 min

* Experiment Part A: Baseline - 10 min

* CW/SA assessment Part A - 15 min

* Experiment Part B: Evaluation - 10 min

<figure><img src="image_6.png" /><figcaption>Figure 5: Experiment setup illustrating the DT104 touch table and the experiment prototype.</figcaption></figure>

* CW/SA assessment Part B - 15 min

* Post experiment interview - 10 min

This experiment was approved by the QUT Ethics Committee and each subject was asked to read and sign the ethics agreement prior to commencing the experiment.

## 3.2 Apparatus

The experiment software prototype was designed in an arcade game-like style for an interactive tabletop device. The prototype was implemented using the Java programming language and the MT4J Java multi-touch software framework [Laufs et al., 2010]. The experiment was conducted on a CircleTwelve DiamondTouch DT104 (figure 5) [Dietz and Leigh, 2001].

## 3.3 Mission Scenario

There are two similar, but disjointed mission scenarios included in this experiment. Both scenarios required the operators (test subjects) to first supervise a group of four UAVs with heterogeneous functional LOAs in their respective search zones, and then carry out a search operation over the zones.
The two tasks were performed in different phases. The objective of phase 1 was to allow the subject to acquire and develop SA about the UAVs and their surroundings. The objective of phase 2 was to monitor the UAVs which were able to autonomously perform a search and for the operator to assist with the less autonomous UAVs with the searching operation.

During phase 1, certain UAVs are expected to encounter obstacles such as hazardous weather and a malfunctioning of a subsystem (ie. low fuel levels). The subject was not only required to acknowledge these obstacles but also be aware of the subsequent evasive actions the UAVs took or failed to take prior to proceeding to phase 2, which was to carry out the search for several target operations.

During phase 2, certain UAVs have less autonomy to perform a search of IOIs than others. At different LOAs, certain UAVs required greater assistance from the subject to search, identify, and confirm the IOIs, while other UAVs were able to carry out this chain of operations completely autonomously. Upon completion of phase 2, the UAVs leave the experiment arena and the scenario ends.

## 3.4 Test Subject

Forty three (43) subjects participated in the experiment. They were recruited primarily from aerospace companies; these included engineers, technicians, as well as a minority of university students and professionals who had minimal to no experience in any form of UAV operations. The operators were given time to familiarise themselves with the experimental platform (section 3.1). The subjects ranged from a variety of ethnic backgrounds and were primarily aged between 24 to 44. One requirement for the experiment was that the subjects not have any form of colour-blindness, as the experiment primarily uses colours to distinguish the UAV system and autonomy transparency states.

## 3.5 Data Collection

The experiment assessed two quantifiable parts and one qualitative part. The two quantifiable parts were the CW - collected on a paper-based rating form, and SA - collected on a web-based questionnaire package at the end of each part of the experiment. The qualitative, or observational measure is a behavioural-type interview conducted prior to the end of the session.

To ensure statistical equivalence, the former half of the test subjects commenced the experiment with the baseline scenario first, followed by the evaluation scenario; while the latter half of the test subjects commenced with the evaluation scenario, followed by the baseline scenario. This method can minimise any effect of learning and familiarisation of the software and apparatus, which can contribute to bias in the data collection process.

The CW was assessed using NASA-TLX (Task Load Index), which considers CW as six dimensions [Hart and Stave-land, 1988]; 1) Mental demand, 2) Physical demand, 3) Tem-

poral demand, 4) Own performance, 5) Effort, and 6) Frustration. The assessment began by laying out 15 combination pairs where each dimension was separately compared to the next, and the subject was asked to circle the dimension in each pair that he/she considered more relevant to the tasks in the experiment, which resulted in a weight associated with each dimension. The subject was then asked to rate each dimension separately on a scale of 1 to 20, where the higher the number, the more CW the subject perceived. After the completion of these two steps, a TLX score in percentage was achieved.

The SA was assessed using the SAGAT, which is an objective measure of the subject's SA [Endsley, 1988]. This technique was tailored to Endsley's three levels of SA; where Level 1) Perception, Level 2) Comprehension, and Level 3) Projection - with the assumption that to satisfy the current level of SA, the level is assumed to be satisfied [Endsley, 1995]. During data collection, a questionnaire (with ten questions) designed specifically for each experiment was administered to the subjects on conclusion of each experiment. The questions were designed based on the UAV's situational status and the environmental impact, such as *Which UAV reported a low fuel level?* and *What was the purpose of UAV2 deviating from its original path?* Only the first two levels of SA were considered during the design of the questionnaires as the aim of this study was to understand how autonomy transparency affects an operators awareness of the UAV's capabilities, rather than the implications of what their capabilities might bring.

The post-experiment interview was conducted in an informal setting, where the experimenter asked the subject broad questions based on their observations during the experiment. The subject was encouraged to talk freely about his/her experience and any thoughts he/she might have accumulated during the experiment. Example questions asked included; *I saw you looked a little confused at a new flight path when the UAV generated it, what was going through your mind at the time?, What was your process of maintaining updated awareness of all of the UAV's searching progress?* From the interviews, the experimenter was able to acquire a greater understanding of the subjects behaviour and their CW and SA reported in the collected quantitative data.

# 4 Results and Analysis

Results of the CW and SA for a sample size of 43 subjects were analysed using the statistical software package SPSS.

## 4.1 Cognitive Workload

The Shapiro-Wilk's test of normality for the data sets revealed that, although the CW data collected during the baseline experiment was considered normally distributed, the data collected from the evaluation experiment was not. Hence, a non-parametric - Wilcoxon Signed-Rank Test was used to determine whether there was significant statistical evidence to
<table><caption>Table 1: Related sample Wilcoxon Signed-Rank Test for the mean CW score with a CI = 95%</caption><thead><tr><td>p</td><td>Decision</td></tr></thead><tbody><tr><td>0.042</td><td>Reject H<sub>0</sub></td></tr></tbody></table>

<table><caption>Table 2: One tailed test for the mean CW score with a CI = 95%</caption><thead><tr><td>p</td><td>x&#772;<sub>base</sub></td><td>x&#772;<sub>eval</sub></td><td>Decision</td></tr></thead><tbody><tr><td>0.021</td><td>55.86</td><td>51.95</td><td>Reject H<sub>0</sub></td></tr></tbody></table>

suggest a reduction in the subject's CW during the evaluation experiment (where autonomy information was available) compared to the CW experienced during the baseline experiment (no autonomy information was available - ie. no autonomy transparency).

The hypothesis states:

* $H_0$: No significant difference between the mean CW score collected from the evaluation experiment and the baseline experiment

* $H_a$: Significant difference found between the mean CW score collected from the evaluation experiment and the baseline experiment

As illustrated in Table 1, with a Confidence Interval (CI) of 95%, $p = 0.042$, one can conclude that there was significant statistical evidence to reject the null hypothesis ($H_0$), that is to say, the subject's CW exhibited significant differences when a UAV's navigation capability was visually presented in the scenario, compared to the scenario where this information was not available. Hence, a one-tailed test to determine the direction of the mean CW score can be performed.

The hypothesis for the one tailed test states:

* $H_0$: CW score for the baseline experiment <evaluation experiment

* $H_a$: CW score for the baseline experiment >evaluation experiment

As illustrated in Table 2, the significance value ($p$) of the one-tailed test is 0.021 and the mean CW of the evaluation experiment is lower than that of the baseline experiment, at CI = 95%. One can conclude that there was significant statistical evidence to reject $H_0$, that is to say, the subject's CW was significantly lower in the scenario where a UAV's navigation autonomy capability was visually represented, compared to the scenario where this information was not available.

## 4.2 Situation Awareness

The SA results were collected through a SAGAT questionnaire with the aim to capture the two levels of SA' level 1 SA (information perception) and level 2 SA (information comprehension).

<table><caption>Table 3: Two tailed t-test statistics output of the mean SA result at N = 43 samples with a CI = 95%</caption><thead><tr><td>SA Level</td><td>p</td><td>Decision</td></tr></thead><tbody><tr><td>Level 1</td><td>0.238</td><td>Accept H<sub>0</sub></td></tr><tr><td>Level 2</td><td>0.000</td><td>Reject H<sub>0</sub></td></tr><tr><td>Overall</td><td>0.000</td><td>Reject H<sub>0</sub></td></tr></tbody></table>

<table><caption>Table 4: One tailed t-test statistics of the mean SA result at N = 43 samples with a CI = 95%</caption><thead><tr><td>SA Level</td><td>x&#772;<sub>base</sub></td><td>x&#772;<sub>eval</sub></td><td>p</td><td>Decision</td></tr></thead><tbody><tr><td>Level 2</td><td>0.1860</td><td>0.5291</td><td>0.000</td><td>Reject H<sub>0</sub></td></tr><tr><td>Overall</td><td>0.2843</td><td>0.5077</td><td>0.000</td><td>Reject H<sub>0</sub></td></tr></tbody></table>

A Student T-Test was conducted to determine whether there was an increase in the SA of the evaluation experiment (where automation transparency was available) compared to the SA of the baseline experiment (where no transparency was available). The use of the T-test to produce reliable results, required that the assumptions for the test were satisfied.

The hypotheses are:

* $H_0$: No significant difference between the results collected from the evaluation experiment and the baseline experiment

* $H_a$: Significant difference found between the results collected from the evaluation experiment and the baseline experiment

Table 3 illustrates a summary of the two tailed test results conducted to compare the differences in the SA mean at the two levels of SA and the overall SA. At CI = 95%, both the level 2 SA and overall SA results exhibited significant differences in their mean, while no significant statistical differences were found at level 1 SA. Therefore, a one-tailed test was performed to assess whether the transparency configuration in the evaluation experiment exhibited greater SA.

The hypothesis for the one-tailed T-test is:

* $H_0$: SA result for the baseline experiment >evaluation experiment

* $H_a$: SA result for the baseline experiment <evaluation experiment

Table 4 illustrates a summary of the one-tailed T-test for level 2 SA and the overall SA results. At CI = 95%, both SA result comparisons rejected the null hypothesis ($H_0$), that is to say, both the level 2 SA and the overall SA, revealed that the SA obtained in the evaluation experiment where autonomy information was available to the subject, was significantly higher than the SA collected from the baseline experiment where no autonomy information was available.
The level 1 SA did not reveal an increase, while level 2 SA did suggest that the subjects were able to perceive changes reasonably well in both experiments. However, with some form of autonomy transparency available, the subjects were able to comprehend the situation with less effort (hence, a reduction in CW was evident also), resulting in a higher mean SA. Due to this increase, the overall SA (associated level 1 and level 2) also yielded a significant increase during the evaluation experiment, where the UAV's path planning autonomous capability was visually represented.

# 5 Discussions

The operators' CW and SA of both the baseline experiment (where no path-planning autonomy transparency was present) and the evaluation experiment (where autonomy transparency was available through graphical representation of flight paths of the UAVs) were collected during the experiment, analysed quantitatively through a Wilcoxon Signed-Rank Test and a Student's T-Test.

Results revealed that at CI = 95%, there was a significant decrease in the mean of the CW scores and a significant increase in the mean of the SA results experienced by the test subject when the transparency of the UAV's autonomous path-planning capability was visually represented on the experiment prototype. This allowed the subject to acquire most instantaneous information related to any changes in the current path to avoid possible hazards or conflict.

Through the post-experiment interview, several interesting observations and reports were made by the subject, in particular, the notion of *predictability*. A large portion of the participants reported that with the flight path information readily available, they were able to better *predict* the intentions of the UAVs and any changes in the flight path instantaneously, as opposed to the experiment that did not have this information. The subjects generally commented that a large interactive surface such as the tabletop used in this experiment was more intuitive and easy-to-use than the traditional desktop computer systems for this scale of multiple UAV management operations, as they had the display real-estate to arrange the interface at their leisure.

Hence, the preliminary results from this study had shown a potential of improving human cognitive performance during the management of multiple heterogeneous UAVs with a certain level of autonomy information transparency. Further studies can be carried out to enhance the understanding of the impact of autonomy transparency.

# 6 Conclusion and Future Work

The advancements of the autonomy capabilities in UAVs have motivated a shift in the supervisory control paradigm of multiple heterogeneous UAVs, managed by a single-operator ratio [Franke et al., 2005] in a human-in-the-loop involvement configuration, where UAVs are viewed as teammates in

any mission rather than a tool [Ososky et al., 2014]. However, without the appropriate support, the aforementioned paradigm of interaction heavily reduces the human cognitive abilities [Chen et al., 2014], hence, the system's autonomy transparency is being studied [Miller, 2014; Chen et al., 2014; Fuchs et al., 2013; Saget et al., 2008].

The preliminary results presented in this paper demonstrated a significant and positive improvement in the human subject's CW and SA when the UAV's navigation LOA is communicated through the continuous visual feedback of the flight path's information. The authors hypothesised a reduction in CW and an increase in SA when the test subjects were given the path information through its visualisation.

An experiment involved 43 non-professional UAV operators who simultaneously managed four heterogeneous UAVs on a hypothetical disaster response search mission, demonstrated that at CI = 95%, both the mean CW of the subject and their mean SA improved significantly. This suggested that through a controlled study, evidence supported the notion that the autonomy transparency of a system is linked to the operator's cognitive performance when managing multiple UAVs.

Further investigation into the communication of the UAV's autonomy capability information (that is not only limited to the navigation autonomy) is currently underway, where the UAV's autonomous status model is represented to the operator in two fold; 1) A graphical representation [Schneiderman, 2000] similar to the method presented in this paper, and 2) A natural language dialogue [Lemon et al., 2002] where autonomous capability status information is communicated to the operator in a message box through simple English. The latter will be a bi-directional communication, where the operators will be asked to physically acknowledge any incoming system messages, and use the information contained in the message to manage and supervise the UAVs.

The authors aim to provide evidence that suggests the criticality for a human operator to possess a mental model of the system's autonomy status, that is, the visualisation of autonomy transparency through graphical [Schneiderman, 2000] and natural language representations [Lemon et al., 2002], can directly influences the human's cognitive performance in terms of CW, SA, and trust [Parasuraman et al., 2008; Legras and Coppin, 2007].

# References

[Baranzadeh et al., 2013] Ahmad Baranzadeh. A decentralized control algorithm for target search by a multi-robot team. In *Proceedings of Australasian Conference of Robotics and Automation*, Sydney, Australia, December 2013.

[Brown et al., 2005] Shaun Brown, Jake Toh, and Salah Sukkarieh. A flexible human-robot team framework for information gathering missions In *Proceedings of Aus-
tralasian Conference of Robotics and Automation, Sydney,
Australia, December 2005.

[Bruni et al., 2012] Sylvain Bruni, Jessica J. Marquez, Amy Brzezinski, Carl Nehme, and Yves Boussemart. Introducing a Human-Automation Collaboration Taxonomy (HACT) in command and control decision-support systems In Proceedings of the 12th International Command and Control Research and Technology Symposium, Newport, RI, June 2007.

[Chaput, 2004] Armand J. Chaput Aircraft design and design laboratory [Online] http://www.ae.utexas.edu/ASE261KChaput/, 2004.

[Chen et al., 2014] Jessie Y. C. Chen, and Michael Barnes. Human-Agent teaming for multirobot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44(1):13–29, February 2014.

[Chen et al., 2014] Ting (B.) Chen, Felipe Gonzalez, Duncan Campbell, and Gilles Coppin. Management of multiple heterogeneous UAVs using capability and autonomy visualisation: Theory, experiment and result In Proceedings of International Conference on Unmanned Aircraft Systems, Orlando, Florida, May 2014.

[Chen et al., 2013] Ting (B.) Chen, Duncan Campbell, Gilles Coppin, and Felipe Gonzalez. Management of heterogeneous UAVs through a capability framework of UAV's functional autonomy In Proceedings of the 15th Australian International Aerospace Congress, Melbourne, Australia, March 2013.

[Chen et al., 2012] Ting (B.) Chen, Duncan Campbell, and Gilles Coppin, Martijn Mooij, and Felipe Gonzalez. A capability framework visualisation for multiple heterogeneous UAVs: From a mission commander's perspective In Proceedings of the 28th International Congress of the Aeronautical Sciences, Brisbane, Australia, October 2012.

[Chen et al., 2011] Jessie Y. C. Chen, Michael J. Barnes, and Michelle Harper-Sciarini Supervisory control of multiple robots: Human-performance issues and user-interface design IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews, 41(4):435–454, July 2011.

[Coppin et al., 2009] Gilles Coppin, François Legras, and Sylvie Saget Supervision of autonomous vehicles: Mutual modeling and interaction management Engineering Psychology and Cognitive Ergonomics, 489–497, May 2009.

[Cummings and Mitchell, 2008] Mary L. Cummings, and Paul J. Mitchell. Predicting control capacity in supervisory control of multiple UAVs IEEE Systems, Man, and Cybernetics: Part A Systems and Humans, 38(2), March 2008.

[Cummings et al., 2007] Missy Cummings, Silvia Bruni, S. Mercier, and P. J. Mitchell. Automation architecture for

single operator, multiple UAV command and control. The International C2 Journal, 1(2):1–24, 2007.

[Dietz and Leigh, 2001] Paul Dietz, and Darren Leigh. Di-amondTouch: a multi-user touch technology In Proceedings of the 14th annual ACM Symposium on User Interface Software and Technology, 219–226, 2001.

[Franke et al., 2005] Jerry L. Franke, Vera Zaychik, Thomas M. Spura, and Erin E. Alves. Inverting the operator/vehicle ratio: Approach to next generation UAV command and control In Association for Unmanned Vehicle Systems International, 2005.

[Endsley, 1997] Raja Parasuraman, and Victor Riley. Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39:230–253, June 1997.

[Endsley, 1995] Mica R. Endsley. Toward a theory of situation awareness in dynamic systems: Situation awareness Human Factors, 37(1):32–64, 1995.

[Endsley, 1988] Mica R. Endsley. Situation Awareness Global Assessment Technique (SAGAT) In Proceedings of the IEEE Aerospace and Electronics Conference, 3:789–795, May 1988.

[Fuchs et al., 2013] Christian Fuchs, Sérgio Ferreira, João Sousa, and Gil Gonçalves Adaptive consoles for supervisory control of multiple unmanned aerial vehicles In Human-Computer Interaction. Interaction Modalities and Techniques, 678–687, 2013.

[Hart and Staveland, 1988] Sandra G. Hart, and Lowell E. Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research Advances in Psychology, 52:139–178, 1988.

[Jarvis and Tang, 2005] Ray Jarvis, and Kai W. Tang Multiple robot path planning strategies for bush fire fighting. In Proceedings of Australasian Conference of Robotics and Automation, Sydney, Australia, December 2005.

[Laufs et al., 2010] Uwe Laufs, Christopher Ruff, and Jan Zibuschka. MT4j - A cross-platform multi-touch development framework. VIMation Journal, 1:58–64, 2010.

[Legras and Coppin, 2007] François Legras, and Gilles Coppin. Autonomy spectrum for a multiple UAVs system. In Proceedings of Cognitive systems with Interactive Sensors 07, Palo Alto, California, November 2007.

[Lemon et al., 2002] Oliver Lemon, Lawrence Cavedon, Alexander Gruenstein, and Stanley Peters. Collaborative Dialogue for Control Autonomous Systems. Technical Report (FS-02-03), Center for the Study of Language and Information, Stanford University, 2002. American Association for Artificial Intelligence.

[Miller, 2014] Christopher A. Miller Delegation and Transparency: Coordinating Interactions So Information Exchange Is No Surprise. Springer International Publishing, Switzerland, 2014.
[Miller et al., 2001] Christopher A. Miller, Robert P. Goldman, Harry B. Funk, Peggy Wu, and Billy B. Pate. A playbook approach to variable autonomy control: Application for control of multiple, heterogeneous unmanned air vehicles In *American Helicopter Society 60th Annual Forum*, Baltimore, Maryland, June 2004.

[Murphy and Burke, 2010] Robin R. Murphy, and Jennifer L. Burke. The safe human-robot ratio. *Human-Robot Interactions in Future Military Operations*. Ashgate, Surrey, U.K., 2010.

[Ososky et al., 2014] Scott Ososky, Trancy Sanders, Florian Jentsch, Peter Hancock, and Jessie Y. C. Chen. Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems In *Inproceeding of Unmanned Systems Technology XVI*, 9084, May 2014.

[Patten et al., 2013] Timothy Patten, Robert Fitch, and Salah Sukkarieh. A flexible human-robot team framework for information gathering missions In *Proceedings of Australasian Conference of Robotics and Automation*, Sydney, Australia, December 2013.

[Parasuraman et al., 2008] Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. In *Journal of Cognitive Engineering and Decision Making*, 2(4):140–160, 2008.

[Parasuraman et al., 2000] Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. A model for types and levels of human interaction with automation. In *IEEE Transactions on Systems, Man and Cybernetics: Part A Systems and Humans*, 30(3):289–297, May 2000.

[Phillips et al., 2011] Elizabeth Phillips, Scott Ososky, Janna Grove, and Florian Jentisch. From tools to teammates: Toward the development of appropriate mental models for intelligent robots. In *Proceedings of the Human Factors and Ergonomics Society Annual Meeting*, 55(1):1491–1495, September 2011.

[Saget et al., 2008] Sylvie Saget, François Legras, and Gilles Coppin. Cooperative interface of a swarm of UAVs. In *Humans Operating Unmanned Systems*, 2008.

[Schneiderman, 2000] Ben Schneiderman. The limits of speech recognition. *Communications of the ACM*, 43(9):63-65, 2000.

[Sheridan and Verplanck et al., 1978] Thomas B. Sheridan, and William L. Verplanck. Human and Computer Control of Undersea Teleoperators. Technical Report (78-08-15-025), Man-Machine Systems Lab, Massachusetts Institute of Technology, Cambridge, 1978.

[Tessier and Dehais, 2012] Catherine Tessier, and Frédéric Dehais. Authority management and conflict solving in human-machine systems. *Journal Aerospace Lab*, 4:1–10, May 2012.