ARL-TR-8292 ● FEB 2018

US Army Research Laboratory

Soldier–Robot Team Communication: An 

Investigation of Exogenous Orienting Visual 

Display Cues and Robot Reporting Preferences

by Daniel J Barber, Julian Abich IV, Andrew B Talone,

Elizabeth Phillips, Florian Jentsch, Rodger Pettitt, and 

Linda R Elliott

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the 

Army position unless so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official 

endorsement or approval of the use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
ARL-TR-8292 ● FEB 2018

US Army Research Laboratory

Soldier–Robot Team Communication: An 

Investigation of Exogenous Orienting Visual 

Display Cues and Robot Reporting Preferences

by Daniel J Barber, Julian Abich IV, Andrew B Talone, 

Elizabeth Phillips, and Florian Jentsch

Institute for Simulation and Training, University of Central Florida

Rodger Pettitt and Linda R Elliott

Human Research and Engineering Directorate, ARL

Approved for public release; distribution is unlimited.

FOR OFFICIAL USE ONLY (delete if not FOUO)
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid 

OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

February 2018

2. REPORT TYPE

Technical Report

3. DATES COVERED (From - To)

1–28 February 2016

4. TITLE AND SUBTITLE

Soldier–Robot Team Communication: An Investigation of Exogenous Orienting 

Visual Display Cues and Robot Reporting Preferences

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Daniel J Barber, Julian Abich IV, Andrew B Talone, Elizabeth Phillips, 

Florian Jentsch, Rodger Pettitt, and Linda R Elliott

5d. PROJECT NUMBER

W911NF-10-2-0016

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

University of Central Florida, Institute for Simulation and Training

3100 Technology Parkway

Orlando, FL 32829

US Army Research Laboratory

ATTN: RDRL-HRB-AB

Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-TR-8292

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

Institute for Simulation and Training, University of Central Florida

14. ABSTRACT

As more focus is directed at technology for the future dismounted warfighter, new autonomous systems are being developed 

to enable bidirectional Soldier–robot dialogue. These autonomous systems implement multimodal communication methods to 

ensure communication robustness across operational environments. This research investigation had 2 major focuses:

1) investigate the use of exogenous orienting visual cues during unidirectional communication from a robot to a Soldier 

through a visual display and 2) investigate Soldier preferences for robot reporting of status information. Results under the first 

focus showed the longer it took to respond to a visual report, the higher the associated perceived workload and the lower the 

usability preference. Results under the second focus showed that the frequency with which participants expected status 

updates differed depending upon the difficulty of the primary task. In addition, most participants (75%) quickly and 

consistently identified a preferred report format (image or text). Recommendations for visual display designs are provided as

well as suggestions for instantiating reporting capabilities within future robot teammates.

15. SUBJECT TERMS

workload, usability, unmanned system, transparency, human–robot teams, multimodal communication

16. SECURITY CLASSIFICATION OF:

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

88

19a. NAME OF RESPONSIBLE PERSON

Daniel Barber

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

407-882-1128

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
Approved for public release; distribution is unlimited.

iii

Contents

List of Figures vi

List of Tables vii

Acknowledgments viii

1. Introduction 1

1.1 Visual Displays for Dismounted Soldier–Robot Teams 2

1.2 Information Sharing within Dismounted Soldier–Robot Teams 4

1.3 Experiment Purpose 5

2. Method 6

2.1 Participants 6

2.2 Questionnaires 7

2.2.1 Ishihara’s Test for Color Deficiency 7

2.2.2. Demographics Questionnaire 7

2.2.3 Cube Comparison Test 8

2.2.4 NASA-TLX 8

2.2.5 SUS 8

2.2.6 Free Response Questionnaire 9

2.2.7 Situation Awareness Global Assessment Technique (SAGAT) 

Probes 9

2.2.8 Adapted SART 10

2.3 Experimental Tasks 10

2.3.1 TD Task 12

2.3.2 RA Task 14

2.3.3 RR Task 16

2.4 Apparatus 20

2.5 Procedures 20

2.5.1 Training 20

2.5.2 Experiment’s Design 21

3. Results 22
Approved for public release; distribution is unlimited.

iv

3.1 RA Task 22

3.1.1 TD Performance 22

3.1.2 RA performance 23

3.1.3 Questionnaires 24

3.1.4 Bivariate Correlations 26

3.2 RR Task 28

3.2.1 Global Workload 29

3.2.2 Number of Reports 30

3.2.3 Report-Format Preference 31

3.2.4 Bivariate Correlations 32

4. Discussion 32

4.1 Robot Assistance 32

4.1.1 Navigational Routes 32

4.1.2 Building Identification 33

4.1.3 Limitations and Future Research 34

4.2 Robot Reporting 35

4.2.1 Task Difficulty 35

4.2.2 Reporting Frequency 35

4.2.3 Format Preference 36

4.2.4 Soldier SA 37

4.2.5 Limitations and Future Research 38

5. Conclusions 38

References 40

Appendix A. Demographics Questionnaire 45

Appendix B. Cube Comparison Questionnaire 51

Appendix C. NASA-Task Load Index (NASA-TLX) 55

Appendix D. System Usability Scale 57

Appendix E. Free Response Questionnaire 61
Approved for public release; distribution is unlimited.

v

Appendix F. Situation Awareness Global Assessment Technique (SAGAT) 

Probes 63

Appendix G. Situation Awareness Rating Technique (SART), Adapted 65

Appendix H. Cheat Sheet for Robot-Reporting Scenarios 69

Appendix I. Informed Consent 71

List of Symbols, Abbreviations, and Acronyms 77

Distribution List 78
Approved for public release; distribution is unlimited.

vi

List of Figures

Fig. 1 Conceptual illustration depicting points at which the simulation was 

paused for each scenario and one or more measures administered .......7

Fig. 2 Example overhead illustration of routes traveled by the Soldier and the 

robot during each scenario; initial starting location and travel direction 

were randomized for each participant but the Soldier and robot always 

started in the same relative location.....................................................11

Fig. 3 Image represents the virtual MMI as a prompt on the screen within the 

MIX environment.................................................................................11

Fig. 4 Representation of the RA-scenario context; the MMI provided a visual 

interface for the robot to send RA queries and for the Soldier to 

respond, creating closed-loop communication ....................................12

Fig. 5 Image represents an alternative use of the virtual MMI, whereby status 

updates can be requested from the robot..............................................12

Fig. 6 Screenshot of the MIX test bed representing the first-person 

perspective of the Soldier traveling through a Middle Eastern urban 

environment .........................................................................................13

Fig. 7 Examples of characters used within the TD task are (l to r) friendly 

Soldier, friendly civilian, enemy soldier, and insurgent (armed 

civilian) ................................................................................................13

Fig. 8 Conceptual representation of how the event rates (low = 15/min, 

moderate = 30/min, high = 60/min) differed across the 3 scenarios....14

Fig. 9 Two images representing the robot’s navigational-assistance-query 

display formats in the MMI; image on the left has extended directional 

arrows (Navigation A) while the image on the right has short, bold 

directional arrows (Navigation B) .......................................................15

Fig. 10 Two images representing the robot’s building-identification-query 

display formats in the MMI; image on the left has highlighted boxes 

around the buildings (Building A) while the image on the right has 

highlighted boxes around the buildings with modifications to the scene 

and aerial images (Building B) ............................................................15

Fig. 11 Zoomed-in view of the portion of the screen dedicated to the RR task, 

with text-report button on the left and image-report button on the 

right......................................................................................................17

Fig. 12 (l to r) Example of an image report containing one threat and 4 

nonthreats and the same report requested in text format .....................18

Fig. 13 Robot’s report shows its immediate environment is unsafe; (left) in 

image report 4 threats are bounded in yellow boxes while (right) text 

report counts the total number of threats .............................................19
Approved for public release; distribution is unlimited.

vii

Fig. 14 Robot’s report shows unsafe immediate environment; (left) in image

report a critical threat is bounded in blue box while (right) the text 

report counts the number of critical threats .........................................19

Fig. 15 The Fort Benning laboratory while 14 Soldiers participate in the

experiment at the same time (no identifiable information can be 

gathered from photograph) ..................................................................21

Fig. 16 Graph depicting TD performance across event-rate conditions for the 

RR scenarios ........................................................................................30

Fig. 17 Graph depicting number of reports across event-rate conditions for the 

RR scenarios ........................................................................................31

List of Tables

Table 1 Results of t-tests for TD task performance for both building and 

navigation display designs with mean (M), standard deviation (SD), 

and Cohen’s d statistical value (d) reported.........................................23

Table 2 Results of t-tests for robot assistance response accuracy for both 

building and navigation display designs..............................................23

Table 3 Results of t-tests for RA response time for both building and 

navigation display designs...................................................................23

Table 4 Results of t-tests for NASA-TLX ratings for both building- and 

navigation-display designs...................................................................24

Table 5 Results of t-tests for usability for both building- and navigationdisplay designs.....................................................................................24

Table 6 Common themes among participant responses to the Free Response 

Questionnaire for RA-display formats; numbers in parentheses next to 

each theme are total participants reporting each theme (an example of 

which is shown) ...................................................................................25

Table 7 Bivariate correlation matrix for statistically significant relationships 

among demographics, task performance, global workload, and 

usability................................................................................................27

Table 8 Bivariate correlation matrix for situation-awareness measures, 

reporting preference, and primary task performance...........................32
Approved for public release; distribution is unlimited.

viii

Acknowledgments

The authors would like to thank and acknowledge the contributions of the project 

managers supporting the effort described in this report, Sue Hill (US Army 

Research Laboratory [ARL] Human Research and Engineering Directorate) and 

Johnathan Bornstein (ARL Autonomous Systems Division, Collaborative Alliance 

Manager), for their ongoing oversight and support for integrated investigations of 

human–robot interaction for the dismounted Soldier. We are particularly indebted 

to the Soldiers who participated readily and steadily throughout this event, 

providing insightful comments as well as active attention.

This research was sponsored by ARL and was accomplished under Cooperative 

Agreement No. W911NF-10-2-0016. The views and conclusions contained in this 

document are those of the authors and should not be interpreted as representing the 

official policies, either expressed or implied, of ARL or the US Government. The 

US Government is authorized to reproduce and distribute reprints for Government 

purposes notwithstanding any copyright notation hereon.
Approved for public release; distribution is unlimited.

1

1. Introduction

The advancement of robot capabilities and functionality has changed the way in 

which Soldiers perform many of their operational tasks. The various unmanned air 

(Valvanis 2008), ground (Zych et al. 2013), and submersible vehicles (Clegg and 

Rodgers 2015) currently deployed have significantly impacted present-day warfare. 

Although many of these systems have shown to be beneficial and effective for 

mission success, traditional control of these systems is through teleoperation (Chen

et al. 2007; Lichiardopol 2007). While teleoperation may be necessary and 

appropriate for situations that may otherwise require soldiers to be exposed to 

hazardous or life-threatening situations, it is not recommended for dismounted 

operations (Chen et al. 2007). Hence, autonomous robots provide a solution that 

takes advantage of current robot sensing and intelligence while reducing the 

cognitive demands on the Soldier, allowing robots to maintain awareness of the 

operational environment (Kott et al. 2015). The implementation of autonomous 

robots within human teams carries with it concern regarding human–robot 

interaction (HRI) and, more specifically, human–robot communication.

Moving beyond teleoperation, military HRI has focused on integrating multimodal 

communication (MMC) methods that leverage the natural ways in which 

human–human interaction takes place and the commonly employed functionality 

for human–computer interaction (Abich et al. 2015). In a general sense, MMC is 

sending and/or receiving information through multiple sensory systems (e.g., 

seeing text information that is also presented auditorily). In terms of benefits for 

signal-communication processing, MMC systems are robust, flexible, efficient, 

intuitive, and redundant (Dumas et al. 2009; Partan and Marler 1999). While many 

robot systems are equipped with multimodal interaction capabilities (Barber et al. 

2013; Harris and Barber 2014), the impact of each communication type on the 

Soldier’s ability to perform task critical operations is not well known. Therefore, 

systematic evaluation of the components that comprise the transactions between 

humans and robots and the way in which information is conveyed is critical prior 

to the deployment of any system to the field.

There were 2 major goals of this experiment. The first was to investigate the effects 

on performance and operator perception of various exogenous orientation design 

cues associated with a visual display in a multimodal interface to facilitate squadlevel communication within a dismounted Soldier–robot team. In particular, this 

goal focused on determining whether the elements of visually displayed robot 

reports provided adequate information about the situational context so the Soldier 

could quickly determine the best course of action the robot should take without 
Approved for public release; distribution is unlimited.

2

being cognitively overloaded. The second goal was to investigate Soldiers’ 

preferences when it came to status updates from a robot teammate (e.g., reporting 

frequency and format). Specifically, this aspect of the experiment focused on 

understanding the relationship between robot-reporting preferences, task 

performance, and situation awareness (SA) with a Soldier population.

1.1 Visual Displays for Dismounted Soldier–Robot Teams

The military has stated its interest in pursuing mobile technology for dismounted 

squads, such as smartphones or tablets, because of the high processing capabilities 

now available in smaller, lighter platforms. In 2014, the Federal Mobile Computing 

Summit held by the Department of Defense solicited a US-based vendor that can 

supply an NSA-compliant device with a 12-24 hour battery life, compatibility with 

multiple commercial networks and the ability to integrate with a planned Defense 

Information Systems Agency application store... (Mazmanian 2014). The Army’s 

Communications-Electronics Research, Development and Engineering Center has 

been working on the Edge-Enabled System with the mission of developing a 

handheld interface for the future warfighter and is currently field testing command, 

control, communications, computers, intelligence, surveillance, and 

reconnaissance—commonly, C4ISR—technology (Jones-Bonbrest 2015; McCall 

2010). Additionally, the Defense Advanced Research Projects Agency’s (DARPA) 

Squad X Core Technologies program is geared toward developing new,

lightweight, energy-efficient technology to support tactical advantages for 

dismounted infantry squads while avoiding negative repercussions such as 

cognitive or physical overloading (DARPA 2015).

Visual displays have the potential to provide a vast amount of information within a 

relatively small space. A single image can convey complex ideas. In a field study 

assessing the effects of different information-presentation types on task 

performance, Soldiers indicated visual displays were “easier to follow” and 

information was “easier to recall.” They also exhibited better SA when using a 

visual display compared to an auditory display, potentially because graphical 

representations might facilitate chunking of information into manageable sizes 

(Glumm et al. 1999). Studies have shown that the decay of information in visual 

working memory can be gradual. Further, the capacity for storing visual images is 

larger compared to auditory working memory. Therefore, the larger working 

memory storage, in addition to the chunking of information, might explain the 

Soldiers’ performance and preference when using visual displays (Card et al. 1983; 

Zhang and Luck 2009). 
Approved for public release; distribution is unlimited.

3

For visual displays to be effective, they must orient the attention of the user (Posner

et al. 1980). Visual perception is driven by both internal (endogenous) and external 

(exogenous) orienting events. Endogenous orientation refers to “purposeful 

allocation of attentional resources” (e.g., scanning an arrival screen at an airport) 

while exogenous orientation refers to “reflexive, automatic responses” such as 

attention captured by a bright blinking light (Mayer et al. 2004). For the purposes 

here, we will be concentrating on exogenous orienting. To support exogenous 

orienting, non-content specific features should be implemented, meaning the cues 

used to attract attention should not be normally present in the environment or signal. 

These are referred to as feature singletons (Mulckhuyse and Theeuwes 2010). 

Highlighting would be considered an exogenous orientation event that provides 

visual cues to attract attention and support detection (Posner 1980). The 

highlighting of navigational routes and objects in an environment is an example of 

exogenous orienting because it elicits a reflexive and automatic response without 

having predetermined biases (Mayer et al. 2004). There are many factors that 

increase the chances of capturing visual attention, such as size, intensity, color, and 

transformation (Wickens and Hollands 2000), but the elements of highlighting 

should contain non-content features. In other words, the graphics (e.g., labels, 

shapes, colors) used should not be found within any of the images or environments 

to reduce any confusion and misunderstandings that could lead to errors (Salcedo

et al. 2014; Wogalter et al. 2002). The colors used in this experiment contrast highly 

with the environment while not being accompanied by common associations, such 

as the colors red indicating a warning or stop, green indicating good or go, and 

yellow indicating caution or slow down (Wogalter et al. 2002). The application of 

such features becomes apparent when developing mobile applications that 

inherently have limited visual space.

Although mobile devices may be an HRI and communication solution for the future 

warfighter, it is imperative such technology not impair primary task performance; 

therefore, particular care must be taken when designing the interface. Chittaro 

(2006) states that simply scaling down visual information from a desktop computer 

or a larger display to fit a mobile device is not as straightforward as it may seem 

because of the many limitations that exist with the mobile device (e.g., limited 

space, display ratio reduction, reduced processing capabilities, and different 

interaction or input techniques). Further, mobile devices are more likely to be used 

in various physical environments (e.g., inside/outside, day/night) and must be 

robust enough to display information in all conditions. Even more of an issue is

safety while performing other activities during interaction with a mobile device—

a major concern because of the demand on the user’s attention, which has limited 

allocation capacity (Kahneman 1973). These factors are crucial for a dismounted 
Approved for public release; distribution is unlimited.

4

Soldier as the increased attentional and cognitive demands related to operating the 

mobile device could lead to catastrophic and life-threatening events. 

1.2 Information Sharing within Dismounted Soldier–Robot 

Teams

To operate effectively as a team, Soldiers should communicate with one another 

before, during, and after completion of a mission. Communication among team 

members can be facilitated through information sharing (e.g., information 

regarding the environment, task progress, or new directives); this is necessary for 

the development of an individual’s SA while working within a team (Salas et al. 

1995). In comparing the current state of the art in communicating within 

human–robot teams (i.e., continuous video feeds) to that of communicating within 

human–human teams (i.e., periodic communication of relevant information), it can 

be concluded that humans and robots have different mental models when it comes 

to engaging in information sharing. In other words, humans and robots each use 

different rules when deciding how often and what information should be shared 

among team members. This issue is less prevalent within human–human teams 

whereby team members draw from shared mental models that inform them of when 

information should be “pushed” to their team members (Johannesen et al. 1994; 

MacMillan et al. 2004). Thus, humans are often aware of their human teammates’ 

informational needs. 

In contrast, current robots are unaware of the informational needs of their Soldier 

team members. To combat this, current robots are designed to feed information to 

Soldiers in a continuous fashion via live video feeds (Scholtz 2003). This method 

of information sharing is not ideal for dismounted operations for 2 major reasons. 

First, it is taxing on the human visual and cognitive processing systems (Burke et 

al. 2004; Casper and Murphy 2003; Yanco and Drury 2007). Second, it requires the 

Soldier–operators to spend most of their time “heads down” (i.e., looking 

downward at a visual display) as they continuously control or monitor their robot 

teammate (RCTA 2014). Thus, this form of human–robot interaction makes it 

difficult for dismounted Soldiers to perform other tasking while managing the 

robot.

Unlike current Soldier–robot teams, whereby a Soldier is responsible for direct 

control of a robot team member, the vision for future Soldier–robot teams is one in 

which robots autonomously accomplish tasks and contribute to team performance 

(RCTA 2014). While this autonomy will allow Soldiers to perform other tasks, it 

will become even more important for robots to proactively push information (in the 

form of updates) to Soldiers in anticipation of their informational needs. What is 
Approved for public release; distribution is unlimited.

5

unknown at this point is how often Soldiers expect to receive these updates and 

how they would like the shared information to be presented to them (e.g., via text, 

imagery, or auditorily). 

1.3 Experiment Purpose

The first part of this study was robot assistance (RA) and focused on the 

unidirectional communication transaction from a robot to a dismounted Soldier 

teammate via a visual display during a surveillance task. Specifically, this explores 

the effects of various exogenous orienting visual display cues within robotgenerated assistance request on participants’ task performance, perceived mental 

workload, and usability preference. Primary task performance was assessed as the 

percentage of correctly detected targets during a signal-detection task; that is, the 

threat detection (TD) task described in Section 2.3.1. Task performance was also 

measured as the time it took participants to respond to reports (i.e., intervention 

response time) and the percentage of correct choices made (i.e., intervention 

accuracy) referring to the content of the RA requests. Two types of RA requests 

were investigated: navigational route and building selection. These requests were 

selected based on current robot capabilities and expected future-use cases in which

a robot needs to address situations where both options have equal probability of 

being correct and the robot has not yet learned how to decide on its own (Barber et 

al. 2015). The mental workload of Soldiers was assessed using a subjective measure 

that captured demands imposed on the participants and on their interaction with the 

visual display. This assessment helped identify the sources of mental workload 

associated with the display designs. Usability preference was also captured using a 

subjective measure that provided a composite score across all items to indicate the

participants’ preference for display configuration. Five central research questions 

were of interest for the RA part of this study:

1) Is performance accuracy of the primary detection task affected by the type 

(navigation or building) or display version of the robot request?

2) Is there a difference in accuracy of correct responses to robot assistance 

requests for both navigational route and building selection depending on the 

type of exogenous visual cues displayed?

3) Is there a difference in response time to visual reports for both navigational

route and building selection depending on the type of exogenous visual cues 

displayed?

4) How is workload affected by the type of exogenous visual cues displayed 

for both navigational route and building selection?
Approved for public release; distribution is unlimited.

6

5) How is usability preference affected by the type of exogenous visual cues 

displayed for both navigational route and building selection?

The second part of this study was robot reporting (RR) and focused on investigating 

Soldier preferences in receiving status updates from a robot teammate. Given that 

there is little research indicating how often Soldiers expect to receive status updates 

from an autonomous robot teammate (i.e., how often they expect the robot to push 

information without being asked), this task attempted to address these questions by 

taking a slightly different approach to the problem. Participants were instructed to 

perform an individual task (i.e., the TD task), with the option of checking in on a 

robot teammate as often as they preferred (via the ability to request either text or 

image reports from the robot). This strategy allowed for an empirical examination 

of Soldier preferences for robot reporting (i.e., number and type of reports) and the 

factors that may affect their preferences (e.g., primary task difficulty). Four central 

research questions were of interest for the RR part of the study:

1) How often do Soldiers expect to receive status reports from a robot 

teammate, and is this preference influenced by the difficulty of a Soldier’s 

primary task?

2) Is there a general preference among Soldiers for text-based or image-based 

reports when seeking updates from a robot teammate?

3) How is SA affected by the number and type of robot reports requested by 

the Soldier?

4) Is there a performance tradeoff associated with robot reports, whereby

primary task performance decreases as more robot reports are received by 

the Soldier?

2. Method

2.1 Participants

Twenty-nine Soldiers from Fort Benning, Georgia’s Officer Candidate School, 22 

males and 7 females (mean age = 26.8, standard deviation [SD] = 3.3), volunteered 

for and completed the experiment. Participation was voluntary, and no 

compensation was awarded. Participants all had normal or corrected-to-normal 

(e.g., glasses, contact lenses) vision and were screened for color-vision deficiency 

(2 males were red/green color deficient but were still included in the analysis since 

they were not classified as data outliers). None of the participants had prior 

experience with the simulator or multimodal interface. Based on responses gathered 
Approved for public release; distribution is unlimited.

7

from the demographics questionnaire, participants indicated they had little 

knowledge of basic robotics technology or experience interacting with robots.

Figure 1 illustrates the points during each scenario when the simulation was paused 

and measures were administered; in addition to situation-awareness probes, the 

NASA Task Load Index (NASA-TLX), System Usability Scale (SUS), and

Situation Awareness Rating Technique (SART) were used. For the purposes of this 

study, we defined a route as the path traveled between 2 street corners. Figure 1

shows most of the measures were administered at the end of each route (i.e., once 

the Soldier and robot reached a corner). During the RR scenarios (i.e., RR1 and 

RR2), however, the simulation randomly paused twice during each route to 

administer SA probes.

Fig. 1 Conceptual illustration depicting points at which the simulation was paused for each 

scenario and one or more measures administered

2.2 Questionnaires

2.2.1 Ishihara’s Test for Color Deficiency

This color deficiency test assessed for red and green reduced sensitivity, which is 

the most common form of color-vision deficiency (Ishihara 2014). Participants

viewed 11 colored plates requiring them to identify numbers or trace a path. If 

participants answered 5 or more incorrectly, they were considered color deficient. 

Participants were not removed if they were assessed to have a color-vision 

deficiency, but it was documented to account for any individual differences.

2.2.2. Demographics Questionnaire

This questionnaire (shown in Appendix A) was developed in-house and gathered 

background information on each participant’s age, gender, visual acuity, and 
Approved for public release; distribution is unlimited.

8

academic level of achievement (and type of degree, if applicable). Information on 

military experience—years in the service, rank, military occupational specialty, 

number of deployments, and mission-related experience—was collected. The 

questionnaire asked about computer use: length of time participants have been 

using a computer and how often as well as the ways they interact with the computer;

further, video-game use, such as how often and what types, as well as any 

experience working with them was recorded. Finally, experience with and 

knowledge of robotics, both military and commercial, was recorded.

2.2.3 Cube Comparison Test

This spatial orientation test (shown in Appendix B) assesses the ability to mentally 

rotate and compare objects in space. The test comprises 2 parts and each part 

consists of 21 items, but only Part 1 was used. This test requires participants to 

compare 2 cubes and determine whether one cube can be rotated to match the other 

cube. Participants have 3 minutes to answer as many items as possible. A 

participant’s score is the number of correct responses minus incorrect responses;

therefore, guessing is not encouraged. 

2.2.4 NASA-TLX 

The NASA-TLX (Hart and Staveland 1988) is a perceived-workload assessment 

comprising 6 subscales: mental demand, physical demand, temporal demand, 

effort, frustration, and performance. A global perceived-workload measure is also 

calculated by averaging the 6 subscales. Each subscale was scored on a 100-point 

scale. For the RA scenario, the unweighted TLX (shown in Appendix C) was 

administered by computer at the end of each route (i.e., 4 times). For the RR 

scenarios, the unweighted TLX was administered after 2 routes had been completed 

(i.e., 2 times: halfway through each RR scenario and at the end of each RR 

scenario). 

2.2.5 SUS

This 10-item questionnaire focused on perceived usability of the system; that is, 

hardware, software, and equipment (Brooke 1996). Ratings were indicated using 

5-point Likert items (1 = strongly disagree; 5 = strongly agree). The composite 

score represents the overall usability of the system and ranges from 0 to 100 with 

higher scores indicating higher usability. Specifically, these questions focused on 

the participants’ interaction with the device during the RA scenario. The 

questionnaire (shown in Appendix D) was administered at the end of each route of 

the RA scenario.
Approved for public release; distribution is unlimited.

9

2.2.6 Free Response Questionnaire

This 6-item, open-ended questionnaire (shown in Appendix E) covered positive and 

negative aspects of the participants’ interaction with the simulated multimodal 

interface device and their preference for display design, and asked them to suggest 

any improvements. Specifically, these questions were focused on the interaction 

with the device during the RA scenario. The questionnaire was presented to 

participants after completing the RA scenario.

2.2.7 Situation Awareness Global Assessment Technique (SAGAT)

Probes 

During RR scenarios RR1 and RR2 a modification of the SAGAT (Endsley 1988) 

was used to assess objectively the participants’ SA at Levels 1 (Perception) and 2 

(Comprehension) of Endsley’s SA model (Endsley 1995). The SAGAT method 

pauses a simulation to ask participants questions about the simulation environment 

and task(s) being performed. For this experiment, SAGAT probes (shown in

Appendix F) were presented to participants as they completed each RR scenario. 

Each time the simulation paused, participants received 2 questions: one about the 

safety of the Soldier’s environment and one about the safety of the robot’s 

environment. (Section 2.3.3.2 details the safety rules relevant to answering each 

question.) Specifically, there were 2 probes presented to participants: SA-1 and 

SA-2. 

SA-1 probes appeared at random, as the Soldier moved along each route. These 

probes assessed Level-1 SA because correctly answering the questions only 

required the participant to have perceived elements within the environment (i.e., 

higher-level comprehension of said elements was not necessary). The same 2

questions were presented each time an SA-1 probe appeared. The first question 

assessed participant SA regarding the Soldier’s environment and the second 

question assessed participant SA regarding the robot’s environment.

In contrast, SA-2 probes appeared at the end of a route (i.e., once a corner was 

reached). These probes assessed Level-2 SA because correctly answering the 

questions required the participant to perceive and comprehend elements within the 

environment (i.e., a conclusion had to be formed based on information obtained 

while traversing the previous route). The same 2 questions were presented each 

time an SA-2 probe appeared. Similar to the SA-1 probes, the first question assessed 

participant SA regarding the Soldier’s environment and the second question 

assessed participant SA regarding the robot’s environment.

These probes served as an objective measure of SA because each question had only 

one correct answer. For each RR scenario, each participant answered 8 SA-1 probes 
Approved for public release; distribution is unlimited.

10

(2 per route) and 4 SA-2 probes (one at the end of each route). For the purposes of 

this experiment, scores on the SA-1 and SA-2 probes were calculated as the 

percentage of correctly answered probes.

2.2.8 Adapted SART

During the RR scenarios, a modification of the original Situation Awareness Rating 

Technique (Taylor 1990) was used to assess participants’ subjective perceptions of 

their own SA during the RR scenarios. The original SART consists of 10 items that 

are divided into 3 subscales: demand on attentional resources, supply of attentional 

resources, and understanding of the situation. Participants responded to the SART 

by rating their level of agreement with each item using a 7-point Likert scale 

ranging from 1 (low) to 7 (high). The SART adapted for this experiment (shown in 

Appendix G) contained the original 10 items as well as 3 additional items specific 

to the participants’ knowledge of their robot teammate. (These latter 3 items 

pertaining to the robot, however, are not included in the results of this report as they 

were for exploratory analytical use.) Scoring of the original 10 items followed the 

procedure used by Endsley et al. (2000). The mean of each subscale was calculated

and then the mean of each subscale was used in the following formula:

(Understanding of the situation) + (Demand on attentional 

resources) – (Supply of attentional resources) = Overall SART 

rating

Higher overall SART ratings (based on the original 10 items) correspond with 

higher subjective perceptions of one’s own SA; lower ratings correspond with 

lower subjective perceptions of one’s own SA.

2.3 Experimental Tasks 

A general overview of the experimental tasks are provided first and then followed 

by a detailed description of each. The experimental scenarios were composed of 3

tasks—TD, RA, and RR—all performed within the Mixed Initiative eXperimental 

test bed (MIX) (Barber et al. 2008; Reinerman-Jones et al. 2010). Given that this 

experiment had 2 focuses, there were 2 types of scenarios: RA and RR. Both 

included a TD task. All scenarios simulated a reconnaissance and surveillance task 

where the Soldier and robot were traveling along separate, non-overlapping routes 

within the same portion of a city. Figure 2 is a conceptual illustration of these 

routes: the left box represents the Soldier’s route and the right box represents the 

robot’s route. Each “side” of the box (left, top, right, bottom) represents a different 

route through the city (i.e., there are 4 routes traveled during each scenario). In each 

scenario, the Soldier and robot both start at the same corner of their respective 

portion of the city (bottom left, top left, top right, or bottom right) and travel the 
Approved for public release; distribution is unlimited.

11

same direction (clockwise or counterclockwise). Both the Soldier and robot travel 

at the same speed; thus, each reaches the beginning/end of each route at the same 

time. For example, if the robot and Soldier start at the bottom-left corner and are 

traveling clockwise, then they will reach the end of each route (i.e., corner) at the 

same time.

Fig. 2 Example overhead illustration of routes traveled by the Soldier and the robot during 

each scenario; initial starting location and travel direction were randomized for each 

participant but the Soldier and robot always started in the same relative location

In the RA scenario, the participant’s role was to identify potential threats in their 

own environment and periodically respond to RA queries displayed on a visual 

prompt (Fig. 3) representing a virtual version of a multimodal interface (MMI). 

Figure 4 shows a conceptual mockup illustrating how the interaction between the 

Soldier and robot took place through the MMI during this scenario. As the robot 

travels through the environment and comes across an uncertain situation, it sends a 

report to a Soldier via the MMI. The Soldier responds to the report and that 

information gets transmitted back to the robot through the MMI.

Fig. 3 Image represents the virtual MMI as a prompt on the screen within the MIX 

environment
Approved for public release; distribution is unlimited.

12

Fig. 4 Representation of the RA-scenario context; the MMI provided a visual interface for 

the robot to send RA queries and for the Soldier to respond, creating closed-loop 

communication

In the RR1 and RR2 scenarios, one of the participant’s roles was to aid in the 

development of a robot’s reporting capabilities by showing the robot how often 

(and in which presentation format) they preferred to receive information from the 

robot. In addition, participants were told they also were helping the robot learn 

which routes through a simulated urban area were safe for travel. For this scenario, 

status updates (i.e., robot reports) from the robot were displayed at the top left of 

the participants’ display (Fig. 5). This scenario, thus, introduced a different use for 

the MMI (i.e., to request information from the robot).

Fig. 5 Image represents an alternative use of the virtual MMI, whereby status updates can 

be requested from the robot

2.3.1 TD Task 

The MIX test bed was customized to represent the first-person perspective of a 

Soldier traveling through a generic Middle Eastern urban environment (Fig. 6). The

Soldier’s route was preplanned and did not require the participants to control the 

Soldier’s movement. The participants’ role was to identify potential threats in the 

environment (i.e., a signal-detection task) by capturing photos to help populate a 
Approved for public release; distribution is unlimited.

13

robot teammate’s database with examples so it could more effectively carry out 

mission tasks autonomously. There were 4 categories of characters (i.e., events) 

within the environment: friendly Soldiers, friendly civilians, enemy soldiers, and 

insurgents (Fig. 7). Each category included at least 5 different types of characters. 

Enemy Soldiers and insurgents were classified as threats (i.e., signals), and an equal 

number of each was presented. An equal number of each category of nonthreats 

was also present. All threats were identified by left-clicking with a mouse on the 

character within the environment. No feedback was provided regarding the 

accuracy of detection, but participants did hear the sound of a camera shutter to 

indicate they were capturing photos. 

Fig. 6 Screenshot of the MIX test bed representing the first-person perspective of the 

Soldier traveling through a Middle Eastern urban environment

Fig. 7 Examples of characters used within the TD task are (l to r) friendly Soldier, friendly 

civilian, enemy soldier, and insurgent (armed civilian)

The event rate of the TD task—speed at which threats and nonthreats were 

presented within the Soldier’s environment—differed among the RA and RR 

scenarios. In the RA scenario, the event rate was presented at 30/min with a signal 

probability of 13.33% based on previous research (Abich et al. 2013). In the RR1 

and RR2 scenarios, the event rate of the TD task varied. Across the 2 RR scenarios, 
Approved for public release; distribution is unlimited.

14

3 event rates were encountered by each participant: low, moderate, and high. In all 

3 event-rate conditions, signal probability (i.e., the ratio of threats to nonthreats 

among the characters) was consistent at 13.33%; however, the number of events 

encountered in each condition differed. The event rate was 15/min in the low eventrate condition, 30/min in the moderate condition, and 60/min in the high condition. 

Figure 8 depicts how the 3 event-rate conditions were distributed across the RA, 

RR1, and RR2 scenarios. The RA and RR1 scenarios both had a constant, moderate 

event rate while the event rate in the RR2 scenario switched from either low to high 

or high to low (depending upon the assigned counterbalancing condition). 

Performance on the TD task was assessed as the overall percentage of correct 

responses. False-positive rates were very low; therefore, signal-detection theory 

indices were not calculated.

Fig. 8 Conceptual representation of how the event rates (low = 15/min, moderate = 30/min, 

high = 60/min) differed across the 3 scenarios

2.3.2 RA Task 

While participants were performing the TD task (i.e., the primary task), visual 

prompts randomly appeared and required participants to respond to RA queries 

generated by a robot teammate (i.e., the secondary task). The visual prompts were 

a virtual representation of the MMI (shown in Fig. 2). The robot teammate was 

never viewed by participants, as the robot was traveling a separate route. At times,

the robot required assistance because it could not deduce the best option based on 

its intelligence capabilities. The types of assistance the robot requested were for 1) 

navigational routes and 2) building identification (Figs. 9 and 10). The 

navigational-assistance requests asked participants to decide the best route for the 

robot to avoid obstacles. The building-identification assistance requests asked 

participants which building a robot should screen (i.e., monitor). The information 

the participants needed to make their decision was gathered from the MMI. Every 
Approved for public release; distribution is unlimited.

15

MMI prompt comprised 2 images: 1) the right image represented the point of view 

(POV) of the robot traveling through the environment and 2) the left image 

represented an aerial view of the operational area (Fig. 3). Both images represented 

the same scene but from different angles; therefore, the RA queries could be

answered by gathering the pertinent information from either image. The research 

interest here lies in evaluating the workload impact and preference of visual display 

format by comparing the types of display formats for each query (i.e., Navigation 

A vs. B and Building A vs. B). Participants responded to the robot’s query by leftclicking on one of the buttons located at the top of the screen to indicate their choice 

(shown in Fig. 3). No feedback was provided to the participants regarding the 

accuracy of their choices. The simulation paused—Soldier’s movement through the 

environment stopped—when each RA query appeared and did not resume until a 

response to the robot’s request was made.

Fig. 9 Two images representing the robot’s navigational-assistance-query display formats 

in the MMI; image on the left has extended directional arrows (Navigation A) while the image 

on the right has short, bold directional arrows (Navigation B)

Fig. 10 Two images representing the robot’s building-identification-query display formats 

in the MMI; image on the left has highlighted boxes around the buildings (Building A) while

the image on the right has highlighted boxes around the buildings with modifications to the 

scene and aerial images (Building B)

The navigational-route RA queries displayed route options as either extended 

segmented arrows (i.e., Navigation A design) or short, bold directional arrows (i.e., 

Navigation B design). The building-identification RA queries displayed 

highlighted boxes around the buildings of interest (i.e., Building A design) or had 

highlighted boxes with enhanced modifications to the images such as grayed-out 

areas in the POV of the robot image and detailed aerial information (i.e., Building 

 

 
Approved for public release; distribution is unlimited.

16

B design). The queries for each type of display format (i.e., navigation and building) 

were the same; the only difference was the way in which the information was 

portrayed on the MMI. The elements comprising exogenous orientation 

highlighting were chosen based on their noncontent features. The labels, shapes, 

and colors had no relation or similarity to elements within the environment. The 

red (R), green (G), and blue (B) color values for the magenta were R: 255, G: 51, 

B: 204 and for the green were R: 0, G: 255, B: 204. The scenario lasted about 10 

min, and there was a total of 32 RA reports (i.e., 8 of each type of display format):

one presented (on average) every 15–20 s. Each route in the scenario was about 2.5 

min and comprised one type of display format. The order in which the display 

formats were presented for each route was randomized using a random-number 

generator to determine the presentation order. At the end of each route, the 

workload and usability questionnaires were presented (depicted in Fig. 1). The 

order in which each participant navigated around the scenario was counterbalanced 

and randomized, meaning participants were randomly assigned starting locations at 

one of the 4 corners of the scenario and either traveled clockwise or 

counter-clockwise (shown in Fig. 2).

2.3.3 RR Task 

Participants were informed their role in the RR scenarios (in addition to performing 

the TD task) was to aid in the development of a robot’s reporting capabilities and 

to aid the robot in determining which navigational routes through a simulated city 

were safe for travel. Similar to the RA task, participants were informed the robot 

was traveling along a series of routes throughout a portion of the city separate from 

the one traveled by the Soldier. Participants were instructed they were responsible 

for helping the robot a) build a database of potential threats in the environment, b) 

decide how often it should send status reports and what type (image or text reports), 

and c) decide which routes through the city were safe for travel. Participants were 

informed that during the RR scenarios they were to identify threats from the 

first-person Soldier perspective (i.e., perform the TD task, the primary) while 

simultaneously performing the RR task (requesting robot reports and assessing the 

safety of the Soldier’s and robot’s routes, the secondary). 

Thus, both the RA and RR scenarios required participants to perform the TD task, 

but differed in terms of the interactions taking place between the Soldier and robot. 

In the RA scenario, the robot pushed information to the Soldier (without prompting) 

and the Soldier could not request information (i.e., reports) from the robot. In 

contrast, in the RR scenarios, the robot only sent information to the Soldier if it was 

requested (via a text or image report).
Approved for public release; distribution is unlimited.

17

2.3.3.1 Robot Reports

During each RR scenario, participants were told they could request reports from the 

robot that informed them of the robot’s current environment. Participants were told 

that due to bandwidth limitations, the robot could only send new reports (i.e., 

containing updated information) every 10 s. Each report included a report number 

that appeared above the requested report to clearly identify it as unique (containing 

new information). These report numbers “reset” at the beginning of each route (i.e., 

whenever the Soldier and robot reached a corner). This made it clear to participants 

that a) each scenario consisted of 4 distinct routes and b) the SA-1 and SA-2 probes 

should be answered based on only the current route (not the previously traveled 

routes). 

Participants had the option to receive these reports in text or image format by 

clicking on one of 2 buttons in the top-left portion of their screen (Fig. 11). When 

an image report was requested, participants were shown an image of what was 

being viewed from a forward-facing camera mounted on the robot (Fig. 12). The 

image showed bounding boxes around both threats and critical threats to assist the 

Soldier in assessing the number of threats present in the robot’s immediate 

environment. Participants could also request text reports from the robot. These 

reports displayed the same information about the robot’s immediate environment 

but in a text format. Specifically, the text reports provided the Soldier with the total 

number of threats, critical threats, and nonthreats present (Fig. 12). The TD task did 

not stop—simulation did not pause—when a robot report was requested. The 

decision to include a continuous TD task was made to assess more accurately the 

effect requesting and reviewing robot reports had on the primary task. In addition, 

participants could not view both an image and text report simultaneously (i.e., have 

both displayed at the same time); however, they could request one right after 

another and view the same information in both report formats if they wanted to.

Fig. 11 Zoomed-in view of the portion of the screen dedicated to the RR task, with textreport button on the left and image-report button on the right
Approved for public release; distribution is unlimited.

18

Fig. 12 (l to r) Example of an image report containing one threat and 4 nonthreats and the 

same report requested in text format

2.3.3.2 Safety Rules

In the RR scenarios, participants were also responsible for assessing the safety of 

their and the robot’s environments. To accomplish this task, participants used a set 

of rules to determine the safety status of a particular environment. Participants 

needed to use these rules to correctly answer 3 of the 4 SA probes (i.e., one SA-1 

probe and both SA-2 probes).

The only rule participants had to use to identify whether the Soldier’s route was 

safe was as follows: “If three or more critical threats are seen along your route, then 

the route is unsafe.” Critical threats differed from other threats in that they were 

armed with a rocket launcher. If 3 or more of these were present along the Soldier’s 

route, the route was classified as “Unsafe”. Participants used this rule to correctly 

answer the following SA-2 probe: “Was the Soldier’s route safe?”

Participants were also responsible for classifying the safety of the robot’s 

environment. There were 3 rules used to classify the robot’s environment as “Safe” 

or “Unsafe”. Two of these rules applied to the safety of the robot’s immediate 

environment (i.e., what is displayed within the robot’s report) while the third 

applied to the safety of the route the robot was traveling. The 2 safety rules relevant 

to the robot’s immediate environment were as follows:

1) If the robot sends a report (either text or image) that includes 3 or more 

threats, the robot’s immediate environment is unsafe.

2) If the robot detects a critical threat, the robot’s immediate environment is 

unsafe.

Figures 13 and 14 illustrate these 2 safety rules in application. 

 
Approved for public release; distribution is unlimited.

19

Fig. 13 Robot’s report shows its immediate environment is unsafe; (left) in image report 4

threats are bounded in yellow boxes while (right) text report counts the total number of threats

Fig. 14 Robot’s report shows unsafe immediate environment; (left) in image report a critical 

threat is bounded in blue box while (right) the text report counts the number of critical threats

Thus, both of these rules were needed to correctly answer the following SA-1 

probe: “Based on the currently available robot report, is the robot’s immediate 

environment safe?”

The safety rule relevant to the robot’s route was, “If the robot sends three 

consecutive unsafe reports while traveling a particular route, then that route is 

unsafe.” For example, if a participant requested reports No. 5–7 and all 3 were 

unsafe, then the entire route the robot was traveling on was unsafe. This rule applied 

even when the 3 reports were a combination of text and image (e.g., Report 5 was

image, Report 6 was text, and Report 7 was image). Participants used this rule to 

correctly answer the following SA-2 probe: “Was the robot’s route safe?”

So participants’ SA would not be influenced by the need to remember these 4 rules, 

they were given a “cheat sheet” to use during the RR scenarios. This cheat sheet 

depicted all 4 safety rules both graphically and in text (see Appendix H). In 

addition, participants were informed each scenario consisted of 4 distinct routes—

each side of the boxes depicted in Figs. 1, 2, and 8 represent a distinct route—and 

they should keep that in mind when considering the safety of a particular route and 

answering the SA questionnaires (SA-1 probes, SA-2 probes, and SART).
Approved for public release; distribution is unlimited.

20

2.4 Apparatus

The simulation was presented using a standard desktop computer or laptop with 

equal specifications (3.2-GHz, Intel Core i7 processor) connected to a 22-inch

(16:10 aspect ratio) monitor. Responses to the tasks were collected using the left 

mouse button and keyboard. Participants wore headphones to reduce any 

interference between them while viewing the narrated training slides and listening 

to any sounds generated from the simulation.

2.5 Procedures 

When participants arrived, they were instructed to first read the informed consent 

(see Appendix I). Upon consent, participants were assessed for color-vision 

deficiency; then, they completed the demographics form and the Cube Comparison 

test. Task training followed completion of the prestudy questionnaires.

2.5.1 Training

A 30-min, narrated PowerPoint presentation was used to support consistency of 

training for each participant. Training was accomplished in 3 phases. The first 

phase was 12 min, during which participants were instructed on the continuous task 

(TD) they were to perform and how to respond to the NASA-TLX. They were then 

given the chance to practice the task for about 1 min and respond to the 

NASA-TLX. 

The next 2 phases of training continued in the same format. The second phase was 

6 min and explained the purpose and elements of the 4 different RA-query display 

formats. Instructions were also provided on how to respond to the usability and 

free-response questionnaires. Participants then practiced responding to 2 display 

formats of RA queries; this illustrated the way queries would be presented and 

ensured the Soldiers understood how to respond to them. The third phase was 12

min and described the RR task and its associated SA questionnaires and probes. 

Participants practiced performing the task for about 1 min and responded to 3 SA 

probes.
Approved for public release; distribution is unlimited.

21

2.5.2 Experiment’s Design 

Following the training, each participant completed 3 experimental scenarios: one 

RA and 2 RR. To reduce the level of confusion regarding the rules for the RA and 

RR tasks, the RR scenarios were always paired in consecutive order (e.g., scenario 

orders RA, RR1, RR2 or RR2, RR1, RA). This resulted in 4 total scenariopresentation orders: 

• RA, RR1, RR2

• RA, RR2, RR1

• RR1, RR2, RA

• RR2, RR1, RA

Therefore, the order of presentation of the 3 experimental scenarios was 

randomized but only semicounterbalanced. In addition, the starting/ending location 

(i.e., starting/ending corner) within each scenario as well as the direction traveled, 

clockwise or counterclockwise, were fully counterbalanced to account for possible 

order effects.

During the study, up to 15 soldiers participated at the same time. Although grouped 

together in the facility (Fig. 15), participants worked independently of each other 

and wore headphones during tasks. 

Fig. 15 The Fort Benning laboratory while 14 Soldiers participate in the experiment at the 

same time (no identifiable information can be gathered from photograph)
Approved for public release; distribution is unlimited.

22

3. Results

3.1 RA Task 

Paired sample t-tests were used to assess the impact of various RA-query designs 

on task performance, perceived workload, and display preference. Cohen’s d effect 

sizes using the conventional scale of 0.2, 0.5, and 0.8 (small, medium, and large, 

respectively) are also reported for specific comparisons of RA-query designs. 

Bivariate correlations were run to find relationships among demographic 

characteristics, visual–spatial skills, performance, perceived workload, and 

usability preference. The sample size for all analyses was n = 29. The analyses were 

used to answer the 5 research questions related to the robot-assistance task, stated 

previously:

1) Is performance accuracy of the threat-detection task affected by the type 

(navigation or building) or display version of the robot’s request?

2) Is there a difference in accuracy of correct responses to RA requests for both 

navigational route and building selection depending on the type of 

exogenous visual cues displayed?

3) Is there a difference in reaction time to respond to visual reports for both 

navigational route and building selection depending on the type of 

exogenous visual cues displayed?

4) How is workload affected by the type of exogenous visual cues displayed 

for both navigational route and building selection?

5) How is usability preference affected by the type of exogenous visual cues 

displayed for both navigational route and building selection?

3.1.1 TD Performance

Two paired sample t-tests were run to assess the effects on TD-task-performance 

accuracy of responding to the different building and navigation RA-report designs. 

Results in Table 1 show no significant differences were found (p > 0.3 and 0.6, 

respectively).
Approved for public release; distribution is unlimited.

23

Table 1 Results of t-tests for TD task performance for both building and navigation display 

designs with mean (M), standard deviation (SD), and Cohen’s d statistical value (d) reported

M SD M SD

Threat detection 

accuracy (% correct)

Building A Building B t-test d

88.51 22.12 91.19 13.41 –0.96 0.05

Navigation A Navigation B

88.51 19.35 89.66 22.01 –0.46 0.15

Note: probability of rejection of the hypothesis (p) > 0.05

3.1.2 RA performance

Two paired sample t-tests were run to assess the effects of RA-report designs on 

RA-response accuracy. Statistically significant results were found for the difference 

in percentage of correct responses for building-identification and navigationalroute selection (Table 2). Participants made significantly more correct choices 

when responding to the Building-A and Navigation-A design.

Table 2 Results of t-tests for robot assistance response accuracy for both building and 

navigation display designs

M SD M SD

Correct response 

accuracy (% correct)

Building A Building B t-test d

57.33 18.15 45.26 16.84 2.27a 0.69

Navigation A Navigation B

71.98 15.54 55.17 22.79 3.28b 0.86

a p < 0.05 b p < 0.01

Two paired sample t-tests were run to assess the effects of RA report designs on 

response time for building identification and navigation route selection. Results 

show that no significant differences were found (Table 3).

Table 3 Results of t-tests for RA response time for both building and navigation display 

designs 

M SD M SD

Average response 

time (s)

Building A Building B t-test d

4.45 1.98 3.80 1.86 1.34 0.34

Navigation A Navigation B

5.10 2.72 4.23 1.85 1.73 0.36

Note: p > 0.05
Approved for public release; distribution is unlimited.

24

3.1.3 Questionnaires

3.1.3.1 NASA-TLX

Two paired sample t-tests were run to assess the effects of RA report designs on 

perceived workload, and statistically significant differences were found between 

report designs for mental and global demand. The Navigation-A design elicited 

higher perceived demand on both nonweighted subscales. Results are reported in 

Table 4.

Table 4 Results of t-tests for NASA-TLX ratings for both building- and navigation-display 

designs

M SD M SD

Building A Building B t-test d

Mental 17.41 22.14 17.07 18.20 0.197 0.02

Physical 7.93 14.73 8.79 13.41 –0.623 0.06

Temporal 14.14 17.12 13.97 16.60 0.108 0.01

Effort 16.03 20.59 14.31 16.84 0.935 0.09

Frustration 11.90 20.59 10.00 16.09 0.938 0.10

Performance 18.97 27.40 13.28 20.58 1.516 0.23

Global 14.40 15.50 12.90 14.36 1.509 0.10

Navigation A Navigation B

Mental 20.86 27.09 13.28 16.05 2.491a 0.34

Physical 9.14 17.06 6.72 13.11 1.545 0.16

Temporal 18.45 23.72 13.79 18.88 1.831 0.22

Effort 16.55 23.11 15.69 20.73 0.604 0.04

Frustration 17.59 25.45 13.28 20.50 1.314 0.19

Performance 15.86 23.19 16.72 22.88 –0.393 0.04

Global 16.41 19.17 13.25 15.29 2.320a 0.18

a p < 0.05

3.1.3.2 SUS

A series of t-tests was run to assess the effects of RA report design on perceived 

usability. Results show a significant difference between building-identification 

designs, but not for navigation design (Table 5). Participants rated the Building B 

higher in usability.

Table 5 Results of t-tests for usability for both building- and navigation-display designs

M SD M SD

SUS rating

Building A Building B t-test d

78.28 18.86 81.41 15.58 –2.051a .18

Navigation A Navigation B

77.66 18.87 79.83 17.10 –1.167 .12

a

 p < 0.05
Approved for public release; distribution is unlimited.

25

3.1.3.3 Free Response

Three independent raters conducted an evaluation of the data for common themes 

across participant responses to each item. Raters began by organizing the textual 

responses into common ideas and then comparing the frequency with which certain 

themes occurred in the text for each item. Common themes that were mentioned by 

at least 3 respondents for each item were retained. The 3 assessments were then 

compared for overlapping patterns identified by all 3 raters. Table 6 shows the 

results of the assessment. Within the table are the common themes, the number of 

participants who made that comment, and an example response.

Table 6 Common themes among participant responses to the Free Response Questionnaire 

for RA-display formats; numbers in parentheses next to each theme are total participants 

reporting each theme (an example of which is shown)

Item/question Theme Example responses

Positive aspects of 

the device

a) Ease of use (12)

b) Multiple views (6)

c) Benefit to Soldier (7)

a) The device was really simple and easy to use.

b) The different perspectives provided for quick analysis 

and decision-making.

c) It doesn’t place any Soldiers in harm’s way by allowing 

areas to be viewed without danger or risk involved.

Negative aspects 

of the device

Diversion (4) I had to divert my attention away from the task to make 

decisions for the robot.

Navigational

information

a) No preference for layout (9)

b) Preferred aerial view (5)

c) Clear and sufficient information 

(5)

a) The device did a very good job providing navigation. I 

did not have a preference.

b) I looked at both but preferred the overhead view because 

I was able to distinguish the area better.

c) The device provided clear and concise information to 

choose the best way for the robot to travel.

Building 

identification

a) No preference for layout (10)

b) Preferred aerial view (5)

a) I liked having both views, both were beneficial.

b) I think the aerial view was the easiest way to gauge how 

to give an answer to the presented question.

The 2 items that were not included in Table 6 probed for elaboration on the first 4

items and asked for suggested improvements. These items were not included 

because the responses did not provide any unique, additional information compared 

to the first 4 items. Few participants suggested any improvements, except one 

participant requested a zoom feature be added to provide a wider perspective of the 

environment.
Approved for public release; distribution is unlimited.

26

3.1.4 Bivariate Correlations

Bivariate correlations were run to find relationships among demographic 

characteristics, spatial skills, task performance, perceived workload, and usability.

To clarify, only the significant variables are listed in Table 7 and they refer to how 

often participants use a computer, how much experience they have playing video 

games, time when responding to the RA reports, global TLX associated with 

responding to RA reports, and RA report-design usability.
Approved for public release; distribution is unlimited.

27

Table 7 Bivariate correlation matrix for statistically significant relationships among demographics, task performance, global workload, and usability

1 2 3 4 5 6 7 8 9 10

1. How often use a computer ... ... ... ... ... ... ... ... ... ...

2. Working with video games 0.363 ... ... ... ... ... ... ... ... ...

3. RA—response time—Navigation A –0.151 0.194 ... ... ... ... ... ... ... ...

4. RA—response time—Navigation B –0.364 –0.013 ... ... ... ... ... ... ... ...

5. RA—response time—Building A –0.062 –0.106 ... ... ... ... ... ... ... ...

6. RA—response time—Building B –0.244 0.105 ... ... ... ... ... ... ... ...

7. NASA-TLX—global—Navigation A –0.399a –0.286 0.429a ... ... ... ... ... ... ...

8. NASA-TLX—global—Navigation B –0.430 a –0.318 ... 0.425a ... ... ... ... ... ...

9. NASA-TLX—global—Building A –0.513b –0.270 ... ... 0.493b ... ... ... ... ...

10. NASA-TLX—global—Building B –0.465a –0.324 ... ... ... 0.180 ... ... ... ...

11. SUS—Navigation A 0.451a 0.382a –0.177 ... ... ... –0.715b ... ... ...

12. SUS—Navigation B 0.331 0.492b ... –0.400a ... ... ... –0.695b ... ...

13. SUS—Building A 0.468a 0.437a ... ... –0.546b ... ... ... –0.607b ...

14. SUS—Building B 0.456a 0.434a ... ... ... –0.301 ... ... ... –0.696b

a p < 0.05 b p < 0.01
Approved for public release; distribution is unlimited.

28

The correlation matrix shows that the more participants used a computer, the less 

perceived workload was elicited during the task; also, they reported a higher 

usability preference. Video-game experience was also shown to have a positive 

relationship with usability preference. For the most part, longer reaction times were 

associated with higher perceived workload, but only reaction time to RA reports of 

the Building-A and Navigation-B designs were negatively associated with usability 

preference at a statistically significant level. Overall, NASA-TLX and usability 

preferences were negatively correlated.

3.2 RR Task 

One-way repeated-measures analyses of variance (ANOVAs), chi-square tests, and 

bivariate correlations were used to analyze the RR data and to address the 4 central 

research questions of the robot-reporting task. As stated previously, our research 

questions were

1) How often do Soldiers expect to receive status reports from a robot 

teammate and is this preference influenced by the difficulty of a Soldier’s 

primary task?

2) Is there a general preference among Soldiers for text-based or image-based

reports when seeking updates from a robot teammate?

3) How is SA affected by the number and type of robot reports requested by 

the Soldier?

4) Is there a performance tradeoff associated with robot reports, whereby 

primary task performance decreases as more robot reports are received by 

the Soldier?

To ensure the data used for the moderate event-rate condition (i.e., RR1 scenario) 

were comparable to data used for the low and high event-rate conditions (i.e., RR2 

scenario), only the measures corresponding to the first half of the RR1 scenario 

(i.e., the first 2 sides of the box) were used for analyses involving rate comparisons. 

This decision was made because participants spent twice as much time performing 

RR tasks in moderate than they did in low or high (Fig. 8). Thus, it was decided 

only those measures corresponding to a participant’s initial experience with the 

moderate condition would be used to ensure the data used for each event-rate 

condition were comparable (i.e., were based on a comparable amount of time spent 

within each condition). 

The sample size for all analyses was n = 29. The effect sizes for the main effect of 

each repeated-measures ANOVA were calculated using partial eta-squared (η2

) and 
Approved for public release; distribution is unlimited.

29

the effect sizes for each post hoc comparison (i.e., paired sample t-test) were 

calculated using Cohen’s d.

3.2.1 Global Workload

A one-way repeated-measures ANOVA was conducted—with event rate (low, 

moderate, or high) as the independent variable and global workload as the 

dependent variable—to assess whether the different event-rate conditions resulted 

in differences in participants’ perceived workload (i.e., global workload) while 

completing the RR scenarios. The goal of this analysis was to identify whether the 

event-rate manipulations resulted in the perception of a more (or less) difficult 

primary task. It was found there was not a significant main effect of event rate on 

global workload: Wilks’ λ = 0.88, F(2, 27) = 1.96, p = 0.161, partial η2 = 0.126.

To further investigate whether the event-rate manipulation resulted in a more 

difficult primary task, we ran a one-way repeated-measures ANOVA with the event 

rate (low, moderate, or high) as the independent variable and TD performance (% 

correct) as the dependent variable. It was found there was a significant main effect 

of event rate on TD performance: Wilks’ λ = 0.71, F(2, 27) = 5.44, p = 0.01, partial 

η2 = 0.287. To further investigate differences in TD performance among the 3

event-rate conditions, post hoc comparisons were conducted using paired-samples 

t-tests and a Bonferroni correction (p < 0.0167 was considered significantly 

different). The post hoc tests indicated participants had significantly worse TD 

performance when the event rate was high—M = 87.93, SD = 15.53—than when 

the event rate was moderate: M = 96.55, SD = 5.64; t(28) = –3.11, p = 0.004, 

Cohen’s d = –0.683. There were, however, no significant differences in TD 

performance between the moderate and low—M = 91.38, SD = 15.75; t(28) = 1.68, 

p = 0.103, Cohen’s d = 0.348—or the high and low conditions: t(28) = –1.80, p = 

0.083, Cohen’s d = –0.333. Figure 16 depicts these relationships.
Approved for public release; distribution is unlimited.

30

Fig. 16 Graph depicting TD performance across event-rate conditions for the RR scenarios

3.2.2 Number of Reports

The total number of reports requested by each participant during each route was 

calculated to identify the average number of robot reports requested during each 

RR route. It was found the Soldiers on average (averaging across all 8 RR routes 

regardless of event rate) requested a robot report 11.3 times during each route or 

once every 12.39 s. 

The same procedure was used for each event-rate condition (i.e., averaging across 

both routes within each condition) to calculate the average number of robot reports 

requested during each RR route for a given event-rate condition. Then, a one-way 

repeated-measures ANOVA was conducted—with event rate (low, moderate, or

high) as the independent variable and the number of reports as the dependent 

variable—to assess whether the number of reports requested in each condition was 

significantly different. It was found there was a significant main effect of event rate 

on the number of reports requested: Wilks’ λ = 0.55, F(2, 27) = 10.98, p < 0.001, 

partial η2 = 0.448. To further investigate differences in the number of reports 

between the 3 event-rate conditions, post hoc comparisons were made using 

paired-samples t-tests and a Bonferroni correction (p < 0.0167 was considered 

significantly different). The post hoc tests indicated participants requested 

significantly fewer robot reports when the event rate was high—M = 9.19, SD 

= 6.74—than when it was moderate: M = 12.90, SD = 9.51; t(28) = –3.12, p = 0.004, 

Cohen’s d = –0.633. Participants also requested significantly fewer robot reports 

when the event rate was high than when the event rate was low: M = 11.67, SD = 

7.27; t(28) = –4.19, p < 0.001, Cohen’s d = –0.789. There was no significant 

difference in the number of requested reports between the moderate and low 
Approved for public release; distribution is unlimited.

31

conditions: t(28) = 1.01, p = 0.322, Cohen’s d = 0.198. Figure 17 depicts these 

relationships.

Fig. 17 Graph depicting number of reports across event-rate conditions for the RR 

scenarios

3.2.3 Report-Format Preference

To identify participants’ preference when it came to report format, counts were 

used to identify the total number of text and image reports requested by each 

participant across the 2 RR scenarios (i.e., all event-rate conditions). Participants 

who requested more of one report format (> 50% of total reports requested) than 

the other were assumed to prefer the given format. For example, if a participant 

requested 80 text reports and 30 image reports, it was assumed the participant 

preferred text reports. 

In total, 17 participants preferred the text reports while 12 preferred the image 

reports. A chi-square test was run to assess whether this distribution was greater 

than chance. It was found the distribution of preferences was not statistically greater 

than chance: χ2

(1) = 0.86, p > 0.05. Furthermore, it was found that 2 of the 

participants exclusively used text reports (never requested an image report) and 2

of the participants exclusively used image reports (never requested a text report). 

Finally, it was found that 22 participants (75%) were consistent in their preference 

for either text or image reports, requesting more of their preferred format than the 

other format during 6 out of the 8 RR routes. An additional chi-square test was run 

to assess whether this finding was greater than chance. It was found the distribution 

was statistically greater than chance: χ2

(1) = 7.76, p = 0.005. Close inspection of 

the data for these participants indicated report-format preference was independent 
Approved for public release; distribution is unlimited.

32

of the event rate (i.e., participants stayed with their preferred format regardless of 

the event rate). 

3.2.4 Bivariate Correlations

A series of bivariate correlations was run to examine 1) the relationship between 3

measures of SA relevant to the Soldier’s environment: SA-1 probe, SA-2 probe, 

and overall SART rating; 2) participants’ RR preferences via the number of reports

requested and report-format preference (text preference was coded as 0; image 

preference was coded as 1); and 3) primary task performance (i.e., TD 

performance). These correlations were based upon the averages across all 8 RR 

routes, regardless of event-rate condition. Table 8 provides the correlations among 

these sets of variables. There was a significant positive correlation between the total 

number of reports requested and performance on the SA-2 probes (r = 0.559, p = 

0.002). Most of the remaining variable pairs were positively correlated but none of 

these correlations reached statistical significance.

Table 8 Bivariate correlation matrix for situation-awareness measures, reporting 

preference, and primary task performance

1 2 3 4 5 6

1. Number of reports ... ... ... ... ... ...

2. Report-format preference 0.293 ... ... ... ... ...

3. SA-1 probes—% correct 0.222 0.059 ... ... ... ...

4. SA-2 probes—% correct 0.559a 0.233 0.323 ... ... ...

5. Overall SART rating 0.117 –0.101 0.221 0.220 ... ...

6. TD performance—% correct 0.074 0.122 0.171 0.242 –0.030 ... a p < 0.01

4. Discussion

4.1 Robot Assistance

The first goal of this study was to explore the use and effects of various exogenous 

orientation visual cues within visual reports generated by a robot on a mobile 

platform to convey squad-level information to a Soldier teammate in a dismounted 

scenario. The findings support expectations that design differences have an impact 

on Solders’ perceived mental workload, usability, and response performance to 

robot reports. Each cue design type (i.e., navigation and building) is discussed 

separately.

4.1.1 Navigational Routes

For TD-task performance, no effect was shown for either type of navigation report. 

This is likely due to the TD task being held at a constant rate. Additionally, when 
Approved for public release; distribution is unlimited.

33

participants responded to the RA reports the simulation paused, which meant the 

cost of looking at the MMI on TD performance was not assessed during this 

experiment. 

The performance for the RA reports shows that participants responded correctly 

more often when viewing the Navigation-A reports (d = 0.86). The exogenous 

orienting visual cues in the Navigation-A design displayed options that showed 

participants not only which way the robot could go, but also whether the robot 

would encounter any obstacles along each route and where the robot would end up. 

The additional information conveyed in the Navigation-A design, compared to the 

Navigation-B design, seemed to support decision making by visually displaying the 

consequences of each option. Further, responding to the Navigation-A reports 

elicited higher perceived mental-demand and global-workload ratings than 

Navigation-B reports, which could be partially explained by the difference in the 

amount of information visually displayed. Interestingly, mean ratings of the 

Navigation-B design were higher in terms of usability, but the effect was fairly low 

(d = 0.12) and not statistically significant; therefore, based on this sample, no 

preference was indicated, which corresponds to the open-ended responses. A key 

point is response accuracy to RA reports might be inherently linked with perceived 

workload, meaning display designs might need to elicit a certain level of perceived 

demand to facilitate decision making and engagement, potentially through more 

content-rich visualization. This is related to past research that found too little mental 

workload (Abich et al. 2013) or too much could lead to declines in task performance 

(Hancock and Warm 1989).

Taking into account the individual differences, it seems the more often a Soldier 

uses a computer, the lower the elicited workload response (as shown in Table 7), 

which makes sense considering the entire experiment was conducted on a computer 

and the MMI is designed on a computer-based platform. Overall, these findings 

show that even though task performance may be better supported by a specific 

visual-display design, if there is an increase in perceived cognitive demand the 

usability ratings of system will decline.

4.1.2 Building Identification

Similar to the navigation-display designs, no differences were found for TD-task 

performance. This could be explained by the same interpretation above. 

Additionally, and similar to the Navigation-A design, results indicate responses to 

Building-A reports had better accuracy (d = 0.69). The design for Building-A 

reports simply placed highlighted boxes around the areas of interest as opposed to 

desaturating the visual information outside the highlighted areas as in the BuildingB design. Therefore, in terms of display-format design, all of the information 
Approved for public release; distribution is unlimited.

34

displayed visually for object identification contributes to task accuracy and should 

be included because it provides contextual information that assists identification 

(De Graef et al. 1990).

When looking at the effects on perceived workload, although not statistically 

significant, reported mean scores show the Building-B design elicited lower ratings, 

which might have contributed to the significant effect on usability (d = 0.18); yet,

the free-response questionnaires showed no stated preference. Further, 

response-accuracy performance was also lower. This could be explained thus:

having too low of a mental load potentially leads to disengagement and poor 

decision making—as discussed previously (Abich et al. 2013). Further, correlations 

showed response times to either building design were positively associated with 

NASA-TLX ratings and negatively related to usability ratings (Table 7), suggesting 

both designs are related to task performance, cognitive demand, and display 

preference.

4.1.3 Limitations and Future Research

When the participants responded to the RA reports, the simulation paused. This was 

a known limitation implemented intentionally to first assess the quality of the RA 

report designs; but, to gain more ecological validity it would be best to have the 

primary task (e.g., TD task) continue as it would in the real world. Manipulation of 

this factor would support the quantified evaluation of the cost associated with MMI 

interaction in terms of task performance, cognitive impact, and SA. (A subsequent 

study is planned to assess this impact on Soldiers receiving robot reports from 

multiple robot teammates using nonvisual displays, such as auditory or 

visual–auditory combination, during a cordon-and-search task.)

Further, the extensive control of the laboratory settings allows for precise 

assessment, but limits the ecological validity of the findings as well. Ultimately, 

the results from this program of study will drive design of deployable products that 

will support Soldiers in a dismounted environment and, therefore, a balance of both 

field and lab studies is necessary to foster a transfer of findings from the lab to the 

field.

The intention of the free-response questions was to probe participants about their 

interactions with the MMI and to provide any suggestions for improvement. 

Although some participants did provide feedback regarding the MMI, it seems 

more responses were directed toward the capabilities of the robot—even though 

participants were told that evaluating the robot was not the focus of the study. In 

fact, the robot was never seen—just reports from the robot were displayed—yet 

participants were still evaluating the (notional) robot. 
Approved for public release; distribution is unlimited.

35

The next step will be to explore the effects of robot reports conveyed through other

modalities, such as auditory or tactile, on dismounted Soldier performance and to 

use a more dynamic approach (i.e., higher-fidelity simulation).

4.2 Robot Reporting

The second goal of this study was to explore Soldier preferences when it came to 

receiving status updates (in the form of robot reports) from a robot teammate. Also 

of interest was the impact these reporting preferences had on participant SA and 

primary task performance.

4.2.1 Task Difficulty

The analysis of both global workload and TD performance served the purpose of 

assessing whether the event-rate manipulation resulted in a more or less difficult 

primary task for participants. We expected to find the low event-rate condition 

would be the least difficult (i.e., lowest global-workload scores and highest TD 

performance), high to be the most difficult (i.e., highest global-workload scores and 

lowest TD performance), and moderate to be between low and high in difficulty in 

terms of both global workload and TD performance. Instead, our results were 

inconclusive (as shown in Figs. 16 and 17). Participants did not perceive the eventrate manipulations as different, from a workload standpoint, and their performance 

was similar across most of the conditions. The only difference found was in TD 

performance between the moderate and high conditions. 

Overall, it appears the TD task may have been a bit too easy for participants. 

Evidence of this comes from the fact that event-rate manipulation did not 

substantially impact participants (from either a workload or a performance 

perspective). Looking at the means for the 3 conditions for both global workload 

and TD performance, we see further evidence of this possible explanation. In all 3

conditions, the mean TD performance (i.e., accuracy) was more than 87%. In 

addition, mean global workload was below 30 (the scale is 0–100) for all 3

conditions. Taken together, this indicates participants tended to perform well at the 

TD task and perceive it to be minimally demanding across all 3 conditions. 

4.2.2 Reporting Frequency

It was anticipated participants would request fewer reports from the robot as the 

event rate increased due to increased demands on the participant. In particular, we 

believed participants would spend more time focused on the TD task (due to the 

increased number of threats in the Soldier’s environment) with less time spent 

requesting reports from the robot. We expected a linear relationship whereby most 
Approved for public release; distribution is unlimited.

36

reports would be requested in the low event-rate condition, the second most in the 

moderate condition, and the least in the high condition. This hypothesis was only 

partially supported. The high condition did result in fewer reports being requested 

than in both the low and moderate conditions, but a similar number of reports were 

requested in both low and moderate (Fig. 17).

Overall, while the pattern of results did not match what we predicted, there were 

differences among the conditions. These findings highlight the importance of 

considering how “busy” one’s team member may be when deciding how often to 

push information to them. Considering that Soldiers must perform a variety of tasks 

that are more or less time/resource intensive, this finding provides initial evidence 

that having a robot send reports to a Soldier teammate based on a strict rule (e.g., 

Once every X seconds) may not be ideal. Instead, it may be more appropriate to 

calibrate sending of reports based on nature of the task the Soldier will be 

performing.

4.2.3 Format Preference

When it came to Soldier preferences for a specific report format, more Soldiers 

preferred text reports than image reports, albeit this finding was not significant (i.e., 

this observed preference could be due to chance). While the Soldiers were not 

formally asked to describe why they chose/preferred one format over the other, 

several Soldiers informally shared their thoughts during (or after completing) the 

experiment. Soldiers who preferred the text reports mentioned the reports helped 

them process the information they needed quicker than image reports. This was 

because the text reports presented participants with the number of threats, critical 

threats, and nonthreats present, whereas the image reports required the participants 

to add up the number of threats and/or critical threats seen in the report. As for the 

image reports, Soldiers in favor of these reports may have found it easier to keep 

track of the reports due to each looking distinctly different from previous reports. 

Some Soldiers mentioned having difficulty quickly identifying the number of 

threats and critical threats with the text reports due to the font being similar for each 

category (i.e., same font style, size, and color).

Seventy-five percent of the participants were consistent in their preference 

throughout most of the RR scenarios; that is, they identified a preferred reporting 

format during the first or second RR route and stuck to that preference throughout 

the remainder of the RR routes, regardless of the event rate. The remaining 

participants may have switched back and forth to experiment with the 2 formats 

before deciding which they preferred. The fact most participants immediately 

decided upon a preferred report format is evidence it may be appropriate to allow 

Soldiers to inform their robot teammate of their preferred reporting format before 
Approved for public release; distribution is unlimited.

37

beginning a mission. Only 4 Soldiers exclusively requested one format throughout 

all RR routes. Thus, 25 out of 29 (86%) of the Soldiers used both report formats in 

some capacity throughout the RR scenarios. This finding is initial evidence that 

despite having a general preference for a specific reporting format, Soldiers may 

still like to have another format available to them. It is possible the Soldiers may 

have wanted to use both report formats to double check the information in one 

report format against the other. Alternatively, at certain times during their tasking, 

having both formats may have helped them feel more comfortable in understanding 

the information that was currently available. However, because the Soldiers were 

not directly asked about their preference, we can only speculate on the reasons why 

both formats were used to such an extent. 

4.2.4 Soldier SA

Despite the fact most linear relationships among the reporting variables (total 

number of reports and report format preference) and Soldier SA variables (SA-1, 

SA-2, and SART) were not strong enough to reach statistical significance, a 

discussion of the data trends is still relevant. In particular, it was expected there 

would be a negative correlation between the total number of reports and 

performance on both the Soldier SA-1 and SA-2 probes. In actuality, these 2 

relationships were positive. This means participants who requested more reports 

from their robot teammate tended to perform better when asked about the Soldier’s 

environment within the MIX simulation. 

This finding is puzzling—it would seem participants who spent more time 

reviewing robot reports would be less able to keep track of the number of critical 

threats identified along the Soldier’s route. One possible explanation for this 

finding may be that participants who requested more reports were more engaged in 

the tasks in general. In other words, participants who were requesting reports more 

frequently may have been more motivated to perform the experimental tasks in 

general and, therefore, kept better track of the total number of critical threats present 

along the Soldier’s route. 

The finding that TD performance was positively correlated to both Soldier SA 

probes (as opposed to negatively correlated) further supports this possible 

explanation. Another interesting finding is report-format preference was positively 

correlated with both the SA-1 and SA-2 probes but negatively correlated with the 

SART. This provides initial evidence that participants who preferred image reports 

tended to perform better on both the SA-1 and SA-2 probes (i.e., had higher SA) 

while simultaneously having lower perceived SA than participants who preferred 

text reports. Taken together, these relationships indicates that participants who 
Approved for public release; distribution is unlimited.

38

preferred image reports may have believed they were less aware of the Soldier’s 

environment than they actually were.

4.2.5 Limitations and Future Research

Three limitations related to the RR task should be considered.

First, despite the fact the event rate of the primary task was manipulated to achieve 

low, moderate, and high task difficulty (based on previous research with an 

undergraduate student population), participants did not necessarily perceive the 

different event-rate conditions as expected (nor did they perform as expected). 

Participants may not have felt the event-rate manipulations were substantial enough 

to change their perceptions of the primary task. Future research should pilot-test 

any difficulty/workload manipulations on the population of interest before full 

implementation with the final sample.

A second limitation related to the RR task is that participant preferences for robotreport format and frequency were completely derived from behavioral data (i.e., 

total number of reports and the number of each type of report requested). While the 

use of behavioral data resulted in interesting insights, the inclusion of surveys or 

questionnaires would have allowed participants to share their opinions and 

elaborate on why they preferred one format to another. Without these subjective 

data, it is difficult to know exactly why a given participant chose one format over 

another. Future research could address this by including a pre- and post-task survey 

that asks participants for their opinions of the 2 different reporting formats. 

Finally, future research should also consider other factors that may influence a 

Soldier’s preference for robot reporting such as reliability, trust, and the nature of 

the task.

5. Conclusions

In general, participants felt the MMI was fairly easy to use, was simple and straight 

forward, and could greatly benefit Soldiers in terms of mission safety by supporting 

remote interaction with a robot teammate. As expected, concerns the visual display 

could distract or allocate the attentional resources of the Soldiers away from their 

primary tasking were expressed in the free-response questionnaire. This is a 

primary reason the US Army is developing visual displays on mobile platforms 

(Young 2014); the intention is these displays will only be used for quick reference

or response and should not require extensive viewing time. Additionally, part of the 

US Army’s Robotics Collaborative Technology Alliance (RCTA 2014) research 
Approved for public release; distribution is unlimited.

39

interests focuses on the use of other modalities that do not require a visual display 

for bidirectional communication between dismounted Soldier–robot teammates. 

In addition, reporting frequency was impacted by primary task difficulty, which 

supports the idea of adaptive automation that manages the frequency of 

robot-to-human information sharing. While more participants showed a preference 

for reports that condensed and summarized information (i.e., reports displayed in a 

text format), participants still liked having both report formats available. For future 

MMI display designs this may mean redundancy in the presentation of information 

will likely be considered valuable to users.
Approved for public release; distribution is unlimited.

40

References

Abich J IV, Reinerman-Jones L, Taylor G. Establishing workload manipulations 

utilizing a simulated environment. In: Shumaker R, editor. Virtual augmented 

and mixed reality. Systems and applications: 5th International Conference, 

VAMR 2013, held as part of HCI International 2013; Las Vegas, NV: Springer 

International Publisher; 2013. vol. 8022, p. 211–220.

Abich J IV, Barber D, Reinerman-Jones L. Experimental environments for 

dismounted human-robot multimodal communications. In: Shumaker R, 

Lackey S, editors. Virtual, augmented and mixed reality: 7th International 

Conference, VAMR 2015, held as part of HCI International 2015; Los Angeles 

(CA): Springer International Publishing; 2015. vol. 9179, p. 165–173.

Barber D, Davis L, Nicholson D, Finkelstein N, Chen JYC. The mixed initiative 

experimental (MIX) testbed for human robot interactions with vaired levels of 

automation. Proceedings of the 26th Army Science Conference; 2008; Orlando 

(FL): US Department of Army.

Barber D, Lackey S, Reinerman-Jones L, Hudson I. Visual and tactile interfaces for 

bi-directional huamn-robot communication. SPIE defense, security, and 

sensing. International Society for Optics and Photonics; 2013. p. 1–11.

Barber D, Abich J IV, Phillips E, Talone A, Jentsch F, Hill S. Field assessment of 

multimodal communication for dismounted human-robot teams. Proceedings 

of the Human Factors and Ergonomics Society Annual Meeting; 2015; Los 

Angeles (CA): SAGE Publications. p. 921–925.

Brooke J. SUS–A quick and dirty usability scale. Usab Eval Ind. 1996;189(194):4–

7.

Burke JL, Murphy RR, Coovert MD, Riddle DL. Moonlight in Miami: a field study 

of human-robot interaction in the context of an urban search and rescue disaster 

response training exercise. Hum-Comp Interact. 2004;19(1–2):85–116.

Card SK, Moran TP, Newell A. The model human processor. In: Boff KR,

Kaufman L, Thomas JP, editors. Handbook of perception and human 

performance. New York (NY): John Wiley and Sons; 1983. p. 1–35.

Casper J, Murphy RR. Human-robot interactions during the robot-assisted urban 

search and rescue response at the World Trade Center. IEEE transactions on 

systems, man, and cybernetics, part B: cybernetics. 2003;33(3):367–385.
Approved for public release; distribution is unlimited.

41

Chen JYC, Haas EC, Barnes MJ. Human performance issues and user interface 

design for teleoperated robots. Systems, man, and cybernetics, part C: 

applications and reviews, IEEE transactions. 2007;37(6):1231–1245.

Chittaro L. Visualizing information on mobile devices. Computer. 2006;39(3):40–

45.

Clegg DL, Rodgers AC. Transitioning unmanned underwater vehicle technology 

from commercial use to naval mine countermeasures operations. OCEANS 

2015 - MTS/IEEE. Washington, DC: IEEE, 2015. p. 1–7.

DARPA. Squad X Core Technologies seeks to bring technological advances to the 

infantry squad. 2015 Feb 9. [accessed 2015 Aug 4]. www.darpa.mil.

http://www.darpa.mil/news-events/2015-02-09.

De Graef P, Christiaens D, d’Ydewalle G. Perceptual effects of scene context on 

object identification. Psych Res. 1990;52:317–329.

Dumas B, Lalanne D, Oviatt S. Multimodal interfaces: a survey of principles, 

models, and frameworks. In: Lalanne D, Kohlas J, editors. Human machine 

interaction: lecture notes in computer science. Berlin: Springer Berlin 

Heidelberg; 2009. p. 3–26

Endsley MR. Situation awareness global assessment technique (SAGAT).

Proceedings of the IEEE 1988 National Aerospace and Electronics 

Conference; 1988; Piscata (NJ): IEEE. p. 789–795.

Endsley MR. Toward a theory of situation awareness in dynamic systems. Hum 

Fac: J Hum Fac Ergo Soc. 1995;37(1):32–64.

Endsley M, Sollenberger RL, Nakata A, Stein ES. Situation awareness in air traffic 

control: enhanced displays for advanced operations. Atlantic City (NJ): 

Federal Aviation Administration (US); 2000 Jan. Report No.: DOT/FAA/CTTN00/01.

Glumm MM, Branscome TA, Patton DJ, Mullins LL, Burton PA. The effects of an 

auditory versus a visual presentation of information on Soldier performance.

Human Factors and Ergonomics Society 43rd Annual Meeting; 1999. p. 1041–

1043.

Hancock PA, Warm JS. A dynamic model of stress and sustained attention. Hum 

Fac. 1989;31(5):519–537.

Harris J, Barber D. Speech and gesture interfaces for squad level human robot 

teaming. Proceedings from SPIE 9084: Unmanned Systems Technology XVI, 

90840B; 2014; Baltimore, MD.
Approved for public release; distribution is unlimited.

42

Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of 

empirical and theoretical research. In: Hancock PA, Meshkati N, editors.

Human mental workload. Amsterdam: North Holland Press; 1988.

Ishihara S. Ishihara’s tests for colour deficiency: concise edition. Tokyo: Kanehara 

Trading Inc.; 2014.

Johannesen LJ, Cook RI, Woods DD. Cooperative communications in dynamic 

fault management. Proceedings of the Human Factors and Ergonomics Society 

Annual Meeting; 1994; 38, no. 4. p. 225–229.

Jones-Bonbrest N. Army S and T community assesses emerging technology at 

field-based event. 2015 Aug 4. [accessed 2015 Aug 4] www.cerdec.army.mil. 

http://www.cerdec.army.mil/news_and_media/Army_S_and_T_community_

assesses_emerging_technology_at_field_based-event/.

Kahneman D. Attention and effort. Ennglewood Cliffs (NJ): Prentice-Hall; 1973.

Kott A, Alberts D, Zalman A, Shakarian P, Maymi F, Wang C, Qu G. Visualizing 

the tactical ground battlefield in the year 2050: workshop report. Adelphi 

Laboratory Center (MD): Army Research Laboratory (US); 2015 Jun. Report 

No.: ARL-SR-0327.

Lichiardopol S. A survey on teleoperation. Technische Universiteit Eindhoven,

Netherlands. 2007 Jan. Report No.: DCT 2007.155.

MacMillan J, Entin EE, Serfaty D. Communication overhead: the hidden cost of 

team cognition. In: Salas E, Fiore SM, editors.Team cognition: understanding 

the factors that drive process and performance. Washington (DC): American 

Psychological Association; 2004. p. 61–82.

Mayer AR, Dorflinger JM, Rao SM, Seidenberg M. Neural networks underlying 

endogenous and exogenous visual–spatial orienting. NeuroImage.

2004;23(1):534–541.

Mazmanian A. DOD seeks ideas on mobile phones of the future. FCW: the business 

of federal technology; 2014 July 7 [accessed 2016 Apr 13]. https://fcw.com

/articles/2014/07/07/dod-seeks-mobile-ideas.aspx 

McCall A. Army’s “edgy” concept: the warfighter of the future; 2010 Aug 31 

[accessed 2015 Aug 4] armylive.com. http://armylive.dodlive.mil/index.php

/2010/08/armys-edgy-concept-the-warfighter-of-the-future/.

Mulckhuyse M, Theeuwes J. Unconscious attentional orienting to exogenous cues: 

a review of the literature. Acta Psych. 2010;134(1):299–309.
Approved for public release; distribution is unlimited.

43

Osborn K. Navy to deploy first underwater drones from submarines. 2015 Apr 13

[accessed 2015 Aug 12] http://www.military.com/daily-news/2015/04/13

/navy-to-deploy-first-underwater-drones-from-submarines.html.

Partan S, Marler P. Communication goes multimodal. Science.

1999;283(5406):1272–1273.

Posner MI. Orienting of attention. Exp Psych. 1980;32(1):3–25.

Posner MI, Snyder CRR, Davidson BJ. Attention and detection of signals. J Exp 

Psych: Gen. 1980;109(2):160–174.

Reinerman-Jones L, Barber D, Lackey S, Nicholson D. Developing methods for 

utilizing physiological measures. In: Tadeusz M, Waldemar K, Rice V, editors. 

Advances in understanding human performance: neuroergonomics, human 

factors design, and special populations. Boca Raton (FL): CRC Press; 2010.

[RCTA] Robotics Collaborative Technology Alliance. FY 2014 annual program 

plan. Contract No. W911NF-10-2-0016; 2014.

Salas E, Prince C, Baker DP, Shrestha L. Situation awareness in team performance: 

implications for measurement and training. Hum Fac: J Hum Fac Ergo Soc.

1995;37(1):123–136.

Salcedo JN, Lackey SJ, Reinerman-Jones L. Massed exposure improves response 

time for detection of nervous human behaviors in robot-aided ISR missions.

MODSIM World Conference 2014; Hampton (VA), 2014. p. 1–10.

Scholtz J. Theory and evaluation of human robot interactions. Proceedings of the 

36th Annual Hawaii International Conference on System Sciences; 2003; 

Piscataway (NJ): IEEE.

Taylor RM. Situation awareness rating technique (SART): the development of a 

tool for aircrew systems design. AGARD Conference Proceedings No 478, 

Situational Awareness in Aerospace Operations; 1990.

Valvanis KP, editor. Advances in unmanned aerial vehicles: state of the art and the 

road to autonomy. Netherlands: Springer; 2008. vol. 33.

Wickens CD, Hollands JG. Attention, time-sharing, and workload. In: Roberts N, 

Webber B, editors. Engineering psychology and human performance, vol. 3. 

Upper Saddle River (NJ): Prentice-Hall Inc.; 2000. p. 439–479.

Wogalter MS, Conzola VC, Smith-Jackson TL. Research-based guidelines for 

warning design and evaluation. App Ergo. 2002;33(1):219–230.
Approved for public release; distribution is unlimited.

44

Yanco HA, Drury JL. Rescuing interfaces: a multi-year study of human-robot 

interaction at the AAAI robot rescue competition. Auton Rob.

2007;22(4):333–352.

Young R. Federal mobile computing summit. www.fedsummits.com. 2014 Aug 20 

[accessed 2015 Aug 4]. http://www.fedsummits.com/wp-content/uploads

/2014/09/Young-slides-Mobile-Summit-2014-08-201.pdf.

Zhang W, Luck SJ. Sudden death and gradual decay in visual working memory.

Psych Sci. 2009;20(4):423–428.

Zych N, Silver D, Stager D, Green C, Pilarski T, Fischer J, Kuntz N, Anderson D, 

Costa A, Gannon J, et al. Achieving integrated convoys: cargo unmanned 

ground vehicle development and experimentation. Proceedings of SPIE 8714, 

Unmanned Systems Technology XV 87401Y. 2013.
Approved for public release; distribution is unlimited.

45

Appendix A. Demographics Questionnaire

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

46

Demographics Questionnaire

1. General Information

a. Age (yrs): _____ b. Gender: ___M ___F

c. Do you have corrected vision? ___None ___Glasses ___Contact Lenses

d. Do you have any type of color blindness/color vision deficiency? ___Yes ___ No

e. What is your native language? _______________

f. Do you speak more than one language? ___Yes ___No

g. If you answered YES to question 1.f, how fluent would you rate your ability to 

speak a secondary language?

1 2 3 4 5 6

Very low 

fluency

Low 

fluency

Moderate 

fluency

Above 

moderate 

fluency

High 

fluency

Very high 

fluency

2. Military Experience

a. How many years have you been in the military? ________ b. Current rank ______

c. What is your MOS? ________________

d. Please list all combat deployments (Iraq, Afghanistan, etc.) and the length (Years / 

Months) of each.

Location Time

_________________________________ ______________________

_________________________________ ______________________

_________________________________ ______________________

_________________________________ ______________________

_________________________________ ______________________

_________________________________ ______________________

e. Do you have operational experience in complex urban terrain? ___Yes ___No

f. Do you have operational experience in reconnaissance situations? ___Yes ___No
Approved for public release; distribution is unlimited.

47

3. Education

a. What is your highest level of education received? Select one. 

____ GED 

____ High School 

____ Some College 

____ Bachelor’s Degree 

____ M.S/M.A 

 ____ Ph.D or other doctorate. 

Other: ______________________

b. If applicable, what subject is your degree in (for example, Criminal justice)? 

________________________________________________________________

4. Computer Experience

a. How long have you been using a computer?

1 2 3 4 5 6

Never Less than 

1 year

1-3 years 4-6 years 7-10 years 10 years or 

more

b. How often do you use a computer? 

1 2 3 4 5 6

Less than 

1 hour a 

day

1-2 hours a 

day

Over 2 

hours a 

day

Weekly Monthly A few

times a 

year

c. For each of the following questions, circle the response that best describes how 

often you:

Use a mouse: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

Use a joystick: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

Use a touch screen: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily
Approved for public release; distribution is unlimited.

48

Use icon-based programs/software: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

Use programs/software with pull-down menus: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

Use a graphics/drawing features in software packages: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

5. Video Game Experience

a. Please indicate how often you play video games: 

1 2 3 4 5 6

Never Rarely Once every 

few 

months

Monthly Weekly Daily

b. Please indicate how you would rate your experience in working with any type of 

video games:

1 2 3 4 5 6

Not at all 

familiar

Somewhat 

familiar

Moderately 

familiar

Above 

moderately 

familiar

Highly 

familiar

Very 

highly

familiar

c. Which type of video game do you play most often?

____Action-adventure

____First person shooters 

____Military-based

____Mobile/cellphone games

____Multiplayer online gaming

____Role playing

____Serious games/Educational

____Simulation

____Strategy

____Sports

____Other, please indicate which one: 

________________________________

6. Robotics Experience

a. Have you any experience with military robots? ___Yes ___No

b. If you answered YES to question 6.a, what type of robots and for what purpose?

Type Purpose
Approved for public release; distribution is unlimited.

49

_________________ ________________________________________________

__________________ ________________________________________________

c. Please indicate how you would rate your level of experience with any robots:

1 2 3 4 5 6

Not at all 

familiar

Somewhat 

familiar

Moderately 

familiar

Above 

moderately 

familiar

Highly 

familiar

Very 

highly

familiar

d. Please indicate how you would rate your level of knowledge regarding robotics 

technology (e.g., pack bot, big dog, talon, AIBO etc.):

1 2 3 4 5 6

Not at all 

familiar

Somewhat 

familiar

Moderately 

familiar

Above 

moderately 

familiar

Highly 

familiar

Very 

highly

familiar
Approved for public release; distribution is unlimited.

50

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

51

Appendix B. Cube Comparison Questionnaire

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

52

Cube Comparison Test
Approved for public release; distribution is unlimited.

53
Approved for public release; distribution is unlimited.

54

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

55

Appendix C. NASA-Task Load Index (NASA-TLX)

____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

56

NASA-TLX Questionnaire

Please rate your overall impression of demands imposed on you during the exercise.

1. Mental Demand: How much mental and perceptual activity was required (e.g., 

thinking, looking, searching, etc.)? Was the task easy or demanding, simple or 

complex, exacting or forgiving?

LOW |-----|----|----|----|----|----|----|----|----| HIGH

 1 2 3 4 5 6 7 8 9 10

2. Physical Demand: How much physical activity was required (e.g., pushing, 

pulling, turning, controlling, activating, etc.)? Was the task easy or demanding, 

slow or brisk, slack or strenuous, restful or laborious?

LOW |-----|----|----|----|----|----|----|----|----| HIGH

 1 2 3 4 5 6 7 8 9 10

3. Temporal Demand: How much time pressure did you feel due to the rate or pace 

at which the task or task elements occurred? Was the pace slow and leisurely or 

rapid and frantic?

LOW |-----|----|----|----|----|----|----|----|----| HIGH

 1 2 3 4 5 6 7 8 9 10

4. Level of Effort: How hard did you have to work (mentally and physically) to 

accomplish your level of performance?

LOW |-----|----|----|----|----|----|----|----|----| HIGH

 1 2 3 4 5 6 7 8 9 10

5. Level of Frustration: How insecure, discouraged, irritated, stressed and annoyed 

versus secure, gratified, content, relaxed and complacent did you feel during the 

task?

LOW |-----|----|----|----|----|----|----|----|----| HIGH

 1 2 3 4 5 6 7 8 9 10

6. Performance: How successful do you think you were in accomplishing the goals 

of the task set by the experimenter (or yourself)? How satisfied were you with your 

performance in accomplishing these goals?

GOOD |-----|----|----|----|----|----|----|----|----| POOR

 1 2 3 4 5 6 7 8 9 10
Approved for public release; distribution is unlimited.

57

Appendix D. System Usability Scale

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

58

Comparison of System Usability Questionnaire

Please circle the response you feel is most accurate

1. I think that I would like to use the device frequently

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

2. I found the device unnecessarily complex

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

3. I thought the device was easy to use

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

4. I think that I would need the support of a technical person to be able to use 

this device

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

5. I found the various functions in this device were well integrated

Strongly 

disagree

Strongly 

agree

1 2 3 4 5
Approved for public release; distribution is unlimited.

59

6. I thought there was too much inconsistency in this device

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

7. I would imagine that the most people would learn to use this device very 

quickly

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

8. I found the device very cumbersome to use

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

9. I felt very confident using the device

Strongly 

disagree

Strongly 

agree

1 2 3 4 5

10. I needed to learn a lot of things before I could get going with this device

Strongly 

disagree

Strongly 

agree

1 2 3 4 5
Approved for public release; distribution is unlimited.

60

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

61

Appendix E. Free Response Questionnaire

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

62

Free Response Questionnaire

Instructions: Please answer the following questions about your experience with 

the device.

1. Please list the most positive aspect(s) of the device:

2. Please list the most negative aspect(s) of the device:

3. What do you think of the way in which the device provided building 

identification information? Did you have a preference?

4. What do you think of the way in which the device provided navigation 

information? Did you have a preference?

5. What else did you think was important, remarkable or surprising about your 

interaction with the robot using the device? Additional comments:

6. Please comment on any features of the device that you think had an impact on 

your interactions with the robot. You can comment on both positive and negative 

features, and suggest improvements.
Approved for public release; distribution is unlimited.

63

Appendix F. Situation Awareness Global Assessment Technique 

(SAGAT) Probes

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

64

SAGAT Probes

SA-1 Probes

“1. Based on the currently available robot report, is the robot’s immediate 

environment safe?”

“2. Up to this point, how many critical threats have been seen on the Soldier’s 

route?”

SA-2 Probes

“1. Was the robot’s route safe?”

“2. Was the Soldier’s route safe?”
Approved for public release; distribution is unlimited.

65

Appendix G. Situation Awareness Rating Technique (SART),

Adapted

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

66

SART (Adapted)

Please rate your awareness of the situation by circling a number, according to 

the following dimensions.

Dimension 1: Demand on attentional resources

Low High

Instability: 

How likely did you feel that the 

demands of this mission and 

anything related to the situation 

would suddenly change?

1 2 3 4 5 6 7

Complexity: 

How complex did you feel this

mission and anything related to 

the situation was?

1 2 3 4 5 6 7

Variability: 

How many elements of the 

mission did you feel were 

changing in this mission and 

anything related to the situation?

1 2 3 4 5 6 7

Dimension 2: Supply of attentional resources

Low High

Arousal: 

How ready did you feel you were 

in completing this mission and 

anything related to the situation?

1 2 3 4 5 6 7

Spare Mental capacity: 

How much attention did you feel 

you had left and could direct to 

other tasks?

1 2 3 4 5 6 7

Concentration: 

Did you feel you were able to 

concentrate on completing this 

mission and anything related to 

the situation?

1 2 3 4 5 6 7

Division of attention: 

How much attention did you 

direct towards this mission and 

anything related to the situation?

1 2 3 4 5 6 7
Approved for public release; distribution is unlimited.

67

Dimension 3: Your understanding of the situation

Low High

Information quantity: 

How much information did you 

feel you understood while 

completing the mission?

1 2 3 4 5 6 7

Information quality: 

How good was the information 

you received while completing 

this mission?

1 2 3 4 5 6 7

Familiarity: 

How much knowledge did you 

feel you had about this mission 

and anything related to the 

situation?

1 2 3 4 5 6 7

Please rate your awareness of the robot’s behaviors by circling a number, 

according to the following dimensions.

Dimension 4: Situation awareness of the robot

Low High

Perception: 

Please rate your knowledge of the 

robot’s location, its surrounding 

environment, and its status during 

the course of the mission.

1 2 3 4 5 6 7

Comprehension: 

Please rate your ability to derive 

meaning from perception of the 

robot (e.g., its location, 

surroundings, and status) while 

completing the mission. 

1 2 3 4 5 6 7

Projection: 

Please rate your ability to predict 

how the robot would behave in 

the near future, while completing 

the mission.

1 2 3 4 5 6 7
Approved for public release; distribution is unlimited.

68

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

69

Appendix H. Cheat Sheet for Robot-Reporting Scenarios

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

70
Approved for public release; distribution is unlimited.

71

Appendix I. Informed Consent

_____________________

This appendix appears in its original form, without editorial change.
Approved for public release; distribution is unlimited.

72
Approved for public release; distribution is unlimited.

73
Approved for public release; distribution is unlimited.

74
Approved for public release; distribution is unlimited.

75
Approved for public release; distribution is unlimited.

76

INTENTIONALLY LEFT BLANK.
Approved for public release; distribution is unlimited.

77

List of Symbols, Abbreviations, and Acronyms

ANOVA analysis of variance

B blue

D Cohen’s d statistical value

DARPA Defense Advanced Research Projects Agency

G green

HRI human–robot interface

M mean

MIX Mixed Initiative eXperimental

MMC multimodal communication

MMI multimodal interface

NASA National Aeronautics and Space Administration

p probability of rejection of the hypothesis

POV point of view

R red

RA robot assistance

RR robot reporting

SA situation awareness

SAGAT Situation Awareness Global Assessment Technique

SART Situation Awareness Rating Technique

SD standard deviation

SUS System Usability Scale

TD threat detection

TLX Task Load Index

XUV eXperimental unmanned vehicle
Approved for public release; distribution is unlimited.

78

1 DEFENSE TECHNICAL

(PDF) INFORMATION CTR

DTIC OCA

2 DIR ARL

(PDF) IMAL HRA

RECORDS MGMT

RDRL DCL

TECH LIB

1 GOVT PRINTG OFC

(PDF) A MALHOTRA

1 DIR USARL

(PDF) RDRL HRB AB

J ABICH

1 ARL

(PDF) RDRL HRB B

T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 

35898-7290

8 ARL

(PDF) SFC PAUL RAY SMITH 

CENTER

RDRL HRO COL H BUHL

RDRL HRF J CHEN

RDRL HRA I MARTINEZ

RDRL HRR R SOTTILARE

RDRL HRA C A RODRIGUEZ

RDRL HRA B G GOODWIN

RDRL HRA A C METEVIER

RDRL HRA D B PETTIT

12423 RESEARCH PARKWAY

ORLANDO FL 32826

1 USA ARMY G1

(PDF) DAPE HSI B KNAPP

300 ARMY PENTAGON 

RM 2C489

WASHINGTON DC 20310-0300

1 USAF 711 HPW

(PDF) 711 HPW/RH K GEISS

2698 G ST BLDG 190 

WRIGHT PATTERSON AFB OH 

45433-7604

1 USN ONR

(PDF) ONR CODE 34 P MASON

875 N RANDOLPH STREET

SUITE 1425 

ARLINGTON VA 22203-1995

1 USN ONR

(PDF) ONR CODE 341 J TANGNEY

875 N RANDOLPH STREET

BLDG 87 

ARLINGTON VA 22203-1986

1 USA HQDA 

(PDF) ARI M SAMS

6000 6TH STREET BLDG 1464 

FT BELVOIR VA 22060

1 USA NSRDEC

(PDF) RDNS D D TAMILIO

10 GENERAL GREENE AVE 

NATICK MA 01760-2642

1 OSD OUSD ATL

(PDF) HPT&B B PETRO

4800 MARK CENTER DRIVE

SUITE 17E08

ALEXANDRIA VA 22350

ABERDEEN PROVING GROUND

11 ARL

(PDF) RDRL HR

J LOCKETT

P FRANASZCZUK

K MCDOWELL

K OIE

RDRL HRB

D HEADLEY

RDRL HRB C

J GRYNOVICKI

RDRL HRB D

C PAULILLO

RDRL HRF A

A DECOSTANZA

RDRL HRF B

A EVANS

RDRL HRF C

J GASTON

RDRL HRF D

A MARATHE