Supervisory Control of Unmanned Vehicles 

by Jessie Y. C. Chen, Michael J. Barnes, and Michelle Harper-Sciarini 

ARL-TR-5136 April 2010 

Approved for public release; distribution is unlimited. 
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position unless 

so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or approval of the 

use thereof. 

Destroy this report when it is no longer needed. Do not return it to the originator. 
Army Research Laboratory 

Aberdeen Proving Ground, MD 21005-5425 

ARL-TR-5136 April 2010 

Supervisory Control of Unmanned Vehicles 

Jessie Y. C. Chen and Michael J. Barnes 

Human Research and Engineering Directorate, ARL 

Michelle Harper-Sciarini

University of Central Florida 

Approved for public release; distribution is unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering 

and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson 

Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to 

comply with a collection of information if it does not display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

April 2010 

2. REPORT TYPE 

Final 

3. DATES COVERED (From - To) 

June 2008–June 2009 

4. TITLE AND SUBTITLE 

Supervisory Control of Unmanned Vehicles 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

5c. PROGRAM ELEMENT NUMBER 

6. AUTHOR(S) 

Jessie Y. C. Chen, Michael J. Barnes, and Michelle Harper-Sciarini

5d. PROJECT NUMBER 

622716H70 

5e. TASK NUMBER 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Laboratory 

ATTN: RDRL-HRM-AT 

Aberdeen Proving Ground, MD 21005-5425 

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-TR-5136 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 

11. SPONSOR/MONITOR'S REPORT 

 NUMBER(S) 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES 



Institute for Simulation and Training, University of Central Florida, 3280 Progress Dr., Orlando, FL 32826 

14. ABSTRACT 

The purpose of this report is to review research pertaining to the limitations and advantages of supervisory control for 

unmanned vehicle (UV) systems. We identify and discuss results showing technologies that mitigate observed problems, such 

as specialized interfaces and adaptive systems. In the report, we first present an overview of definitions and important terms of 

supervisory control and human-agent teaming. We then discuss human performance issues in supervisory control of UVs with 

regard to operator multitasking performance, trust in automation, situation awareness, and operator workload. In the following

sections, we review research findings for specific areas of supervisory control of unmanned air vehicles, unmanned ground 

vehicles, and heterogeneous UVs (i.e., using different types of UVs in the same mission). In the last section, we review 

innovative techniques and technologies designed to enhance operator performance and reduce potential performance 

degradations identified in the literature. 

15. SUBJECT TERMS

performance, user interface design, supervisory control, unmanned vehicle, robotics operator 

16. SECURITY CLASSIFICATION OF: 

17. LIMITATION 

OF ABSTRACT 

UU 

18. NUMBER 

OF PAGES 

60 

19a. NAME OF RESPONSIBLE PERSON

Jessie Y. C. Chen 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

19b. TELEPHONE NUMBER (Include area code) 

407-384-5435 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18 
iii 

Contents 

List of Figures v

List of Tables vi

Acknowledgments vii

1. Introduction 1

1.1 Supervisory Control - Overview .....................................................................................1

1.1.1 Level of Autonomy .............................................................................................2

1.1.2 Human-Automation Interactions .........................................................................3

2. Operator Performance Issues in Supervisory Control of UVs 4

2.1 Operator Multitasking Performance ................................................................................4

2.2 Trust in Automation ........................................................................................................5

2.3 Situation Awareness ........................................................................................................8

2.3.1 Task Switching ....................................................................................................9

2.3.2 Error Diagnosis and Recovery ............................................................................9

2.4 Operator Workload ........................................................................................................10

3. Supervisory Control of Unmanned Aerial Vehicles 10

4. Supervisory Control of Unmanned Ground Vehicles 13

5. Supervisory Control of Heterogeneous Unmanned Vehicles 15

6. Interface Designs for Supervisory Control of Unmanned Vehicles 16

6.1 Multimodal Displays/Controls ......................................................................................18

6.2 Planning Tools ...............................................................................................................18

6.2.1 Schedule Management Aid ...............................................................................19

6.3 Visualization Tools........................................................................................................20

6.3.1 Augmented Reality ............................................................................................21

6.3.2 Ecological Interface Design ..............................................................................22

6.4 Attention Management Tools (Interruption Recovery Aid) ..........................................24
iv 

6.5 Trust Calibration Tools .................................................................................................26

6.6 Adaptive Automation ....................................................................................................27

6.7 Intelligent Agents and Human-Robot Teaming ............................................................31

7. Conclusions 32

8. References 33

List of Symbols, Abbreviations, and Acronyms 47

Distribution List 48
v 

List of Figures 

Figure 1. Timeline display and configural display (right) (adapted from Cummings et al. 

[2007], with permission). .........................................................................................................20

Figure 2. Visualization tool (from left to right): individual robots, semitransparent, and solid 

(adapted from Humphrey et al. [2006], with permission). ......................................................21

Figure 3. Synthetic vision symbology added to simulated UAV gimbal camera video, 

marking threat, landmarks, areas of interest, and runway (symbology generated with 

LandForm SmartCam3D, Rapid Imaging Software, Inc.; adapted from Calhoun and 

Draper [2006], with permission). .............................................................................................22

Figure 4. EID display that shows the intention of the automated system (dotted lines indicate 

the automation’s intentions) (adapted from Furukawa et al. [2004], with permission). ..........23

Figure 5. EID display (the three panels in the center) for a partially automated process 

control simulation. ...................................................................................................................23

Figure 6. Supervisory control of four UAVs—configural display is the center panel with 

three green triangles indicating the costs of planned operations (image adapted from 

Cummings and Bruni [2009], with permission). .....................................................................24

Figure 7. Interruption recovery aid—textual event history list (CHEX, upper right) (adapted 

from St. John et al. [2005], with permission). .........................................................................25

Figure 8. The interruption assistance interface (adapted from Scott et al. [2006], with 

permission). ..............................................................................................................................25
vi 

List of Tables 

Table 1. The 10 LOAs of decision and action selection (modified from table 1 in 

Parasuraman et al. [2000]). ........................................................................................................2
vii 

Acknowledgments 

This report was sponsored by the Safe Operations of Unmanned Systems for Reconnaissance in 

Complex Environments Army Technology Objective of the U.S. Army. The authors wish to 

thank Mr. Bryan Clark for his assistance in preparing the manuscript. The authors also wish to 

thank the reviewers for their helpful comments. 
viii 

INTENTIONALLY LEFT BLANK. 
1 

1. Introduction 

Unmanned vehicles (UVs), including unmanned air vehicles (UAVs) and unmanned ground 

vehicles (UGVs), are becoming an essential part of the battlefield. The Armed Forces have large 

programs for developing robotic systems (Barnes, in press) that encompass aerial, sea, ground, 

and subterranean applications. Future warfare will depend on collaboration among UVs, not 

only within services but also between services, and eventually among allied partners. Battlefield 

collaborations will involve hundreds of UVs as well as an equal number of manned systems 

requiring novel techniques, such as call center approaches, to monitor the unfolding decision 

environment. The complexity and sheer number of mixed assets in future operations will require 

increased autonomy and problem-solving capabilities for unmanned systems (Lewis et al., 2006). 

Furthermore, to maximize human resources, it will be desirable to designate a single operator to 

supervise multiple UVs, adding to his or her already heavy task load (Barnes et al., 2006b; Chen 

et al., 2008). However, having systems with automated behaviors introduces its own set of 

problems, including overreliance on the automated systems, potential situation awareness (SA) 

degradations, and possible loss of skills to perform the tasks manually when automation fails 

(Klein et al., 2004; Parasuraman and Riley, 1997; Parasuraman et al., 2000). 

The purpose of this report is to review research pertaining to the limitations and advantages of 

supervisory control for UV systems. More importantly, we identify and discuss results showing 

technologies that mitigate observed problems, such as specialized interfaces and adaptive 

systems. In the following section, we first present an overview of definitions and important 

terms of supervisory control and human-agent teaming. We then discuss human performance 

issues in supervisory control of UVs with regard to operator multitasking performance, trust in 

automation, SA, and operator workload. In sections 3–5, we review research findings for 

specific areas of supervisory control of UAVs, UGVs, and heterogeneous UVs (i.e., using 

different types of UVs in the same mission). Finally, in section 6, we review innovative 

techniques and technologies designed to enhance operator performance and reduce potential 

performance degradations identified in the literature. 

1.1 Supervisory Control - Overview 

Supervisory control of technology may be defined in terms of human information processing and 

the operator’s role in a given task, or from the perspective of the level of automation employed 

and the types of operator interactions with the automated technology. Humans play a variety of 

roles in supervisory tasks, including planning, teaching, monitoring, intervening, and learning 

(Sheridan, 2002). These roles typically occur in the temporal order described and may repeat 

throughout a supervisory task. Sheridan (2002) describes these roles in sequence as (1) planning 

the course of action before the automation is activated, (2) instructing the computerized 

technology to perform a task in a particular manner, (3) monitoring the instructed automation to 
2 

be sure it goes as planned, (4) intervening, when necessary, to adjust or correct the automation, 

and (5) learning from the performance and outcomes of the automation in order to improve 

planning for future interactions. As task complexity increases, there is greater need for planning 

and teaching. In addition, the need for monitoring and intervening depend on the quality of 

planning and instruction. 

1.1.1 Level of Autonomy 

When implementing a supervisory control task, the amount and types of human interaction with 

the automated technology must be considered in order to determine the appropriate level of 

automation to employ. Parasuraman et al. (2000) have defined human interaction with 

automated technology in terms of 10 “levels of automation of decision and action selection” that 

are based on four stages of human information processing: (1) information acquisition, (2) 

information analysis, (3) decision and action selection, and (4) action implementation. The 10 

levels of automation are defined in table 1, which has been modified from the original table 1 

appearing in Parasuraman et al. (2000). There are risks, benefits, and consequences associated 

with each level of automation in terms of the associated mental workload, reliance on the 

automation (which introduces the issues of trust and reliability of the automation), and the 

human operator’s level of SA (see Parasuraman et al., 2000). Therefore, it is important to plan 

for a level of automation (LOA) that provides a balance of human workload that is challenging to 

the operator, yet manageable, and that also provides a level of SA sufficient to meet task 

performance goals. For example, automation level 10 excludes the human operator from making 

decisions and taking actions; thus, the supervisory monitoring role of the human becomes a 

vigilance task, which can lead to operator complacency, resulting in human performance errors. 

Table 1. The 10 LOAs of decision and action selection (modified from table 1 in Parasuraman et al. [2000]).

High 10 Full autonomy: the computer makes all decisions, acts autonomously, and ignores the human 

operator. 

9 The computer informs the human operator, only if it “decides” to. 

8 The computer informs the human operator, only if asked to. 

7 The computer executes automatically, then informs the human operator as necessary. 

6 Allows the human a limited amount of time to veto an action before it is automatically 

executed. 

5 The computer executes the suggestion with approval from the human operator. 

4 The computer suggests one alternative. 

3 The computer narrows the decision making to a few selections. 

2 The computer offers a complete set of decision/action alternatives. 

Low 1 Manual operation: the computer offers no assistance to the human operator, who must make 

all decisions and take all actions. 
3 

1.1.2 Human-Automation Interactions 

Supervisory control may also be defined in terms of the coordinated interactions that occur 

between the human and the automation, referred to here as human-agent (H-A) teaming. 

Researchers have referred to H-A teaming as team play (Dekker and Woods, 2002; Klein et al., 

2004; Sarter and Woods, 2000). These researchers argue that the interactions between the 

operator and the automation should be the focus of supervisory task designs. Researchers have 

empirically investigated how constructs associated with team play (e.g., common ground) may 

be facilitated by the characteristics of the automation during an exploration task. Stubbs et al. 

(2008), for example, studied how common ground could be facilitated among a (simulated) 

globally distributed team of operators and exploration robots. More specifically, Stubbs and her 

colleagues investigated how introducing a robot proxy would influence the degree of 

collaboration the operators perceived they were engaged in during the task. The robot proxy 

assisted the operator with planning and supervised the less autonomous robots. Results indicated 

that those who were assisted by the robot proxy reported higher perceptions of collaboration than 

those who did not use the robot proxy. Furthermore, those who used the robot proxy had 

improved performance and more accurate mental models of the robot team’s capabilities, and 

were more efficient at the exploration task. 

Concepts such as common ground and other human-team processes (e.g., coordination and 

communication) may provide a compressive understanding of H-A coordination (Fiore et al., 

2008; Stubbs et al., 2008). H-A coordination is similar to human-team coordination in that they 

both address interdependency and dynamic interplay among team members. Given these 

similarities, the processes that occur during human-team coordination may very likely emerge 

during H-A coordination (Cuevas et al., 2007). For example, communication, a team process, 

emerges from the exchange of information between the operator (who may, for example, request 

information) and the automation (which may provide information about the state of the system 

and the operating environment). In sum, the team behaviors that occur during a supervisory task 

will illuminate relevant team processes. Knowing what team processes are necessary and 

understanding how they will affect the outcome of H-A coordination can inform the design of 

supervisory tasks. 

The following section discusses in detail some of the major human performance issues 

associated with supervisory control of multiple UVs. Particularly, we discuss operator 

multitasking performance, trust in automation, SA (issues associated with tasking switching and 

error diagnosis and recovery), and operator workload. 
4 

2. Operator Performance Issues in Supervisory Control of UVs 

2.1 Operator Multitasking Performance 

In the future battlefield, Soldiers will very likely be expected to perform other tasks concurrently 

while operating a robot (or multiple robots) (Mitchell and Chen, 2006). In addition to the 

multitasking requirement, operators’ overall performance may be affected by other factors, such 

as situation understanding, mental workload, trust in the automated systems with less than 

perfect reliability, and tasking environment and stress (e.g., time constraints and fatigue). For 

example, Manzey et al. (2008) investigated performance consequences of automated aids (for 

fault identification and management) that are occasionally unreliable in a simulated multitasking 

supervisory control task. The results showed that automation benefits both primary (i.e., fault 

diagnosis) and secondary task (i.e., response to communications) performance. However, a 

significant automation bias effect was observed. About half of the participants followed a wrong 

diagnosis generated by the automated system. Additionally, with the highest LOA, participants 

showed degraded “return-to-manual” performance. The authors, therefore, recommended that 

medium levels of automation be used if manual skills need to be maintained. 

Recent multitasking studies have investigated combining navigation tasks concurrently with 

targeting and firing (search and destroy) tasks and intermittent communication tasks (Chen and 

Joyner, 2009; Cummings and Guerlain, 2007). Typical navigation task manipulations have 

included perception (egocentric vs. exocentric), attention (number of robots to control 

simultaneously), and communication with teammates. To increase task complexity, search and 

destroy tasks have manipulated attentional resources by increasing the number of targets or 

gunner stations to command. The multitasking study by Chen and Joyner (2009) required an 

operator to perform a gunner task, detecting and firing upon targets, while simultaneously either 

managing a semi-autonomous UGV or teleoperating or monitoring a UGV. A third intermittent 

task required participants to simulate communication with a gunner crew. Results showed that 

overloading the operator’s mental capacity with multiple complex tasks simultaneously led to 

performance decrements; however, when (semi-) autonomy was given to robotic entities, the 

operator could focus cognitive resources on complex tasks while simply monitoring (rather than 

controlling) the semi-autonomous robots, intervening only when necessary. 

Cummings and Guerlain (2007) tried to determine the approximate number of autonomous 

vehicles that one operator could efficiently command and control simultaneously, under a lowor high-tempo multitask condition. Their study involved commanding missile launches while 

being distracted by intermittent communication messages. Participants in this study 

demonstrated that 16 missiles were the limit to control before performance degradation was 

observed. Cummings and Guerlain cited that their findings corresponded with air traffic control 

(ATC) studies, where 17 managed aircraft were the limit that operators could handle well. Other 
5 

studies have concluded that a fewer number of UAVs could be controlled simultaneously 

(Miller, 2004); however, it is important to consider the levels of automation applied to the 

autonomous entities. In many cases, the more manual operation that was demanded, the fewer 

entities that could effectively be supervised (Ruff et al., 2002). 

Factors that impact supervisory control of decision-making tasks include degrees of coherence 

(comprehensiveness, consistency, and rational soundness) and correspondence (accuracy) of the 

information available for these tasks (Mosier et al., 2007). Since coherence is a prerequisite to 

correspondence, Mosier et al. (2007) studied the effects of coherence (i.e., information 

congruence) in the context of time pressure and operator confidence. In their study, airplane 

pilots were found to make quicker, less accurate decisions under time pressure, as they did not 

utilize all of the cues available to them. Instead, they tended to use cognitive heuristics such as 

anchoring the most salient cues to satisfice rather than take the time necessary to make a betterinformed decision. Satisficing was also observed in the supervisory command and control study 

by Cummings and Guerlain (2007) in which participants, who were under a time constraint, had 

to select one of several missiles to redirect toward a target. Participants often redirected a 

missile that was suboptimal, yet sufficient, to complete the mission. 

2.2 Trust in Automation 

Trust in automation is probably misleading because trust has the connotation of a prescribed 

behavior. Calibration is a more fitting term because it suggests that operators intervene only 

when they have reason to believe their own decisions (od) are superior to the automated system’s 

decisions (ad). In decision theoretic terms, operators choose to use automated systems when the 

probability of a correct decision (Pc) meets the following criterion (Dzindolet et al., 2001a): 

 Intervene if P(c) od > P(c) ad 

However, operators have a difficult time assessing their own accuracy. In general, humans tend 

to be poorly calibrated, often overestimating their own abilities (Fischhoff et al., 1977). This 

implies that humans will overvalue their own decisions in comparison to automated solutions. 

However, the opposite tendency for humans to over-rely on automated systems has been shown 

by a number of researchers (Chen and Terrence, 2008, 2009; Dzindolet et al., 2000; Mosier and 

Skitka, 1996; Parasuraman et al., 1993; Thomas and Wickens, 2000; Young and Stanton, 2007). 

The psychological context of the decision determines the tendency of the operator to disuse 

(under-rely on) or misuse (over-rely on) automated systems (Parasuraman and Riley, 1997). In a 

series of experiments, Dzindolet et al. (2001b) showed that by a simple change in decision order, 

disuse of an automated target recognition device changed to misuse. If participants made a 

decision before being informed of the automated solution, they tended to rely on their own 

decisions even when they were suboptimal; whereas, in a related experiment using a similar 

paradigm, participants tended to over-rely on the device whenever the automated solution was 

presented at the same time as the target scene. One explanation is that participants were 

attempting to reduce their cognitive load to maintain reasonable timeliness. 
6 

However, such a strategy was not useful in the first experiment (automated solutions being 

shown after the participants made their decisions) because considering an alternative would 

increase workload by requiring operators to reconsider their original decisions (Barnes et al., 

2006a). The workload hypothesis is supported by compliancy research that indicates operators 

misuse automation in a multitasking environment but not in single-task environment 

(Parasuraman et al., 1993). 

Another distinction important to “trust” decisions is the difference between evaluation errors and 

intent errors. For evaluation errors, the operator misperceives the optimal solution and commits 

a calibration error. For intent errors, the operator is aware of the aid’s superiority but still 

chooses to “disuse” automation in order to maintain control over the decision environment. For 

example, intent errors occurred when participants were aware of the superiority of an automated 

target detection aid, but 84% of them still tended to disuse the aid leading to suboptimal targeting 

decisions (Beck et al., 2007). 

The reliability of the automated system also impacts operator calibration. A number of 

researchers reported that target cueing is ineffective below a certain level of reliability 

(~60%–70%) (Wickens and Dixon, 2005; Wickens et al., 2006). In their meta-analytic study, 

Wickens and Dixon found that “a reliability of 0.70 was the ‘crossover point’ below which 

unreliable automation was worse than no automation at all.” Reliable automation enhances 

performance (Rovira et al., 2007); however, even reliable aids tend to be disused if the few 

mistakes the aid makes are obvious, such as failing to detect a target when it is plainly visible 

(Dzindolet et al., 2006). However, the converse, automation paradox, is also evident; operators 

tend to over-rely more on highly accurate aids when advisories are incorrect. That is, operators 

are more complacent when they develop trust in automation with high reliability because the 

possibility that the aid will mislead them is perceived as minimal (Parasuraman et al., 2007; 

Rovira et al., 2007). 

The type of unreliability has an important impact on the operator’s perception of and response to 

system alerts. Cueing systems for the automated systems are often false-alarm prone (FAP) or 

miss-prone (MP), based on the threshold settings of the alert. Wickens et al. (2005b) showed 

that the operator’s automated task (i.e., system failure monitoring) performance degraded when 

the false alarm (FA) rate of the alerts for the automated task was high. In other words, high FA 

rate reduced the operator’s compliance with automation (compliance was defined as “the 

tendency to agree with an automated aid when it provides an alert” [Levinthal and Wickens, 

2006]). Conversely, when the miss rate was high, performance on a concurrent task was affected 

more than the automated task, because operator had to allocate more visual attention to monitor 

the automated task. In other words, high miss rate reduced the operator’s reliance on automation 

(reliance was defined as “the operator’s assumption that a system is functioning normally while 

the alert is silent” [Levinthal and Wickens, 2006]). Similarly, Dixon and Wickens (2006) showed 

that FAs and misses affected compliance and reliance, respectively, and their effects appeared to 

be relatively independent of each other. In contrast, Dixon et al. (2007) showed that FAP 
7 

automation hurt “performance more on the automated task than did miss-prone automation, (e.g., 

the “cry wolf” effect) and hurt performance (both speed and accuracy) at least as much as MP 

automation on the concurrent task.” FAP automation was found to affect both operator 

compliance and reliance, while MP automation affected only operator reliance. The authors 

suggested that the FAP automation had a negative impact on reliance because of the operator’s 

overall reduced trust in the automated system. Similarly, Wickens et al. (2005a) demonstrated a 

greater cost associated with FAP automation (than with MP automation), which affected both the 

automated and concurrent tasks. 

Lees and Lee (2007) showed that “unnecessary alarms” (i.e., alerts that are legitimate but, due to 

the peculiarities of the situation, do not require operator’s compliance) foster the operators’ trust 

in the automated system and actually enhance their compliance with the alerts rather than 

reducing it. Lees and Lee’s data suggest that the three dimensions of trust (i.e., utility, 

predictability, and intent) need to be considered beside the traditional description of alarms 

according to signal detection theory (i.e., FAP vs. MP). Additionally, Chen and Terrence (2009) 

showed that there is a strong interaction between the type of automation unreliability and 

participants’ self-assessed attentional control. Overall, it appears that for high-attentional control 

participants, FAP alerts were more detrimental than MP alerts due to disuse of automation. For 

low-attentional control participants, conversely, MP automation was more harmful than FAP 

automation due to misuse of (i.e., over-reliance on) automation. Their results are consistent with 

past research (Lee and Moray, 1992; Lee and See, 2004) that self-confidence is a critical factor 

in mediating the effect of trust (in automation) on reliance (on the automatic system). Lee and 

Moray found that when self-confidence exceeded trust, operators tended to use manual control. 

When trust exceeded self-confidence, automation was used more. 

Lee and See (2004) characterize human-agent teams in terms of how human teams interact. They 

define trust as the “attitude that an agent will help achieve an individual’s goal in a situation 

characterized by uncertainty and vulnerability,” implying that humans perceive automated 

systems as a team member, and attitudes of misuse or disuse develop as operators become 

familiar with the system. This implies that trust is an intrinsic quality depending on the attitude 

of humans toward automated systems. Much of the previously mentioned results suggest that 

trust (or calibration) is an extrinsic factor determined by operators’ perception of the task 

environment. However, the research on intent errors and false alarms indicates “trust” decisions 

are also based on the operators’ belief in the importance of personally controlling decision 

outcomes (Beck et al., 2007; Chen and Terrence, 2009). On a positive note, Beck et al. indicate 

that a combination of performance feedback and scenario training can reduce both appraisal and 

intent errors suggesting that different training regimes may be necessary for extrinsic (feedback) 

and intrinsic (scenario training) sources of automation usage errors. 

The intrinsic definition has interesting implications suggesting that humans view automated 

systems anthropomorphically. This begs the question whether we can design systems not only to 

perform human-like tasks, but to actually act like humans. For example, if robots can be made to 
8 

be perceived as human, then teaming relations would not only be more natural but also 

instinctive and synergistic like a well-honed basketball team. Research to date is sparse but 

suggestive. Based on their research, Arkin (2007) and his colleagues posit that, in the future, 

robots can be made not only to react to human cues but to do so in a predictable and human-like 

manner—even to the extent of developing ethical behaviors. This would allow humans and 

robots to share mental models of not only the external environment but also each others’ models 

(Endsley and Riley, 2004). However, the research in this area is not mature; hominid robots or 

even robots that act like dogs can operate successfully only in constrained laboratory or 

commercial environments. Currently, humans control or monitor unmanned systems as 

sophisticated equipment rather than as team members, showing more trust in a perceived human 

peer than a machine peer with the same capabilities (Dzindolet et al., 2001a). 

2.3 Situation Awareness 

One of the most critical factors for achieving effective supervisory control of multiple UVs is 

maintaining adequate SA of the overall tasking environment as well as individual UVs. There 

are three levels of SA as defined by Endsley (1995): (1) perception of data and environmental 

elements, (2) comprehension of the current situation, and (3) projection of future states and 

events. Changes in the environment that may affect the plans for the robots need to be detected, 

and the plans need to be modified in a timely fashion. While information updates are needed, 

recent studies have also shown that interrupting a primary task (i.e., supervisory control of UVs) 

with an intermittent task (e.g., communication messages) can have a negative impact on SA 

(Cummings, 2004; Dorneich et al., 2006). For example, Cummings (2004) found that instant 

messages diverted participants’ attention from their primary task (i.e., simulated supervisory 

control of Tomahawk missiles), thus reducing their SA when returning their focus to the primary 

task. Therefore, the challenge is to identify the means by which operators can most effectively 

maintain SA. 

SA may also be affected perceptually as a result of the change blindness phenomenon, which is 

the inability to perceptually attend to a change in one’s environment. Parasuraman et al. (2009) 

examined change blindness in the context of a supervisory control task, in which participants 

were asked to monitor a UAV and a UGV video feed in a reconnaissance tasking environment. 

Participants performed four tasks in this experiment, including target detection and routeplanning (primary tasks), a communications task to evaluate SA, and a change detection task. 

The routes for the UAV and the UGV were preplanned, and the only time the participant 

controlled the UGV was when it was necessary to navigate around an obstacle in the 

environment. The primary tasks were interrupted by both the verbal communication task and the 

change detection task. The latter required participants to indicate each time they noticed that an 

icon of a target they had previously detected had unexpectedly changed position on a map grid. 

Half of these changes occurred during a “transient event,” when the UGV stopped and its status 

bar flashed, while the remaining four changes occurred while participants were focused on the 

UAV monitoring task. Parasuraman et al. found that participants’ accuracy at detecting changes 
9 

related to the position of the target icons was very low, especially during the transient events. 

Their results suggest that change blindness occurred most frequently when a distracter (a 

transient event) was present, but it also occurred while participants shifted their attention from 

the UAV monitoring task to the UGV monitoring task. These results also suggest that task 

switching during a robot supervisory task may incur change blindness, which by its very nature 

affects an operator’s SA (additional information on tasking switching is provided in the next 

paragraph). According to Norman’s (1986) seven stages of user activity, interruptions incur the 

greatest cognitive costs during the planning phases (i.e., intention forming and action planning) 

as well as the evaluation phases (i.e., outcome interpretation and assessment). Thus, interface 

designers should account for this, so that primary tasks are only interrupted during emergency 

situations or during moments of low workload (e.g., after evaluation is completed or before 

initiating a new plan). However, alerts should be provided to the operator indicating the changes 

to the interface and the degree of importance of the changes (Sarter et al., 2007). 

2.3.1 Task Switching 

Simultaneous control of multiple UVs may require the operator to switch attention/control 

among the vehicles from time to time. Basic research on the costs of task switching consistently 

shows that people’s responses tend to be substantially slower and more error-prone after task 

switching (Monsell, 2003; Rubinstein et al., 2001). There is some evidence that this cost may be 

reduced if the participants have a chance to prepare for the switch or receive task-switching cues 

(Monsell, 2003; Rubinstein et al., 2001). In the context of human-robot interaction (HRI), 

research has been conducted to investigate the effects of task switching on SA and operator 

performance (Crandall et al., 2005; Squire et al., 2006; Wang and Lewis, 2007). Squire et al. 

(2006) studied operators’ response time as it related to interface type (options available to the 

user) and task switching, as well as strategy switching (offensive vs. defensive, in the context of 

a RoboFlag simulated game). Task switching was shown to slow response time by several 

seconds, especially when automation was involved. Likewise, Squire et al. found that response 

time increased by several seconds when participants switched between offensive and defensive 

strategies. When participants were provided an interface that allowed the flexibility to choose 

between a fixed sequence of automated actions or selectable waypoint-to-waypoint movement, 

mission time was reduced in spite of task or strategy switching. Crandall et al. (2005) examined 

the amount of interaction time vs. noninteraction time, or “neglect tolerance,” to develop a 

predictive model to determine the number of robots (heterogeneous or homogeneous) that may 

be effectively monitored simultaneously by a single operator, given the requirements of a 

particular interactive task. Crandall et al. suggested that a predictive analysis such as theirs 

might be useful in the context of task switching. 

2.3.2 Error Diagnosis and Recovery 

Frequently, changes in the environment may require the operator to modify his/her plans for the 

UVs. Muthard and Wickens (2002) evaluated the effects of automation on pilot’s performance 
10 

in plan monitoring and revision. They found that pilots only detected about 30% of the 

experimenter-induced changes, which should have resulted in flight plan revisions. Mumaw et 

al. (2001) showed an even more alarming inadequacy in monitoring performance. In their study, 

pilots only detected about 3% (1 of 32 total cases) of unexpected changes to the mode of an 

automation aid. Indeed, it has been well documented that operators are frequently unaware of 

mode changes when interacting with automation systems and, therefore, are confused about the 

systems’ behaviors (Sarter and Woods, 1995). In fact, data show that even if changes in the 

environment are detected, operators may have difficulty interpreting the relevance of the changes 

and their effects on the existing plans (Endsley, 1995). According to the National Transportation 

Safety Board (1994, as cited in Muthard and Wickens, 2002), nearly two-thirds of aviation 

accidents caused by faulty human decision making can be attributed to pilots’ failure to revise 

their flight plans. In Muthard and Wickens (2002), pilots failed to modify their plans based on 

environmental changes on nearly one-third of the trials. Plan continuation errors were especially 

pronounced when there was an unreliable automation aid as compared with no aid present. 

2.4 Operator Workload 

Perceived workload can impact operator performance in an UV supervisory task relative to the 

level of automation assigned to the robotic entities being supervised. The level of automation of 

a robotic entity further influences operator compliance and reliance on the automated robots. 

Ruff et al. (2002) examined perceived workload and found that when given complex tasks, 

operators tended to prefer autonomy that provided a LOA exhibiting “management-byexception” (~level 6 in table 1). Although this LOA produced performance decrements, it 

lowered perceived workload as participants demonstrated satisficing. In contrast to 

“management-by-exception,” Ruff et al. found that perceived workload increased when 

participants were subjected to automation that was less than perfectly accurate. Their lack of 

trust in the automation led participants to take extra time to double-check the system’s accuracy 

in a “management-by-consent” condition (~LOA 3, 4, or 5 in table 1). There are also individual 

differences associated with perceived workload. For example, Mogford et al. (1995) found that 

individual operator differences were a mediating factor of perceived mental workload, along 

with equipment quality and ATC operator strategies, in a model of ATC. 

The following sections review research findings in specific areas of supervisory control of 

UAVs, UGVs, and heterogeneous UVs (i.e., using different types of UVs in the same mission). 

3. Supervisory Control of Unmanned Aerial Vehicles 

Two often-used methods of supervisory control of UAVs are management by consent (MBC) 

and management by exception (MBE), which have been researched extensively in the manned 

aviation environment (Olson and Sarter, 2001). MBC requires the automation to ask for explicit 
11 

consent from the human operator before taking any actions; MBE, on the other hand, allows the 

automation to initiate/perform actions unless overruled by the human operator. In their survey of 

airline pilots, Olson and Sarter (1998) found that MBC was preferred by most pilots under 

normal circumstances, as it gave them more control over the automated systems. However, 

MBE was preferred when the pilots were under time pressure and higher workload, or when the 

task was highly complex or of low criticality. Olson and Sarter (2001) conducted a simulation 

experiment to further examine pilot performance when interacting with MBC systems. The 

results showed that, under time constraints, pilots often had difficulty detecting conflicts before 

giving consent to the automation, especially when the automation did more than expected by the 

pilots. 

Comparisons of the effectiveness of MBC and MBE were also conducted in the unmanned 

aviation environment (Cummings and Mitchell, 2006; Ruff et al., 2002, 2004). Ruff et al. (2002) 

evaluated the effects of LOA, decision-aid reliability, and number of UAVs under control (one, 

two, or four) on the human operator’s target acquisition task performance. They found that, 

overall, an MBC-type decision aid provided the highest mission efficiency (i.e., total number of 

targets destroyed divided by the total number of missiles fired) and resulted in the best 

survivability performance (i.e., number of UAV hit points sustained). However, the authors 

reported that automation level interacted with number of UAVs controlled and reliability of 

decision aid for the other measures such as SA, perceived workload, and trust in the automation 

system. They found that SA was highest in the MBC condition, followed by manual operation, 

and finally the MBE condition. As the number of controlled UAVs increased, SA degraded for 

all three LOA conditions, with the manual condition having the most severe degradation. 

Furthermore, the higher the LOA, the more benefit of vigilance and workload relief it provided 

during complex tasks (when the number of UAVs increased and/or when reliability decreased). 

For example, in the MBC condition at 95% reliability, participants experienced higher workload 

(compared with the MBE condition) as a result of double-checking to be sure the automation was 

accurate. In the context of trust in the system, researchers found that it increased in both the 

MBC and the manual conditions as the number of UAVs increased when reliability was perfect, 

with the MBC providing the most trust. However, when the reliability was not perfect (i.e., 95% 

reliable), both MBC and MBE resulted in lower trust ratings as the number of UAVs increased. 

Ruff et al. (2004) conducted another study to compare the effectiveness of MBC and MBE 

systems in a UAV control setting (two or four UAVs). In this study, the MBC system proposed 

route replans and target identifications. The MBE system, on the other hand, automatically 

implemented proposed actions after a predetermined time period, unless the operator overruled. 

Overall, regardless of experimental condition, participants did not utilize the automation aiding 

system much at all. When under greater time constraints, they utilized it more often but still 

rarely. As contrary to the Ruff et al. (2002) study, MBC did not result in significantly better 

performance than MBE. However, with MBE, participants’ perceived workload increased as 
12 

time pressure increased. With MBC, in contrast, perceived workload remained at a similar level 

with an increased time constraint. In other words, although MBE was hypothesized to reduce 

workload under high time pressure, the opposite was found. 

Cummings and Mitchell (2006) investigated LOA in the context of controlling four UAVs for 

target acquisition tasks. The LOAs employed in the study roughly corresponded to levels 1, 2, 4, 

and 6 in Parasuraman et al. (2000). The level of replanning required in reaction to unexpected 

events was also manipulated (high vs. low). They found that under the high replanning 

condition, operators performed the worst with the “active” level automation (corresponding most 

closely to MBC). The authors described this performance decrement as “cognitive saturation” 

and observed that operators were unable to correctly assimilate data from various sources, weigh 

uncertainty in the solution space, and prioritize their tasks. Operators had the highest level of 

level-1 SA (i.e., perception) and level-2 SA (i.e., comprehension) with the “super-active” level 

automation (i.e., MBE) but not the level-3 SA (i.e., future projection). In fact, operators 

exhibited automation complacency and erroneously destroyed more targets than they did when 

using automation with lower LOAs. This finding of complacency is consistent with what has 

been reported in the literature and is an important issue that user interface designers of 

automation systems need to consider (Chen and Joyner, 2009; Parasuraman et al., 1993; Thomas 

and Wickens, 2000; Young and Stanton, 2007). 

Levinthal and Wickens (2006) also investigated the effects of imperfect automation on robotics 

operator’s target detection performance when controlling multiple (two or four) UAVs. This 

study compared Meyer’s (2001) independence hypothesis (which posited that FAP and MP 

automations have qualitatively different effects on operator dependence on the automated 

systems) to the model that FAP automation hurts more than MP automation. Their results were 

in favor of the independence hypothesis as participants in an MP condition (i.e., 60% reliability 

with 3:1 likelihood of misses over FAs) showed an increase in compliance and a decrease in 

reliance whereas participants in an FAP condition (i.e., 60% reliability with 3:1 likelihood of 

false alarms over misses) showed a decrease in compliance and an increase in reliance. 

However, it was found that FAP aids resulted in delayed response times compared to the MP 

aids and 90% reliability aids. The authors, therefore, concluded that FAP aids, overall, were 

more disruptive to operator performance than were MP aids. 

Some researchers examined operator performance and workload in control of large numbers of 

UAVs (Galster et al., 2006). For example, Miller (2004) modeled the operator workload whose 

task is to authorize weapon release for targets identified by UAVs. His model shows that, under 

anticipated target densities, the operator would become overloaded when controlling 13 UAVs, 

even if the weapon release authorization is the only task the operator has to perform. In an 

empirical study, Galster et al. (2006) examined operator performance in a supervisory control 

task of four, six, or eight UAVs. Overall, participants performed well on their primary task 

(selecting the correct targets and the highest-priority targets to process). Results suggested that 

although the increase in number of UAVs slightly impacted performance (negatively), the 
13 

number of targets (not the number of UAVs) had a higher impact on perceived workload. 

Secondary (monitoring) task performance, on the other hand, was significantly worse for the 

eight UAV conditions than for the four and six UAV conditions. 

4. Supervisory Control of Unmanned Ground Vehicles 

Supervisory control of multiple UGVs has been examined in contexts of search and rescue tasks 

(Wang and Lewis, 2006; Wang et al., 2009) and other exploration and reconnaissance tasks 

(Trouvain et al., 2003; Trouvain and Wolf, 2002). Olsen and Wood (2004) proposed an equation 

to calculate fan-out (FO: the number of robots a human operator can control simultaneously 

[i.e., span of control]) based on activity time (AT: the time that a robot is active) and interaction 

time (IT: the time that it takes for a human operator to interact with a robot). The authors also 

conducted four simulated experiments to test the equation. Results showed that when 18 robots 

were available for a target search task, FO for the type of robots with the highest autonomy was 

as high as 9 for the less cluttered tasking environment (compared to 5 in the more cluttered 

environment). Olsen and Wood concluded that their FO equation could accurately model 

operator span of control in many cases; however, FO might change due to other task 

characteristics and complexities in the environment. 

Riley and Strater (2006) investigated navigating two UGVs through a maze while manipulating 

the control mode (serial, parallel, and two parallel conditions involving manual control of one 

robot and a varying LOA in the second, supervised robot). SA, workload, and performance 

navigating the robots as well as monitoring performance were assessed. Monitoring 

performance was positively correlated with SA scores, and participants in the serial control 

condition exhibited slightly higher SA scores than the parallel conditions. However, the parallel 

control condition resulted in the best navigation performance and the lowest perceived workload. 

Participants in Chadwick’s (2006) study were responsible for controlling one, two, or four semiautonomous UGVs simultaneously. Operators were assessed on monitoring, responding to cued 

decision requests, and detecting contextual errors, the latter of which occurred when the robot 

was performing correctly but inappropriately, given contextual or environmental factors. In the 

case of a contextual error, the robot cannot recognize such a malfunction, so the operator must 

take notice of it; one example may be failed navigation, in which the robot would need to be 

redirected to a more optimal path. Participants were required to monitor and recharge robots’ 

batteries, attend to targets, and detect and redirect navigation errors. Operator performance 

varied depending upon the cognitive demands of the various tasks. Degradation was found to 

come from “event time-line conflicts” and attentional limitations, the latter of which were most 

prevalent in the contextual error identification and redirecting task. 
14 

In Trouvain and Wolf (2002), participants used either two, four, or eight simulated UGVs to 

perform inspection tasks at the predesignated inspection locations. Results showed that 

participants’ overall task performance (i.e., number of inspections completed) improved from the 

two- to four-robot condition but not from the four- to eight-robot condition. However, their data 

also showed that when the number of robots increased from two to four, the average inspection 

delay (i.e., amount of time robots spent on waiting for human inspection after reaching the 

inspection locations) also more than doubled. In fact, it was found that participants had 

difficulties keeping more than two robots active at the same time. The subjective workload data 

also showed that participants experienced slightly higher workload when the number of robots 

increased from two to four; however, they experienced significantly higher workload when the 

number of robots was eight. 

Wang and Lewis (2006) explored operator demands of autonomous robot teams vs. manually 

controlled robot teams in a building search task. In this within-subjects experiment, three robots 

were controlled serially, so only the selected robot would accept commands. Participants 

controlled waypoint movements, manually teleoperated the robots, and operated a camera 

(panning and tilting it to achieve an optimal view). Waypoint and teleoperation control 

improved in the autonomous condition, but camera control did not differ. Participants also 

switched control between the three robots more frequently in the autonomous mode than in the 

manual control mode. Results further showed that autonomous cooperation between robots 

aided operator performance. Wang and Lewis believed that this was because their search task 

consisted of several subtasks, which participants had to interact with during the windows of their 

“neglect time” (see Crandall et al., 2005). Switching control between the robots more frequently 

(as in the autonomous mode) gave participants more time to finish the subtasks involved (Wang 

and Lewis, 2006). 

In a follow-on study, Lewis and his colleagues (Wang et al., 2009) investigated operators’ 

performance when they controlled 4, 8, or 12 robots (within-subject variable) for a victim-search 

task. The results showed that in the full-task control condition (participants implemented 

waypoints for the robots and controlled the cameras), operators performed better (i.e., found 

more victims) with 8 robots compared with 4 and 12 robots. Operators’ perceived workload, 

however, increased monotonically with the number of robots. Wang et al. (2009), therefore, 

concluded that somewhere between 8–12 robots seemed to be the limit for direct human control. 

The authors also suggested that automation of navigation-related tasks (e.g., path-planning) 

seemed to be more important than “efforts to improve automation for target recognition and 

cueing” in the context of controlling a large team of robots.
15 

5. Supervisory Control of Heterogeneous Unmanned Vehicles 

Future warfare employing the Future Combat System (FCS) may need to integrate information 

from multiple platforms, potentially from both aerial and ground sources. UAVs generally 

provide exocentric views of the problem space (i.e., the battlefield), while UGVs present 

viewpoints that are egocentric and immersed in the environment. The ideal view depends on the 

task; overall awareness and pattern recognition are optimized by exocentric views, whereas the 

immediate environment is often viewed better egocentrically. Displays for integrating 

information from different frames of reference (e.g., exocentric and egocentric) present potential 

human performance issues that need to be carefully evaluated (Thomas and Wickens, 2000). 

Research has shown that integrating information across egocentric and exocentric views can be 

challenging for the operator (Olmos et al., 2000; Thomas and Wickens, 2001). Essentially, dual 

displays with both frames of reference require effective scanning of the displays and integrating 

information from two different perspectives to form an accurate assessment of the situation. 

Furthermore, operators may be susceptible to a saliency effect and anchoring heuristic/bias 

(Thomas and Wickens, 2000). In other words, salient information on one display may catch 

most of the operator’s attention, and the operator may form an inaccurate judgment because 

information from the other sources are not properly attended to and integrated. In Thomas and 

Wickens (2000), participants were found to tunnel their attention into the egocentric view to the 

exclusion of information from the exocentric view. 

Chen et al. (2008) simulated an FCS-like command vehicle environment and had the participants 

perform a target designation task with a semi-autonomous UAV and a semi-autonomous UGV, 

teleoperate a UGV, or, in the mixed condition, control all three assets. They found that when the 

operator could use all three assets, they tended to ignore the equally efficacious UGV and relied 

on the UAV. This agreed with other U.S. Army–sponsored research using gaming technology 

showing the counterintuitive result that even when participants performed better with UGVs, 

they still preferred UAVs (Luck et al., 2006). This may indicate a generalized preference for the 

god’s-eye exocentric view afforded by the UAVs in comparison to the egocentric views obtained 

from the UGVs. 

Billman and Steinberg (2007) described a set of HRI performance metrics for the evaluation of 

mixed-initiative heterogeneous robots (i.e., 5–10 air, sea, and undersea UVs). They found that 

these metrics were useful in understanding collaboration between human operators and 

heterogeneous robots. The metrics included planning time, task time, SA, operator workload, 

operator assessment of the usability of the systems, and mental model mapping (between reality 

and the operator’s temporal and spatial mental model of the system state). Lessons learned from 

a series of human-in-the-loop experiments were documented, and recommended modifications of 

the metrics were also presented. 
16 

In the following section, we review innovative techniques and technologies designed to enhance 

operator performance of supervisory control of UVs. 

6. Interface Designs for Supervisory Control of Unmanned Vehicles 

When conducting supervisory control research, the interface design must support effective 

interactions and provide good usability, or the design itself may affect perceived workload and 

overall performance. For instance, Cummings (2004) found that intermittent messages from a 

chat communication tool interrupted primary task performance. This finding demonstrates how 

the presentation of alerts and alarms can be important in both experimental and “real world” 

system interfaces, as alerts and alarms may impact overall operator performance in unexpected 

and negative ways if they are designed as independent features of the system. Some 

methodologies for interface design have been investigated in the context of complex supervisory 

control tasks, including abstraction hierarchy and cognitive task analysis (Linegang et al., 2006; 

Nehme et al., 2006). For example, Linegang et al. (2006) applied the abstraction hierarchy 

approach to design a novel system. This approach allows an interface designer to decompose a 

system into subcomponents that may each be analyzed in terms of how to best apply automation 

to best aid the user. Additionally, Nehme et al. (2006) examined how cognitive task analysis 

could be mixed with SA criteria to determine a list of requirements for interfaces of future 

technologies. 

Olson and Wuennenberg (2001) presented a list of recommended user interface design guidelines 

for supervisory control of UAVs. The recommendations included the following: 

1. Automation behavior (e.g., system status/mode, system goals, and flight control 

functions) should be highly visible to the operator. (Rationale: operators tend to be “out 

of the loop” when systems are highly automated, and they often find it difficult to 

understand system behavior due to system complexity, coupling, and autonomy.) 

2. It should be easy for the operator to extract meaning from the display quickly—minimize 

information access costs by highlighting relevant information, integrating dimensions, 

and displaying information in appropriate formats. (Rationale: operators often find it 

difficult to understand system behavior due to system complexity, coupling, and 

autonomy.) 

3. Display/highlight projected changes and predicted information based on operator inputs, 

and direct operator attention to the relevant areas. (Rationale: changes to system 

behavior may be difficult for the operator to visualize or detect.)
17 

4. Provide a quick and easy way to reinstruct automated systems. (Rationale: highly 

automated systems tend to be difficult to reprogram, and manual control may not be an 

option.) 

In a recent study, Nam et al. (2009) applied human-computer interaction principles to the design 

of a multiagent system interface. When a user-centered design approach was used, three 

interface types were designed and tested. Each interface satisfied a set of design goals, including 

(1) one window display of relevant information, (2) user-friendly layout, (3) displaying only 

essential information for decision making, (4) providing rapid user interaction, (5) providing 

necessary and essential information for dynamic role changing, and (6) providing feedback. 

Nam et al. concluded that interfaces for multiagent systems are efficient when they are compact 

and when they accentuate information that aids in the decision-making process. Based on the 

results, guidelines for multiagent interface design were proposed. 

Adaptive automation can be a very useful tool to aid operators with supervisory control tasks as 

it can “leverage the strengths of humans and computers to improve overall system performance 

while mitigating the negative aspects of both” (Cummings, 2004). More information on adaptive 

automation and other potential user interface designs for supervisory control of unmanned 

vehicles will be discussed in section 5.6. 

Finally, within the context of H-A teaming, interface design techniques should focus on 

supporting team processes between the human supervisor and the UVs. As mentioned 

previously, team processes can be supported by identifying team behaviors that will occur during 

the task. For interface design, the goal would be to identify characteristics of the interface that 

could support the team processes that emerge from the H-A team’s behaviors. For example, if 

shared awareness emerges, then the interface should support communication methods that will 

facilitate a shared understanding among the team. This approach to interface design could be 

referred to as a team-centered interface design approach. Designing an interface using a teamcentered approach may enhance the robustness of the interface and facilitate the management of 

workload through a balance between task and interaction workload (Saget et al., 2008). 

It is important to note that designing an agent’s interface that supports team processes will not 

alone lead to effective team interactions; both the human and the agent are jointly responsible for 

developing team play (Endsley, 1995; Fan et al., 2008). In Fan et al.’s (2008) collaboration 

model of an H-A team, agents can work autonomously until they run into a problem, at which 

time they could query the human. The active role of the human within the H-A team, therefore, 

requires attributes that allow the human to accept/exploit an agent’s capabilities. For example, 

an agent may support coordination among team members, yet whether the human accepts the 

agent’s attempt to coordinate may depend on the human’s trust in the agent. Trust may be 

established through an understanding of the agent’s abilities and reliability (Lee and See, 2004; 

also see section 2.2). Therefore, developing team play within H-A teams requires consideration 

of not only the agent’s capabilities and attributes, but also the human’s knowledge, skills, 

abilities, and attitudes.
18 

6.1 Multimodal Displays/Controls 

Remote perception is essential for effective teleoperation. In teleoperating environments, human 

perception is compromised because the natural perceptual processing is decoupled from the 

physical environment. This decoupling affects people’s perception of affordances in the remote 

scene and can have a detrimental effect on situational awareness and, therefore, the overall 

effectiveness of the robotic’s task performance (Woods et al., 2004). The performance 

degradations can be further exacerbated due to less-than-ideal video quality (Chen et al., 2007; 

Chen and Thropp, 2007). Simpson et al. (2004) proposed using a spatial audio display to 

augment UAV operator’s SA and discussed its utility for each of the three SA levels. They 

recommended that both visual and spatial audio information should be presented concurrently. 

They also suggested that presenting the audio information spatially may enhance UAV 

operator’s sense of presence (i.e., telepresence). However, the link between presence and task 

performance has yet to be established. Additionally, spatial audio displays can be ineffective in 

a noisy environment, and the user may experience front-back confusion. Tactile displays, 

therefore, present a viable alternative to audio displays. Haas et al. (2009) designed a Soldierswarm display (the swarm consisted of 40 ducted fan aircraft) and compared its effectiveness 

when information was displayed using visual, auditory, and tactile combinations. The 

multimodal displays (compared with the visual display baseline) significantly improved Marine 

participants’ detection performance (i.e., reduction in response times) and decreased their 

perceived workload. In another study, Gunn et al. (2005) compared the effectiveness of a 

sensory display and a cognitive display for presenting warning information in a UAV control 

task. They found that the sensory display (which presented the information by changing the 

physical attributes of stimuli) resulted in more threat detections, fewer false alarms, faster 

response time, and a lower perceived workload than did the cognitive display (which required 

symbolic manipulations to define critical signals). Gunn et al. also compared the utility of 

different types of directional cueing interfaces (visual, spatial audio, and haptic) and found no 

differences in their effectiveness. For a comprehensive review of multimodal displays and 

controls for robotics control, please see Chen et al. (2007). 

6.2 Planning Tools 

Planning (e.g., route-planning task) is a vital part of human-robot interaction. It is increasingly 

common for plans to be generated by automated/intelligent systems in mixed-initiative 

operations. However, lessons learned from a U.S. Naval Intelligent Autonomy program 

indicated that human operators sometimes questioned the accuracy and effectiveness of 

automated plans (Linegang et al., 2006). Specifically, some human operators indicated that they 

had difficulties understanding the rationales for some plans and how the plans were generated 

(Linegang et al., 2006). Additionally, some operators reported that they had difficulties trying to 

specify mission parameters (e.g., goals and constraints) in the way required by the automated 

planning system (Linegang et al., 2006). Furthermore, real-time development on the battlefield 
19 

may require the human operator to change the plan for the robot team and/or for the individual 

robots. Therefore, effective communication between the human operator and the robots is 

critical in ensuring planning effectiveness and mission successes. 

Research has been conducted on ways to enhance human-robot communication (Stubbs et al., 

2008). For example, Stubbs et al. demonstrated the effectiveness of a robot proxy to enhance 

shared understanding between the human operator and the robot in an exploration task. The 

communication mechanism was based on a common ground collaboration model and was able to 

improve the human operator performance in the following areas: more accurate plans, more 

efficient planning (fewer times of replanning), more efficient and faster task performance, and 

better mental model of the robots’ capabilities. The U.S. Navy has also sponsored a research and 

development program to design a mission-planning tool for human operators controlling a 

heterogeneous group of robots (Linegang et al., 2006). The tool, Mission Displays for 

Autonomous Systems (MiDAS), uses an ecological approach to reduce conflicts between human 

operators and the automated planning system. 

6.2.1 Schedule Management Aid 

When controlling multiple semi-autonomous UVs, each executing its own predetermined plan, 

the operator may experience high workload when more than one UV needs his/her attention at 

the same time. It is therefore beneficial to let the operator know when potential processing 

bottlenecks may occur so that they can make arrangements to mitigate the bottleneck if 

necessary. Cummings et al. (2007) designed an intelligent schedule management aid and tested 

its effectiveness in a simulated suppression of enemy air defenses mission environment with four 

UAVs. The aid incorporated timeline displays that show potential bottlenecks, and it also 

presented potential scheduling conflicts using configurable displays (figure 1). Overall, both the 

timeline and the configural displays were effective. However, the authors cautioned that in order 

for the displays to be effective, the user interface designers need to ensure the following: 

1. The information’s importance matches its representation salience. (Rationale: overly 

salient representations may cause the operator to fixate on something not important while 

overlooking more serious issues.) 

2. The aid presents useful solutions to emerging problems rather than simply providing 

visualizations of potential problems. (Rationale: Cummings et al. showed that only 

presenting visualizations without solutions was not more effective than no 

visualizations.) 

Dorneich et al. (2006) took a different approach and used neurophysiological sensors 

(electroencephalography [EEG]) to detect cognitive workload in a navigation task (i.e., walking 

along a familiar route) with an intermittent communication task and mathematical interruption
20 

Figure 1. Timeline display and configural display (right) (adapted from Cummings et al. [2007], with 

permission). 

task. Throughout the experiment, EEG sensors accurately evaluated cognitive workload about 

70% of the time, and a communication scheduler adapted task scheduling information by 

informing the participant what to focus on next. The communication scheduler positively 

impacted participants’ performance by rescheduling their priorities, resulting in only a temporary 

loss of SA for low-priority messages (Dorneich et al., 2006). 

6.3 Visualization Tools 

Humphrey et al. (2006) examined the use of visualizations as an aid for operators responsible for 

controlling a team of robots. Previous studies examined color-coding robot status and team 

associations; this study added shapes to represent an individual robot (condition 1), or teams of 

robots (condition 2 [semitransparent] and condition 3 [solid]) (figure 2). Participants were 

tasked with selecting specific robots relative to other robots or environmental characteristics, and 

to verbally confirm or deny accuracy of statements regarding a robot’s status. Investigators were 

interested in the level of operator engagement, as indicated by the frequency with which 

participants utilized the visualizations. Results indicated that the semitransparent and solid 

visualizations were preferred and utilized significantly more than the individual visualization. 

Participants slightly preferred (in a posttrial questionnaire) the presentation of both an individual 

and semitransparent visualization (in the same trial). The authors plan to continue this research 

with the hypothesis that an operator’s SA will increase when visualization techniques are applied 

to managing teams of robots; this technique may also help an operator supervise a larger team of 

robots. 
21 

Figure 2. Visualization tool (from left to right): individual robots, 

semitransparent, and solid (adapted from Humphrey et al. 

[2006], with permission). 

6.3.1 Augmented Reality 

Augmented reality (AR; also known as synthetic vision) has been found to be an effective means 

to enhance pilot/UAV operator SA by portraying a more veridical view of the combat 

environment (Calhoun and Draper, 2006; Calhoun et al., 2006; Draper et al., 2006; Kramer et al., 

2005). For example, the U.S. Air Force has identified several candidate synthetic vision overlay 

concepts for UAV applications (Calhoun and Draper, 2006) (figure 3). The following 

information (potentially) can be overlaid graphically on the streaming video: maps and other 

synthetically generated symbology, photoimagery, terrain elevation, laser range scans, past and 

potential future robot paths, updates via networked communication with other sources, and other 

vital statistical data (Baker et al., 2004; Calhoun and Draper, 2006; Collett and MacDonald, 

2006; Daily et al., 2003; Keskinpala and Adams, 2004). However, large amounts of information, 

although helpful in reducing the operator’s scanning effort by providing more data in a 

centralized area (i.e., the video), can create visual clutter and degrade operator’s information 

processing (Calhoun et al., 2005). Thus, it is important that a declutter capability be provided so 

the operator can customize the overlaid information presentation according to the situation and 

tasks. 

More research is needed to determine the optimal approach to decluttering (i.e., global approach 

by deselecting classes of information or local approach by deselecting individual symbology 

[Calhoun et al., 2005]). Additionally, overlaying information on a video feed can potentially 

lead to cognitive tunneling, as the operator’s attention can be captured by the overlaid data while 

important elements/developments in the video might be overlooked (Kramer et al., 2005; 

Tufano, 1997; Wickens, 2005). However, a more recent study by Iani and Wickens (2007) 

indicated that the attentional tunneling effect of the AR displays may not be as pronounced as 

previously suggested. The tradeoff between adding information to the video feed and cognitive 

tunneling needs to be more systematically evaluated. A list of human factors issues with UAV 

AR systems and their potential solutions can be found in Calhoun et al. (2005). 
22 

Figure 3. Synthetic vision symbology added to simulated UAV 

gimbal camera video, marking threat, landmarks, areas 

of interest, and runway (symbology generated with 

LandForm SmartCam3D, Rapid Imaging Software, Inc.; 

adapted from Calhoun and Draper [2006], with 

permission). 

6.3.2 Ecological Interface Design 

Ecological interface design (EID) is a user interface design technique that conveys the 

constraints in the tasking environment, usually visually via emergent patterns, so the operator 

can intuitively perceive and solve the problem (Vincente and Rasmussen, 1992). Furukawa and 

Parasuraman (2003) demonstrated that EID was beneficial for enhancing human operators’ 

detection of automation errors as well as their comprehension of system states. In their first 

experiment, Furukawa and Parasuraman showed that human operators, using an EID display 

showing an emergent perceptual feature, were able to detect significantly more system errors 

than when they used a nonintegrated display (i.e., they showed significantly less automationinduced complacency). More strikingly, the operators were able to achieve better performance 

even though their visual attention to the EID display was significantly less, according to an eye 

movement analysis, indicating that their monitoring was more efficient. In the second 

experiment, Furukawa and Parasuraman (2003) showed the effectiveness of an EID display that 

portrays graphically the intention of the automated system. Their results showed that this 

visualization tool helped the human operators to achieve a better mental model of the system, 

which enabled them to make better decisions. In yet another study, Furukawa et al. (2004) 

integrated the intention-represented EID display in a partially automated process control 

simulation (figure 4) and compared its effectiveness with that of an EID display without 

intention indicators of the automated system (figure 5). Results showed that the intentionrepresented EID display was able to enhance the operators’ predictions of the actions and 

behaviors of the automated system and therefore was able to improve the operator’s action 
23 

Figure 4. EID display that shows the intention of 

the automated system (dotted lines 

indicate the automation’s intentions) 

(adapted from Furukawa et al. [2004], 

with permission). 

Figure 5. EID display (the three panels in the center) for a partially automated 

process control simulation. 

planning and decision making. Additionally, the benefits were demonstrated in novel scenarios, 

suggesting that the operators had a better mental model of the automated system with the 

intention-represented EID display than with the EID display without the intention indicators. 

Cummings and Bruni (2009) designed a user interface that supports a single operator’s ability to 

control four UAVs simultaneously. They utilized visualization techniques in designing a tool 

that helps the operator replan the mission. For example, they used a configural display to depict 

the overall cost of the revised plan (figure 6). Furukawa (2008) also incorporated an EID display 

into a RoboFlag simulation program. The results showed that the EID display enhanced the 

human operator’s performance of supervising a team of robots by portraying essential functional 

information (i.e., graphical representation of the offensive and defensive functions for the 

RoboFlag simulated game). A detailed review of the EID—the empirical evidence of the 

efficacy of EID, why EID improves operator performance, how EID can be leveraged, and 

challenges of implementing EID—can be found in Vincente (2002). 
24 

Figure 6. Supervisory control of four UAVs—configural display is 

the center panel with three green triangles indicating the 

costs of planned operations (image adapted from 

Cummings and Bruni [2009], with permission).

6.4 Attention Management Tools (Interruption Recovery Aid) 

When controlling multiple robots at the same time, it is inevitable that the operator will tend to 

some aspects of the environment (e.g., one of the robots) before resuming his/her monitoring of 

all the robots. Techniques that facilitate task resumption have been proposed and tested in 

various tasking environments (Ratwani et al., 2007; Scott et al., 2006; St. John et al., 2005). 

Some techniques focus on reminding the operator where he or she was before the interruption 

(Ratwani et al., 2007), while others present aids for the operator to quickly review what 

happened during the interruption (Scott et al., 2006; St. John et al., 2005). Ratwani et al. 

demonstrated that by simply reducing the size (by about 75%) of the window for the interrupting 

task (i.e., reducing the occlusion of the primary task screen by the interrupting task window), 

participants were able to resume their primary task significantly faster. Eye-tracking data also 

showed that participants were more accurate at returning to where they left off. Other more 

sophisticated techniques to facilitate recovery from interruptions have also been developed. For 

example, St. John et al. discussed the utility of an SA recovery tool (named CHEX, figure 7) that 

displayed a textual event history list in a naval air warfare environment (i.e., monitoring a 

geoplot of an airspace that contained ownship and ~50 other aircraft). St. John et al. found that 

CHEX was more effective in helping the operator resume his or her task after interruptions (i.e., 

blank screen lasting either 30 or 120 s) than a video replay tool, which was worse than the 

baseline condition (i.e., no aid). CHEX presented changes in a table and linked them to the map 

when the operator selected a change from the table. However, Scott et al. argued that the 

ineffectiveness of the video replay tool shown in St. John et al. might be improved if a better 

design had been adopted. 
25 

Figure 7. Interruption recovery aid—textual event history list 

(CHEX, upper right) (adapted from St. John et al. 

[2005], with permission). 

Scott et al. presented two types of replay tools—one replayed the events at a 10 × real-time 

speed, and the other presented bookmarks on the event timelines and the operator could view the 

replay by selecting the bookmarks (figure 8). Results showed that both replay techniques were 

effective, especially when the tasking environment was challenging. Based on the results, the 

authors recommended the following design guidelines for interruption assistance interfaces: 

1. Enable user control of event replay. 

2. Provide visual summary of critical events. 

3. Limit visual summary to goal-related events. 

4. Clearly indicate relationships between past and current system state. 

Figure 8. The interruption assistance interface (adapted 

from Scott et al. [2006], with permission). 
26 

6.5 Trust Calibration Tools 

Lee and See (2004) recommended that the capabilities and limitations of the automated systems 

be conveyed to the operator, when feasible, in order for the operator to develop appropriate trust 

and reliance. Bagheri and Jamieson (2004) demonstrated that when operators were aware of the 

context-related nature of automation reliability, their detection rate of automation failures 

increased significantly without affecting their concurrent tracking and system management tasks. 

The authors attributed this improvement in performance to a more effective attentional allocation 

strategy. Rovira et al. (2007) investigated the differential effect of automation reliability and 

different types of automation (i.e., decision support and information support) on performance 

during a command and control task. Their results confirmed a differential cost of automation 

unreliability for three forms of decision automation, as compared with information automation 

when the overall automation reliability was at 80%. At 60% overall automation reliability, 

however, there was a reduction in performance for both information and decision automation 

during unreliable trials. This finding suggests that the type of automation employed is irrelevant 

when automation reliability is below a certain threshold. Based on their findings on the different 

types of automation, Rovira et al. suggested that decrements in decision-making performance 

will be lower when operators can query the automation, inspect raw information sources, and 

verify or negate the automated advice. 

In another study, Seppelt and Lee (2007) designed a display based on EID that portrayed 

graphically the capabilities and limitations of the adaptive cruise control of a vehicle for a 

driving task in different traffic and weather conditions. They found that drivers’ reliance on the 

automation (i.e., cruise control) was more appropriate when the display was present than when it 

was not. Lee and See (2004) developed the following guidelines on designing systems that 

promote appropriate trust in the automation: 

1. Design for appropriate trust, not greater trust. 

2. Show the past performance of the automation. 

3. Show the process and algorithms of the automation by revealing intermediate results in a 

way that is comprehensible to the operators. 

4. Simplify the algorithms and operation of the automation to make it more understandable. 

5. Show the purpose of the automation, design basis, and range of applications in a way that 

relates to the user’s goals. 

6. Train operators regarding its expected reliability, the mechanisms governing its behavior, 

and its intended use. 

7. Carefully evaluate any anthropomorphizing of the automation, such as using speech to 

create a synthetic conversational partner, to ensure appropriate trust.
27 

6.6 Adaptive Automation 

Adaptive systems were developed more than 50 years ago to aid the operator by keeping 

performance constant as a function of task difficulty (Kelley, 1968). One of the early paradigms 

used queuing theory as an allocation system to service aviation tasks on which the operator was 

performing poorly during high workload situations. More recently, adaptive automation has 

been proposed as a technique to keep operators engaged in a multitasking situation without 

overwhelming them during peak workload situations (Barnes et al., 2006a; Rouse, 1988; 

Parasuraman et al., 1992). These systems, a middle ground between fully automated and manual 

control, provide a means of keeping operators in the loop. The basic concept consists of 

decisions rules for the computer invoking automation during high workload and for reinvoking 

manual control during lower workload mission segments. The purpose of the adaptive 

automation is to keep operator performance within acceptable levels while maintaining 

attentional focus on important tasks. Decision tasks, in particular, should be completely 

automated with extreme care because even with perfectly automated tasks, the operator may lose 

SA if the environment changes or unexpected emergencies occur (Barnes et al., 2006a; 

Parasuraman et al., 2007; Rovira et al., 2007). For imperfect automation, the situation is worse. 

Even for highly accurate automated aids, operators tend to over-rely on the automated solution, 

which can cause even obvious errors to be missed (i.e., automation paradox [Rovira et al., 

2007]). Consequently, with less accurate aids, automation itself can be a detriment (Wickens et 

al., 2006). Therefore, important issues with adaptive systems such as type of invocation process, 

switching rate, and engagement time must be taken into account in the design of these systems. 

Invocation rules (i.e., when to invoke automated or manual control) can be based on models, 

performance levels, physiological changes, or some mixture of them (Parasuraman et al., 2007). 

A number of physiological measures have been used to covertly monitor operator’s workload 

state: EEG, event-related potential, heart rate variability, functional magnetic resonance imaging 

(fMRI), and combinations of the measures (Barnes et al., 2006a; Scerbo et al., 2001). Freeman 

et al. (1999) showed the potential of using an adaptive system for human tracking performance 

using three neurophysiological EEG indices for automation invocation. To test whether EEG 

was a good indicator of performance, the authors compared two methods for invoking 

automation: high arousal components of the EEG (high ratios of the beta wave components 

divided by theta and/or alpha components) and components associated with low arousal (higher 

ratios of alpha and theta wave components). As predicted, conditions that used EEGs with high 

beta ratios resulted in better tracking performance indicating that neurophysiological indices 

could provide a potential means of covertly measuring the operator’s arousal level for adaptive 

processes. Milkulka et al. (2002) showed similar patterns of performance for a vigilance task 

using the high beta weightings of the EEG to control adaptive event rate procedures. However, 

because yoked participants (adaptive processes not based on the operator’s EEG) performed as 

well as participants in the high beta ratios conditions, it is possible that task difficulty measures 

may be as efficient as EEG indices for invoking automated processes. They also reported that an 
28

automation switching rate of about 15 s between conditions was sufficient for operators to be 

able to focus on the current task and to adjust to new adaptive conditions. 

Wilson and Russell (2007) developed a more sophisticated adaptive automation invocation 

method using multiple physiological indices that were performance weighted using a neural 

network–based algorithm. The adaptive conditions resulted in better performance than manual 

conditions. More importantly, the weights based on the individual’s past performance resulted in 

better performance than group-weighted invocation methods indicating their indices were 

sensitive to individual differences. All three studies (Freeman et al., 1999; Milkulka et al., 2002; 

Wilson and Russell, 2007) showed that physiological indices were potential invocation methods 

for adaptive processes; they also indicated that fairly short switching rates were feasible. 

Moreover, the Wilson and Russell study indicated that the measures should be tailored to 

individual differences (see also the Dorneich et al. [2006] study discussed previously). However, 

none of these studies actually showed an advantage for adaptive processes compared to static 

automation, nor did they compare physiological to nonphysiological indices. 

Parasuraman et al. (1996) demonstrated the efficacy of adaptive processes in a multitasking 

aviation environment. There were three conditions: static automation, model-based adaptive 

automation (mission segments with high task difficulty), and performance-based adaptive 

automation (operator error rate–determined invocation). Manual control was invoked in the 

middle of the 90-min experiment for the two adaptive conditions. The adaptive system increased 

the SA of the operators and showed significantly better instrument monitoring performance 

compared to the static automated conditions for the sessions immediately following the manual 

conditions. The authors interpreted this as indicating that putting operators in the loop, even 

briefly during a 90-min experiment, increased their awareness compared to automating a system 

for the entire 90 min. Also, the experiment demonstrated that either model-based or 

performance-based rules could be used as invocation triggers for automation. 

Parasuraman et al. (2009) compared performance-based adaptive automation, static automation, 

and manual target identification using an automatic target recognition (device in the automated 

conditions. Their study employed a multitasking environment involving targeting with UAVs 

and planning UGV routes. The trigger for adaptive automation was the number of missed 

changes in a situation map, which suggests that adaptive processes can be invoked from 

secondary, as well as primary, task difficulties. The results were dramatic—SA, workload, and 

performance on the secondary task were all improved by automation, but more importantly, 

performance gains for the adaptive (performance-based) vs. static (model-based) automation 

were substantial. Based on this study and a review of the literature, Parasuraman et al. (2007) 

suggested the following guidelines for adaptive automation: 
29 

1. Information displays should adapt to the changing military environment. 

2. Software should be developed to allow the operator to allocate automation under specified 

conditions before the mission (as in the Rotorcraft Pilot’s Associate [Dornheim, 1999]). 

3. At least initially, adaptive systems that do not take decision authority away from the 

operator should be evaluated. This can be accomplished in two ways: 

a. An advisory asking permission to invoke automation (i.e., management by consent). 

b. An advisory that automation will be invoked unless overridden (i.e., management by 

exception). 

4. For safety or crew protection situations, specific tactical or safety responses can be invoked 

without crew permission. 

Steinhauser et al. (2009) also provided several design guidelines for implementing adaptive 

automation based on their review of empirical research on adaptive automation and aiding in the 

past three decades. The guidelines are as follows: 

1. Adaptive function allocation to the operator should be used intermittently. Intermittent 

allocation can improve performance in monitoring tasks. 

2. Energetic human qualities should be considered in design. For example, degrees of 

challenge can be automatically adjusted with artificial tasks. 

3. Emotional requirements of the human operator must be considered. The human 

operator should not feel unnecessary to the system as a whole. 

4. The system should be calibrated to the individual operating it. Individual differences 

factor into the human operator portion of a human-system pairing and thus should be 

incorporated into the design. 

5. Task transformation should be used to simplify tasks for operators. A task that is 

partitioned and transformed can be handled piecemeal instead of as a whole. 

6. The environmental context of the system should be used to determine allocation. 

Environmental stressors such as heat, vibration, and gravitational force affect human 

performance and should be addressed. 

7. Tasks should be partitioned when both the human and the system can contribute 

effectively. A true human-system collaboration operates as a pairing instead of a 

dichotomy of effort. Performance is improved when the most effective attributes of 

each part are employed.
30 

8. Adaptation should be controlled by the system but be open to human intervention when 

the system fails to recognize new conditions or demands. In order to reduce task load 

on the human operator and improve general performance, the system should allocate 

tasks. To improve satisfaction and motivation, the human operator should retain 

control, or perceived control, of the system. 

Hou and his colleagues developed an intelligent adaptive system for controlling multiple UAVs 

and demonstrated that the system significantly reduced operators’ workload, improved their 

situational awareness, and facilitated the operators’ ability to work under high time pressure 

(Hou et al., 2007b). Hou et al. (2007a) also presented a framework, based on their review of 

more than 200 articles, for designing intelligent adaptive systems. A number of useful guidelines 

for designing intelligent adaptive systems were provided in this article. 

Adaptable systems and adjustable automation are two related schemas for combining automated 

and manual control. Adaptable systems allow the operator to elicit preprogrammed behaviors 

during system use similar to the way a coach would send in various “plays” from a playbook 

during a football game. The advantage of adaptable systems is the introduction of more flexible 

roles for supervisory control that allow the operator to command complex behaviors without 

requiring complicated communication protocols (Miller and Parasuraman, 2007). Adjustable 

automation allows operators to fix the level of system autonomy, allowing them to decide, 

depending on the changing situation, how much decision authority to reserve for themselves and 

how much autonomy to delegate (Goodrich, in press; Sellner et al., 2006). The best type of 

adaptive system depends on the task environment, the amount of complexity required, and the 

ability of the operator to delegate autonomy during high-workload environments. However, it 

should be kept in mind that delegation and choosing an appropriate autonomy level can be very 

high-workload decisions in themselves. Hardin and Goodrich (2009) demonstrated that a mixedinitiative control scheme (i.e., allowing the human operator and the intelligent robotic agents to 

jointly decide the appropriate level of autonomy) resulted in better overall performance than did 

adaptive autonomy and adjustable autonomy in a simulated target search task. The authors, 

however, caution that mixed-initiative control schemes should meet the following requirements 

in order to be effective: 

1. The human operator and the agents should have complementary abilities. 

2. Agents should have the capabilities to progress without waiting for commands from the 

human operator. 

3. The human operator must be able to interact with multiple agents efficiently and 

simultaneously. 

The next section further discusses teaming between human and intelligent agents.
31 

6.7 Intelligent Agents and Human-Robot Teaming 

The necessity for more powerful intelligent agents that can interact with human operators in 

increasingly sophisticated ways will require that current performance-improving techniques and 

technologies be augmented with techniques and technologies that facilitate effective H-A team 

interactions (Billings, 1996; Bradshaw et al., 2004; Green et al., 2008; Lewis, 1998). In recent 

years, researchers have designed agents that can support team cognition. Dias et al. (2006), for 

example, presented the architecture of a dynamically formed human-robot team that performs 

coordinated tasks. The architecture was developed based on the vision that H-A teams will 

dynamically form to solve complex tasks by coordinating their complementary capabilities. The 

architecture included TraderBot (Dias et al., 2004), TeamTalk (Harris et al., 2004), and Plays 

(Bowling et al., 2004). These components were used to support the development of coordination 

among both H-A teams and agent-agent teams. 

TraderBot supports team coordination by allocating tasks to other agents using a market-based 

coordination approach. This approach is analogues to and is described within the context of 

market trading (Dias et al., 2006). Market-based approaches can distribute much of the planning 

and execution of an operation over the team, gather information about the team, and distribute 

resources all within a team aware context. Within the architecture described by Dias et al., each 

team member is assigned an interface agent, or trader (see Zlot and Stentz, 2006). This agent is 

the team members’ interface to the market and simplifies the communication process among 

human and agent team members. The use of TeamTalk, a multiagent, multimodal dialog system, 

further simplifies the communication between humans and agents. Finally, the architecture 

supports shared environment awareness by broadcasting all user input to all robots on the team. 

Therefore, coordination, communication, and shared awareness are all supported within the 

architecture. 

Yen et al. (2006) presented research on agents that can support the development of shared mental 

models. These researchers discussed the use of the Collaborative Agents for Simulating 

Teamwork (CAST) model, which enables agents and humans to anticipate potential information 

needs and proactively exchange information between team members. Components of the CAST 

model were tested in a simulated battlefield over two experiments. The ability of CAST to 

support communication and decision making was investigated. For the scenario used in the first 

study, the human was considered a virtual agent and invisible to other team agents. Human team 

members could affect team activities by directly adjusting their agent’s behaviors and strategies. 

For this team, the decision-making process was supported by the interactions (e.g., 

communication) between the human and the agents; the human provided domain expertise, while 

the agents gathered relevant information by collaborating with other agents. In experiment one, 

the use of a CAST-supported communication model (i.e., decision-theoretic) was compared to a 

non-CAST-supported communication model. Results of the simulation indicated that using the 
32 

decision-theoretic approach, which analyzes the cost benefit ratio of communicating with other 

team members vs. not communicating with other team members, produced better decision 

making than when team members communicated every time. 

In the simulated scenario used in the second experiment, humans had an assistant agent that was 

invisible to the other team members. The assistant agent supported the human by fusing and 

filtering information received from other agents, tracking the human team member’s mental 

model, and tracking the mental model shared among the team. Here, both the interactions 

between the human and the agents, and the interaction between the human and the assistant agent 

supported decision making. For this experiment, the use of information fusion provided by the 

assistant agent was compared to not using information fusion (i.e., no assistant agent) by 

assessing the accuracy of decision making, particularly when the human’s cognitive capacity was 

taxed. The results indicated that the simulated fusion team, in general, performed better than the 

nonfusion team. Furthermore, the disparity in performance between the two simulated teams 

increased as cognitive capacity decreased. 

The results of the two experiments by Yen et al. (2006) suggest that using collaboration models 

such as CAST to form H-A teams may lead to better performance. However, future research 

should determine whether the increase in performance was a result of the CAST approach 

facilitating team cognition. For example, did the decision-theoretic approach support team 

decision-making processes and/or did fusion lead to an increase in shared mental models among 

team members? Investigating these questions along with investigating factors that may influence 

an operator’s decision to accept an agent’s attempt to establish team cognition (see section 1.2) 

just begins to open the door to the design of future human-robot team systems. 

7. Conclusions 

This report examined human performance issues in supervisory control of unmanned vehicles 

(i.e., robotics) and reviewed user interface solutions that could potentially address those issues. 

As robotics become increasingly prevalent in both military and civilian operations, it is important 

to understand HRI and its associated limitations and potentials. In the foreseeable future, it will 

be more common for humans to work with robots as a team to perform tasks that humans cannot 

realistically accomplish alone. Research programs, such as the U.S. Army’s Safe Operations of 

Unmanned Systems for Reconnaissance in Complex Environments Army Technology Objective, 

were also initiated to explore how to enhance operator performance by employing advanced 

technologies and user interface design concepts. These solutions, and other innovative user 

interface designs reviewed in this report, can hopefully improve HRI and therefore reduce the 

current challenges of operators’ robotic control tasks. 
33 

8. References 

Arkin, R. C. Governing Lethal Behavior: Embedded Ethics in Hybrid Deliberative Reactive 

Robot Architecture; GIT-GVU-07-11; Georgia Institute of Technology: Atlanta, GA, 2007. 

Bagheri, N.; Jamieson, G. A. The Impact of Context-Related Reliability on Automation Failure 

Detection and Scanning Behavior. Proceedings of IEEE International Conference on 

Systems, Man, and Cybernetics; The Hague, The Netherlands, 10–13 October 2004; 

pp 212–217. 

Baker, M.; Casey, R.; Keyes, B.; Yanco, H. A. Improved Interfaces for Human-Robot 

Interaction in Urban Search and Rescue. Proceedings of 2004 IEEE International 

Conference on Systems, Man and Cybernetics, 2004; http://www.cs.uml.edu/~holly 

/papers/baker-casey-keyes-yanco-smc04.pdf (accessed 5 April 2006). 

Barnes, M. J. Soldier-Robot Teams: An Overview. In Human-Robot Interaction in Future 

Military Applications; Barnes, M. J., Jentsch, F. G., Eds.; Ashgate Publishing: Hampshire, 

England, in press. 

Barnes, M. J.; Parasuraman, R.; Cosenzo, K. A. Adaptive Automation for Military Robotic 

Systems. In Uninhabited Military Vehicles (UMVs): Human Factors Issues in Augmenting 

the Force; RTO-TR-HFM-078; RTO/NATO Research and Technology Organisation: 

Neuilly-Sur-Seine Cedex, France, 2006a; pp 420–440. 

Barnes, M. J.; Chen, J. Y. C.; Jentsch, F.; Haas, E. Soldier Interactions With Aerial and Ground 

Robotic Systems in Future Military Environments. Proceedings of NATO Conference on the 

Human Factors of Uninhabited Military Systems (NATO Task Group HFM-135), Biarritz, 

France, 10–12 October 2006b. 

Beck, H. P.; Dzindolet, M. T.; Pierce, L. G. Automation Usage Decisions: Controlling Intent 

and Appraisal Errors in a Target Detection Task. Human Factors 2007, 49, 429–437. 

Billings, C. E. Aviation Automation: The Search for a Human-Centered Approach; Lawrence 

Erlbaum Associates: Mahwah, NJ, 1996. 

Billman, L.; Steinberg, M. Human System Performance Metrics for Evaluation of MixedInitiative Heterogeneous Autonomous Systems. Proceedings of PerMIS 07, Gaithersburg, 

MD, 28–30 August 2007; pp 120–126. 

Bowling, M.; Browning, B.; Veloso, M. Plays as Effective Multiagent Plans Enabling 

Opponent-Adaptive Play Selection. Proceedings of the International Conference on 

Automated Planning and Scheduling, Whistler, British Columbia, Canada, 3–7 June 2004. 
34 

Bradshaw, J. M.; Acquisti, A.; Allen, J.; Breedy, M. R.; Bunch, L.; Chambers, N.; Galescu, L.; 

Goodrich, M.; Jeffers, R.; Johnson, M.; Jung, H.; Kulkarui, S.; Lott, J.; Olsen, D.; Sierhuis, 

M.; Suri, N.; Taysom, W.; Tonti, G.; Uszok, A.; van Hoof, R. Teamwork-Centered 

Autonomy for Extended Human-Agent Interaction in Space Applications. Proceedings of 

the AAAI 2004 Spring Symposium; AAAI Press: Stanford University, CA, 2004. 

Calhoun, G. L.; Draper, M. H. Multi-Sensory Interfaces for Remotely Operated Vehicles. In 

Human Factors of Remotely Operated Vehicles; Cooke, N. J., Pringle, H. L., Pedersen, 

H. K., Connor, O., Eds.; Elsevier: Oxford, UK, 2006; pp 149–163. 

Calhoun, G. L.; Draper, M. H.; Nelson, J. T.; Ruff, H. A. Advanced Display Concepts for UAV 

Sensor Operations: Landmark Cues and Picture-in-Picture. Proceedings of the Human 

Factors and Ergonomics Society 50th Annual Meeting, San Francisco, CA, 16–20 October 

2006; pp 121–125. 

Calhoun, G. L.; Draper, M. H.; Abernathy, M. F.; Delgado, F.; Patzek, M. Synthetic Vision 

System for Improving Unmanned Aerial Vehicle Operator Situation Awareness. 

Proceedings of SPIE, 2005; Vol. 5802, pp 219–230. http://www.hec.afrl.af.mil/Publications 

/05SPIE5.pdf (accessed 12 June 2007). 

Chadwick, R. A. Operating Multiple Semi-Autonomous Robots: Monitoring, Responding, 

Detecting. Proceedings of the Human Factors and Ergonomics Society 50th Annual 

Meeting, San Francisco, CA, 16–20 October 2006; pp 329–333. 

Chen, J. Y. C.; Thropp, J. E. Review of Low Frame Rate Effects on Human Performance. IEEE 

Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 2007, 37 

(6), 1063–1076. 

Chen, J. Y. C.; Terrence, P. I. Effects of Tactile Cueing on Concurrent Performance of Military 

and Robotics Tasks in a Simulated Multi-Tasking Environment. Ergonomics 2008, 51 (8), 

1137–1152. 

Chen, J. Y. C.; Joyner, C. T. Concurrent Performance of Gunner’s and Robotic Operator’s Tasks 

in a Multi-Tasking Environment. Military Psychology 2009, 21 (1), 98–113. 

Chen, J. Y. C.; Terrence, P. I. Effects of Imperfect Automation on Concurrent Performance of 

Military and Robotics Tasks in a Simulated Multi-Tasking Environment. Ergonomics 2009, 

52 (8), 907–920. 

Chen, J. Y. C.; Haas, E. C.; Barnes, M. J. Human Performance Issues and User Interface Design 

for Teleoperated Robots. IEEE Transactions on Systems, Man, and Cybernetics—Part C: 

Applications and Reviews 2007, 37 (6), 1231–1245.
35 

Chen, J. Y. C.; Durlach, J. P.; Sloan, J. A.; Bowens, L. D. Human Robot Interaction in the 

Context of Simulated Route Reconnaissance Missions. Military Psychology 2008, 20 (3), 

135–149. 

Collett, T. H. J.; MacDonald, B. A. Developer Oriented Visualisation of a Robot Program: An 

Augmented Reality Approach. Proceedings of 2006 ACM Conference on Human-Robot 

Interaction, Salt Lake City, UT, 2–4 March 2006; pp 49–56. 

Crandall, J. W.; Goodrich, M. A.; Olsen, D. R.; Nielsen, C. W. Validating Human-Robot 

Systems in Multi-Tasking Environments. IEEE Transactions on Systems, Man, and 

Cybernetics-Part A: Systems and Humans 2005, 35 (4), 438–449. 

Cuevas, H. M.; Fiore, S. M.; Caldwell, B. S.; Strater, L. Augmenting Team Cognition in 

Human-Automation Teams Performing in Complex Operational Environments. Aviation, 

Space, and Environmental Medicine 2007, 78, B63–B70. 

Cummings, M. L. The Need for Command and Control Instant Message Adaptiveinterfaces: 

Lessons Learned From Tactical Tomahawk Human-in-the-Loop Simulations. 

CyberPsychology and Behavior 2004, 7 (6), 653–661. 

Cummings, M. L.; Mitchell, P. M. Managing Multiple UAVs Through a Timeline Display. 

AIAA Journal of Aerospace Computing, Information, and Communication 2006, 3, 294–308. 

Cummings, M. L.; Guerlain, S. Developing Operator Capacity Estimates for Supervisory 

Control of Autonomous Vehicles. Human Factors 2007, 49 (1), 1–15. 

Cummings, M. L.; Bruni, S. Replanning Tools for Supervisory Control of Multiple UAVs. 

Manuscript in preparation, 2009. 

Cummings, M. L.; Brzezinski, A. S.; Lee, J. D. The Impact of Intelligent Aiding for Multiple 

Unmanned Aerial Vehicle Schedule Management. IEEE Intelligent Systems (Special Issue 

on Interacting With Autonomy) 2007, 22 (2), 52–59. 

Daily, M.; Cho, Y.; Martin, K.; Payton, D. World Embedded Interfaces for Human-Robot 

Interaction. Proceedings of the 36th Hawaii International Conference on System Sciences, 

2003; pp 125–130; http://csdl.computer.org/comp/proceedings/hicss/2003/1874/05 

/1874toc.htm (accessed 27 October 2003). 

Dekker, S. W. A.; Woods, D. D. Progress on human-automation coordination. Cognition, 

Technology, and Work 2002, 4, 240–244. 

Dias, B. M.; Zinck, M.; Zlot, R.; Stentz, A. Robust Multirobot Coordination in Dynamic 

Environments. IEEE International Conference on Robotics and Automation 2004, 4, 

3435–3442.
36 

Dias, M. B.; Harris, T. K.; Browning, B.; Jones, E. G.; Argall, B.; Veloso, M. Dynamically 

Formed Human-Robot Teams Performing Coordinated Tasks. Proceedings of the 2006 

AAAI Spring Symposium, Stanford University, Palo Alto, CA, 27–29 March 2006. 

Dixon, S. R.; Wickens, C. D. Automation Reliability in Unmanned Aerial Vehicle Control: A 

Reliance-Compliance Model of Automation Dependence in High Workload. Human 

Factors 2006, 48, 474–486. 

Dixon, S. R.; Wickens, C. D.; McCarley, J. S. On the Independence of Compliance and 

Reliance: Are Automation False Alarms Worse Than Misses? Human Factors 2007, 49, 

564–572. 

Dorneich, M. C.; Ververs, P. M.; Whitlow, S. D.; Mathan, S.; Carciofini, J.; Reusser, T. NeuroPhysiologically-Driven Adaptive Automation to Improve Decision Making Under Stress. 

Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, San 

Francisco, CA, 16–20 October 2006; pp 410–414. 

Dornheim, M. A. Apache Tests Power of New Cockpit Tool. Aviation Week and Space 

Technology 1999, 151 (16), 46–49. 

Draper, M.; Calhoun, G.; Nelson, J.; Ruff, H. Evaluation of Synthetic Visionoverlay Concepts 

for UAV Sensor Operations: Landmark Cues and Picture-in-Picture; AFRL-HE-WP-TP2006-0038; U.S. Air Force Research Laboratory: Wright-Patterson Air Force Base, OH, 

2006; http://stinet.dtic.mil/oai/oai?andverb=getRecordandmetadataPrefix=htmlandidentifier 

=ADA444719 (accessed 18 April 2007). 

Dzindolet, M. T.; Beck, H. P.; Pierce, L. G. Adaptive Automation: Building Flexibility Into 

Human-Machine Systems. In Understanding Adaptability: A Prerequisite for Effective 

Performance Within Complex Environments (Advances in Human Factors and Cognitive 

Engineering); Salas, E., Burke, C. S., Pierce, L., Eds.; Elsevier Press: Boston, MA, 2006; 

Vol. 6, pp 213–245. 

Dzindolet, M. T.; Pierce, L. G.; Beck, H. P.; Dawe, L. A. A Framework for Automation Use; 

ARL-TR-2412; U.S. Army Research Laboratory: Aberdeen Proving Ground, MD, 2001a. 

Dzindolet, M. T.; Pierce, L. G.; Beck, H. P.; Dawe, L. A.; Anderson, B. W. Misuse of an 

Automated Decision Making System. Proceedings of the 5th International Conference on 

Human Interaction With Complex Systems, Human Interaction With Complex Systems 

(HICS), Urbana, IL, 30 April–2 May 2000; pp 81–85. 

Dzindolet, M. T.; Pierce, L. G.; Beck, H. P.; Dawe, L. A.; Anderson, B. W. Predicting Misuse 

and Disuse of Combat Identification Systems. Military Psychology 2001b, 13, 147–164. 

Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors

1995, 37 (1), 32–64.
37 

Endsley, M. R.; Riley, J. M. Supporting Situation Awareness in Human Operator – UAV/UGV 

Collaboration. Presentation to the Army AMRDEC Science and Technology Workshop, 

Moffet Field Air Force Base, CA, 2004. 

Fan, X.; Oh, S.; McNeese, M.; Yen, J.; Cuevas, H.; Strater, L; Endsley, M. R. The Influence of 

Agent Reliability on Trust in Human-Agent Collaboration. Proceedings of ECCE, Madeira, 

Portugal, 16–19 September 2008. 

Fiore, S. M.; Elias, J.; Gallagher, S.; Jentsch, F. Cognition and Coordination: Applying 

Cognitive Science to Understand Macrocognition in Human-Agent Teams. Proceedings of 

the 8th Annual Symposium on Human Interaction With Complex Systems, Norfolk, VA, 

2008. 

Fischhoff, B.; Slovic, P.; Lichtenstein, S. Knowing With Certainty: The Appropriateness of 

Extreme Confidence. Journal of Experimental Psychology: Human Perception and 

Performance 1977, 3, 552–564. 

Freeman, F. G.; Mikulka, P. J.; Prinzel, L. J.; Scerbo, M. W. Evaluation of an Adaptive 

Automation System Using Three EEG Indices With a Visual Tracking Task. Biological 

Psychology 1999, 50, 61–76. 

Furukawa, H. An Empirical Study on Ecological Interface Design for Multiple Robot 

Operations: Feasibility, Efficacy, and Issues. In Recent Advances in Multi Robot Systems; 

Lazinica, A., Ed.; I-Tech Education and Publishing: Vienna, Austria, 2008; pp 15–32. 

Furukawa, H.; Parasuraman, R. Supporting System-Centered View of Operators Through 

Ecological Interface Design: Two Experiments on Human-Centered Automation. 

Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting, Denver, 

CO, 13–17 October 2003; pp 567–571. 

Furukawa, H.; Nakatani, H.; Inagaki, T. Intention-Represented Ecological Interface Design for 

Supporting Collaboration With Automation: Situation Awareness and Control in 

Inexperienced Scenarios. Proceedings of Human Performance, Situation Awareness and 

Automation II; Vincenzi, D. A., Mouloua, M., Hancock, P. A., Eds.; Lawrence Erlbaum 

Associates: Mahwah, NJ, 2004; Vol. 1, pp 49–55. 

Galster, S. M.; Knott, B. A.; Brown, R. D. Managing Multiple UAVs: Are We Asking the Right 

Questions? Proceedings of the Human Factors and Ergonomics Society 50th Annual 

Meeting, San Francisco, CA, 16–20 October 2006; pp 545–549. 

Goodrich, M. A. On Maximizing Fan-Out: Controlling Multiple Unmanned Vehicles. In 

Human-Robot Interaction in Future Military Applications; Barnes, M. J., Jentsch, F. G., 

Eds.; Ashgate Publishing, Hampshire, England, in press. 
38 

Green, S. A.; Billinghurst, M.; Chen, X.; Chase, J. G. Human-Robot Collaboration: A 

Literature Review and Augmented Reality Approach in Design. International Journal of 

Advanced Robotic Systems, 2008, 5 (1), 1–18. 

Gunn, D. V.; Warm, J. S.; Nelson, W. T.; Bolia, R. S.; Schumsky, D. A.; Corcoran, K. J. Target 

Acquisition With UAVs: Vigilance Displays and Advanced Cuing Interfaces. Human 

Factors 2005, 47, 488–497. 

Haas, E.; Hill, S.; Stachowiak, C.; Fields, M. Extreme Scalability: Designing Interfaces and 

Algorithms for Soldier-Robotic Swarm Interaction; ARL-TR-4800; U.S. Army Research 

Laboratory: Aberdeen Proving Ground, MD, 2009. 

Hardin, B.; Goodrich, M. A. On Using Mixed-Initiative Control: A Perspective for Managing 

Large-Scale Robotic Teams. Proceedings of the 4th ACM/IEEE International Conference on 

Human-Robot Interaction, La Jolla, CA, 11–13 March 2009; pp 165–172. 

Harris, T. K.; Banerjee, S.; Rudnicky, A.; Sison, J.; Bodine, K.; Black, A. A Research Platform 

for Multi-Agent Dialogue Dynamics. IEEE International Workshop on Robotics and Human 

Interactive Communication (RO-MAN), Osaka, Japan, 21–22 September 2004; pp 497–502. 

Hou, M.; Gauthier, M. S.; Banbury, S. Development of a Generic Design Framework for 

Intelligent Adaptive Systems. In Human-Computer Interaction: HCI Intelligent Multimodal 

Interaction Environments; Jacko, J., Ed.; Springer-Verlag, New York, 2007a; Part III, 

pp 313–320. 

Hou, M.; Kobierski, R. D.; Brown, M. Intelligent Adaptive Interfaces for the Control of 

Multiple UAVs. Journal of Cognitive Engineering and Decision Making 2007b, 1 (3), 

327–362. 

Humphrey, C. M.; Gordon, S. M.; Adams, J. A. Visualization of Multiple Robots During Team 

Activities. Proceedings of the Human Factors and Ergonomics Society 50th Annual 

Meeting, San Francisco, CA, 16–20 October 2006; pp 651–655. 

Iani, C.; Wickens, C. D. Factors Affecting Task Management in Aviation. Human Factors

2007, 49, 16–24. 

Kelley, C. R. Manual and Automatic Control; Wiley: New York, 1968. 

Keskinpala, H. K.; Adams, J. A. Objective Data Analysis for a PDA-Based Human-Robot 

Interface. Proceedings of IEEE International Conference on Systems, Man, and 

Cybernetics. Hague, The Netherlands, 10–13 October 2004; http://eecs.vanderbilt.edu 

/CIS/pubs/2004-Hande-SMC.pdf (accessed 27 April 2006). 

Klein, G.; Woods, D. D.; Bradshaw, J. M.; Hoffman, R. R.; Feltovich, P. J. Ten Challenges for 

Making Automation a “Team Player” in Joint Human-Agent Activity. IEEE Intelligent 

Systems 2004, 19 (6), 91–95.
39 

Kramer, L.; Prinzel, L.; Arthur, J.; Bailey, R. Advanced Pathway Guidance Evaluations on a 

Synthetic Vision Head-Up Display; NASA/TP-2005-213782; NASA Langley: Hampton, 

VA, 2005. 

Lee, J. D.; Moray, N. Trust, Control Strategies and Allocation of Function in Human-Machine 

Systems. Ergonomics 1992, 35, 1243–1270. 

Lee, J. D.; See, K. A. Trust in Technology: Designing for Appropriate Reliance. Human 

Factors 2004, 46, 50–80. 

Lees, M. N.; Lee, J. D. The Influence of Distraction and Driving Context on Driver Response to 

Imperfect Collision Warning Systems. Ergonomics 2007, 50, 1264–1286. 

Levinthal, B. R.; Wickens, C. D. Management of Multiple UAVs With Imperfect Automation. 

Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, San 

Francisco, CA, 16–20 October 2006; pp 1941–1944. 

Lewis, M. Designing for Human-Agent Interaction. AI Magazine 1998, 19 (2), 67–78. 

Lewis, M.; Wang, J.; Scerri, P. Teamwork Coordination for Realistically Complex Multi Robot 

Systems. Proceedings of NATO Symposium on Human Factors of Uninhabited Military 

Vehicles as Force Multipliers, Biarritz, France, 2006. 

Linegang, M. P.; Stoner, H. A.; Patterson, M. J.; Seppelt, B. D.; Hoffman, J. D.; Crittendon, 

Z. B.; Lee, J. D. Human-Automation Collaboration in Dynamic Mission Planning: A 

Challenge Requiring an Ecological Approach. Proceedings of the Human Factors and 

Ergonomics Society 50th Annual Meeting, San Francisco, CA, 16–20 October 2006; 

pp 2482–2486. 

Luck, J. P.; McDermott, P. L.; Allender, L.; Fisher, A. Advantages of Co-Location for Effective 

Human to Human Communication of Information Provided by an Unmanned Vehicle. 

Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, San 

Francisco, CA, 16–20 October 2006; pp 550–554. 

Manzey, D.; Reichenbach, J.; Onnasch, L. Performance Consequences of Automated Aids in 

Supervisory Control: The Impact of Function Allocation. Proceedings of the Human 

Factors and Ergonomics Society 52nd Annual Meeting, New York, NY, 22–26 September 

2008; pp 297–301. 

Meyer, J. Effects of Warning Validity and Proximity on Responses to Warnings. Human 

Factors 2001, 43, 356–572. 

Milkulka, P. J.; Scerbo, M. W.; Freeman, F. G. Effects of a Biocybernetic System on Vigilance 

Performance. Human Factors 2002, 44, 654–664.
40 

Miller, C. Modeling Human Workload Limitations on Multiple UAV Control. Proceedings of 

the Human Factors and Ergonomics Society 48th Annual Meeting, New Orleans, LA, 20–24 

September 2004; pp 526–527. 

Miller, C.; Parasuraman, R. Designing for Flexible Interaction Between Humans and 

Automation: Delegation Interfaces for Supervisory Control. Human Factors 2007, 49, 

57–75. 

Mitchell, D. K.; Chen, J. Y. C. Impacting System Design With Human Performance Modeling 

and Experiment: Another Success Story. Proceedings of the Human Factors and 

Ergonomics Society 50th Annual Meeting, San Francisco, CA, 16–20 October 2006; 

 pp 2477–2481. 

Mogford, R. H.; Guttman, J. A.; Morrow, S. L.; Kopardekar, P. The Complexity Construct in Air 

Traffic Control: A Review and Synthesis of the Literature; DOT/FAA/CT-TN95/22; U.S. 

Department of Transportation, Federal Aviation Administration: Washington, DC, 1995. 

Monsell, S. Task Switching. TRENDS in Cognitive Sciences 2003, 7 (3), 134–140. 

Mosier, K. L.; Skitka, L. J. Automated Decision Makers and Automated Decision Aids: Made 

for Each Other. In Automation and Human Performance: Theory and Applications; 

Parasuraman, R., Mouloua, M., Eds.; Lawrence Erlbaum Associates: Mahwah, NJ, 1996. 

Mosier, K. L.; Sethi, N.; McCauley, S.; Khoo, L.; Orasanu, J. M. What You Don't Know Can 

Hurt You: Factors Impacting Diagnosis in the Automated Cockpit. Human Factors 2007, 

49 (2), 300–310. 

Mumaw, R. J.; Sarter, N. B.; Wickens, C. D. Analysis of Pilots’ Monitoring and Performance on 

an Automated Flight Deck. Proceedings of the 11th International Symposium on Aviation 

Psychology; The Ohio State University: Columbus, OH, 2001; http://www.humanfactors. 

uiuc.edu/ReportsandPapersPDFs/isap01/starter.pdf (accessed 9 August 2007). 

Muthard, E. K.; Wickens, C. D. Factors That Mediate Flight Plan Monitoring and Errors in 

Plan Revision: An Examination of Planning Under Automated Conditions; AFHD02-11/NASA-02-8; University of Illinois: Urbana-Champaign, IL, 2002. 

Nam, C. S.; Johnson, S.; Li, Y.; Seong, S. Evaluation of Human-Agent User Interfaces in MultiAgent Systems. International Journal of Industrial Ergonomics 2009, 39 (1), 192–201. 

Nehme, C. E.; Scott, S. D.; Cummings, M. L.; Furusho, C. Y. Generating Requirements for 

Futuristic Heterogeneous Unmanned Systems. Proceedings of the Human Factors and 

Ergonomics Society 50th Annual Meeting, San Francisco, CA, 16–20 October 2006; 

pp 235–239. 
41 

Norman, D. Cognitive Engineering. In User Centered System Design: New Perspectives on 

Human-Computer Interaction; Norman, D., Draper, S., Eds.; Lawrence Erlbaum: Hillsdale, 

NJ, 1986; pp 31–61. 

Olmos, O.; Wickens, C. D.; Chudy, A. Tactical Displays for Combat Awareness: An 

Examination of Dimensionality and Frame of Reference Concepts and the Application of 

Cognitive Engineering. The International Journal of Aviation Psychology 2000, 10 (3), 

247–271. 

Olsen, D. R.; Wood, S. B. Fan-Out: Measuring Human Control of Multiple Robots. 

Proceedings of SIGCHI Conference on Human Factors in Computing Systems, Vienna, 

Austria, 24–29 April 2004; pp 231–238. 

Olson, W. A.; Sarter, N. B. As Long as I’m in Control: Pilot Preferences for and Experiences 

With Different Approaches to Automation Management. IEEE Proceedings of the 4th 

Symposium on Human Interaction With Complex Systems, Los Alamitos, CA, 1998; 

pp 63–72. 

Olson, W. A.; Sarter, N. B. Management by Consent in Human-Machine Systems: When and 

Why it Breaks Down. Human Factors 2001, 43, 255–266. 

Olson, W. A.; Wuennenberg, M. G. Autonomy Based Human-Vehicle Interface Standards for 

Remotely Operated Aircraft. Proceedings of the 20th Digital Avionics Systems Conference, 

Daytona Beach, FL, 14–18 October 2001. 

Parasuraman, R.; Riley, V. Humans and Automation: Use, Misuse, Disuse, Abuse. Human 

Factors 1997, 39, 230–253. 

Parasuraman, R.; Molloy, R.; Singh, I. Performance Consequences of Automation-Induced 

‘Complacency’. International Journal of Aviation Psychology 1993, 3, 1–23. 

Parasuraman, R.; Mouloua, M.; Molloy, R. Effects of Adaptive Task Allocation on Monitoring 

Automated Systems. Human Factors 1996, 38, 665–679. 

Parasuraman, R.; Sheridan, T. B.; Wickens, C. D. A Model for Types and Levels of Human 

Interaction With Automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A: 

Systems and Humans 2000, 30 (3), 286–297. 

Parasuraman, R.; Barnes, M. J.; Cosenzo, K. A. Adaptive Automation for Human-Robot 

Teaming in Future Command and Control Systems. The International C2 Journal 2007, 1

(2), 43–68. 

Parasuraman, R.; Cosenzo, K. A.; de Visser, E. Adaptive Automation for Human Supervision of 

Multiple Uninhabited Vehicles: Effects on Change Detection, Situation Awareness, and 

Mental Workload. Military Psychology 2009, 21 (2), 270–297.
42 

Parasuraman, R.; Bahri, T.; Deaton, J.; Morrison, J.; Barnes, M. J. Theory and Design of 

Adaptive Automation in Aviation Systems; NAWCADWAR-92033-60; U.S. Naval Air 

Warfare Center, Aircraft Division: Warminster, PA, 1992. 

Ratwani, R. M.; Andrews, A. E.; McCurry, M.; Trafton, J. G.; Peterson, M. S. Using Peripheral 

Processing and Spatial Memory to Facilitate Task Resumption. Proceedings of the Human 

Factors and Ergonomics Society 51st Annual Meeting, Baltimore, MD, 1–5 October 2007; 

pp 244–248. 

Riley, J. M.; Strater, L. D. Effects of Robot Control Mode on Situation Awareness and 

Performance in a Navigation Task. Proceedings of the Human Factors and Ergonomics 

Society 50th Annual Meeting, San Francisco, CA, 16–20 October 2006; pp 540–544. 

Rouse, W. B. Adaptive Aiding for Human/Computer Control. Human Factors 1988, 30, 

431–438. 

Rovira, E.; McGarry, K.; Parasuraman, R. Effects of Imperfect Automation on Decision Making 

in a Simulated Command and Control Task. Human Factors 2007, 49, 76–87. 

Rubinstein, J. S.; Meyer, D. E.; Evans, J. E. Executive Control of Cognitive Processes in Task 

Switching. Journal of Experimental Psychology: Human Perception and Performance

2001, 27, 763–797. 

Ruff, H. A.; Narayanan, S.; Draper, M. H. Human Interaction With Levels of Automation and 

Decision-Aid Fidelity in the Supervisory Control of Multiple Simulated Unmanned Air 

Vehicles. Presence 2002, 11 (4), 335–351. 

Ruff, H. A.; Calhoun, G.; Draper, M.; Fontejon, J.; Guilfoos, B. Exploring Automation Issues in 

Supervisory Control of Multiple UAVs. Proceedings of Human Performance, Situation 

Awareness and Automation Conference; Lawrence Erlbaum: Mahwah, NJ, 2004; 

pp 218–223. 

Saget, S.; Legras, F.; Coppin, G. Collaborative Model of Interaction and Unmanned Vehicle 

Systems’ Interface. HCP Workshop on Supervisory Control in Critical Systems 

Management, 3rd International Conference on Human Centered Processes, Delft, The 

Netherlands, 2008. 

Sarter, N. B.; Woods, D. D. How in the World Did We Get Into That Mode? Mode Error and 

Awareness in Supervisory Control. Human Factors 1995, 37, 5–19. 

Sarter, N. B.; Woods, D. D. Teamplay With a Powerful and Independent Agent: A Full-Mission 

Simulation Study. Human Factors 2000, 42 (3), 390–402. 

Sarter, N. B.; Mumaw, R. J.; Wickens, C. D. Pilots’ Monitoring Strategies and Performance on 

Automated Flight Decks: An Empirical Study Combining Behavioral and Eye-Tracking 

Data. Human Factors 2007, 49 (3), 347–357.
43 

Scerbo, M. S.; Freeman, F. G.; Milkulka, P. J.; Parasuraman, R.; Dinocera, F.; Prinzel, L. J. The 

Efficacy of Psychophysiological Measure for Implementing Adaptive Technology; 

NASA/TP-2001-211018; NASA Langley Research Center: Hampton, VA, 2001. 

Scott, S. D.; Mercier, S.; Cummings, M. L.; Wang, E. Assisting Interruption Recovery in 

Supervisory Control of Multiple UAVs. Proceedings of the Human Factors and Ergonomics 

Society 50th Annual Meeting, San Francisco, CA, 16–20 October 2006; pp 699–703. 

Sellner, B. P.; Heger, F. W.; Hiatt, L. M.; Simmons, R.; Singh, S. Coordinated Multi-Agent 

Teams and Sliding Autonomy for Large-Scale Assembly. Proceedings of the IEEE 2006, 94

(7), 1425–1444; http://www.ri.cmu.edu/pub_files/pub4/sellner_brennan_peter_2006_2 

/sellner_brennan_peter_2006_2.pdf (accessed 28 May 2008). 

Seppelt, B. D.; Lee, J. D. Making Adaptive Cruise Control (ACC) Limits Visible. International 

Journal of Human-Computer Studies 2007, 65, 192–205. 

Sheridan, T. B. Supervisory Control. In Humans and Automation; Human Factors and 

Ergonomics Society: Santa Monica, CA, 2002; pp 115–129. 

Simpson, B. D.; Bolia, R. S.; Draper, M. H. Spatial Auditory Display Concepts Supporting 

Situation Awareness for Operators of Unmanned Aerial Vehicles. In Human Performance, 

Situation Awareness, and Automation: Current Research and Trends; Vincenzi, D. A., 

Mouloua, M., Hancock, P. A., Eds.; Erlbaum: Mahwah, NJ, 2004; Vol. 1, pp 61–65. 

Squire, P.; Trafton, G.; Parasuraman, R. Human Control of Multiple Unmanned Vehicles: 

Effects of Interface Type on Execution and Task Switching Times. Proceedings of 2006 

ACM Conference on Human-Robot Interaction, Salt Lake City, UT, 2–4 March 2006; 

pp 26–32. 

St. John, M.; Smallman, H. S.; Manes, D. J. Recovery From Interruptions to a Dynamic 

Monitoring Task: The Beguiling Utility of Instant Replay. Proceedings of the Human 

Factors and Ergonomics Society 49th Annual Meeting, Orlando, FL, 26–30 September 2005; 

pp 473–477. 

Steinhauser, N. B.; Pavlas, D.; Hancock, P. A. Design Principles for Adaptive Automation and 

Aiding. Ergonomics in Design 2009, 17 (2), 6–10. 

Stubbs, K.; Wettergreen, D.; Nourbakhsh, I. Using a Robot Proxy to Create Common Ground in 

Exploration Tasks. Proceedings of 3rd International Conference on Human-Robot 

Interaction, Amsterdam, The Netherlands, 12–15 March 2008; pp 375–382. 

Thomas, L. C.; Wickens, C. D. Effects of Display Frames of Reference on Spatial Judgments 

and Change Detection; ARL-00-14/FED-LAB-00-4; University of Illinois: UrbanaChampaign, IL, 2000. 
44 

Thomas, L. C.; Wickens, C. D. Visual Displays and Cognitive Tunneling: Frames of Reference 

Effects on Spatial Judgments and Change Detection. Proceedings of Human Factors and 

Ergonomics Society 45th Annual Meeting, Minneapolis, MN, 8–12 October 2001; 

pp 336–340. 

Trouvain, B.; Wolf, H. Evaluation of Multi-Robot Control and Monitoring Performance. 

Proceedings of the IEEE International Workshop on Robot and Human Interactive 

Communication (RO-MAN), Berlin, Germany, 25–27 September 2002; pp 111–116. 

Trouvain, B.; Schlick, C.; Mevert, M. Comparison of a Map- vs. Camera-Based User Interface 

in a Multi-Robot Navigation Task. Proceedings of the International Conference on Robotics 

and Automation, Taipei, Taiwan, 14–19 September 2003; pp 3224–3231. 

Tufano, D. Automotive HUDs: The Overlooked Safety Issues. Human Factors 1997, 39, 

303–311. 

Vincente, K. J. Ecological Interface Design: Progress and Challenges. Human Factors 2002, 

44, 62–78. 

Vincente, K. J.; Rasmussen, J. Ecological Interface Design: Theoretical Foundations. IEEE 

Transactions on Systems, Man, and Cybernetics 1992, 22 (4), 589–606. 

Wang, J.; Lewis, M. Autonomy in Human-Robot Team Control. Proceedings of the Human 

Factors and Ergonomics Society 50th Annual Meeting, San Francisco, CA, 16–20 October 

2006; pp 525–529. 

Wang, J.; Lewis, M. Human Control for Cooperating Robot Teams. Proceedings of the 2nd 

ACM/IEEE Conference on Human-Robot Interaction, 2007; http://usl.sis.pitt.edu 

/ulab/hri222-wang.pdf (accessed 8 August 2007). 

Wang, H.; Lewis, M.; Velagapudi, P.; Scerri, P.; Sycara, K. How Search and Its Subtasks Scales 

in N Robots. Proceedings of the 4th ACM/IEEE Conference on Human-Robot Interaction, 

La Jolla, CA, 11–13 March 2009; pp 141–147. 

Wickens, C. D. Attentional Tunneling and Task Management. Proceedings of the 13th 

International Symposium on Aviation Psychology, Dayton, OH, 18–21 April 2005. 

Wickens, C. D.; Dixon, S. R. Is There a Magic Number 7 (to the Minus 1)? The Benefits of 

Imperfect Diagnostic Automation: A Synthesis of the Literature; AHFD-05-01/MAAD-05-

01; University of Illinois: Urbana-Champaign, IL, 2005; http://www.humanfactors.uiuc.edu 

/ReportsandPapersPDFs/TechReport/05-01.pdf (accessed 15 May 2006). 

Wickens, C. D.; Dixon, S. R.; Johnson, N. UAV Automation: Influence of Task Priorities and 

Automation Imperfection in a Difficult Surveillance Task; AHFD-05-20/MAAD-05-6; 

University of Illinois: Urbana-Champaign, IL, 2005a; http://www.humanfactors.uiuc.edu 

/ReportsandPapersPDFs/TechReport/05-20.pdf (accessed 15 December 2005).
45 

Wickens, C. D.; Dixon, S. R.; Ambinder, M. Workload and Automation Reliability in 

Unmanned Air Vehicles. In Human Factors of Remotely Operated Vehicles; Cooke, N. J., 

Pringle, H. L., Pederson, H. K., Connor, O., Eds.; Elsevier: Amsterdam, The Netherlands, 

2006; pp 209–222. 

Wickens, C. D.; Dixon, S. R.; Goh, J.; Hammer, B. Pilot Dependence on Imperfect Diagnostic 

Automation in Simulated UAV Flights: An Attentional Visual Scanning Analysis; AHFD05-02/MAAD-05-02; University of Illinois: Urbana-Champaign, IL, 2005b; 

http://www.humanfactors.uiuc.edu/ReportsandPapersPDFs/TechReport/05-02.pdf (accessed 

15 December 2005). 

Wilson, G.; Russell, C. A. Performance Enhancement in a UAV Task Using 

Psychophysiologically Determined Adaptive Aiding. Human Factors 2007, 50 (3), 

468–474. 

Woods, D. D.; Tittle, J.; Feil, M.; Roesler, A. Envisioning Human-Robot Coordination in Future 

Operations. IEEE Transactions on Systems, Man and Cybernetics: Part C: Special Issue 

on Human-Robot Interaction, 2004. http://csel.eng.ohio-state.edu/woods/papers 

/woods_hri_IEEE.pdf (accessed 21 April 2004). 

Yen, J.; Fan, X.; Sun, S.; Hanratty, T.; Dumer, J. Agents With Shared Mental Models for 

Enhancing Team Decision-Makings. Decision Support Systems 2006, 41 (3), 634–653. 

Young, M. S.; Stanton, N. A. Back to the Future: Brake Reaction Times for Manual and 

Automated Vehicles. Ergonomics 2007, 50, 46–58. 

Zlot, R.; Stentz, A. Market-Based Multi Robot Coordination for Complex Tasks. International 

Journal of Robotics Research 2006, 25 (1), 73–101.
46 

INTENTIONALLY LEFT BLANK. 
47 

List of Symbols, Abbreviations, and Acronyms 

AR augmented reality 

AT activity time 

ATC air traffic control 

CAST Collaborative Agents for Simulating Teamwork 

EEG electroencephalography 

EID ecological interface design 

FA false alarm 

FAP false-alarm prone 

FCS Future Combat System 

fMRI functional magnetic resonance imaging 

FO fan-out 

H-A human-agent 

HRI human-robot interaction 

IT interaction time 

LOA level of automation 

MBC management by consent 

MBE management by exception 

MiDAS Mission Displays for Autonomous Systems 

MP miss prone 

SA situation awareness 

UV unmanned vehicle 

UAV unmanned air vehicle 

UGV unmanned ground vehicle
NO. OF 

COPIES ORGANIZATION 

48 

 1 DEFENSE TECHNICAL 

 (PDF INFORMATION CTR 

 only) DTIC OCA 

 8725 JOHN J KINGMAN RD 

 STE 0944 

 FORT BELVOIR VA 22060-6218 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 IMNE ALC HRR 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 RDRL CIM L 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 RDRL CIM P 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 RDRL D 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

ABERDEEN PROVING GROUND

 1 DIR USARL 

 RDRL CIM G (BLDG 4600) 
NO. OF NO. OF 

COPIES ORGANIZATION COPIES ORGANIZATION

49 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM A J MARTIN 

 MYER CENTER BLDG 2700 RM 2D311 

 FORT MONMOUTH NJ 07703-5601 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM C A DAVISON 

 320 MANSCEN LOOP STE 115 

 FORT LEONARD WOOD MO 65473 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM DI T DAVIS 

 BLDG 5400 RM C242 

 REDSTONE ARSENAL AL 35898-7290 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRS EA DR V J RICE 

 BLDG 4011 RM 217 

 1750 GREELEY RD 

 FORT SAM HOUSTON TX 78234-5002 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM DG R SPINE 

 BLDG 333 

 PICATINNY ARSENAL NJ 07806-5000 

 1 ARMY RSCH LABORATORY – HRED 

 ARMC FIELD ELEMENT 

 RDRL HRM CH C BURNS 

 THIRD AVE BLDG 1467B RM 336 

 FORT KNOX KY 40121 

 1 ARMY RSCH LABORATORY – HRED 

 AWC FIELD ELEMENT 

 RDRL HRM DJ D DURBIN 

 BLDG 4506 (DCD) RM 107 

 FORT RUCKER AL 36362-5000 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM CK J REINHART 

 10125 KINGMAN RD 

 FORT BELVOIR VA 22060-5828 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AY M BARNES 

 2520 HEALY AVE 

 STE 1172 BLDG 51005 

 FORT HUACHUCA AZ 85613-7069 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HR MP D UNGVARSKY 

 POPE HALL BLDG 470 

 BCBL 806 HARRISON DR 

 FORT LEAVENWORTH KS 66027-2302

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AJ J HANSBERGER 

 JFCOM FE 

 115 LAKEVIEW PKWY STE B 

 SUFFOLK VA 23435 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM DQ M R FLETCHER 

 NATICK SOLDIER CTR 

 AMSRD NSC WS E BLDG 3 RM 343 

 NATICK MA 01760-5020 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AT J CHEN 

 12350 RESEARCH PKWY 

 ORLANDO FL 32826-3276 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AT C KORTENHAUS 

 12350 RESEARCH PKWY 

 ORLANDO FL 32826 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AS C MANASCO 

 SIGNAL TOWERS 

 BLDG 29808A RM 303A 

 FORT GORDON GA 30905-5233 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM CU 

 6501 E 11 MILE RD MS 284 

 BLDG 200A 2ND FL RM 2104 

 WARREN MI 48397-5000 

 1 ARMY RSCH LABORATORY – HRED 

 FIRES CTR OF EXCELLENCE 

 FIELD ELEMENT 

 RDRL HRM AF C HERNANDEZ 

 3040 NW AUSTIN RD RM 221 

 FORT SILL OK 73503-9043 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM AV S MIDDLEBROOKS 

 91012 STATION AVE RM 348 

 FORT HOOD TX 76544-5073 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM CN R SPENCER 

 DCSFDI HF 

 HQ USASOC BLDG E2929 

 FORT BRAGG NC 28310-5000 
NO. OF 

COPIES ORGANIZATION 

50 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM DW E REDDEN 

 BLDG 4 CL 60 

 FORT BENNING GA 31905-5400 

 1 ARMY G1 

 (CD DAPE MR B KNAPP 

 only) 300 ARMY PENTAGON RM 2C489 

 WASHINGTON DC 20310-0300 

 1 ARMY RSCH LABORATORY – HRED 

 RDRL HRM D T DAVIS 

 BLDG 5400 RM C242 

 REDSTONE ARSENAL AL 35898-7290 

ABERDEEN PROVING GROUND

 5 DIR USARL 

 RDRL CIM G 

 L ALLENDER 

 S FOPPIANO 

 RDRL HR 

 T LETOWSKI 

 RDRL HRM B 

 J LOCKETT 

 RDRL HRS D 

 B AMREIN