The Effects of Imperfect Automation on Concurrent 

Performance of Gunner’s and Robotic Operator’s Tasks 

in a Simulated Mounted Environment 

by Jessie Y.C. Chen and Peter I. Terrence 

ARL-TR-4455 May 2008 

Approved for public release; distribution is unlimited. 
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

DESTRUCTION NOTICE⎯Destroy this report when it is no longer needed. Do not return it to 

the originator. 
Army Research Laboratory 

Aberdeen Proving Ground, MD 21005-5425 

ARL-TR-4455 May 2008 

The Effects of Imperfect Automation on Concurrent 

Performance of Gunner’s and Robotic Operator’s Tasks 

in a Simulated Mounted Environment 

Jessie Y.C. Chen 

Human Research and Engineering Directorate, ARL 

Peter I. Terrence 

University of Central Florida 

Approved for public release; distribution is unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining 

the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for 

reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, 

VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not 

display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

 May 2008 

2. REPORT TYPE 

Final 

3. DATES COVERED (From - To) 

 June 2007 to February 2008 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

 The Effects of Imperfect Automation on Concurrent Performance of 

 Gunner’s and Robotics Operator’s Tasks in a Simulated Mounted 

 Environment 5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

 62716AH70 

5e. TASK NUMBER 

6. AUTHOR(S) 

 Jessie Y.C. Chen (ARL); Peter I. Terrence (UCF) 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

 U.S. Army Research Laboratory 

 Human Research and Engineering Directorate 

 Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-TR-4455

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 

11. SPONSOR/MONITOR’S REPORT NUMBER(S 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

In this study, we simulated a generic mounted environment and conducted an experiment to examine the performance and 

workload of the combined position of gunner and robotics operator. Aided target recognition (AiTR) (via tactile and visual 

cueing) with imperfect reliability (false alarm-prone versus miss-prone) was provided to the participants to aid their gunnery 

task. Besides the gunnery task, participants performed robotics and communication tasks concurrently. Results show that 

when the robotics task was simply monitoring the video feed, participants had the best performance in the other two 

concurrent tasks and the lowest perceived workload, compared with the other robotics tasking conditions. Our data also 

show that there is a strong interaction between the type of AiTR unreliability and participants’ perceived attentional control. 

Overall, it appears that for high attentional control participants, false alarm-prone alerts were more detrimental than missprone alerts. For low attentional control participants, conversely, miss-prone automation was more harmful than false 

alarm-prone automation. Additionally, low spatial ability participants preferred visual cueing over tactile cueing, and high 

spatial ability participants favored tactile cueing over visual cueing.

15. SUBJECT TERMS

 multi-tasking; multimodal; robotic operator performance; simulation; tactile display 

16. SECURITY CLASSIFICATION OF:

19a. NAME OF RESPONSIBLE PERSON

 Jessie Y.C. Chen 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

17. LIMITATION 

OF ABSTRACT 

SAR 

18. NUMBER 

OF PAGES 

50 19b. TELEPHONE NUMBER (Include area code) 

 407-384-5435 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18
iii 

Contents 

List of Figures v

List of Tables v

Acknowledgments vi

1. Introduction 1

1.1 Purpose ............................................................................................................................1 

1.2 Background .....................................................................................................................1 

1.3 Imperfect Automation and Multi-tasking Performance ..................................................2 

1.4 Individual Differences.....................................................................................................3 

1.5 Current Study ..................................................................................................................4 

2. Method 4

2.1 Participants ......................................................................................................................4 

2.2 Apparatus.........................................................................................................................5 

2.2.1 Simulators............................................................................................................5 

2.2.2 Communication Task Materials ..........................................................................7 

2.2.3 Questionnaires and Spatial Tests.........................................................................7 

2.3 Experimental Design .......................................................................................................8

2.4 Procedure.........................................................................................................................9 

2.5 Measures........................................................................................................................10 

3. Results 10

3.1 Target Detection Performance.......................................................................................10 

3.1.1 Gunnery Tasks...................................................................................................10 

3.1.2 Robotics Tasks ..................................................................................................15 

3.2 Communication Task Performance...............................................................................16 

3.3 Perceived Workload ......................................................................................................16 

3.4 AiTR Display Assessment.............................................................................................18 

4. Discussion 18
iv 

4.1 Gunnery Task Performance, Hostile Targets ................................................................18 

4.2 Gunnery Task Performance, Neutral Targets................................................................19 

4.3 Robotics Task Performance...........................................................................................20 

4.4 Communication Task Performance...............................................................................21 

4.5 Perceived Workload ......................................................................................................21 

4.6 AiTR Display Preference ..............................................................................................21 

5. Conclusions 22

6. References 23

Appendix A. Demographic Questionnaire 27

Appendix B. Attentional Control Survey 29

Appendix C. NASA TLX Questionnaire 31

Appendix D. Usability Questionnaire 33

Appendix E. Complete Gunnery Performance Data (hostile targets) 37

Acronyms 40

Distribution List 42 
v 

List of Figures 

Figure 1. User interface of RCTA TCU......................................................................................... 5

Figure 2. TCU (left) and gunnery station (gunner’s OTW view) (right)....................................... 6 

Figure 3. Visual cueing (red dot indicating the direction of hostile target).................................. 6 

Figure 4. Tactile system................................................................................................................. 7 

Figure 5. Gunnery task performance, hostile targets only........................................................... 11 

Figure 6. Effects of spatial ability on gunnery task performance, hostile targets only................ 12 

Figure 7. Effects of perceived attentional control on gunnery task performance, hostile 

targets only............................................................................................................................ 12 

Figure 8. Effects of perceived attentional control on gunnery task performance (hostile 

targets only) in MP conditions.............................................................................................. 13 

Figure 9. Gunner’s neutral target detection performance. ........................................................... 14 

Figure 10. Effects of perceived attentional control on gunner’s neutral target detection 

performance. ......................................................................................................................... 14 

Figure 11. Robotics task performance. ........................................................................................ 15 

Figure 12. Effects of perceived attentional control on robotics task performance. ..................... 16 

Figure 13. Communication task performance.............................................................................. 17 

Figure 14. Perceived workload. ................................................................................................... 17 

List of Tables 

Table 1. Operator task performance (mean percent of targets detected for the gunnery and 

robotics tasks and percent correct for the communication task ............................................ 10 
vi 

Acknowledgments 

This project was funded by the U.S. Army’s Robotics Collaboration Army Technology Objective. 

The authors wish to thank Michael Barnes of the U.S. Army Research Laboratory’s (ARL’s) 

Human Research and Engineering Directorate for his guidance throughout the process of this 

research project. We would also like to thank Richard Gilson of the University of Central Florida 

(UCF) for his valuable input on the tactile system. 

The authors wish to thank Jon Bornstein of ARL and his group for providing the system integration 

laboratory (SIL) for our experiment. We would also like to acknowledge Raymond Compton and 

Henry Marshall of the U.S. Army Research, Development, and Engineering Command, Simulation 

and Training Technology Center, for all their efforts in acquiring the SIL. We further thank Brian 

Plamondon of UCF’s Institute for Simulation and Training and his group for their support. In 

particular, Andrew Houchin was an integral part of this research effort and his excellent work 

made our data collection and extraction process a very smooth one. 

We would like to thank Paula Durlach of U.S. Army Research Institute for Behavioral and Social 

Sciences-Simulator Systems Research Unit for providing the Spatial Orientation Test to us. 

Finally, we would like to thank our reviewers for their helpful comments. 
1 

1. Introduction 

1.1 Purpose 

The goal of this research was to examine the effects of unreliable alerts on gunners’ concurrent 

performance of gunnery, robotics, and communication tasks. We simulated a generic mounted 

(i.e., tank) crew station environment and incorporated tactile and visual displays to provide 

directional cueing for the gunnery targeting task (based on simulated aided target recognition 

[AiTR] capability). Two types of imperfect AiTR were simulated: false-alarm prone (FAP) and 

miss-prone (MP). Effects of individual differences such as spatial ability (SpA) and perceived 

attentional control (PAC) were also evaluated. We were also interested in investigating discrepancies in previous research related to compliance and reliance effects as a function of type of AiTR 

error. 

1.2 Background 

The current U.S. Army Future Combat Systems (FCS) concept for the Mounted Combat System 

(MCS) is that it will be operated with a crew of three Soldiers: a vehicle commander, driver, and 

gunner; however, there is no dedicated operator to control the platoon’s unmanned ground vehicle 

(UGV) (TRADOC, 2003). To examine how the MCS crew can effectively employ its UGV and 

which crew member should serve the role as the robotics operator, Mitchell (2005) conducted a 

workload analysis, using the Improved Performance Research Integration Tool (IMPRINT) to 

model the workload of each crew member if s/he has to concurrently operate a UGV. Mitchell 

found that the gunner had the fewest instances of overload and could assume control of the UGV 

and its associated tasks. However, there were instances in the model when the gunner dropped his 

or her primary tasks of detecting and engaging targets to perform robotics control tasks, which 

could be catastrophic for the team and mission during a real operation. Additionally, in scenarios 

when the UGV required tele-operation, the gunner was found to be consistently in overload, making concurrent performance of other tasks even more challenging. 

In order to further investigate issues derived from Mitchell’s IMPRINT analysis and to verify its 

results, we conducted a series of three simulation experiments to examine the mental workload and 

performance of the combined position of robotics operator and gunner. In the first study, we found 

that when the robotics operator must perform robot targeting and local security (i.e., gunner’s 

tasks) at the same time, both workload and performance degraded (Chen & Joyner, 2006). Our 

results showed that a gunner’s target detection performance degraded significantly when participants had to concurrently monitor, manage, and tele-operate a UGV, compared with the baseline 

condition when they only had to perform the gunnery task. As the robotics task became more 

challenging, participants’ gunnery performance degraded accordingly. Their gunnery performance 
2 

was worst when they had to tele-operate a robot simultaneously. For the robotics tasks, participants’ performance was lowest when they controlled a semi-autonomous UGV (only 53% of the 

targets were detected). Participants’ perceived workload was lowest in the single task condition. 

There were three types of robotics tasks: Monitor, Auto, and Teleop. As the robotics task became 

more difficult, participants’ workload increased accordingly, with the Teleop condition being the 

highest. 

In the second study (Chen & Terrence, 2007), we examined if and how reliable tactile cueing, 

which delivered AiTR information to aid the operator’s gunnery task, enhanced a gunner’s performance in a multi-tasking environment similar to Chen and Joyner’s (2006). We found that 

participants’ gunnery task performance improved significantly when it was assisted by AiTR. 

Those participants with higher SpA outperformed those with lower SpA, especially when there was 

no AiTR. It was also found that significantly fewer neutral targets (which were not cued) in the 

gunnery environment were detected (implying less visual attention being devoted to the gunnery 

station) when participants concurrently tele-operated a robotic asset or when the gunnery task was 

assisted by AiTR. The most likely reason for this was that AiTR cueing increased task switching 

efficiency between the primary (gunnery) and secondary (robotics) tasks. However, this improved 

effectiveness for the gunnery and robotics tasks had a deleterious effect on the uncued neutral 

targets. Participants’ concurrent task performance (i.e., robotics and communication) improved 

significantly when the gunnery task was assisted by AiTR. It was also found that the overall performance gap (including gunnery and robotics) between those participants with higher and lower 

SpA appeared to be narrower when the AiTR was available. A similar pattern was also observed 

for the PAC factor. Finally, participants’ perceived workload was significantly influenced by the 

type of robotics tasks and whether the gunnery task was assisted by AiTR. Partici-pants’ perceived 

workload was significantly higher when they tele-operated a robotic asset and when their gunnery 

task was unassisted. In a post-experimental survey, 65% of the participants indicated that they 

relied predominantly on the tactile cues when tactile and visual displays were available; only 15% 

said they relied primarily on the visual cues. Those who preferred visual cueing tended to have 

lower SpA, and their gunnery and robotics task performance tended to be inferior. 

1.3 Imperfect Automation and Multi-tasking Performance 

As described previously, Chen and Terrence (2007) demonstrated the utility of reliable AiTR for 

enhancing the operator’s automated and concurrent tasks. However, since no AiTR systems can 

have perfect reliability, Chen and Terrence (2007) can only provide us an estimate of optimal 

condition when the alert is highly reliable. In the real world, cueing systems are often FAP or MP, 

based on the threshold settings of the alert. Wickens, Dixon, Goh, and Hammer (2005) showed 

that the operator’s automated task (i.e., system failure monitoring) performance degraded when the 

false alarm (FA) rate of the alerts for the automated task was high. In other words, high FA rate 

reduced the operator’s compliance with automation (compliance defined as taking actions based on 

the alerts). Conversely, when the miss rate was high, the concurrent task performance was affected 
3 

more than the automated task because the operator had to allocate more visual attention to monitor 

the automated task. In other words, a high miss rate reduced the operator’s reliance on automation 

(reliance defined as failure to take precautious actions when there is no alert). Similarly, Dixon 

and Wickens (2006) showed that FAs and misses affected compliance and reliance, respectively, 

and their effects appeared to be relatively independent of each other. In contrast, Dixon, Wickens, 

and McCarley (2007) showed that FAP automation hurt “performance more on the automated task 

than did MP automation, (e.g., the “cry wolf” effect) and hurt performance (both speed and accuracy) at least as much as MP automation on the concurrent task (p. 570-571).” FAP automation 

was found to affect operator compliance and reliance, while MP automation affected only operator 

reliance. The authors suggested that the FAP automation had a negative impact on reliance because of the operator’s overall reduced trust in the automated system. Similarly, Wickens, Dixon, 

and Johnson (2005) demonstrated a greater cost associated with FAP automation (than with MP 

automation), which affected the automated and concurrent tasks. 

Furthermore, Wickens and Dixon (2005) demonstrated that when the reliability level is below 

about 70%, operators often ignore the alerts. In their meta-analytic study, Wickens and Dixon 

found that “a reliability of 0.70 was the ‘cross-over point’ below which unreliable automation was 

worse than no automation at all.” Although Wickens and his colleagues have done extensive 

research in this area, their studies were done in a different environment (unmanned aerial vehicle 

control display monitoring), and they did not use tactile cueing. The current study was the first 

one to examine these issues in the context of combined roles of gunner and robotics operator. 

Since an AiTR cannot have a perfect reliability rate in foreseeable real-world operations, the data 

from this study should provide useful information to the design community of future systems such 

as the FCS, in which AiTR will play an integral role (see Chen & Terrence, 2007, for a detailed 

discussion of the utility of tactile display for AiTR cueing). 

1.4 Individual Differences 

In the current study, we sought to investigate the effects of individual differences factors such as 

SpA and PAC on the operators’ concurrent performance. Previous studies (Chen et al., in press; 

Chen & Joyner, 2006) found that SpA was correlated with robotics and gunnery task performance. 

Lathan and Tracey (2002) demonstrated that people with higher SpA performed better in a teleoperation task through a maze. They finished their tasks faster and had fewer errors. Lathan and 

Tracey suggested that military missions can benefit from selecting personnel with higher SpA to 

operate robotic devices. In addition to SpA, we examined the relationship between attentional 

control and multi-tasking performance. Derryberry and Reed (2002) showed that those with 

higher attentional control could switch their attention among tasks more effectively and this was 

partially confirmed by Chen and Joyner (2006) and Chen and Terrence (2007). There is some 

evidence that attention-switching flexibility can predict performance of such diverse tasks as flight 

training and bus driving (Kahneman, Ben-Ishai, & Lotan, 1973). Several studies show that there 

are individual differences in multi-tasking performance, and some people are less prone to 
4 

performance degradation during multi-tasking conditions (Rubinstein, Meyer, & Evans, 2001; 

Schumacher et al., 2001). 

1.5 Current Study 

In the current study, we replicated the conditions of Chen and Joyner (2006) and manipulated the 

reliability of the AiTR cueing system (i.e., false alarm and miss rates). Based on the data from 

Wickens, Dixon, Goh, and Hammer (2005), we expected that the operator’s gunnery (automated) 

task performance would degrade if the FA rate of the AiTR for the gunnery system were high 

because of reduced compliance with the automation. Conversely, if the cueing were MP, the 

operator’s robotics (concurrent) task performance would be affected more than the gunnery task 

because of reduced reliance on the automation. More mental and visual resources would be 

devoted to checking the raw data for the automated task, and therefore, the performance of the 

concurrent task would be degraded. On the other hand, there was evidence that FAP automation 

was more detrimental to the automated and concurrent tasks than MP automation (Dixon et al., 

2007). Therefore, it is likely that FAP automation would have a more negative impact on the 

overall performance than would MP automation. In other words, there have been conflicting 

results in the literature regarding the independence of the effects of FAP and MP automation on 

operator compliance and reliance. It is possible that individual differences may be responsible for 

some of the observed differences in the literature. Therefore, we investigated the effects of 

individual differences on FAP and MP conditions as a possible explanation for the discrepancies. 

Additionally, in Chen and Terrence (2007), we observed that low SpA participants indicated that 

they preferred visual cueing when visual and tactile cueing displays were available, whereas high 

SpA participants favored tactile cueing over visual cueing. In the current study, visual and tactile 

cueing were presented. The relationship between SpA and preference of AiTR display modality 

was examined. 

2. Method 

2.1 Participants 

A total of 24 students (4 females and 20 males) was recruited from the University of Central 

Florida and participated in the study. The ages of the participants ranged from 18 to 34 (mean [M] 

= 22.25, standard deviation [SD] = 4.0). Participants were compensated $15/hr and/or class credit 

for their participation in the experiment. 
5 

2.2 Apparatus 

2.2.1 Simulators 

2.2.1.1 Tactile Control Unit (TCU) 

The experiment was conducted with a TCU developed by the U.S. Army Research Laboratory’s 

(ARL’s) Robotics Collaborative Technology Alliance (RCTA) for the robotic control tasks. The 

TCU is a one-person crew station from which the operator can control several simulated robotic 

assets that can perform their tasks semi-autonomously or be tele-operated (see figure 1). The 

operator switched operation modes and display modes through the use of a 19-inch touch-screen 

display. A joystick was used to manipulate the direction in which the unmanned vehicles moved 

when in Teleop mode. The UGV simulated in our study is the experimental unmanned vehicle 

developed by ARL. The simulation program used in this study was rSAF (robotic semi-automated 

forces), which is a version of OneSAF for robotics simulation. 

Figure 1. User interface of RCTA TCU. 

2.2.1.2 Gunnery Station 

The gunnery component was implemented with an additional screen and controls to simulate the 

out-the-window (OTW) view and line-of-sight (LOS) fire capabilities. The interface consisted of a 

15-inch flat panel monitor and a joystick (see figure 2). Participants used the joystick to rotate the 

sensors 360 degrees, zoom in and out, switch between firing modes, and engage targets. In each 

scenario, there were 10 hostile targets and 10 neutral targets (i.e., civilians) scattered throughout 

the simulated environment. 

OCU with RSTA view
6 

Figure 2. TCU (left) and gunnery station (gunner’s OTW view) (right). 

2.2.1.3 AiTR Systems - Tactile and Visual Alerts 

To augment target detection in the gunnery component, visual and tactile alerts were used to cue 

the participant to the direction of a target, as determined by the AiTR. Visually, the targets consisted of icons presented around the overhead view diagram of the participant vehicle in the lower 

right area of the screen (see figure 3). The target icon appeared in one of eight possible locations 

around the gunner, corresponding to 45-degree increments along a 360-degree azimuth. As the 

gunner rotated the view, the turret portion of the vehicle diagram moved along the eight possible 

orientations to allow the gunner to place his/her field of view (FOV) on the cued target. Tactually, 

target positions relative to the gunner were presented via eight electromechanical transducers 

known as “tactors” (see figure 4). More details about the tactile display are given in Chen and 

Terrence (2007). 

Figure 3. Visual cueing (red dot indicating the 

direction of hostile target). 
7 

Figure 4. Tactile system. 

2.2.2 Communication Task Materials 

The communication task was administered concurrently with the experimental scenarios. The 

questions included simple military-related reasoning tests and simple memory tests. The inclusion 

of these cognitive tasks was for simulating an environment where the gunner was communicating 

with fellow crew members in the vehicle. For the reasoning tests, there were questions such as “if 

the enemy is to our left, and our UGV is to our right, what direction is the enemy to the UGV?” 

For the memory tests, the participants were asked to repeat some short statements or keep track of 

three radio call signs (e.g., Bravo 83) and they had to report to the experimenter whether the call 

signs they heard were one of those they were keeping track. Test questions were delivered by a 

synthetic speech program, DECTalk1. The questions were pre-recorded by a male speaker and 

presented at the rate of one question every 33 seconds. 

2.2.3 Questionnaires and Spatial Tests 

A demographics questionnaire was administered at the beginning of the training session (appendix A). A questionnaire about attentional control (appendix B) (Derryberry & Reed, 2002) was 

used to evaluate participants’ PAC. The Attentional Control Survey (ACS) consists of 21 items 

and measures attention focus and shifting. The scale has been shown to have good internal reliability (α = .88). Derryberry and Reed conducted an experiment to examine the relationship between self-reported (i.e., ACS score) and actual attentional control. They found that participants 

with a high ACS score could better resist interference in a Stroop-like spatial conflict task. In our 

previous studies (Chen & Joyner, 2006; Chen & Terrence, 2007), we observed a positive, although 

some-what weak, relationship between ACS score and some multi-tasking performance measures. 

 1DECtalk is a registered trademark of Digital Equipment Corporation. 
8 

The Cube Comparison (Cube for short) and the Hidden Patterns tests (Educational Testing Service, 

2007a&b) as well as the Spatial Orientation Test (Orientation for short) were used to assess participants’ SpA. The Cube test requires participants to compare, in 3 minutes, 21 pairs of six-sided 

cubes and determine if the rotated cubes are the same or different. The Hidden Patterns test 

measures flexibility of closure and involves identifying specific patterns or shapes embedded 

within distracting information. The Orientation test, constructed by Dr. Paula Durlach of the U.S. 

Army Research Institute, is modeled after the Cardinal Direction test developed by Gugerty and 

his colleagues (Gugerty & Brooks, 2004) and is a computerized test consisting of a brief training 

segment and 32 test questions. Accuracy and response time were automatically captured by the 

program. 

Participants’ perceived workload was evaluated with the computer-based version of National 

Aeronautics and Space Administration Task Load Index (NASA TLX) questionnaire (appendix C) 

(Hart & Staveland, 1988). The NASA TLX is a self-reported questionnaire of perceived demands 

in six areas: mental, physical, temporal, effort (mental and physical), frustration, and performance. 

Participants were asked to evaluate their perceived workload level in these areas on 10-point scales. 

They also assessed the contribution (i.e., weight) of each factor to the perceived workload by comparing the 15 possible pairs of the six factors. According to Noyes and Bruneau (2007), computerbased NASA TLX tends to generate higher workload ratings compared with the traditional paperbased survey. However, since the ratings were used to compare the workload levels across the 

experimental conditions, the elevated ratings should not affect these comparisons. 

Finally, a usability questionnaire about the tactile/visual AiTR displays was constructed (see 

appendix D). Participants indicated their level of reliance on tactile and/or visual cueing for the 

gunnery task when both types of alerts were available. They also indicated their perceived 

usability of the AiTR displays. Participants were also asked to evaluate their trust in the AiTR 

system using a modified survey by Jian, Bisantz, and Drury (2000) (appendix D, items 22-33). 

2.3 Experimental Design 

The overall design of the study is a 2 x 3 mixed design. The between-subject variable is AiTR 

type (FAP versus MP). The within-subject variable is Robotics Task type (Monitor versus Auto 

versus Teleop). 

The reliability level of the FAP and MP alerts was 60%. The FAP condition consisted of 10 hits 

(i.e., alerts when there were targets), eight FAs (i.e., alerts when there were no targets), no misses 

(i.e., no alerts when there were targets), and two correct rejections (CRs) (i.e., no alerts when there 

were no targets). The MP condition consisted of two hits, no FAs, eight misses, and 10 CRs. 

Additionally, only hostile targets were cued, not the neutral targets. The participants were 

instructed to detect the neutral targets independently. 
9 

2.4 Procedure 

After being briefed about the purpose of the study, the tasks for the experiment, and any risks 

involved, participants read and signed a consent form. They then answered the attentional control 

survey and were administered the spatial ability tests (i.e., Cube, Hidden Patterns, and Orientation). 

After these tests, participants received training, which was self-paced and was delivered by PowerPoint2 slides showing the elements of the TCU, steps for completing various tasks, several miniexercises for practicing the steps, and two exercises for performing the robotic control tasks (one 

for practicing the tele-operation task and one for practicing the auto control tasks). After the 

tutorial on TCU, participants were trained in the gunnery tasks. Participants had to demonstrate 

that they could perform the tasks without any help. The entire training session lasted about 2.5 

hours. 

Before the experimental session began, participants changed into one of the laboratory cotton 

T-shirts in order to standardize how the tactors were applied to the skin. They chose a size and 

then were escorted to the restroom where they could change privately. After this was done, the 

experimenter asked to measure the participant around the abdomen just above navel height so 

that the tactile display could be custom fitted. After taking this measurement, the experimenter 

arranged the tactors so that they were equidistant for the participant’s abdomen. Once fitted 

with the tactile display, the participant was seated in front of the gunner monitor. A test pattern 

confirmed that all eight tactors were working properly and that the participant could readily 

perceive the stimuli. The experimenter then explained the nature of the AiTR system and the 

corresponding visual or tactile cues that would be provided. 

In the experimental trials, participants’ tasks were to use their robotic asset to locate targets (i.e., 

enemy dismounted Soldiers) in the remote environment and to find targets in their immediate (i.e., 

MCS) environment. The MCS was simulated as traveling along a designated route, which was 

approximately 4.3 km and lasted about 15 minutes. There were 10 hostile and 10 neutral targets 

(i.e., civilians) along the route in each gunnery scenario. Participants were instructed to engage the 

hostile targets and verbally report spotting the neutral targets. In total, there were three 15-minute 

scenarios, corresponding to the three within-subject experimental conditions, the order of which 

was counterbalanced across participants. 

There were three types of robotics tasks: Monitor, Auto, and Teleop. The Monitor task required 

the operator to continuously monitor the video feed as the UGV travels autonomously and to 

verbally report detection of targets. There were 20 targets (5 hostile and 15 neutral) along the 

route. The Auto control task required the operator to monitor the video feed as the UGV traveled 

autonomously, examine still images generated from the RSTA (reconnaissance, surveillance, and 

target acquisition) scans, and detect targets. The Teleop task required the operator to manually 

manipulate and drive the UGV (using a joystick) along a predetermined route using the TCU to 

detect targets. For both the Auto and Teleop tasks, there were, on average, two to four targets (one 

 2PowerPoint is a trademark of Microsoft Corporation. 
10 

hostile plus one to three neutral targets) at each checkpoint. There were five checkpoints along the 

designated route. Upon detecting a target, participants needed to place the target on the map, label 

the target, and then send a spot report. 

While the participants were performing their gunnery and robotics control tasks, they simultaneously performed the communication task by answering questions delivered to them via DECtalk. 

There were 2-minute breaks between experimental scenarios. Participants assessed their workload 

using the NASA TLX after they completed each scenario. The entire experimental session lasted 

about 1 hour. 

2.5 Measures 

The dependent measures include mission performance (i.e., number of targets detected in the 

remote environment with the robotic asset and number of hostile/neutral targets detected in the 

immediate environment), communication task performance, and perceived workload. 

3. Results 

3.1 Target Detection Performance 

3.1.1 Gunnery Tasks 

3.1.1.1 Hostile Targets 

Table 1 lists several measures relating to operator task performance. A mixed analysis of variance 

(ANOVA) was performed to examine the effects of the concurrent robotic control tasks on the 

gunnery task performance (percentage of hostile targets detected), with the AiTR condition (FAP 

versus MP) being the between-subject factor and the Robotics Task condition (Monitor versus Auto 

versus Teleop) as the within-subject factor (see appendix E for the complete data). The analysis 

revealed that the Robotics condition significantly affected number of targets detected, F(2, 15) = 

4.578, p < .05 (figure 5). Post hoc tests (least significant difference [LSD]) showed that target 

detection in the Monitor condition was significantly higher than in the Auto and Teleop conditions. 

Table 1. Operator task performance (mean percent of targets detected for the gunnery and robotics tasks and 

percent correct for the communication task. (Standard deviations are presented in parentheses.) 

Measures FAP MP 

Monitor Auto Teleop Monitor Auto Teleop 

Gunnery Task 

(Hostile only) 

.775

(.238) 

.551

(.302) 

.644

(.256) 

.608

(.156) 

.533

(.227) 

.500

(.268) 

Gunnery Task 

(Neutral only) 

.325

(.166) 

.442

(.178) 

.319

(.171) 

.583

(.255) 

.467

(.267) 

.413

(.226) 

Robotics Task .908 

(.104) 

.639 

(.187) 

.770 

(.134) 

.906

(.102) 

.742

(.158) 

.637

(.128) 

Communication 

Task 

.802

(.118) 

.773

(.112) 

.777

(.133) 

.870

(.089) 

.812

(.117) 

.805

(.100) 
11 

Figure 5. Gunnery task performance, hostile targets only. 

The effects of participants’ SpA and PAC were also evaluated. The analysis showed that participants with higher SpA had a significantly higher gunnery task performance than did those with 

lower SpA, F(1, 16) = 6.311, p < .05 (figure 6). 

There was also a significant AiTR x PAC interaction, F(1, 16) = 7.358, p < .05 (figure 7). Those 

with a lower PAC had a better performance with the FAP cueing, and those with a higher PAC had 

a better performance with the MP cueing. 

In order to further examine the effect of task load on reliance of AiTR, the data of the MP condition were analyzed separately. Because of the small sample size (N = 12), no significant differences were found between those with a high versus low PAC. However, the trend was evident that 

as task load became heavier (i.e., Teleop > Auto > Monitor, based on Chen & Joyner, 2006), those 

with a low PAC became increasingly reliant on the AiTR, while those with a high PAC maintained 

a fairly stable level of reliance throughout the experimental conditions (figure 8). 
12 

Figure 6. Effects of spatial ability on gunnery task performance, hostile targets only. 

Figure 7. Effects of perceived attentional control on gunnery task performance, hostile targets only. 
13 

Figure 8. Effects of perceived attentional control on gunnery task performance (hostile 

targets only) in MP conditions. 

3.1.1.2 Neutral Targets 

Participants’ detection of neutral targets was also assessed. Since the AiTR only alerted the participants when hostile targets were present, the neutral target detection could be used to indicate 

how much visual attention was devoted to the gunnery station. A mixed ANOVA revealed a 

significant main effect for Robotics, F(2,15) = 4.362, p < .05 (figure 9). Post hoc tests (LSD) 

showed that neutral target detection in the Teleop condition was significantly lower than in the 

Auto condition. There was also a significant AiTR x PAC interaction, F(1, 16) = 3.602, p < .05 

(figure 10). Those with a lower PAC performed at about the same level, regardless of the AiTR 

type, while those with a higher PAC had a significantly better performance with the MP cueing. 
14 

Figure 9. Gunner’s neutral target detection performance. 

Figure 10. Effects of perceived attentional control on gunner’s neutral target detection 

performance. 
15 

3.1.2 Robotics Tasks 

A mixed ANOVA revealed that there was a significant main effect for Robotics, F(2,15) = 25.357, 

p < .001 (figure 11). It was found that the Monitor condition was significantly higher than both the 

Auto and the Teleop conditions in terms of percentage of targets detected. There was a significant 

Robotics x AiTR interaction, F(2,32) = 3.96, p < .05. The Monitor task performance stayed at the 

same level, regardless of the AiTR types. The Auto task performance was higher with the MP 

cueing, while the Teleop task performance was higher with the FAP cueing. 

Figure 11. Robotics task performance. 

There was also a significant AiTR x PAC interaction, F(1,16) = 4.801, p < .05 (figure 12). Those 

with a lower PAC had a better performance with the FAP cueing, while those with a higher PAC 

had a better performance with the MP cueing. 
16 

Figure 12. Effects of perceived attentional control on robotics task performance. 

3.2 Communication Task Performance 

A mixed ANOVA revealed that there was a significant main effect for Robotics, F(2,44) = 3.247, 

p < .05 (figure 13). It was found that the Monitor condition was significantly higher than the 

Teleop conditions. 

3.3 Perceived Workload 

Weighted ratings of the scales of the NASA TLX were used for this analysis. Participants’ selfassessment of workload was significantly affected by Robotic condition, F(2,15) = 25.057, 

p < .001 (figure 14). The perceived workload was significantly higher in the Teleop condition 

(M = 77.71) than in the Auto condition (M = 69.61) and the Monitor condition (M = 61.06). The 

difference between Auto and Monitor was also significant. There was a significant Robotics x 

AiTR interaction, F(2,15) = 5.464, p < .05. 
17 

Figure 13. Communication task performance. 

Figure 14. Perceived workload.
18 

3.4 AiTR Display Assessment 

A usability questionnaire captured participant preferences for the presentation of AiTR information. Following their interaction with the AiTR systems, 41% of participants responded that they 

relied predominantly or entirely on the tactile AiTR display, while 36% responded that they relied 

predominantly or entirely on the visual AiTR display. AiTR preference was also significantly 

correlated with SpA (r = .507, p < .01). Those with higher SpA tended to prefer tactile cueing over 

visual cueing. Conversely, those with lower SpA favored visual cueing over tactile cueing. 

We also evaluated participants’ self-assessed trust in the AiTR system and compared those with 

higher and lower PAC. There was no significant difference between these two groups. 

4. Discussion 

In this study, we simulated a generic mounted environment and conducted an experiment to 

examine the performance and workload of the combined position of gunner and robotics operator. 

More specifically, we investigated the effects of AiTR with imperfect reliability (FAP versus MP) 

on the operator’s performance of the automated (i.e., gunnery) task as well as the concurrent tasks 

(i.e., robotics and communication). Our most intriguing findings were related to the relationship of 

individual differences to predicted compliance and reliance performance. In order to simplify our 

discussion, we discuss effects of PAC differences on gunner performance for FAP and MP conditions and then discuss how these differences affected neutral targeting and secondary task 

performance (robotics and communication). 

4.1 Gunnery Task Performance, Hostile Targets 

Our results showed that the operator’s gunnery task performance in detecting hostile targets was 

significantly better in the Monitor condition than in the other two robotics task conditions, consistent with the findings of Chen and Joyner (2006). In Chen and Joyner (2006) and the current 

study, the workload associated with the Monitor condition was significantly lower than the other 

robotics conditions (see section 4.5 for more discussion of workload). These results suggest that 

the operator had more visual and mental resources for the gunnery task when the robotics task was 

simply monitoring the video feed, compared with the other two robotics conditions. 

Also consistent with the previous findings (Chen & Joyner, 2006; Chen & Terrence, 2007), 

participants’ SpA was found to be an accurate predictor of their gunnery performance. Thomas 

and Wickens (2004) showed that there were individual differences in scanning effectiveness and 

its associated target detection performance. However, Thomas and Wickens did not examine the 

characteristics of those participants who had more effective scanning strategies. The findings of 
19 

the current study along with the previous two studies indicate that SpA may be an important factor 

for determining scanning effectiveness. 

Our results also showed that there was a significant interaction between types of unreliable AiTR 

and participants’ PAC. For those with high PAC, our data are consistent with the notion that 

operator reliance on and compliance with automation are independent constructs and are separately 

affected by system misses and false alarms. Based on figure 7, it is evident that high PAC participants did not comply with alerts in the FAP condition. Since the FAP AiTR had a 0% miss rate, a 

full compliance should result in a detection rate over 80%, as reported in Chen and Terrence (in 

which a perfectly reliable AiTR was used). As predicted, figure 7 shows that in MP conditions, 

high PAC participants did not rely on the AiTR and detected more targets than were cued. However, an examination of the data for the low PAC participants revealed a completely opposite 

trend. Specifically, with the FAP condition, low PAC participants showed a strong compliance 

with the alerts, which resulted in a good performance in target detection (at a similar level as in 

Chen and Terrence). With the MP condition, however, low PAC participants evidently overly 

relied on the automation and therefore had a very poor performance. Figure 8 shows that as task 

load became heavier, those with low PAC became increasingly reliant on the AiTR, while those 

with high PAC maintained a fairly stable level of reliance throughout the experimental conditions. 

According to Biros, Daly, and Gunsch (2004), higher task loads tend to induce a higher level of 

reliance on automated systems. Our present data suggest that this heightened level of reliance is 

also moderated by PAC. More specifically, only those with low PAC tend to exhibit over-reliance 

on automation under a heavy task load. Because of the small sample size, this observation could 

not be confirmed based on statistical tests. Future research should further examine this relationship between PAC, task load, and reliance on automation. 

4.2 Gunnery Task Performance, Neutral Targets 

It was found that the gunner’s detection of neutral targets (which was not aided by AiTR) was 

significantly worse when s/he had to tele-operate a robotic asset (versus when the asset was semiautonomous). This finding is consistent with previous research (Chen & Joyner, 2006; Chen & 

Terrence, 2007) and suggests that participants devoted significantly less visual attention to the 

gunnery station when their robotic asset required tele-operation. Our data also showed that those 

with a lower PAC performed at about the same level, regardless of the AiTR type, while those 

with a higher PAC had a significantly better performance with the MP cueing. This suggests that 

higher PAC participants devoted more visual attention to the gunnery station (implying a reduced 

reliance on automation for the gunnery task) when the AiTR was MP than when the AiTR was 

FAP. Although we did not measure participants’ scanning behaviors, the detection rate of neutral 

targets on the gunnery station provides an estimate of the amount of operator’s visual attention on 

the automated task environment. Again, the data of high PAC participants seem to support the 

hypothesis that MP automation reduces operator reliance. However, the same phenomenon was 

not observed for the low PAC participants. 
20 

4.3 Robotics Task Performance 

For the robotics tasks, the data showed that participants had the best performance when the task 

was only monitoring the video feed. Moreover, the Monitor task performance stayed at the same 

level, regardless of the AiTR types. On the other hand, the Auto task performance was higher 

with the MP cueing, while the Teleop task performance was higher with the FAP cueing. The 

data of the Teleop performance are consistent with previous studies that MP automation degrades 

concurrent task performance more than FAP (Dixon & Wickens, 2006; Wickens, Dixon, Goh et 

al., 2005). However, the same trend was not observed for the other two robotics tasks, which were 

less challenging than the Teleop task. Therefore, it appears that the adverse effect of MP automation on concurrent tasks is only manifest in more challenging task conditions. Our data also 

showed that again, there was a significant AiTR Type x PAC interaction. Consistent with the 

previous two performance measures (gunnery-hostile and gunnery-neutral), the low PAC participants exhibited a higher level of performance degradation with the MP conditions. The performance of the high PAC participants, on the other hand, showed a completely opposite trend. 

These results suggest that the high PAC participants’ reduced compliance with the FAP alerts did 

not help them with their concurrent task, compared with the MP conditions; in fact, it resulted in a 

poorer targeting performance. Conversely, their reduced reliance on the MP alerts did not hurt 

their performance as did the FAP alerts. Overall, the low PAC participants showed the most pronounced adverse effect of MP alerts on concurrent performance. Conversely, the FAP alerts not 

only helped them with their automated task but also their concurrent task. 

Taking the three main performance measures together (i.e., Gunnery task, Hostile Targets, Gunner 

task, Neutral Targets, and Robotics task), it appears that overall, for high PAC participants, FAP 

alerts were more detrimental than MP alerts. FAP alerts affected their automated task and the 

concurrent task. This finding is consistent with the conclusion of Dixon et al. (2007) that FAP 

degraded overall performance more than MP automation. However, it is worth noting that for low 

PAC participants, we observed the opposite pattern: MP automation was more harmful than FAP 

automation. The overall data suggest that low PAC participants had a higher trust in the automation system than did high PAC participants. It is likely that low PAC participants had more 

difficulty in performing multiple tasks concurrently and had to rely on automation when available. 

High PAC participants, on the other hand, tended to rely on their own multi-tasking ability to 

perform the tasks. It is interesting to note that there was no significant difference in the participants’ self assessment of their trust in the AiTR system between high PAC and low PAC groups. 

Our results are consistent with past research (Lee & Moray, 1994; Lee & See, 2004) that selfconfidence is a critical factor in mediating the effect of trust (in automation) on reliance (on the 

automatic system). Lee and Moray found that when self-confidence exceeded trust, operators 

tended to use manual control. When trust exceeded self-confidence, automation was used more. 

Further research should be conducted to examine the relationships between PAC, self-confidence, 

and trust in automation. 
21 

4.4 Communication Task Performance 

Participants’ communication task performance was significantly better when their robotics task 

was Monitor than when it was Teleop. This finding is consistent with Chen and Joyner (2006). 

According to Naveh-Benjamin, Craik, Perretta, and Tonev (2000), information-encoding processes 

require more attention than retrieval and are more prone to the effects of competing demands of 

multi-tasking. It is therefore likely that the information-encoding process of the communication 

task in our study was more disrupted by the more challenging Teleop task than by the Monitor 

task. 

4.5 Perceived Workload 

Participants’ perceived workload was found to be affected by the type of concurrent robotics task. 

The workload was significantly higher in the Teleop condition than in the Auto condition, which 

in turn was significantly higher than in the Monitor condition. These results are consistent with 

Chen and Joyner (2006) and Schipani (2003), which evaluated robotic operator workload in a field 

setting. Although many of the ground robotic assets in the Army’s FCS program will be semiautonomous, tele-operation will still be an important part of any missions involving robotic assets 

(e.g., when robots encounter obstacles or other problems). The higher workload associated with 

tele-operation needs to be taken into account when one is designing the user interfaces for the 

robotic assets (see Chen, Haas, & Barnes, 2007, for a review of user interface designs for teleoperated robots).

4.6 AiTR Display Preference 

The significant positive correlation of AiTR preference with SpA is interesting. It appeared that as 

AiTR ratings tended toward considerable reliance on the tactile display, there was a concurrent 

shift with higher performance on the spatial tests. This confirmed a trend that was first observed in 

Chen and Terrence (2007). Perhaps those with higher SpA can more easily employ the spatial 

tactile signals in the dual task setting and therefore have a stronger preference for something that 

makes the gunner task easier to complete. Individuals with lower SpA, on the other hand, may 

have not used the spatial tactile cues to their full extent and therefore continued to prefer the visual 

AiTR display. According to Kozhevnikov Hegarty, and Mayer (2002), visualizers with lower SpA 

tend to rely on iconic imagery while those with higher SpA tend to prefer using spatial-schematic 

imagery to solve problems. Therefore, it is likely that in our study, those who preferred visual 

AiTR displays might be more iconic in their mental representations. However, this preference 

may have caused degraded target detection performance because more visual attention was 

devoted to the visual AiTR display than to the simulated environment. In contrast, those who 

were more spatial relied on the directional information of the tactile display to help them with the 

visually demanding tasks, resulting in a more effective performance. 
22 

5. Conclusions 

In this study, we simulated a generic mounted environment and conducted an experiment to 

examine the performance and workload of the combined position of gunner and robotics operator. 

More specifically, we investigated the effects of AiTR with imperfect reliability (FAP versus MP) 

on the operator’s performance of the automated (i.e., gunnery) task as well as the concurrent tasks 

(i.e., robotics and communication). Our data suggest that there is a strong interaction between the 

type of AiTR unreliability and participants’ PAC for almost all the performance measures (except 

for the communication task, which showed the same trend but the interaction was not statistically 

significant). Overall, it appears that for high PAC participants, FAP alerts were more detrimental 

than MP alerts. FAP alerts affected not only their automated task but also the concurrent task. 

However, for low PAC participants, MP automation was more harmful than FAP automation. 

Future research should incorporate performance-based measures of attentional shifting effectiveness (e.g., Synthetic Work Environment or SYNWORK) in addition to surveys such as the ACS 

(Branscome & Grynovicki, 2007). In the area of SpA, the current study replicated the finding of 

Chen and Terrence (2007) that the operator’s preference of AiTR display (i.e., visual versus 

tactile) is correlated with his or her SpA. Low SpA individuals prefer visual cueing over tactile 

cueing, although tactile display would be more effective in highly visual environments (so visual 

attention can be devoted to the tasks, not to the cues). These findings may have important 

implications for personnel selection, system designs, and training development. For example, to 

better enhance the task performance for low SpA individuals, the visual cueing display should be 

more integrated with the visual scene. Augmented reality (i.e., visual overlays) is a potential 

technique to embed directional information onto the video (Calhoun & Draper, 2006). Additionally, the capabilities and limits of the automated systems should be conveyed to the operator, when 

feasible, in order for the operator to develop appropriate trust and reliance (Lee & See, 2004). 
23 

6. References 

Biros, D. P.; Daly, M.; Gunsch, G. The Influence of Task Load and Automation Trust on 

Deception Detection. Group Decision and Negotiation 2004, 13, 173-189. 

Branscome, T. A.; Grynovicki, J. O. An Investigation of Factors Affecting Multi-task 

Performance in an Immersive Environment; ARL-TR-4325; U.S. Army Research 

Laboratory: Aberdeen Proving Ground, MD, 2007. 

Calhoun, G.; Draper, M. Multi-Sensory Interfaces for Remotely Operated Vehicles. In N. 

Cooke, H. Pringle, H. Pedersen, & O. Connor (Eds.), Human factors of remotely operated 

vehicles (pp. 149-163), Oxford, UK: Elsevier, 2006. 

Chen, J. Y. C.; Durlach, P.; Sloan, J.; Bowens, L. Human Robot Interaction in the Context of 

Simulated Route Reconnaissance Missions. Military Psychology, in press. 

Chen, J. Y. C.; Haas, E. C.; Barnes, M. J. Human Performance Issues and User Interface Design 

for Teleoperated Robots. IEEE Transactions on Systems, Man, and Cybernetics--Part C: 

Applications and Reviews 2007, 37, 1231-1245. 

Chen, J. Y. C.; Joyner, C. T. Concurrent Performance of Gunner’s and Robotic Operator’s 

Tasks in a Simulated MCS Environment; ARL-TR-3815; U.S. Army Research Laboratory: 

Aberdeen Proving Ground, MD, 2006. 

Chen, J. Y. C.; Terrence, P. I. Effects of Tactile Alerts on Concurrent Performance of the 

Gunner’s and Robotic Operator’s Tasks in a Simulated Mounted Environment; ARL-TR4227; U.S. Army Research Laboratory: Aberdeen Proving Ground, MD, 2007. 

Derryberry, D.; Reed, M. A. Anxiety-Related Attentional Biases and Their Regulation by 

Attentional Control. Journal of Abnormal Psychology 2002, 111 (2), 225-236. 

Dixon, S. R.; Wickens, C. D. Automation Reliability in Unmanned Aerial Vehicle Control: A 

Reliance-Compliance Model of Automation Dependence in High Workload. Human 

Factors 2006, 48, 474-486. 

Dixon, S. R.; Wickens, C. D.; McCarley, J. S. On the Independence of Compliance and 

Reliance: Are Automation False Alarms Worse Than Misses? Human Factors 2007, 49, 

564-572. 

Educational Testing Service. Cube Comparison Test. Princeton, NJ: ETS, 2007a. 

Educational Testing Service. Hidden Figures and Hidden Patterns Tests. Princeton, NJ: ETS, 

2007b. 
24 

Gugerty, L.; Brooks, J. Reference-Frame Misalignment and Cardinal Direction Judgments: 

Group Differences and Strategies. Journal of Experimental Psychology: Applied 2004, 10

(2), 75-88. 

Hart, S.; Staveland, L. Development of NASA TLX (Task Load Index): Results of Empirical and 

Theoretical Research. In P. Hancock & N. Meshkati (Eds.), Human Mental Workload (pp. 

139-183). Amsterdam: Elsevier, 1988. 

Jian, J. Y.; Bisantz, A. M.; Drury, C. G. Foundations for An Empirically Determined Scale of 

Trust in Automated Systems. International Journal of Cognitive Ergonomics 2000, 4, 53-

71. 

Kahneman, D., Ben-Ishai, R.; Lotan, M. Relation of a Test of Attention to Road Accidents. 

Journal of Applied Psychology 1973, 58, 113-115. 

Kozhevnikov, M.; Hegarty, M.; Mayer, R. E. Revising the Visualizer-Verbalizer Dimension: 

Evidence for Two Types of Visualizers. Cognition & Instruction 2002, 20 (1), 47-77. 

Lathan, C.E.; Tracey, M. The Effects of Operator Spatial Perception and Sensory Feedback on 

Human-Robot Teleoperation Performance. Presence 2002, 11, 368-377. 

Lee, J. D.; Moray, N. Trust, Self-Confidence, and Operators’ Adaptation to Automation. 

International Journal of Human-Computer Studies 1994, 40, 153-184. 

Lee, J. D.; See. K. A. Trust in Automation: Designing for Appropriate Reliance. Human 

Factors 2004, 46, 50-80. 

Mitchell, D. K. Soldier Workload Analysis of the Mounted Combat System (MCS) Platoon’s Use 

of Unmanned Assets; ARL-TR-3476; U.S. Army Research Laboratory: Aberdeen Proving 

Ground, MD, 2005. 

Naveh-benjamin, M.; Craik, F. I. M.; Perretta, J. G.; Tonev, S. T. The Effects of Divided 

Attention on Encoding and Retrieval Processes: The resiliency of retrieval processes. The 

Quarterly Journal of Experimental Psychology 2000, 53A, 609-625. 

Noyes, J. M.; Bruneau, D. P. A Self-Analysis of the NASA-TLX Workload Measure. 

Ergonomics 2007, 50, 514-519. 

Rubinstein, J.S.; Meyer, D.E.; Evans, J.E. Executive Control of Cognitive Processes in Task 

Switching. Journal of Experimental Psychology: Human Perception and Performance

2001, 27, 763-797. 

Schipani, S. An Evaluation of Operator Workload During Partially Autonomous Vehicle 

Operations. Proceedings of PerMIS 2003 2003. 
25 

Schumacher E.H.; Seymour, T.L.; Glass, J. M.; Fencsik, D.E.; Lauber, E.J.; Kieras, D.E. et al. 

2001, Virtually Perfect Time Sharing in Dual-Task Performance: Uncorking the Central 

Cognitive Bottleneck. Psychological Science 2001, 12, 101-108. 

Thomas, L.; Wickens, C. Eye-Tracking and Individual Differences in Off-Normal Event 

Detection When Flying With a Synthetic Vision System Display. Proceedings of the Human 

Factors and Ergonomics Society 48th Meeting (pp. 223-227), 2004. 

TRADOC. The United States Army Objective Force Operational and Organizational Plan 

Maneuver Unit of Action; TRADOC Pamphlet 525-3-90 O&O; Unit of Action Maneuver 

Battle Lab: Fort Knox, KY, 2003. 

Wickens, C. D.; Dixon, S. R. Is There a Magic Number 7 (to the Minus 1)? The Benefits of 

Imperfect Diagnostic Automation: A Synthesis of the Literature;Tech. Report AHFD-05-

01/MAAD-05-01; University of Illinois, Institute of Aviation, Aviation Human Factors 

Division: Urbana-Champaign, IL, 2005. Retrieved May 15, 2006, from http://www.human 

factors.uiuc.edu/Reports&PapersPDFs/TechReport/05-01.pdf. 

Wickens, C. D.; Dixon, S. R.; Goh, J.; Hammer, B. Pilot Dependence on Imperfect Diagnostic 

Automation in Simulated UAV Flights: An Attentional Visual Scanning Analysis; Tech. 

Report AHFD-05-02/MAAD-05-02; University of Illinois, Institute of Aviation, Aviation 

Human Factors Division: Urbana-Champaign, IL, 2005. Retrieved December 15, 2005, 

from http://www.humanfactors.uiuc.edu/Reports&PapersPDFs/TechReport/05-02.pdf. 

Wickens, C. D.; Dixon, S. R.; Johnson N. UAV Automation: Influence of Task Priorities and 

Automation Imperfection in a Difficult Surveillance Task; Tech. Report AHFD-05-20/ 

MAAD-05-6; University of Illinois, Institute of Aviation, Aviation Human Factors Division: 

Urbana-Champaign, IL, 2005. Retrieved December 15, 2005, from http://www.human 

factors.uiuc.edu/Reports&PapersPDFs/TechReport/05-20.pdf. 
26 

INTENTIONALLY LEFT BLANK
27 

Appendix A. Demographic Questionnaire 

Participant # _______ Age ______ Major ________________ Date ___________ Gender ___ 

1. What is the highest level of education you have had? 

Less than 4 yrs of college ____ Completed 4 yrs of college ____ Other ____ 

2. When did you use computers in your education? (Circle all that apply) 

Grade School Jr. High High School 

Technical School College Did Not Use 

3. Where do you currently use a computer? (Circle all that apply) 

Home Work Library Other________ Do Not Use 

4. For each of the following questions, circle the response that best describes you. 

How often do you: 

Use a mouse? Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use a joystick? Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use a touch screen? Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use icon-based programs/software? 

 Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use programs/software with pull-down menus? 

 Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use graphics/drawing features in software packages? 

 Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Use E-mail? Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Operate a radio controlled vehicle (car, boat, or plane)? 

 Daily, Weekly, Monthly, Once every few months, Rarely, Never 

Play computer/video games? 

 Daily, Weekly, Monthly, Once every few months, Rarely, Never 

5. Which type(s) of computer/video games do you most often play if you play at least once every few months? 

6. Which of the following best describes your expertise with computer? (check √ one) 

_____ Novice 

_____ Good with one type of software package (such as word processing or slides) 

_____ Good with several software packages 

_____ Can program in one language and use several software packages 

_____ Can program in several languages and use several software packages 

7. Are you in your usual state of health physically? YES NO 

 If NO, please briefly explain: 

8. How many hours of sleep did you get last night? ______ hours 

9. Do you have normal color vision? YES NO 

10. Do you have prior military service? YES NO If Yes, how long __________ 
28 

INTENTIONALLY LEFT BLANK
29 

Appendix B. Attentional Control Survey 

For each of the following questions, circle the response that best describes you. 

It is very hard for me to concentrate on a difficult task when there are noises around. 

 Almost never, Sometimes, Often, Always 

When I need to concentrate and solve a problem, I have trouble focusing my attention. 

 Almost never, Sometimes, Often, Always 

When I am working hard on something, I still get distracted by events around me. 

 Almost never, Sometimes, Often, Always 

My concentration is good even if there is music in the room around me. 

 Almost never, Sometimes, Often, Always 

When concentrating, I can focus my attention so that I become unaware of what’s going on in the room around 

me. Almost never, Sometimes, Often, Always 

When I am reading or studying, I am easily distracted if there are people talking in the same room. 

 Almost never, Sometimes, Often, Always 

When trying to focus my attention on something, I have difficulty blocking out distracting thoughts. 

 Almost never, Sometimes, Often, Always 

I have a hard time concentrating when I’m excited about something. 

 Almost never, Sometimes, Often, Always 

When concentrating, I ignore feelings of hunger or thirst. Almost never, Sometimes, Often, Always 

 

I can quickly switch from one task to another. Almost never, Sometimes, Often, Always 

It takes me a while to get really involved in a new task. Almost never, Sometimes, Often, Always 

It is difficult for me to coordinate my attention between the listening and writing required when taking notes 

during lectures. Almost never, Sometimes, Often, Always 

I can become interested in a new topic very quickly when I need to. 

 Almost never, Sometimes, Often, Always 

It is easy for me to read or write while I’m also talking on the phone. 

 Almost never, Sometimes, Often, Always 

I have trouble carrying on two conversations at once. Almost never, Sometimes, Often, Always 

 

I have a hard time coming up with new ideas quickly. Almost never, Sometimes, Often, Always 

After being interrupted or distracted, I can easily shift my attention back to what I was doing before. 

 Almost never, Sometimes, Often, Always 

When a distracting thought comes to mind, it is easy for me to shift my attention away from it. 

 Almost never, Sometimes, Often, Always 

It is easy for me to alternate between two different tasks. Almost never, Sometimes, Often, Always 

It is hard for me to break from one way of thinking about something and look at it from another point of view. 

 Almost never, Sometimes, Often, Always 
30 

INTENTIONALLY LEFT BLANK
31 

Appendix C. NASA TLX Questionnaire 

Please rate your overall impression of demands imposed on you during the exercise. 

1. Mental Demand: How much mental and perceptual activity was required (e.g., thinking, looking, 

searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 

2. Physical Demand: How much physical activity was required (e.g., pushing, pulling, turning, 

controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or 

laborious? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 

3. Temporal Demand: How much time pressure did you feel due to the rate or pace at which the task or 

task elements occurred? Was the pace slow and leisurely or rapid and frantic? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 

4. Level of Effort: How hard did you have to work (mentally and physically) to accomplish your level of 

performance? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 

5. Level of Frustration: How insecure, discouraged, irritated, stressed and annoyed versus secure, 

gratified, content, relaxed and complacent did you feel during the task? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 

6. Performance: How successful do you think you were in accomplishing the goals of the task set by the 

experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals? 

LOW |---|---|---|---|---|---|---|---|---| HIGH 

 1 2 3 4 5 6 7 8 9 10 
32 

INTENTIONALLY LEFT BLANK
33 

Appendix D. Usability Questionnaire 

1. The gunner station should only have the visual ATR display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

 

Comments 

2. The gunner station should only have the tactile ATR display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

 

3. I made use of both the visual and tactile ATR displays. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

 

4. I sometimes felt ‘lost’ using the tactile ATR display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

5. I sometimes felt ‘lost’ using the visual+tactile display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

6. The tactile ATR display was intuitive and made it easy to 

determine the direction of targets. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

7. The visual+tactile ATR display was intuitive and made it easy to 

determine the direction of targets. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 
34 

8. I do not feel either ATR display (tactile or visual+tactile) was 

helpful in the gunner task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

9. The tactile ATR display was helpful when I had to teleoperate the 

UGV. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

10. I relied heavily on the tactile ATR display for the gunner task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

11. I relied heavily on the visual+tactile ATR display for the gunner 

task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

12. The tactile ATR display was helpful when the UGV was semiautonomous. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

13. The visual+tactile ATR display was helpful when I had to 

teleoperate the UGV. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

14. The visual+tactile ATR display was helpful when the UGV was 

semi-autonomous. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

15. The gunner station should not have an ATR display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

16. The tactile ATR display was confusing. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

17. The visual+tactile ATR display was confusing. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

18. The tactile ATR display was annoying. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

19. The visual+tactile ATR display was annoying. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 
35 

20. The tactile ATR display improved my performance on the gunner 

task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

21. The visual+tactile ATR display improved my performance on the 

gunner task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

22. The visual+tactile ATR display can be deceptive. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

23. The visual+tactile ATR display sometimes behaves in an 

unpredictable manner.

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

24. I am often suspicious of the visual+tactile ATR system’s intent, 

action, or outputs. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

25. I am sometimes unsure of the visual+tactile ATR system.

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

26. The visual+tactile ATR system may have harmful effects on the 

gunnery task.

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

27. I am confident in the visual+tactile ATR system.

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 
36 

28. The visual+tactile ATR system can provide security. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

29. The visual+tactile ATR system has integrity. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

30. The visual+tactile ATR system is dependable. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

31. The visual+tactile ATR system is consistent. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

32. I can trust the visual+tactile ATR system. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

33. I am familiar with the visual+tactile ATR display. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

34. The visual+tactile ATR display improved my performance on the 

gunner task. 

Strongly DISAGREE |----|----|----|----|----|----| Strongly AGREE N/A 

 1 2 3 4 5 6 7 

Which of the following best describes your source of AITR information when you had access to 

both the visual and the tactile displays (please circle ONE answer only): 

1. entirely visual 

2. predominately visual 

3. both visual and tactile 

4. predominately tactile 

5. entirely tactile 
37 

Appendix E. Complete Gunnery Performance Data (hostile targets) 

AiTR SpA PAC Mean Std. Deviation N 

Low .5750 .09574 4

High .2500 .07071 2

 Low 

Total .4667 .18619 6

Low 1.0000 . 1

High .5622 .38144 5

 High 

Total .6352 .38515 6

Low .6600 .20736 5

High .4730 .34790 7

FAP 

Total 

Total .5509 .30154 12

Low .6500 .35355 2

High .5500 .28868 4

 Low 

Total .5833 .27869 6

Low .4200 .08367 5

High .8000 . 1

 High 

Total .4833 .17224 6

Low .4857 .19518 7

High .6000 .27386 5

MP 

Total 

Total .5333 .22697 12

Low .6000 .17889 6

High .4500 .27386 6

 Low 

Total .5250 .23404 12

Low .5167 .24833 6

High .6018 .35471 6

 High 

Total .5592 .29530 12

Low .5583 .21088 12

High .5259 .31236 12

Gunnery Task (Hostile 

Targets)- Auto Condition 

Total 

Total 

Total .5421 .26116 24

Gunnery Task (Hostile FAP Low Low .6444 .18592 4
38 

High .3750 .17678 2

Total .5546 .21529 6

 Low 1.0000 . 1

High .6800 .27749 5

 High 

Total .7333 .28048 6

 Low .7156 .22630 5

High .5929 .28052 7

Total 

Total .6440 .25600 12

 Low .3000 .28284 2

High .5750 .33040 4

 Low 

Total .4833 .31885 6

 Low .4400 .16066 5

High .9000 . 1

 High 

Total .5167 .23647 6

 Low .4000 .18764 7

High .6400 .32094 5

MP 

Total 

Total .5000 .26820 12

 Low .5296 .26149 6

High .5083 .28708 6

 Low 

Total .5190 .26204 12

 Low .5333 .27003 6

High .7167 .26394 6

 High 

Total .6250 .27199 12

 Low .5315 .25343 12

High .6125 .28455 12

Total 

Total 

Total .5720 .26675 24

Low .7750 .32016 4

High .5000 .00000 2

 Low 

Total .6833 .28577 6

Low 1.0000 . 1

2 .8400 .15166 5

Gunnery Task (Hostile 

Targets)- Monitor ConditionFAP 

 High 

Total .8667 .15055 6
39 

Low .8200 .29496 5

High .7429 .20702 7

Total 

Total .7750 .23789 12

Low .5000 .14142 2

High .5750 .17078 4

 Low 

Total .5500 .15166 6

Low .6400 .15166 5

High .8000 . 1

 High 

Total .6667 .15055 6

Low .6000 .15275 7

High .6200 .17889 5

MP 

Total 

Total .6083 .15643 12

Low .6833 .29269 6

High .5500 .13784 6

 Low 

Total .6167 .22896 12

Low .7000 .20000 6

High .8333 .13663 6

 High 

Total .7667 .17753 12

Low .6917 .23916 12

High .6917 .19752 12

Total 

Total 

Total .6917 .21451 24
40 

Acronyms 

ANOVA analysis of variance 

ARL Army Research Laboratory 

AiTR aided target recognition 

CR correct rejection 

FA false alarm 

FAP false alarm prone 

FCS Future Combat System 

FOV field of view 

IMPRINT Improved Performance Research Integration Tool 

LOS line of sight 

LSD least significant difference 

MCS mounted combat system 

MP miss prone 

NASA TLX National Aeronautics and Space Administration Task Load Index 

OCU operator control unit 

OTW out the window 

PAC perceived attentional control 

RCTA Robotics Collaborative Technology Alliance 

RSTA reconnaissance, surveillance, and target acquisition 

SIL system integration laboratory 

SpA spatial ability 

SSQ Simulator Sickness Questionnaire 

TCU tactical control unit 

TS total severity 
41 

TSS Total Severity Score 

UCD unmanned combat demonstration 

UCF University of Central Florida 

UGV unmanned ground vehicle 

XUV experimental unmanned vehicle 
42 

NO. OF 

COPIES ORGANIZATION

 1 DEFENSE TECHNICAL 

 (PDF INFORMATION CTR 

 ONLY) DTIC OCA 

 8725 JOHN J KINGMAN RD 

 STE 0944 

 FORT BELVOIR VA 22060-6218 

 1 US ARMY RSRCH DEV & ENGRG CMD 

 SYSTEMS OF SYSTEMS INTEGRATION 

 AMSRD SS T 

 6000 6TH ST STE 100 

 FORT BELVOIR VA 22060-5608 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 IMNE ALC IMS 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 AMSRD ARL CI OK TL 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 DIRECTOR 

 US ARMY RESEARCH LAB 

 AMSRD ARL CS OK T 

 2800 POWDER MILL RD 

 ADELPHI MD 20783-1197 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR ML J MARTIN 

 MYER CENTER RM 2D311 

 FT MONMOUTH NJ 07703-5601 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MZ A DAVISON 

 320 MANSCEN LOOP STE 115 

 FT LEONARD WOOD MO 65473 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MD T COOK 

 BLDG 5400 RM C242 

 REDSTONE ARSENAL AL 35898-7290 

 1 COMMANDANT USAADASCH 

 ATTN ATSA CD 

 ATTN AMSRD ARL HR ME DR HAWLEY 

 5800 CARTER RD 

 FT BLISS TX 79916-3802 

NO. OF 

COPIES ORGANIZATION

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MM DR V J RICE 

 BLDG 4011 RM 217 

 1750 GREELEY RD 

 FT SAM HOUSTON TX 78234-5002 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MG R SPINE 

 BUILDING 333 

 PICATINNY ARSENAL NJ 07806-5000 

 1 ARL HRED ARMC FLD ELMT 

 ATTN AMSRD ARL HR MH C BURNS 

 BLDG 1467B ROOM 336 

 THIRD AVENUE 

 FT KNOX KY 40121 

 1 ARMY RSCH LABORATORY - HRED 

 AWC FIELD ELEMENT 

 ATTN AMSRD ARL HR MJ D DURBIN 

 BLDG 4506 (DCD) RM 107 

 FT RUCKER AL 36362-5000 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MK MR J REINHART 

 10125 KINGMAN RD 

 FT BELVOIR VA 22060-5828 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MV HQ USAOTC 

 S MIDDLEBROOKS 

 91012 STATION AVE ROOM 348 

 FT HOOD TX 76544-5073 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MY M BARNES 

 2520 HEALY AVE STE 1172 BLDG 51005 

 FT HUACHUCA AZ 85613-7069 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MP D UNGVARSKY 

 POPE HALL BLDG 470 BCBL 

 806 HARRISON DR 

 FT LEAVENWORTH KS 66027-2302 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MJF J HANSBERGER 

 JFCOM JOINT EXPERIMENTATION J9 

 JOINT FUTURES LAB 

 115 LAKEVIEW PARKWAY SUITE B 

 SUFFOLK VA 23435 
43 

NO. OF 

COPIES ORGANIZATION

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MQ M R FLETCHER 

 US ARMY SBCCOM NATICK SOLDIER CTR 

 AMSRD NSC WS E BLDG 3 RM 343 

 NATICK MA 01760-5020 

 10 ARMY RSCH LABORATORY-HRED 

 ATTN AMSRD ARL HR MT J CHEN 

 12423 RESEARCH PARKWAY 

 ORLANDO FL 32826 

 1 ARMY RSCH LABORATORY-HRED 

 ATTN AMSRD ARL HR MT C KORTENHAUS 

 12350 RESEARCH PARKWAY 

 ORLANDO FL 32826 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MS C MANASCO 

 SIGNAL TOWERS BLDG 29808A RM 303 

 FORT GORDON GA 30905-5233 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MU M SINGAPORE 

 6501 E 11 MILE RD MAIL STOP 284 

 BLDG 200A 2ND FL RM 2104 

 WARREN MI 48397-5000 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MF C HERNANDEZ 

 2421 NW AUSTIN RD STE 220 

 FORT SILL OK 73503-9042 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MW E REDDEN 

 BLDG 4 ROOM 332 

 FT BENNING GA 31905-5400 

 1 ARMY RSCH LABORATORY - HRED 

 ATTN AMSRD ARL HR MN R SPENCER 

 DCSFDI HF 

 HQ USASOC BLDG E2929 

 FORT BRAGG NC 28310-5000 

 1 ARMY G1 

 ATTN DAPE MR B KNAPP 

 300 ARMY PENTAGON ROOM 2C489 

 WASHINGTON DC 20310-0300 

 1 ARL-HRED LIAISON 

 PHYSICAL SCIENCES LAB 

 PO BOX 30002 

 LAS CRUCES NM 88003-8002 

NO. OF 

COPIES ORGANIZATION

 1 DIRECTOR 

 UNIT OF ACTION MANEUVER BATTLE LAB 

 ATTN ATZK UA 

 BLDG 1101 

 FORT KNOX KY 40121 

 1 DIR FOR PERS TECHNOLOGIES 

 DPY CHIEF OF STAFF PERS 

 300 ARMY PENTAGON 2C733 

 WASHINGTON DC 20310-0300 

 1 CODE 1142PS 

 OFC OF NAVAL RSCH 

 800 N QUINCY STREET 

 ARLINGTON VA 22217-5000 

 1 CDR 

 USA AEROMEDICAL RSCH LAB 

 ATTN LIBRARY 

 FORT RUCKER AL 36362-5292 

 1 US ARMY NATICK RD&E CTR 

 ATTN STRNC YBA 

 NATICK MA 01760-5020 

 1 PEO STRI 

 12350 RSCH PARKWAY 

 ORLANDO FL 32826-3276 

 1 GOVT PUBLICATIONS LIB 

 409 WILSON M 

 UNIVERSITY OF MINNESOTA 

 MINNEAPOLIS MN 55455 

 1 HUMAN FACTORS ENG PROGRAM 

 DEPT OF BIOMEDICAL ENGNG 

 COLLEGE OF ENG & COMPUTER SCIENCE 

 WRIGHT STATE UNIVERSITY 

 DAYTON OH 45435 

 1 DIR AMC-FIELD ASSIST IN 

 SCIENCE & TECHNOLOGY 

 ATTN AMC-FAST 

 FT BELVOIR VA 22060-5606 

 ABERDEEN PROVING GROUND

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL CI OK (TECH LIB) 

 BLDG 4600 
44 

NO. OF 

COPIES ORGANIZATION

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL CI OK S FOPPIANO 

 BLDG 459 

 1 DIRECTOR 

 US ARMY RSCH LABORATORY 

 ATTN AMSRD ARL HR MR F PARAGALLO 

 BLDG 459