Experimental Evaluation of Computer-Aided

Tele-operation (CATO) and Computer-Aided Robotic 

Manipulation (CARMAN) Technology

by Regina A. Pomranky, Keryl Cosenzo, Andrew Bodenhamer,

and Brad Pettijohn

ARL-TR-5257 August 2010

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the Army position unless 

so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official endorsement or approval of the 

use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
Army Research Laboratory

Aberdeen Proving Ground, MD 21005-5425

ARL-TR-5257 August 2010

Experimental Evaluation of Computer-Aided

Tele-operation (CATO) and Computer-Aided Robotic 

Manipulation (CARMAN) Technology

Regina A. Pomranky, Keryl Cosenzo, Andrew Bodenhamer,

and Brad Pettijohn

Human Research and Engineering Directorate, ARL

Approved for public release; distribution is unlimited.
ii

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering 

and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson 

Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to 

comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

August 2010

2. REPORT TYPE

Final

3. DATES COVERED (From - To)

July 2008–November 2008

4. TITLE AND SUBTITLE

Experimental Evaluation of Computer-Aided Tele-operation (CATO) and Computer-Aided 

Robotic Manipulation (CARMAN) Technology

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Regina A. Pomranky, Keryl Cosenzo, Andrew Bodenhamer, and Brad Pettijohn

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Laboratory

ATTN: RDRL-HRM-D

Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION

 REPORT NUMBER

ARL-TR-5257

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT

The objective of this research was to experimentally evaluate the effect of Computer-Aided Robotic Manipulation (CARMAN) and Computer 

Aided Tele-operation (CATO) technologies on manipulator activity and tele-operation, respectively. These technologies were assessed to 

gauge their effectiveness relative to standard operation of a TALON IIIB. The CARMAN experiment consisted of a light board task with 

targets. CARMAN has two technologies that were evaluated in this experiment for manipulator control; Point and Click and Fly-To. These 

technologies were compared to the baseline configuration (ALTON). The Point and Click mode allows the operator to select a point on the 

video from the Operator Control Unit (OCU), which will direct the manipulator arm to move to that general area. The Fly-To mode allows 

the operator precise control over the manipulator arm, via a joystick on the OCU, to direct the end effector to a specific point. Results showed 

that performance with the Fly-To configuration was superior to the other two configurations. The CATO Experiment consisted of a pathfollowing course and an obstacle negotiation course. CATO has three technologies that were evaluated in this experiment for tele-operation, 

Elevated Camera, Projected Path, and Waypoint. These technologies were compared to the baseline configuration (TALON). Elevated 

Camera was best suited for completing the obstacle course both in terms of speed and minimizing collisions. Projected Path was found to be 

best for following a designated path and was also well suited for avoiding collisions with obstacles. Both Elevated Camera and Projected 

Path were found to significantly reduce the reported frustration of operating the SUGV for these tasks.

15. SUBJECT TERMS

TALON, ALTON, CATO, CARMAN, aided tele-operation

16. SECURITY CLASSIFICATION OF: 

17. LIMITATION

OF ABSTRACT

UU

18. NUMBER

OF PAGES

54

19a. NAME OF RESPONSIBLE PERSON

Regina A. Pomranky

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified

19b. TELEPHONE NUMBER (Include area code)

(334) 255-2135

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18
iii

Contents

List of Figures v

List of Tables vi

1. Introduction 1

1.1 Background .....................................................................................................................1

1.2 CARMAN .......................................................................................................................1

1.3 CATO..............................................................................................................................2

1.3.1 Collaborative UAV Driving Camera...................................................................2

1.3.2 Projected Path Display Driving...........................................................................2

1.3.3 Operator-Selected Waypoint Driving..................................................................3

1.3.4 Elevated Rear-Mounted Driving Camera............................................................3

2. Method 3

2.1 Participants......................................................................................................................3

2.2 Apparatus.........................................................................................................................4

2.2.1 Hardware .............................................................................................................4

2.2.2 Screening and Demographic Measures...............................................................6

2.2.3 General Procedures..............................................................................................7

2.3 Experiment I—CARMAN Procedures............................................................................7

2.3.1 Experiment I—Experimental Design ..................................................................9

2.4 Experiment II—CATO Procedures...............................................................................11

2.4.1 Experimental Design .........................................................................................13

3. Results 15

3.1 CARMAN Results.........................................................................................................15

3.2 CATO Results...............................................................................................................20

4. Conclusions 27

4.1 CARMAN .....................................................................................................................27

4.2 CATO............................................................................................................................27

4.3 Summary .......................................................................................................................28
iv

Appendix A. Informed Consent Form 29

Appendix B. Demographics Questionnaire 35

Appendix C. NASA TLX Questionnaire 39

Appendix D. Symptom Questionnaire 41

List of Symbols, Abbreviations, and Acronyms 43

Distribution List 44
v

List of Figures

Figure 1. TALON IIIB and OCU....................................................................................................4

Figure 2. Operator control unit. ......................................................................................................5

Figure 3. Light board task...............................................................................................................6

Figure 4. Manipulator pointing at a target on the light task............................................................8

Figure 5. Participant controlling the manipulator arm through the operator control unit...............9

Figure 6. Sketch of CATO driving course....................................................................................12

Figure 7. TALON on path following course.................................................................................12

Figure 8. ALTON negotiating obstacles.......................................................................................13

Figure 9. Graph of the percentage of inadvertent contacts for CARMAN configurations...........15

Figure 10. Graph of mean (standard error of the mean) missed targets for CARMAN 

configurations; percent change relative to baseline (ALTON) in missed targets is 

represented above each bar. .....................................................................................................16

Figure 11. Graph of mean (standard error of the mean) time per target for CARMAN 

configurations. .........................................................................................................................17

Figure 12. Graph of mean (standard error of the mean) total task time (seconds) for 

CARMAN configurations........................................................................................................18

Figure 13. Graph of mean (standard error of the mean) subjective workload for CARMAN 

configurations. .........................................................................................................................19

Figure 14. Graph of mean (standard error of the mean) path following completion time for 

CATO configurations...............................................................................................................21

Figure 15. Graph of mean (standard error of the mean) path following deviation for CATO 

configurations. .........................................................................................................................21

Figure 16. Graph of mean (standard error of the mean) obstacle course completion time for 

CATO configurations...............................................................................................................22

Figure 17. Graph of mean (standard error of the mean) obstacle collisions for CATO 

configurations. .........................................................................................................................23

Figure 18. Graph of mean (standard error of the mean) power consumption rates for CATO 

configurations. .........................................................................................................................24

Figure 19. Graph of mean (standard error of the mean) subjective workload for CATO 

configurations. .........................................................................................................................25
vi

List of Tables

Table 1. Order of conditions for experiment I. .............................................................................10

Table 2. Order of conditions for experiment II.............................................................................14

Table 3. Table of percentages of workload from baseline (ALTON) by configuration. ..............19

Table 4. Table of percentages of workload from baseline (TALON) by configuration. ..............26
1

1. Introduction

The objective of this research was to experimentally evaluate the effect of Computer-Aided 

Robotic Manipulation (CARMAN) and Computer Aided Tele-operation (CATO) technologies 

on manipulator activity and tele-operation, respectively. These technologies were assessed to 

gauge their effectiveness relative to standard operation of a TALONIIIB. A secondary 

objective was to develop a set of metrics that will test the tele-operated robotic manipulation and 

mobility of the CARMAN and CATO technologies and may have applications to other robotic 

technologies.

1.1 Background

The Battlefield Automation Team (BAT) of the Aviation and Missile Research, Development 

and Engineering Center (AMRDEC) Software Engineering Directorate (SED) has two projects 

entitled CARMAN and CATO. CARMAN involves the development of technologies to reduce

robotic manipulator task times and improve the precision of robotic manipulator placement. 

CATO involves the development of technologies to reduce tele-operated robotic mobility task 

times and improve the precision of tele-operated driving. The first goal for this experiment was 

to operationally and technically measure CARMAN and CATO technology performance.

The unmanned systems community did not have a set of accepted metrics that measured robotic 

performance for these tasks, particularly the manipulator placement task. Currently, measures of 

effectiveness for robotic systems include parameters such as weight, size, packability, usability, 

mean time between failures (MTBF) and mean time to repair (MTTR). Although these measures 

need to be collected and validated to ensure effectiveness of the system, they do not measure 

operational performance. A second goal for this study was to develop a set of robotic metrics 

that are suitable for tele-operated robotic manipulation and mobility, and then to use these 

metrics to evaluate the performance of both CARMAN and CATO.

1.2 CARMAN

Tele-operated unmanned ground vehicle (UGV) manipulators currently deployed in support of 

contingency operations provide limited sensory data to the operator. Visual data relayed to the 

operator fails to provide accurate depth perception and an adequate field of view while tactile 

and auditory data are typically nonexistent. The resulting shortfall in situational awareness 

fundamentally limits manipulator system effectiveness in terms of spatial precision and the time 

required to perform manipulation tasks. Compounding these limitations is an inadequate match 

between machine characteristics and operator capabilities. The Operator Control Units (OCU) 

supplied with current manipulator systems either burden the operator with low-level tasks, such 

 

 TALON is a registered trademark of Foster-Miller, Inc. – QinetiQ North America, Waltham, MA.
2

as individually controlling manipulator joints, or presents an overwhelming set of operating.

modes designed for a multitude of manipulation tasks. In either case, current manipulation 

systems consume an undue amount of an operator’s attention, detracting from mission focus. 

This in turn unnecessarily lengthens task completion and operator exposure time in hostile 

environments and increases the number of assets needed to support a given workload. 

CARMAN has two technologies that were evaluated in this experiment for manipulator control; 

Point and Click and Fly-To. The Point and Click mode allows the operator to select a point on 

the video from the OCU that will direct the manipulator arm to move to that general area. The 

Fly-To mode allows the operator precise control over the manipulator arm via a joystick on the 

OCU, to direct the end effector to a specific point. These modes are in contrast to the current 

manipulator implementation strategy, where the operator actively moves the arm to the location 

via joystick control manipulating each joint movement independently.

1.3 CATO

Soldiers tele-operate small UGVs in the current force by looking at a video display or line of 

sight of the robot and moving a joystick. CATO was developed to bridge the gap between 

current and future UGVs where there will be increased autonomy. There are four technologies

for CATO; some are only software, some only hardware, and some a combination of both. The 

technologies that were developed were collaborative unmanned aerial vehicle (UAV) driving 

camera, projected path display driving, short distance operator selected waypoint driving, and 

elevated rear mounted driving camera. 

1.3.1 Collaborative UAV Driving Camera

The collaborative UAV driving camera is teamed with a UAV and the ground vehicle. The 

UAV camera will give an overhead visual perspective to the operator in conjunction with the 

regular UGV driving camera(s). This view is also intended to increase the view of the area 

surrounding the UGV, increasing the operator’s situational awareness. The UAV is positioned 

collaboratively above and slightly behind the UGV. It is commanded by the operator to position 

itself based on the location and heading of the UGV. This eliminates the need for the operator to 

command the UAV. The notion is that the UAV driving camera would be added as an additional 

camera selection of the UGV so the operator can select it or any of the other UGV cameras as 

necessary to display current video. Due to technical challenges with the UAV driving camera 

configuration, this CATO technology was not included in this experiment. 

1.3.2 Projected Path Display Driving

The projected path display driving uses the normal UGV driving camera to project a virtual path

on the operator control display. The semi-transparent path is projected in front of the actual 

UGV. The projected path indicates where the UGV will travel if the control input remains 

constant. This technology was largely a software solution. 
3

1.3.3 Operator-Selected Waypoint Driving

The short distance operator-selected waypoint driving provides the operator the ability to 

visually identify and then select waypoints on the OCU display using existing UGV cameras. 

The operator views the display and identifies desired waypoints using an input device. The UGV 

follows the waypoints at a selected speed and automatically stops at the last specified waypoint. 

Waypoint following allows the vehicle to move forward in the battle without continuous operator 

interaction with the robot. As a result the operator can move his position or fight while using a 

robot. 

1.3.4 Elevated Rear-Mounted Driving Camera

The elevated rear-mounted driving camera is a modification in which the driving camera is

behind and above the UGV. This perspective gives a third person view of the UGV and

increases the view of the surrounding area. This is primarily a hardware solution and required

minimal modification to existing systems. 

The three available CATO technologies were assessed to gauge their effectiveness relative to 

standard tele-operation of a small unmanned ground vehicle (SUGV). CARMAN and CATO 

were implemented on the TALON IIIB platform and OCU.

This report documents two independent experiments. The first experiment tested the CARMAN 

technology and was conducted using a light board and the ALTON (a modified TALON IIIB)

platform with and without Point and Click and Fly-To technologies. The second experiment was

conducted on an obstacle course that was run using the TALON IIIB platform, with and without 

the CATO Projected Path Display Driving, Operator Selected Waypoint Driving, and Elevated 

Rear Driving Camera technologies. 

2. Method

2.1 Participants

Participants were 12 civilian men and women ranging in age from 18 to 25 years old, recruited 

from the AMRDEC workforce. Participants had roughly the same level of experience with 

operation of unmanned systems, which was determined from recruiting techniques and verified 

by the demographics questionnaire. Participants completed an Informed Consent form (appendix

A) and a Demographics Questionnaire (appendix B) prior to the beginning the experiment. 
4

2.2 Apparatus

2.2.1 Hardware

2.2.1.1 The TALON IIIB

The TALON IIIB (figure 1) is a tracked UGV that is widely used for explosive ordnance 

disposal (EOD), reconnaissance, communications, hazmat, security, defense and rescue 

applications. The TALON weighs 115 lb and has a top speed of 5.2 mph. The manipulator arm 

has three degrees of freedom and cannot rotate independently of the robot body. The TALON

IIIB is the primary robot used for improvised explosive device (IED) interrogation by the U.S.

Army Engineers.

Figure 1. TALON IIIB and OCU.

2.2.1.2 ALTON 

The ALTON robot is a customized version of the TALON IIIB robot that has been upgraded to 

improve sensor accuracy and data rates. In order to realize these improvements in performance, 

it was necessary to upgrade the internal electronics and software in both the robot and the 

operator control unit. The resulting increase in sensor accuracy and data rates allows smoother,

more controlled motion of the platform and its manipulator. Although ALTON has new internal 

electronics, the mechanical chassis and motors are identical to those of a TALON IIIB.

2.2.1.3 OCU

The OCU, shown in figure 2, is an encased control unit with a display and joysticks that control 

driving, manipulator arm, and payloads (i.e., cameras). The OCU used in these experiments 

physically mirrors the external appearance of the Foster Miller–TALON IIIB OCU with the 
5

exception of a second two-axis controller and an extra mode Miller–TALON IIIB OCU with the 

exception of a second two-axis controller and an extra mode switch button on the right hand side 

of the OCU face. Internally, the OCU has an onboard computer and custom electronics to 

accommodate the interface. The modifications to the TALON IIIB OCU did not affect how the 

operator interacted with the baseline TALON IIIB capabilities. The modified TALON IIIB OCU 

with or without the CARMAN and CATO technologies is referred to as ALTON in the 

remainder of this document.

Figure 2. Operator control unit.

2.2.1.4 GPS Data Logger

The data logger is a custom data acquisition system that recorded GPS position data for the UGV 

under test. The data logger uses a Novatel GPS receiver in RT2 mode with an accuracy of 2 cm. 

The data logger is self-contained and powered by its own battery. It is contained in an aluminum 

enclosure with dimensions (in inches) 5.125×6.25×3.125, which allows it to be mounted to the 

UGV with minimal impact on weight and volume.

2.2.1.5 Light Board

A 3×4 ft rear-projection light board was used for the CARMAN tasks (see figure 3). The MagicTouch light board is a positional measurement tool designed to report the X and Y position of an 

object penetrating the area enclosed in the frame. The light board is able to determine the 

position of penetration by using an overlapping array of Infrared LEDs. Once an object breaks 

the beam it is recorded on a computer. Three-dimensional (3-D) perception is known to be hard 

for the robotic operator using the standard TALON vision system, especially for novice 

operators. In order to achieve ―clean‖ results, a standard vision system was used to minimize the 

variable of assessing depth, hopefully to reduce the variability that would be induced by relying 

on the operator's ability to perceive depth from cues that are irrelevant to the technologies being 

analyzed. Two operationally relevant actions were assessed in the touch task: (1) Pan/Tilt of the 
6

arm to achieve ―bore-sight‖ alignment of the gripper with the target and, (2) Maintaining steady

trajectory as the gripper approaches the target. These two steps (along with the use of shadows 

to assess depth) are generally how the robotic manipulator is used in real environments. 

Additionally the operator could push through the screen because it is paper, thus the paper can 

easily be repaired/replaced. By using the Magic-Touch light board the sensors registered a ―hit‖

before the pointer touched the paper. 

Figure 3. Light board task.

2.2.2 Screening and Demographic Measures

2.2.2.1 Demographic and Computer Experience Questionnaire

The Demographic and Computer Experience Questionnaire (appendix B) is a 12-item 

questionnaire that requests information regarding age, vision, hearing, and computer experience. 

This questionnaire was used to gain basic demographic information about the participant sample. 

2.2.2.2 The NASA-Task Load Index (NASA-TLX)

Participants were given the NASA-TLX subjective workload rating at the end of each task

(appendix C, Hart and Staveland, 1987).1 The NASA-TLX is a multi-dimensional rating 

procedure that derives an overall workload score based on a weighted average of ratings on six 

subscales (Mental Demand, Physical Demand, Temporal Demand, Own Performance, Effort, 

and Frustration). Scores range from 0 (no workload) to 100 (extremely high workload).

2.2.2.3 Motion Symptom Questionnaire

This questionnaire provides a subjective rating on the participant’s perceived levels of motion 

sickness before and after the experiment (appendix D, Gianaros, Muth, Mordkoff, Levine, and 

 

1Hart, S. G.; Staveland, L. E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical 

Research. In P.A. Hancock and N. Meshkati Eds., Human Mental Workload. Amsterdam: Elsevier, 1987.
7

Stern, 2001).2 A comparison of the pre- and post-experimental response patterns across the 

questionnaire was conducted to evaluate motion sickness. Any subjects showing elevated scores 

on any of the items was held at the research site until their symptoms abated. No motion 

sickness was reported during the experiments.

2.2.3 General Procedures

Prior to the start of the experiment, the experimenter briefed the participants on the purpose and 

procedures of the experiment. Participants who agreed to take part in the study signed the

Volunteer Agreement Affidavit (appendix A) and were given the required briefing regarding 

confidentiality as indicated on DA Form 5303-R. In anticipation of possible concerns regarding 

personal answers on some of the questionnaires, the investigators also described the deliberate 

actions taken when handling research data. In order to ensure that individual data was not 

reported or revealed to anyone, each form was reviewed upon receipt by one of the investigators. 

If any identifying information appeared on the questionnaires (such as name, social security 

number, birth date, etc.), the investigators deleted the identifying information and replaced it 

with a neutral code number. This code number became the participant identification number 

used in data files.

Participants were told that they would complete two experiments in an 8 h block, over the course 

of one day. In experiment I, participants manipulated the ALTON manipulator arm with and 

without the CARMAN Point and Click and Fly-To technologies. In experiment II, the 

participant tele-operated the TALON IIIB through an outdoor obstacle course with and without 

the CATO Elevated Rear-Mounted Driving Camera Mode, Projected Path Display Driving 

Mode, and Operator-Selected Waypoint Driving Mode technologies. There was a 30 min break 

between each experiment and 45 min given for lunch. One-half of the participants completed 

experiment I and then experiment II. The other half completed experiment II first followed by 

experiment I.

2.3 Experiment I—CARMAN Procedures

After a description of the purpose, participants were given an overview of and training on the 

baseline configuration of the CARMAN configurations. The experimenter presented the 

functionality of the CARMAN OCU. The participant completed a training trial. They had the 

opportunity to repeat the training run again until they were comfortable controlling the robot. 

Criteria for efficient training were determined by a subject matter expert on the platform who 

was present during the training mission. These subject matter experts were the engineers of the 

CARMAN and CATO software and they used their knowledge of the system to assess training 

effectiveness. The objective of the CARMAN task was to manipulate the manipulator arm to hit 

20 targets that were displayed on the rear-projection light board. The participants were 

 

2Gianaros, P. J.; Muth, E. R.; Mordkoff, J. T.; Levine, M. E.; Stern, R. M. A Questionnaire For the Assessment of the 

Multiple Dimensions of Motion Sickness. Aviation Space Environmental Medicine 2001 72 (2), 115–9.
8

instructed to perform this task as quickly and accurately as possible. Figure 4 shows the 

manipulator pointing at a target on the light board.

Figure 4. Manipulator pointing at a target on the light board task.

All targets were the same size bull’s-eye-shaped figures that were projected in three random 

patterns on the light board. Each pattern contained 20 targets that were evenly distributed 

throughout the target board, meaning that in each pattern, special care was taken to ensure that 

equal numbers of quadrants were used and that the space between targets was similar. 

Participants completed two trials per configuration, baseline configuration (i.e., ALTON), 

CARMAN Point and Click, and Fly-To configuration. Participants were instructed to use the 

manipulator arm to touch 20 targets that appeared in succession on the light board as quickly and 

accurately as possible. Each target appeared and stayed lit until the participant touched the 

manipulator end point somewhere in the target or the time for a target hit ran out (20 s). If the 

manipulator arm touched some place other than inside the target, it was considered an 

inadvertent contact. Accuracy was measured by distance from the center of the bull’s-eye. Once 

a hit occurred the target disappeared and the next target appeared. The participant continued

until all 20 targets were hit. Each participant completed six trials using the ALTON OCU (see 

figure 5).
9

Figure 5. Participant controlling the manipulator arm through the operator 

control unit.

Participants completed the task with only visual sightings of the target board through the OCU. 

A tarp separated the participant from the TALON platform rendering the task as non-line-ofsight. Additionally, participants could not see the entire target board through the OCU. As a 

result, participants often had to back up the manipulator arm to get a wider view of the target 

board, which allowed them to identify the target area. Speed and accuracy were stressed. 

Subject matter experts observed the trials for technical difficulties. After each CARMAN task 

trial, the participants completed the NASA-TLX and were given the opportunity to take a thirtyminute break before beginning the next experiment. 

2.3.1 Experiment I—Experimental Design

The experimental design of experiment I was a 3×2 within subjects design. There were two

independent variables, configuration type with three levels (baseline [i.e., ALTON], Point and 

Click, and Fly-To) and trial with two levels (trial 1 and trial 2). Thus, the participant completed

two trials in the baseline configuration, two in Point and Click, and the two in the Fly-To 

configuration. In all, each participant completed six trials. In each trial there were 20 targets to 

touch with the manipulator on the light board. Task completion time for experiment I was 

approximately 2 h.
10

Configuration Type: 

• A=Baseline (ALTON)

• B=CARMAN with Point and Click 

• C=CARMAN with Fly-To 

Repetition:

• 1 and 2

Target Order:

• (1), (2), (3)

Order of the conditions was counterbalanced using a Williams Square Design (see table 1).

Table 1. Order of conditions for experiment I.

Participant Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6

1 A1(2) B1(3) C1(1) A2(3) B2(1) C2(2)

2 B1(1) C1(2) A1(3) B2(2) C2(3) A2(1)

3 C1(3) A1(1) B1(2) C2(1) A2(2) B2(3)

4 A1(1) C1(2) B1(3) A2(2) C2(3) B2(1)

5 B1(2) A1(3) C1(1) B2(3) A2(1) C2(2)

6 C1(3) B1(1) A1(2) C2(1) B2(2) A2(3)

7 A1(1) B1(2) C1(3) A2(2) B2(3) C2(1)

8 B1(2) C1(3) A1(1) B2(3) C2(1) A2(2)

9 C1(3) A1(1) B1(2) C2(1) A2(2) B2(3)

10 A1(1) C1(2) B1(3) A2(2) C2(3) B2(1)

11 B1(2) A1(3) C1(1) B2(3) A2(1) C2(2)

12 C1(3) B1(1) A1(2) C2(1) B2(2) A2(3)

Dependent Variables:

The following dependent variables were measured by automatic data collection and survey 

instruments during experiment I.

Time to complete (milliseconds) (digitally recorded with embedded automatic data collection 

software):

• Average Time per Trial

• Average Time per Target

Precision and Errors frequency (digitally recorded with embedded automatic data collection 

software):

• Inadvertent Contacts

• Missed Targets (Timed out)
11

Subjective Responses (surveys): 

• Workload was measured with the NASA-TLX. 

Analyses for experiment I:

• Means and standard errors were calculated for each dependent variable. 

To examine the effects of CARMAN configuration type on operator performance (e.g., time, 

precision), a mixed linear model analysis was conducted. Subsequent pairwise comparisons 

were conducted for the significant dependent variable(s). Effects showing significance of 

p<0.05 were considered statistically significant.

To examine the effects of CARMAN configuration type on workload, a MANOVA was

conducted with scores on the six subscales of the NASA-TLX (Mental Demand, Physical 

Demand, Temporal Demand, Own Performance, Effort, and Frustration) as the dependent 

variables. Effects showing significance of p≤0.05 were considered statistically significant. 

Subsequent repeated measures ANOVAs were conducted for the significant dependent 

variable(s).

2.4 Experiment II—CATO Procedures

After a description of the purpose, participants were given an overview of the CATO and 

training on the baseline CATO configuration. The experimenter presented the functionality of 

the OCU for CATO. The participant completed a training trial. Criteria for efficient training 

were determined by a subject matter expert on the platform who was present during the training 

mission. These subject matter experts were the engineers of the CARMAN and CATO software 

and used their knowledge of the system to assess training effectiveness. After training, the 

participant completed the CATO tasks, the objective was to maneuver the unmanned system 

through the obstacle course.

The obstacle course consisted of two segments: path following and obstacle negotiation. The 

path following segment consisted of following a painted orange line through seven corners and 

curves. The obstacle segment consisted of traveling through four gates (2 widths×2 approach 

orientations), a ramp, a narrow passageway, a tightly-spaced five cone slalom, a gate on a curve, 

and a four cone slalom (a total of 16 obstacles). For each trial, the operator navigated the course 

in one continuous run, with individual segment times being recorded by stopwatch. Collisions 

with obstacles were noted by a data collector. All tele-operation was conducted ―non-line-ofsight‖ from a nearby climate controlled shelter; this means that at no time was the participant 

able to directly view the robot. Participants were instructed to stress accuracy (precision path 

following and avoiding obstacle collisions), but to also drive as quickly as they felt they could 

while remaining accurate in their maneuvers. Subject matter experts observed the operators’ use

of the equipment during the trials for usability issues. After each CATO task trial, the participant 
12

completed the NASA-TLX and was given the opportunity to take a 30-min break before 

beginning the next experiment. The driving course is shown in figures 6–8.

Figure 6. Sketch of CATO driving course.

Figure 7. TALON on path following course.
13

Figure 8. ALTON negotiating obstacles.

2.4.1 Experimental Design

The experimental design of experiment II was a 4×2 within subjects design. There were two

independent variables, configuration type with four levels (baseline [i.e., TALON], Elevated 

Rear Mounted Driving Camera, Projected Path Display Driving, and Operator Selected 

Waypoint Driving) and repetition with two levels (trial 1 and trial 2). Thus, the participant

completed two trials in the baseline configuration, two in the CATO Elevated Rear Mounted 

Driving Camera configuration, two in the CATO Projected Path Display Driving configuration, 

and two in the CATO Operator Selected Waypoint Driving configuration. In all, each participant 

completed eight trials. The obstacle course remained in a static configuration, trials alternated 

running the course from ―start to finish‖ and from ―finish to start‖ to maintain consistent course 

length but reduce learning effect. Task completion time for experiment II was approximately 

3 h.

Configuration Type: 

• A=Baseline (ALTON)

• B=CATO with Elevated Rear Mounted Driving Camera

• C=CATO with Projected Path Display Driving 

• D=CATO with Operator Selected Waypoint Driving 

Trial:

• 1 and 2
14

Order of the conditions was counterbalanced using a Williams Square Design as shown in table

2.

Table 2. Order of conditions for experiment II.

Participant Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Trial 7 Trial 8

1 A1 D1 B1 C1 A2 D2 B2 C2

2 B1 A1 C1 D1 B2 A2 C2 D2

3 C1 B1 D1 A1 C2 B2 D2 A2

4 D1 C1 A1 B1 D2 C2 A2 B2

5 C1 B1 D1 A1 C2 B2 D2 A2

6 D1 C1 A1 B1 D2 C2 A2 B2

7 A1 D1 B1 C1 A2 D2 B2 C2

8 B1 A1 C1 D1 B2 A2 C2 D2

9 A1 B1 C1 D1 A2 B2 C2 D2

10 B1 D1 A1 C1 B2 D2 A2 C2

11 C1 A1 D1 B1 C2 A2 D2 B2

12 D1 C1 B1 A1 D2 C2 B2 A2

Dependent Variables:

The following dependent variables were measured during experiment II.

Time in seconds (recorded by the U.S. Army Research Laboratory [ARL] data collector with 

stopwatch):

• Time to complete path following segment

• Time to complete obstacle segment

Precision and Errors:

• Path following precision (measured by GPS data logger)

• Number of obstacle collisions (measured by ARL data collector)

Power consumption used by the Robot (recorded by ARL data collector using in-line power 

consumption meter)

Subjective Responses (surveys): 

• Workload was measured with the NASA-TLX. 

Analyses for experiment II:

• Means and standard errors were calculated for each dependent variable. 

Path following precision was evaluated by calculating the area of deviation between the path 

taken and the true course. The output is a deviation score in pixels determined by the program 

that calculates the area between the lines. This method was chosen for its simplicity and 
15

insensitivity to path following time. To examine the effects of CATO configuration type on 

operator performance (e.g., time, precision), a mixed linear model analysis was conducted.

Subsequent pairwise comparisons were conducted for the significant dependent variable(s). 

Effects showing significance of p≤0.05 was considered statistically significant.

To examine the effects of CATO configuration type on workload, a MANOVA was conducted 

with scores on the six subscales of the NASA-TLX (Mental Demand, Physical Demand, 

Temporal Demand, Own Performance, Effort, and Frustration) as the dependent variables. 

Effects showing significance of p≤0.05 was considered statistically significant. Subsequent 

repeated measures ANOVAs were conducted for the significant dependent variable(s).

3. Results

3.1 CARMAN Results 

To examine the effects of the three CARMAN configurations (ALTON, Point and Click, and 

Fly-To) and Repetition (1 and 2) on performance in the light board task, repeated measures 

ANOVAs and subsequent pairwise comparisons were conducted.

Figure 9 presents a graph of inadvertent contacts on the light board task. Results for the light 

board task showed that the number of inadvertent contacts was highest in the ALTON

configuration. There was an 80% and 43% average reduction in inadvertent contacts with the

Fly-To and Point and the Click configurations, respectively.

Note: * = a significant difference.

Figure 9. Graph of the percentage of inadvertent contacts for CARMAN configurations. 

0

0.2

0.4

0.6

0.8

1

1.2

1.4

(Baseline) ALTON Fly-To Point and Click

Inadvertent Contacts

Configuration

-80%*

-43%*

0% 
16

A repeated measures ANOVA revealed a significant main effect of CARMAN configuration for 

inadvertent contacts, F (2,22) = 18.01, p<0.01. There was no difference in missed targets 

between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) = 0.04, p<0.83 

and F (2,22) = 0.44, p<0.64, respectively. To explain the main effect of configuration on missed 

targets, pairwise comparisons were conducted. Results showed that significantly more targets 

were missed in the ALTON than the Point and Click or Fly-To configurations, p<0.01 for each. 

Furthermore, more targets were missed in the Point and Click than the Fly-To configuration, 

p<0.01. 

Figure 10 presents a graph of Mean (Standard Error of the Mean [error bars]) Missed Targets for 

CARMAN configurations on the light board task. Results for the light board task showed that 

the number of misses was higher in the Point and Click configuration than in the Baseline 

(ALTON) configuration. The number of misses in the Fly-To configuration was less than a third 

of the number of misses in the Baseline (ALTON) configuration. 

Note: * = p<0.01. 

Figure 10. Graph of mean (standard error of the mean) missed targets for CARMAN 

configurations; percent change relative to baseline (ALTON) in missed targets 

is represented above each bar. 

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Baseline (ALTON) Fly-To Point & Click

Missed Targets

Configuration

0% 

-87.5%*

8.3% 
17

A repeated measures ANOVA revealed a significant main effect of CARMAN configuration for 

number of missed targets (out of twenty), F (2, 22) = 4.35, p<0.02. There was no difference in 

missed targets between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) =

3.98, p<0.07 and F (2,22) = 2.46, p<0.10, respectively. To explain the main effect of 

configuration on missed targets, pairwise comparisons were conducted. Results showed that 

more targets were missed in the Point and Click than the Fly-To configurations, p<0.01. No 

other comparisons were statistically significant. Although there was a large difference between 

the Fly-To configuration and Baseline, this difference was not statistically significant. It is 

important to note that an 87.5% reduction in missed targets is notable and has practical 

significance.

Figure 11 presents a graph of mean task completion time (seconds) on the light board task. 

Results for the light board task showed that task time was shortest in the Fly-To mode. 

Furthermore, task time was slightly shorter in the ALTON configuration than the Point and the 

Click configuration. 

Note: * = p <0.01.

Figure 11. Graph of mean (standard error of the mean) time per target for CARMAN configurations. 

A repeated measures ANOVA revealed a significant main effect of CARMAN configuration for 

task completion time, F (2,22) = 93.79, p<0. 01. There was no significant difference in task 

completion time between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) 

= 2.96, p<0.10 and F (2,22) = 0.448, p<0.64, respectively. To explain the main effect of 

configuration on task time, pairwise comparisons were conducted. Results showed that task time 

was significantly shorter in the Fly-To configuration than the Point and Click or the ALTON 

configuration, p<0.01 for each. Task time was significantly shorter in the ALTON than in the 

Point and Click configuration, p<0.01. 

0

10

20

30

40

50

60

70

ALTON Fly-To Point and Click

Time per Target (secs)

Configuration

-35%*

35% *

0% 
18

Figure 12 presents a graph of mean total time to complete the light board task. Results for the 

light board task showed that the Point and Click configuration took the longest to complete. The 

Fly-To configuration had the shortest task completion time. 

Note: * = p<0.01.

Figure 12. Graph of mean (standard error of the mean) total task time (seconds) for 

CARMAN configurations.

A repeated measures ANOVA revealed a significant main effect of CARMAN configuration for 

total task time, F (2, 22) = 92.43, p<0.01. There was no difference in time between trial 1 and 

trial 2 or an interaction of trial with configuration, F (1,11) = 3.52, p<0.08 and F (2,22) = 0.37, 

p<0.69, respectively. To explain the main effect of configuration on total task time, pairwise 

comparisons were conducted. Results showed that task time was significantly longer in the 

ALTON and Point and Click configurations than the Fly-To configuration, p<0.01 for each. 

Furthermore, task time was significantly longer in the Point and Click than the ALTON, p<0.01. 

To examine the effects of the three CARMAN configurations (ALTON, Point and Click, and 

Fly-To) on subjective workload, a Multivariate Analysis of Variance (MANOVA) was 

conducted. Subsequent ANOVAs and pairwise comparisons were conducted for the significant 

workload subscales.

0

200

400

600

800

1000

1200

1400

ALTON Fly-To Point and Click

Total Task Time (sec)

Configuration

-35% *

35% *

0% 
19

Figure 13 presents a graph of mean workload across the CARMAN configurations. Results 

showed that subjective workload, specifically mental demand, physical demand, temporal 

demand, and frustration, were highest in the ALTON. Furthermore, workload on those subscales 

was lowest in the Fly-To configuration (also shown by percentages in table 3). 

Figure 13. Graph of mean (standard error of the mean) subjective workload for CARMAN 

configurations.

Table 3. Table of percentages of workload from baseline (ALTON) by configuration.

a

Significant comparison between configuration and baseline (ALTON), p<0.05.

A repeated measures MANOVA revealed a significant main effect of CARMAN configuration 

on workload, F (12, 32) = 2.92, p<.01. There was no difference in workload between trial 1 and 

trial 2 or an interaction of trial with configuration, F (6,5) = 1.10, p = 0.46 and F (12,32) = 1.04, 

p = 0.43, respectively. Subsequent ANOVAs were conducted to determine which subscales of 

0

10

20

30

40

50

60

70

80

90

100

Mental Physical Temporal Performance Effort Frustration

Workload

Score

ALTON

Fly-To

Point and Click

Mental Physical Temporal Performance Effort Frustration Overall 

Workload

Baseline 

(ALTON)

0% 0% 0% 0% 0% 0% 0%

Fly-To –29.6%

a

–39.8 a–22.0%

a

–26.0% –5.2% –47.1%

a

–25.1%

a

Point and Click –29.6%

a

–38.5%

a

–11.8% –9.8% –3.5% –15.7%

a

–16.2%

a
20

the NASA-TLX workload scale contributed to this main effect. There were significant main 

effects for mental demand, physical demand, and frustration, F (2,20) = 9.65, p<0.01, F (2,20) =

7.53, p<0.01, and F (2,20) = 5.37, p<0.01, respectively. Temporal demand was marginally 

significant, F (2,20) = 3.24, p<0.06. No other subscales were significant. To explain the main 

effect of configuration on the significant subscales, pairwise comparisons were conducted. 

Results showed that mental demand was significantly higher in ALTON than Fly-To or Point 

and Click configurations. Physical demand was significantly higher in the ALTON than the FlyTo or Point and Click configurations, p<0.02 and p<0.01, respectively. Temporal demand was 

significantly higher in the ALTON configuration than the Fly-To configuration, p<0.01. 

Frustration was significantly higher in the ALTON and Point and Click configurations than the 

Fly-To configuration, p<0.03 and p<0.01, respectively. Although performance was not 

statistically significant, the differences between configurations were practically significant. 

Perceived performance was highest in Fly-To than in all other configurations.

A second repeated measures ANOVA was conducted for overall workload. There was 

significant main effect of CARMAN configurations for overall workload, F (2,20) = 7.65, 

p <0.01. There was no difference in workload between trial 1 and trial 2 or an interaction of trial 

with configuration, F (1,10) = 2.63, p <0.13 and F (2,20) = 0.09, p = 0.91, respectively. To 

explain the main effect of configuration on overall workload, pairwise comparisons were 

conducted. Results showed that overall workload was significantly higher in the ALTON than 

the Fly-To or Point and Click configurations, p<0.01. 

3.2 CATO Results 

Repeated measures ANOVA and pairwise comparisons were used to analyze the performance 

differences of the four CATO configurations (Baseline (TALON)), Projected Path, Elevated 

Camera, and Waypoint)

Figure 14 presents the mean path following time (seconds) for the four CATO configurations. 

The percent change from Baseline (TALON) is listed over each bar. Significant differences 

(p’s≤0.05) from Baseline are marked with an asterisk. For the path following course, Baseline 

was the quickest while Waypoint mode was the slowest.

A mixed linear model analysis revealed a significant main effect of CATO configuration for path 

following time, F (3,33) = 24.02, p<0.01. There was a significant effect in path following time 

between trial 1 and trial 2, F (1,11) = 5.11, p<0.05, but there was no interaction of trial with 

configuration, F (3,33) = 0.99, p<0.40. To explain the main effect of configuration on path 

following time, pairwise comparisons were conducted. Results showed that significantly more 

time was needed to complete the path following portion for the Waypoint configuration than the 

other three configurations, p<0.01 for each. There were no significant differences between the 

other three modes. 
21

Note: * = a significant difference.

Figure 14. Graph of mean (standard error of the mean) path following completion time for 

CATO configurations. 

Figure 15 presents the mean path deviation (pixels) for the four CATO configurations. The 

percent change from Baseline (TALON) is listed over each bar. Significant differences (p<0.05) 

from Baseline are marked with an asterisk. Projected Path was the most precise at path 

following and Waypoint was the least precise.

Note: * = a significant difference.

Figure 15. Graph of mean (standard error of the mean) path following deviation 

for CATO configurations. 

0

50

100

150

200

250

Baseline (TALON) Elevated Camera Projected Path Waypoint

Mode

Time (s)

3.6%

0%

9.0%

93.7%*

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Baseline (TALON) Elevated Camera Projected Path Waypoint

Deviation (pixels)

Mode

0% 10.8%

-35.0%*

108.3%*
22

A repeated measures ANOVA revealed a significant main effect of CATO configuration for path 

following deviation, F (3,33) = 9.81, p<0.01. There was no significant effect in path following 

deviation between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) = 0.44, 

p<0.52 and F (3,33) = 1.28, p<0.29, respectively. To explain the main effect of configuration on 

path following deviation, pairwise comparisons were conducted. Results showed that Projected 

Path configuration was significantly more precise than the other three configurations, p’s<0.01. 

Results also showed that the Waypoint configuration was significantly less precise than the other 

three configurations, p’s<0.03. There was no significant difference between the Baseline and 

Elevated Camera configuration.

Figure 16 presents the mean obstacle course completion time (seconds) for the four CATO 

configurations. The percent change from Baseline (TALON) is listed over each bar. Significant 

differences (p’s<0.05) from Baseline are marked with an asterisk. Elevated Camera was the 

fastest mode for this course and Waypoint was the slowest.

Note: * = a significant difference.

Figure 16. Graph of mean (standard error of the mean) obstacle course completion time for CATO configurations. 

0

100

200

300

400

500

600

700

800

Baseline (TALON) Elevated Camera Projected Path Waypoint

Time (s)

Mode

0%

-18.0%* -10.9%

189.4%*
23

A repeated measures ANOVA revealed a significant main effect of CATO configuration for 

obstacle course time, F (3,33) = 52.15, p<0.01. There was no significant effect in obstacle 

course time between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) 

= 4.33, p<0.062 and F (3,33) = 0.95, p<0.42, respectively. To explain the main effect of 

configuration on obstacle course time, pairwise comparisons were conducted. Results showed 

that Elevated Camera configuration was significantly faster than the Baseline configuration, 

p<0.02. Results also showed that significantly more time was needed to complete the obstacle 

course for the Waypoint configuration than the other three configurations, p’s<0.01. There were 

no significant differences between Projected Path and Elevated Camera or between Projected 

Path and Baseline for obstacle course completion time. 

Figure 17 presents the mean number of obstacle collisions for the four CATO configurations. 

The percent change from Baseline (TALON) is listed over each bar. Significant differences 

(p’s<0.05) from Baseline are marked with an asterisk. Elevated Camera resulted in the least 

number of collisions and Waypoint resulted in the most.

Note: * = a significant difference.

Figure 17. Graph of mean (standard error of the mean) obstacle collisions 

for CATO configurations. 

A repeated measures ANOVA revealed a significant main effect of CATO configuration for 

obstacle collisions, F (3,33) = 8.07, p<0.01. There was no significant effect in collisions 

between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) = 1.96, p<0.19 

and F (3,33) = 1.79, p<0.42, respectively. To explain the main effect of configuration on path 

following time, pairwise comparisons were conducted. Results showed that Elevated Camera 

and Projected Path configurations yielded significantly less collisions than the Baseline 

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Baseline (TALON) Elevated Camera Projected Path Waypoint

Collisions

Mode

0%

-55.1%*

-43.7%*

14.2%
24

configuration, p<0.01 and p<0.05, respectively. Results also showed that significantly more 

collisions resulted from the Waypoint configuration than the Elevated Camera or Projected Path 

configurations, p’s<0.01. There was no significant difference between the Baseline and 

Waypoint modes or between Elevated Camera and Projected Path for number of collisions.

Figure 18 presents the mean power usage rate for the four CATO configurations. The percent 

change from Baseline (TALON) is listed over each bar. Significant differences (p’s<0.05) from 

Baseline are marked with an asterisk. Despite taking longer to complete both the path following 

and obstacle sections of the course, Waypoint uses the least power per minute of operation. 

Elevated Camera was found to use the most power per minute of operation.

Note: * = a significant difference.

Figure 18. Graph of mean (standard error of the mean) power consumption rates for CATO 

configurations. 

A repeated measures ANOVA revealed a significant main effect of CATO configuration for rate 

of power consumption, F (3,33) = 16.32, p<0.01. There was no significant effect in collisions 

between trial 1 and trial 2 or an interaction of trial with configuration, F (1,11) = 3.33, p<0.09 

and F (3,33) = 2.10, p<0.12, respectively. To explain the main effect of configuration on path 

following time, pairwise comparisons were conducted. Results showed that the Waypoint 

configuration yielded significantly less rate of power consumption than the other three 

configurations, p’s<0.03. Results also showed that significantly less power per minute was used

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Baseline (TALON) Elevated Camera Projected Path Waypoint

Power Usage Rate (mAh/min)

Mode

0%

26.6%* 25.2%*

-11.5%*
25

by the Baseline configuration than the Elevated Camera and Projected Path, p<0.01 and p<0.02,

respectively. There was no significant difference between the Elevated Camera and Projected 

Path modes for the rate of power consumption. 

Power Consumption is not correlated with any individual metric. However there are some 

interesting observations. First, although overall completion time is similar for Baseline, Elevated 

Camera, and Projected Path, the Baseline mode uses significantly less power. Second, while 

Elevated Camera and Projected Path modes perform better in terms of path precision and 

avoiding collisions than Baseline, the cost is higher power consumption.

A MANOVA was conducted to examine the effects of the four CATO configurations (BaselineTALON, Elevated Camera, Projected Path, and Waypoint) on subjective workload. Subsequent 

ANOVAs and pairwise comparisons were conducted for the significant workload subscales.

Figure 19 presents a graph of mean workload across the CATO configurations. Results showed 

that subjective workloads were higher in the Waypoint and TALON configurations than the 

Elevated Camera and Projected Path for all six subscales. Table 4 lists the percent change in 

workload from Baseline for the three CATO modes. 

Figure 19. Graph of mean (standard error of the mean) subjective workload for 

CATO configurations.
26

Table 4. Table of percentages of workload from baseline (TALON) by configuration.

a

Significant comparison between configuration and baseline (TALON), p<0.05.

A repeated measures MANOVA revealed a main effect of CATO configuration on workload, 

Wilks λ (21, 69) = 2.31, p<0.01. There was a significant difference in workload between trial 1 

and trial 2, F (7,4) = 6.86, p<0.041. There was not an interaction of trial with configuration,

F (21,69) = 1.06, p<0.41. Subsequent ANOVAs were conducted to determine which subscales 

of the NASA-TLX workload scale contributed to this main effect. There were significant main 

effects of CATO configuration for all six subscales—Mental: F (3,30) = 8.05, p<0.01; Physical: 

F (3,30) = 6.44, p<0.01; Temporal: F (3,30) = 4.47, p<0.01; Performance: F (3,30) = 7.76, 

p<0.01; Effort: F (3,30) = 5.65, p<0.01; Fatigue: F (3,30) = 10.38, p<0.01. To explain the main 

effect of configuration on the significant subscales, pairwise comparisons were conducted. 

Results showed that there were no significant differences between Elevated Camera and 

Projected Path for any of the subscale ratings. There were no significant differences between 

TALON and Waypoint, except for a significantly higher mental demand for Waypoint 

configuration, p<0.02. Waypoint yielded significantly higher workload than both Elevated 

Camera and Projected Path for all six subscales, p’s <0.01. TALON had significantly higher 

workload than Elevated Camera for performance and frustration, p<0.05 and p<0.01, 

respectively. TALON configuration had significantly higher workload than Projected Path for 

physical and frustration.

A second repeated measures ANOVA was conducted for overall workload. There was a 

significant main effect between CATO configurations for overall workload, F (3,30) = 9.86, 

p<0.01. To explain the main effect of configuration on the overall workload, pairwise 

comparisons were conducted. Waypoint had significantly greater overall workload than the 

other three configurations, p’s<0.01. There were no other significant differences between the 

configurations for total workload.

Mental Physical Temporal Performance Effort Frustration Overall

Baseline (TALON) 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

Elevated Camera –10.7% –23.5% –22.1% –22.3%a

–18.9% –35.2%a–19.3%

Projected Path –15.3% –29.5%a

–12.2% –22.6% –11.6% –38.3%a–14.9%

Waypoint 39.1%a

32.4% 22.9% 25.9% 19.5% 27.2% 30%a
27

4. Conclusions

The objective of this research was to experimentally assess the value added of the CARMAN 

and CATO technologies to manipulation and tele-operation, respectively. To achieve this goal, 

ARL-Human Research and Engineering Directorate (HRED) in collaboration with AMRDEC 

developed an experiment plan. AMRDEC provided the hardware and software necessary to 

complete the experiment. ARL conducted the experiment and analyses described in this report.

4.1 CARMAN

For CARMAN, a light board was built. Targets that varied in size and location were projected 

on the board. Participants used the CARMAN technologies to move the manipulator arm to the 

target. The CARMAN technologies used were the Baseline configuration (ALTON), Fly-To, 

and Point and Click configuration. Results showed that performance with the Fly-To 

configuration was superior to the other two configurations. More specifically, fewer targets were 

missed in the Fly-To configuration than either the Baseline (ALTON) or the Point and Click

configuration. Although Point and Click was developed to aid object manipulation, it did not 

improve performance relative to the Baseline (ALTON). Interestingly, task time was longer for 

Point and Click than Baseline, which was contrary to our prediction. In addition to objective 

performance, we assessed operator workload after each experimental trial. The subjective data 

complemented the objective results. In the Point and Click mode participants reported more time 

pressure, irritation, and stress during the task than in the Baseline configuration. The reason for 

these objective and subjective findings may be in the implementation of Point and Click

configuration. In this mode the participant lined up the manipulator with the target and then the 

arm moved close to the location. The participant then had to switch to the basic mode and 

complete the manipulation task manually. In contrast, the Fly-To configuration did not require 

this last-minute manual adjustment by the operator. Thus, the Fly-To technologies successfully 

improved object manipulation, moving the manipulator arm to a target of interest. This 

technology improved the speed and accuracy with which the operator reached a target, with 

lower reported workload ratings.

4.2 CATO 

For the CATO task an obstacle course was created outside. The course included a painted path 

to follow, gates, slaloms, and obstacles. Participants used the CATO technologies to maneuver 

through the course. The course was primarily divided into two portions; a path following section 

and an obstacle course section. The CATO technologies were the Baseline configuration 

(TALON IIIB), Elevated Camera, Projected Path, and Waypoint configuration. Across all tasks, 

Waypoint configuration was found to be the least suitable. This is not surprising since the 
28

proposed application of Waypoint configuration is to allow the operator to move in a straight line 

at a constant speed while performing a visual search or other task. The CATO driving course 

was not designed to test that type of UGV mission. While the three remaining configurations 

allowed completion of the path-following section at similar speeds, the Projected Path 

configuration was superior in minimizing deviation from the path to be followed. This verified 

the primary purpose of the Projected Path configuration; to align the robot with a chosen path. It 

is interesting that the operators did not slow down when they were having difficulty following 

the path. The operators generally maintained a constant speed and did their best to apply a 

―closed-loop‖ course correction to the path as they progressed. While both Projected Path and 

Elevated Camera showed improvement from the Baseline configuration during the obstacle 

course segment, the Elevated Camera was best at both speed and accuracy in completing the 

obstacle course. The extra-wide field of view was instrumental in assisting the operator to pass 

though narrow openings (such as the slalom and gate obstacles) in a quick and efficient manner. 

The projected path configuration did allow the operators to align the robot with the opening, but 

the reduction in field of view was a detriment to performance in that configuration. In an 

informal survey, the participants reported an overwhelming preference for using Projected Path 

for the path following segment and a nearly even split for preferring Projected Path or Elevated 

Camera for the obstacle course. This preference was echoed in the decreased workload scores 

for Elevated Camera and Projected Path compared to Baseline or Waypoint configurations.

4.3 Summary 

The objective of this research was to experimentally evaluate the effect of CARMAN and CATO 

technologies on manipulator activity and tele-operation, respectively. These technologies were

assessed to gauge their effectiveness relative to standard operation of a TALON IIIB. 

CARMAN involves the development of technologies to reduce robotic manipulator task times

and improve the precision of robotic manipulator placement. CATO involves the development 

of technologies to reduce tele-operated robotic mobility task times and improve the precision of 

tele-operated driving. The results of this technical test indicate that not only do CARMAN and 

CATO technologies increase accuracy and precision with respect to manipulator activity and 

tele-operation, but also significantly decrease mission times.
29

Appendix A. Informed Consent Form

 

This appendix appears in its original form without editorial change. 
30

VOLUNTEER AGREEMENT AFFIDAVIT:

ARL-HRED Local Adaptation of DA Form 5303-R. For use of this form, see AR 70-25 or AR 40-38

The proponent for this research is: U.S. Army Research Laboratory

Human Research and Engineering 

Directorate

Aberdeen Proving Ground, MD 21005

Authority:

Privacy Act of 1974, 10 U.S.C. 3013, [Subject to the authority, direction, and control of the 

Secretary of Defense and subject to the provisions of chapter 6 of this title, the Secretary of the 

Army is responsible for, and has the authority necessary to conduct, all affairs of the Department of 

the Army, including the following functions: (4) Equipping (including research and development), 

44 USC 3101 [The head of each Federal agency shall make and preserve records containing 

adequate and proper documentation of the organization, functions, policies, decisions, procedures, 

and essential transactions of the agency and designed to furnish the information necessary to protect 

the legal and financial rights of the Government and of persons directly affected by the agency's 

activities]

Principal purpose: To document voluntary participation in the Research program.

Routine Uses:

The SSN and home address will be used for identification and locating purposes. Information 

derived from the project will be used for documentation, adjudication of claims, and mandatory 

reporting of medical conditions as required by law. Information may be furnished to Federal, State, 

and local agencies.

Disclosure:

The furnishing of your SSN and home address is mandatory and necessary to provide identification 

and to contact you if future information indicates that your health may be adversely affected. 

Failure to provide the information may preclude your voluntary participation in this data collection.

Part A • Volunteer agreement affidavit for subjects in approved Department of Army research projects

Note: Volunteers are authorized medical care for any injury or disease that is the direct result of participating in this 

project (under the provisions of AR 40-38 and AR 70-25).

Title of Research Project: Technical Testing of the CARMAN and CATO technologies

Human Use Protocol Log # 

Number: ARL-20098-08035

Principal Investigator:

Regina Pomranky 

U.S. Army Research Laboratory 

Human Factors Integration Division

Fort Rucker Field Element

Phone: 334-255-2135

E-Mail: rpomranky@arl.army.mil

Associate Investigator(s)

Keryl A. Cosenzo, Ph. D

U.S. Army Research Laboratory 

Soldier Performance Division

Crew Station Branch

Phone: 410-278-5885

E-Mail: kcosenzo@arl.army.mil

Associate Investigator(s)

Brad Pettijohn

U.S. Army Research Laboratory 

Human Factors Integration Division

Fort Leonard Wood Field Element

Phone: 573-563-5326

E-Mail: brad.pettijohn@us.army.mil

Associate Investigator(s)

Andrew Bodenhamer

U.S. Army Research Laboratory 

Human Factors Integration Division

Fort Leonard Wood Field Element

Phone: 573-563-6031

E-Mail: 

andrew.s.bodenhamer@us.army.mil

Location of Research: ARMDEC Huntsville, AL

Dates of Participation: November 2008 
31

Part B • To be completed by the Principal Investigator

Note: Instruction for elements of the informed consent provided as detailed explanation in accordance with

Appendix C, AR 40-38 or AR 70-25.

Purpose of the Research

The purpose of this research is to test the performance effects of the Computer Aided Teleoperation (CATO) and Computer Aided Robotic Manipulation (CARMAN) technologies. The 

CATO technology is a driving aids software. The CARMAN technology is a manipulator control 

software package. We are evaluating the new experimental driving aid and manipulator control

software to assess the performance effects of operating ground vehicles by remote control with 

and without these new technologies.

Procedures

This experiment is divided into three experiments during the day. In Experiment I, you will control the TALONIIIB 

manipulator arm. The objective will be to manipulate the manipulator arm to hit 20 targets that will be displayed on 

a light board as quickly and accurately as possible. In Experiment II, you will remotely drive the TALONIIIB 

through an outdoor obstacle course. The objective will be to maneuver the unmanned system through the obstacle 

course as quickly and accurately as possible. In Experiment III, you will remotely drive a HMMWV through an 

outdoor obstacle course. The objective will be to maneuver the unmanned system through the obstacle course as

quickly and accurately as possible. Experimental I will take place in an indoor climate-controlled laboratory. 

Experiment II will take place outdoors on a mowed field on which our obstacle course laid out. Experiment III will 

take place outdoors on an airfield airstrip. 

You will complete Experiment I, II, and III in one eight hour day. A 30 minute break will be taken between 

each experiment and 45 minutes will be given for lunch. At the start of each experiment you will be given an 

overview of and training on the baseline and advanced configurations for the technology. You will then complete a 

training run. You will have the opportunity to repeat the training run until you are comfortable controlling the robot. 

Criteria for efficient training will be determined by a Subject Matter Expert on the platform who will be present 

during the training mission. These subject matter experts are the engineers of the CARMAN and CATO software 

and will use their knowledge of the system to assess training effectiveness. After training you will complete the 

experimental runs. At the end of each run you will complete the NASA-TLX to assess you workload during the task. 

Total participation time will be approximately eight hours.

Benefits

You will receive the personal satisfaction of providing valuable information to the Army’s unmanned 

systems research. 

Risks

The risks that will be encountered in this study are minimal and typical of the everyday risks encountered by 

military and civilian personnel performing office duties using their computers. Mild motion sickness may be 

experienced due to operation of an operator control unit to control an unmanned system. A motion sickness 

questionnaire will be used to assess this risk. A comparison of the pre- and post-experimental response patterns 

across the questionnaire will be conducted to evaluate motion sickness. If you show elevated scores on any of the 

items, you will be held at the research site until their symptoms abate. 
32

Photography

Photographs and video may be recorded during the experiment. Photographic or video images of you 

taken during this data collection will not be identified with any of your personal information (name, 

rank, or status). 

We would like your permission to take pictures/videotape/audio record during the experimental session. The 

pictures/recording may be used in reports or presentations of this work. Please indicate below if you will agree to 

allow us to record you. You can still be in the study if you prefer not to be recorded.

I give consent to be audio taped during this study: ____Yes ____No please initial:____

I give consent to be videotaped during this study: ____Yes ____No please initial:____

I give consent to be photographed during this study: ____Yes ____No please initial:____

Confidentiality

All data and information obtained about you will be considered privileged and held in confidence. All examinations 

will be recorded using a volunteer identifier code and a separate file with your consent form and the Principal 

Investigator will keep your assigned volunteer identifier code in a locked cabinet. Complete confidentiality cannot 

be promised, particularly if you are a military service member, because information bearing on your health may be 

required to be reported to appropriate medical or command authorities. In addition, applicable regulations note the 

possibility that the U.S. Army Human Research Protection Office officials may inspect the records. In order to 

ensure that your data will not be reported or revealed to anyone, each form will be reviewed upon receipt by one of 

the investigators. If any identifying information appears on the questionnaires (such as name, social security 

number, birth date, etc.), the investigators will delete the identifying information and replace it with a neutral code 

number. All investigators perusing the questionnaires and forms for any sensitive subject information put on the 

forms have taken Human Use Training. 

Disposition of Volunteer Agreement Affidavit

The Principal Investigator will retain the original signed Volunteer Agreement Affidavit and forward a photocopy of 

it to the Chair of the Human Use Committee after the data collection. The Principal Investigator will provide a copy 

of the signed and initialed Affidavit to you.

Contacts for Additional Assistance

If you have questions concerning your rights on research-related injury, or if you have any complaints about your 

treatment while participating in this research, you can contact:

Chair, Human Use Committee OR Office of the Chief Counsel

U.S. Army Research Laboratory U.S. Army Research Laboratory

Human Research and Engineering Directorate 2800 Powder Mill Road

Aberdeen Proving Ground, MD 21005 Adelphi, MD 20783-1197

(410)278-5992 (301) 394-1070 or (DSN) 290-1070
33

I do hereby volunteer to participate in the research project described in this document. I have full capacity to consent 

and have attained my 18th birthday. The implications of my voluntary participation, duration, and purpose of the 

research project, the methods and means by which it is to be conducted, and the inconveniences and hazards that 

may reasonably be expected have been explained to me. I have been given an opportunity to ask questions 

concerning this research project. Any such questions were answered to my full and complete satisfaction. Should 

any further questions arise concerning my rights or project related injury, I may contact the ARL-HRED Human 

Use Committee Chairperson at Aberdeen Proving Ground, Maryland, USA by telephone at 410-278-5992 or 

DSN 298-5992. I understand that any published data will not reveal my identity. If I choose not to participate, or 

later wish to withdraw from any portion of it, I may do so without penalty. I understand that military personnel are 

not subject to punishment under the Uniform Code of Military Justice for choosing not to take part as human 

volunteers and that no administrative sanctions can be given me for choosing not to participate. I may at any time 

during the course of the project revoke my consent and withdraw without penalty or loss of benefits. However, I 

may be required (military volunteer) or requested (civilian volunteer) to undergo certain examinations if, in the 

opinion of an attending physician, such examinations are necessary for my health and well-being.

Printed Name Of Volunteer (First, MI., Last)

Social Security Number (SSN) Permanent Address Of Volunteer

Date Of Birth

(Month, Day, Year)

Today’s Date

(Month, Day, Year)

Signature Of Volunteer
34

INTENTIONALLY LEFT BLANK.
35

Appendix B. Demographics Questionnaire

 

This appendix appears in its original form without editorial change. 
36

Participant ID# __________

Demographic and Computer Experience Questionnaire

1. AGE: _____

2. GENDER: ___Male ___ Female

3. Do you wear glasses? ___ Yes ___ No

4. Do you have any reason to believe that you have a hearing impairment? ___Yes ___ No

5. Please indicate your highest level of education:

___ High School Diploma

___ Undergraduate Degree 

 ___ Some graduate courses 

 ___ Graduate Degree 

 ___ Other

 

6. Are you or have you been in the military? ___Yes ___No If yes, what Branch? 

_________________

For how many years? ___Less than 5 years ___5-10 years ___ 11-15 years ___16-20 years ___ 20 

years or more

7. Does your job require you to use a computer on a regular basis? ___Yes ___No

8. How long have you been using a computer? 

__Less than 1 year ___ 1-3 years ___4-6 years ___7-10 years ___10 years or more

9. How often do you use a computer? 

___Daily ___Weekly ___Monthly ___Once or twice a year

10. Do you have a computer in your house? ___Yes ___No

11. Do you use the computer to play games? ___Yes ___No

 

 If yes, how often? ___Daily ___Weekly ___Monthly ___Once or twice a year

12. What is your level of experience with the operation of unmanned systems?

No previous experience _____

Some experience (1-3 times) _____

Substantial experience (More than 3 times)________

If you have experience, with what systems do you have experience?
37

For what application(s) did you use unmanned systems? 

______________________________________________________________________________

______________________________________________________________________________
38

INTENTIONALLY LEFT BLANK.
39

Appendix C. NASA TLX Questionnaire

 

This appendix appears in its original form without editorial change. 
40

NASA TLX Questionnaire

Participant ID:________________

TLX Workload Scale

Please rate your workload by putting a mark on each of the six scales at the point which matches your experience.

Mental Demand

Physical Demand

Temporal Demand

Performance

Effort

Frustration

Good Poor

p

Low High

Low High

Low High

Low High

Low High
41

Appendix D. Symptom Questionnaire

 

This appendix appears in its original form without editorial change. 
42

Symptom Questionnaire Participant ID: ________________

Condition:____________ Date: _____________________ Time: __________________

Using the scale below, please rate how accurately the following statements describe your 

experience.

1. I feel sick to my stomach Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

2. I feel faint-like Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

3. I feel annoyed / irritated Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

4. I feel sweaty Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

5. I feel queasy Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

6. I feel lightheaded Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

7. I feel drowsy Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

8. I feel clammy / cold sweat Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

9. I feel disoriented Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

10. I feel tired / fatigued Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

11. I feel nauseated Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

12. I feel hot / warm Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

13. I feel dizzy Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

14. I feel like I was spinning Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

15. I feel as if I may vomit Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9

16. I feel uneasy Not at all Severely

 1-------2-------3-------4-------5-------6-------7-------8-------9
43

List of Symbols, Abbreviations, and Acronyms

3-D three-dimensional

AMRDEC Aviation and Missile Research, Development and Engineering Center

ARL U.S. Army Research Laboratory

BAT Battlefield Automation Team

CARMAN Computer Aided Robotic Manipulation

CATO Computer Aided Tele-operation

EOD explosive ordnance disposal

HRED Human Research and Engineering Directorate

IED improvised explosive device

MTBF mean time between failures

MTTR mean time to repair

NASA-TLX NASA-Task Load Index

OCU Operator Control Units

SED Software Engineering Directorate

SUGV small unmanned ground vehicle

UAV unmanned aerial vehicle

UGV unmanned ground vehicle
NO. OF

COPIES ORGANIZATION

44

1 DEFENSE TECHNICAL

(PDF INFORMATION CTR

only) DTIC OCA

8725 JOHN J KINGMAN RD

STE 0944

FORT BELVOIR VA 22060-6218

1 DIRECTOR

US ARMY RESEARCH LAB

IMNE ALC HRR

2800 POWDER MILL RD

ADELPHI MD 20783-1197

1 DIRECTOR

US ARMY RESEARCH LAB

RDRL CIM L

2800 POWDER MILL RD

ADELPHI MD 20783-1197

1 DIRECTOR

US ARMY RESEARCH LAB

RDRL CIM P

2800 POWDER MILL RD

ADELPHI MD 20783-1197

1 DIRECTOR

US ARMY RESEARCH LAB

RDRL D

2800 POWDER MILL RD

ADELPHI MD 20783-1197

ABERDEEN PROVING GROUND

1 DIR USARL

RDRL CIM G (BLDG 4600)
NO. OF NO. OF

COPIES ORGANIZATION COPIES ORGANIZATION

45

1 ARMY RSCH LABORATORY – HRED

RDRL HRM A J MARTIN

MYER CENTER BLDG 2700 RM 2D311

FORT MONMOUTH NJ 07703-5601

1 ARMY RSCH LABORATORY – HRED

RDRL HRM C A DAVISON

320 MANSCEN LOOP STE 115

FORT LEONARD WOOD MO 65473

1 ARMY RSCH LABORATORY – HRED

RDRL HRM DI T DAVIS

BLDG 5400 RM C242

REDSTONE ARSENAL AL 35898-7290

1 ARMY RSCH LABORATORY – HRED

RDRL HRS EA DR V J RICE

BLDG 4011 RM 217

1750 GREELEY RD

FORT SAM HOUSTON TX 78234-5002

1 ARMY RSCH LABORATORY – HRED

RDRL HRM DG R SPINE

BLDG 333

PICATINNY ARSENAL NJ 07806-5000

1 ARMY RSCH LABORATORY – HRED

ARMC FIELD ELEMENT

RDRL HRM CH C BURNS

THIRD AVE BLDG 1467B RM 336

FORT KNOX KY 40121

1 ARMY RSCH LABORATORY – HRED

AWC FIELD ELEMENT

RDRL HRM DJ D DURBIN

BLDG 4506 (DCD) RM 107

FORT RUCKER AL 36362-5000 

1 ARMY RSCH LABORATORY – HRED

RDRL HRM CK J REINHART

10125 KINGMAN RD

FORT BELVOIR VA 22060-5828

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AY M BARNES

2520 HEALY AVE 

STE 1172 BLDG 51005

FORT HUACHUCA AZ 85613-7069

1 ARMY RSCH LABORATORY – HRED

RDRL HR MP D UNGVARSKY

POPE HALL BLDG 470 

BCBL 806 HARRISON DR

FORT LEAVENWORTH KS 66027-2302

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AJ J HANSBERGER

JFCOM FE

115 LAKEVIEW PKWY STE B

SUFFOLK VA 23435

1 ARMY RSCH LABORATORY – HRED

RDRL HRM DQ M R FLETCHER

NATICK SOLDIER CTR

AMSRD NSC WS E BLDG 3 RM 343

NATICK MA 01760-5020

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AT J CHEN

12350 RESEARCH PKWY

ORLANDO FL 32826-3276

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AT C KORTENHAUS

12350 RESEARCH PKWY

ORLANDO FL 32826

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AS C MANASCO

SIGNAL TOWERS

BLDG 29808A RM 303A

FORT GORDON GA 30905-5233

1 ARMY RSCH LABORATORY – HRED

RDRL HRM CU

6501 E 11 MILE RD MS 284

BLDG 200A 2ND FL RM 2104

WARREN MI 48397-5000

1 ARMY RSCH LABORATORY – HRED

FIRES CTR OF EXCELLENCE 

FIELD ELEMENT

RDRL HRM AF C HERNANDEZ

3040 NW AUSTIN RD RM 221

FORT SILL OK 73503-9043
NO. OF

COPIES ORGANIZATION

46

1 ARMY RSCH LABORATORY – HRED

RDRL HRM AV S MIDDLEBROOKS

91012 STATION AVE RM 348

FORT HOOD TX 76544-5073

1 ARMY RSCH LABORATORY – HRED

RDRL HRM CN R SPENCER

DCSFDI HF

HQ USASOC BLDG E2929

FORT BRAGG NC 28310-5000

1 ARMY RSCH LABORATORY – HRED

RDRL HRM DW E REDDEN

BLDG 4 CL 60

FORT BENNING GA 31905-5400

1 ARMY G1

(CD DAPE MR B KNAPP

only) 300 ARMY PENTAGON RM 2C489

WASHINGTON DC 20310-0300

ABERDEEN PROVING GROUND

5 DIR USARL

RDRL CIM G

S FOPPIANO

RDRL HR

L ALLENDER

T LETOWSKI

RDRL HRM B

J LOCKETT

RDRL HRS D

B AMREIN