military experience and training, familiarity with items of
equipment, and their ability to focus attention on various
activities. (Focus items came from the Immersive Tendencies
Questionnaire, Witmer and Singer, 1994.) This questionnaire was
administered on the first day, prior to the first excursion.

The Health Questionnaire (Appendix C) obtained information
on the participants’ current state of health. It was administered
on the first day of the excursions, after initial training but
prior to the first excursion, and on each subsequent day prior to
the start of the excursions.

A potential problem with application of simulation
technology in the context of military concept development is that
participants may experience side and aftereffects similar to, but
not limited to, symptoms of motion sickness. Simulator sickness
is a concern because it can potentially distract or in other ways
degrade the performance of the participants. The Simulator
Sickness Questionnaire (SSQ) (Appendix D) was administered prior
to the first excursion and after the last excursion each day
(before and after immersion) to take pre-existing symptoms into
account. (See Lampton, Kraemer, Kolasinski & Knerr (1995) and
Kennedy, Hettinger, & Lilienthal, (1988) for further
information.)

The Human Factors Equipment and Functions Questionnaire
(Appendix E) determined the frequency with which participants
performed various tasks and functions and the degree of
difficulty they had performing those tasks. It was administered
at the end of each day. The Learning Questionnaire (Appendix F)
was used to determine difficulty participants had learning to
perform various tasks. It also was administered at the end of
each day.

The Simulator Questionnaire (Appendix G) asked the extent to
which the participants felt immersed in the simulation. The
items were taken from a Presence Questionnaire developed by
Witmer and Singer (1994). The Simulator Questionnaire was
administered after the last excursion on the second and fourth
days of week two.

After Action Reviews and Interviews

After Action Reviews (AARs). At the conclusion of each
excursion, the Exercise Controller conducted an AAR. The purpose
of the AAR was to solicit comments, lessons learned, and other
observations. While each AAR was attended by the scenario role
players, observers, and some technical support personnel in
addition to the exercise participants themselves, the primary
focus was on obtaining information from the participants. Each



result, the data collected probably underrepresented performance
that might actually be achieved.

In the independent small team environment, soldier selection
and training are critical. Issues such as workload, fatigue,
boredom, and stress need to be addressed. With respect to task
overload, there clearly were times when the soldiers could not
keep up with the action. This was partially due to the
awkwardness of the equipment but even if it had worked well, they
would have reached overload fairly quickly without an automated
fire-and-forget system.

The amount of information available to a team needs to be
evaluated and it will be important to make some decisions about
use of technology. Simply automating and speeding up current
procedures may just make errors faster and more frequent. Just
because information is available does not guarantee that it is
useful.

The intended precise element by element timeline -- how long
it takes for an observer to receive information, call for fire,
and get munitions on a target -- was not obtained. But the

simulation showed that the mission could be accomplished by the
soldiers, and rough estimates derived. The concept of a small
team portal appears viable, and the simulation facility proved to
be an appropriate place to demonstrate it.

30



also need a predetermined between-group plan for target hand-off,
if a target passes from one sector to another.

The tendency of the participants to lose local situational
awareness, particularly in high stress situations, points out the
need for a separate element of the team to maintain local
security. The fatiguing nature of the tasks suggests a need for
multiple shifts of operators.

Workload and performance issues are largely personality
driven. There were apparent differences between the groups where
there was almost no chatter, and those where it was common. When
there was little to do, each of the soldiers acted bored or
complacent and occasionally lost focus. Although some did better
with pressure, others were sometimes apparently overloaded. The
small remotely placed team may require a special kind of personal
discipline to ensure that attention and awareness are maintained
throughout a period of inactivity. Some personnel selection and
qualification issues will have to be addressed.

Lessons Learned: Observer and Participant Training

Training is critical for equipment, tactics, techniques and
procedures. It must include not only learning to use new
equipment, but a chance to practice or experiment with it.
Observers must also be trained, both on the hardware and on
observation techniques. Observers need to hear team cross talk
and radio traffic, and share the soldier’s field of view. Only
in this way can an observer evaluate operator performance. For
example, the Grunt observer monitor was not useful because the
process of creating messages prior to their display on the screen
was not visible. The same observers also need to be available
every day to note posture changes, signs of fatigue, eyestrain,
etc., and to get a feel for the simulation and for player
personalities.

Although radio traffic should be downplayed, personnel need
instruction on communication procedures, call signs, etc. For
example, a training deficiency became apparent with the radio’s
voice activated mike. Only the experienced soldiers knew to
start messages with double words (“shooter, shooter”), the first
word to activate the mike, the second to start the message.

Just as there are conventions for use of radios, soldiers
need SOPs (acronyms, abbreviations, formats) to be used in the
simulation. Perhaps the most important skill requirement for
team members will be a relatively high degree of computer
literacy. All of the devices were slower to perform than typical
office equipment. Operator unfamiliarity with system speed
caused extra work, either through resending messages, Or

26



3. When watching sports, do you ever become so involved in the game
that you react as if you were one of the players?

{ | | | | I | |
NEVER OCCASIONALLY OFTEN

4. When playing sports, do you become so involved in the game that you
lose track of time?

| “I \ \ I I \ |
NEVER OCCASIONALLY OFTEN

5. How well do you concentrate on enjoyable activities?

i i | I | | | {
NOT AT ALL MODERATELY VERY WELL
WELL

6. How often do you play arcade or video games? (OFTEN should be taken
to mean every day or every two days, on average.)

| I I | | 1 i |
NEVER OCCASIONALLY OFTEN

7. Do you ever become so involved in doing something that you lose all
track of time?

| I { | | ! | |
NEVER OCCASIONALLY OFTEN

8. How easily can you switch attention from the activity in which you
are currently involved to a new and completely different activity?

| | | | I | i |
NOT SO FAIRLY QUITE
EASILY EASILY EASILY

9. How often do you play games on computers?

| i I | | | | {
NOT AT ALL OCCASIONALLY FREQUENTLY

10. How many different video, computer, or arcade games have you become
reasonably good at playing?

| | 1 | | | { |
NONE ONE TWO THREE FOUR FIVE SIX OR MORE

11. Have you ever felt completely caught up in an experience, aware of
everything going on and completely open to all of it?

i | | | | | | |
NEVER OCCASIONALLY FREQUENTLY

12. Are you easily disturbed when involved in an activity or working on
a task?

| | | | i I |
NEVER OCCASIONALLY OFTEN



more subjective reports by subject matter experts about problems
likely to be encountered in a field situation. Virtual
simulations permit this information to be obtained prior to the
development of a complete physical prototype.

Information about how humans perform in these virtual
simulations can be used in several ways. First, it can help to
assess the overall feasibility of the concept itself. (For
example, can human operators detect targets and submit fire
requests rapidly enough to accomplish the mission successfully?)
Second, it can help to refine the system functional requirements.
(Elevated sensor magnification must be at least 6X or targets
cannot be detected beyond 2 km.) Third, it can help design tasks
and organizations, assign system functions to human operators or
automated components, and organize human functions into duty
positions. (Each element requires three personnel for effective
operations.) Fourth, it can help identify the necessary
abilities and prerequisite skills required to perform the jobs
successfully. (All operators require a high degree of general
computer literacy and understanding of friendly and enemy
battalion-level tactics.) Finally, it can identify training
needs and support the development and evaluation of operator
training.

Approach and Procedures

Simulation Center Facility

The IDA Simulation Center, conceptualized as an integrating
center, permits a variety of excursions and explorations.
Commercial off-the-shelf equipment and prototype specialty
equipment are integrated as needed for concept demonstrations and
experimentation. The facility is permanent, but on-site
equipment varies with the specific area of investigation, thus
the concept of an integrating, rather than dedicated facility.

Participants

Two U.S. Marine Corps captains served as Fire Direction
Center (FDC) controllers. During the first week of testing, two
Army National Guard captains, supplemented by IDA civilian
personnel, participated as test subjects for early operational
trials. In the second week, six Marine second lieutenants
participated in various team configurations. For example, in one
configuration, two were on one team, three on another, and one
served as the intermediate team leader.



Appendix H

Results of the Participant Immersive Tendencies
Questionnaire

1. Do you ever become so involved in a video game that it is as
if you are inside the game rather than moving a joystick and
watching the screen?

{Xx of xX__t I xX__t |_ XxX |
NEVER OCCASIONALLY OFTEN

2. How good are you at blocking out external distractions when
you are involved in something?

| | xX__i | 1 XX__|__XXX__I
NOT VERY SOMEWHAT VERY GOOD
GOOD GOOD

3. When watching sports, do you ever become so involved in the
game that you react as if you were one of the players?

| x | XXX__| x | x | | |

NEVER OCCASIONALLY OFTEN

4. When playing sports, do you become so involved in the game
that you lose track of time?

| xX__d XX__| I xt xt xX

NEVER OCCASIONALLY OFTEN

5. How well do you concentrate on enjoyable activities?

| | | a a |_XXx_ | XX
NOT AT ALL MODERATELY - VERY WELL
WELL

6. How often do you play arcade or video games? (OFTEN should
be taken to mean every day or every two days, on average.)

| X__|__Xxx_| xt | xX |
NEVER OCCASIONALLY OFTEN

7. Do you ever become so involved in doing something that you
lose all track of time?

| |___ Xxx | an | aa
NEVER OCCASIONALLY OFTEN



7. How completely were you able to actively survey or search the
environment using vision?

| i ! | | { I |
NOT AT ALL SOMEWHAT COMPLETELY

8. How compelling was your sense of moving around inside the virtual
environment?

| | | I | 1 I !
NOT MODERATELY VERY
COMPELLING COMPELLING COMPELLING

9. How closely were you able to examine objects?

| | | | | i | I
NOT AT ALL PRETTY VERY
CLOSELY CLOSELY

10. How well could you examine objects from multiple viewpoints?

| | | | i | | |
NOT AT ALL SOMEWHAT EXTENSIVELY

11. How involved were you in the virtual environment experience?

I | | I ! | _ |
NOT MILDLY COMPLETELY
INVOLVED INVOLVED ENGROSSED

12. How much delay did you experience between your actions and expected
outcomes?

| l I [ i | | |
NO DELAYS MODERATE LONG
DELAYS DELAYS

13. How quickly did you adjust to the virtual environment experience?

I { l | | | | |
NOT AT ALL SLOWLY LESS THAN
ONE MINUTE

14. How proficient in moving and interacting with the virtual
environment did you feel at the end of the experience?

| | | | | | i |
NOT REASONABLY VERY
PROFICIENT PROFICIENT PROFICIENT



Table I-2

Distribution of Immersion Questionnaire Responses
(Administrations 1 And 2 Combined)

1. How natural did your interactions with the environment seem?

} |__XXXX |XX | XKK_|_ XXX | \
EXTREMELY BORDERLINE

COMPLETELY

ARTIFICIAL NATURAL

2. How much did the visual aspects of the environment involve
you?

| | xl X___|__XXXXX_]___XXX__|__ XX \
NOT AT ALL SOMEWHAT COMPLETELY

3. How natural was the mechanism which controlled movement
through the environment?

{__ xx |__XXXX__[|__XxXX___|__XXX__ \ \
EXTREMELY BORDERLINE COMPLETELY
ARTIFICIAL NATURAL

4. How compelling was your sense of objects moving through
space?

{ | XX |__XXX__I_ XX XK XXXX__ |
NOT AT ALL MODERATELY VERY
COMPELLING COMPELLING

5. How much did your experiences in the virtual environment
seem consistent with your real world experiences?

\ X_{__XXX_|__XXX__|_XXXXX_I xt \ |
NOT MODERATELY VERY
CONSISTENT CONSISTENT CONSISTENT

6. Were you able to anticipate what would happen next in
response to the actions that you performed?

| {xXx |XX |__XXXXX_|_XXX_ |
NOT AT ALL SOMEWHAT COMPLETELY

1-3



Please use the following scale.

Appendix F

Learning to Use the Equipment

below.

How difficult was it to learn to use the equipmen

1 Very Easy

2 Easy

3 Neither Easy nor Difficult
4 Difficult

5 Very Difficult

Indicate your answer in the table

Equipment Name

Difficulty

Dial a Tank

Drive the vehicle

Maintain orientation

Navigate in virtual environment

Treadport

Change orientation

Keep your balance

Navigate

of the Future

Maintain map orientation

Keep track of things on map

Learn the menus

Respond to a message

Call for fire/send message

Grunt II PDA

Understand menu

Realize a message is waiting

Remember PDA functions

Compose text messages

Make PDA do what you wanted

Adjust fires with PDA

GATH/Melios

Detect targets

Identify targets (name them)

Designate targets (laze) a

Perform BDA

COVER

Detect targets

Identify targets (name them)

Designate targets (laze)

Perform BDA

Use the joystick controller

Maintain orientation

t or perform the task?



Table 2.

Ratings of COVER Tasks (Means)

Task Difficulty Frequency Learning
Difficulty

Distinguish friendly and 3.70 2.90 NA

neutral

Making joystick work 3.33 5.00 NA

right

Designate targets 3.00 3.53 2.13

Identify targets (name 2.97 3.70 2.63

them)

Using the joystick 2.93 4.83 3.20

controller

Lazing 2.63 4.37 2.13

Detect targets 2.50 4.10 2.77

Maintaining orientation 2.50 4.70 2.10

Prioritize targets 2.33 2.67 NA

Perform BDA* 2.20 2.67 2.30

N= 9 *Battle damage assessment

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-Very Difficult

Making the joystick work right, designating targets and
using and learning to use the joystick controller were also
difficult. Designating a target required placing a cross-hair on
the center of the COVER display over a (sometimes moving) target
using the joystick controller. The joystick had some undesirable
handling characteristics: a wide area of insensitivity around the
neutral position, with high sensitivity beyond that band made it
easy to overshoot the intended destination.

In the group interview, observing a single target with the
COVER was identified as one of the easiest individual tasks while
tracking multiple targets was one of the hardest. The viewing
conditions of the COVER and the joystick appeared to be the
cause. Viewing with the COVER could be done at either normal
resolution or 6X magnification. Our crude measurements indicated
that the horizontal field of view (FOV) of COVER was 47 degrees
at normal magnification, and 4 degrees at 6X. The resolution of
the monitor made it difficult to detect distant targets, so
participants did most of their searching and tracking at 6X.

This was somewhat similar to viewing the world through a soda
straw, and made it difficult to switch back and forth between two

13



References

Coe, G. Q., Madden J. L., Mengel, L. W., & Wright, R. K., (1996).
Small team portal into the 21st century (STP21) simulation
exercise report (Report for the Defense Advanced Research
Projects Agency). Alexandria, VA: Institute for Defense
Analyses.

Endsley, M. R. (1995). Toward a theory of situation awareness in
dynamic systems. Human Factors, 37(1), 32-64.

Kennedy, R. S., Hettinger, L. J., & Lilienthal, M. G. (1988).
Simulator sickness. In G. H. Crampton (Ed.), Motion and space
sickness, Chapter 15, 317-341. Boca Raton, FL: CRC Press.

Lampton, D. R., Kraemer, R. E., Kolasinski, E. M., & Knerr, B. W.
(1995). An investigation of simulator sickness in a tank

driver trainer (ARI Research Report 1684). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences.

Lampton, D. R., Kolasinski, E. M., Knerr, B. W., Bliss, J. P.,
Bailey, J. H., & Witmer, B. G. (1994). Side effects and
after effects of immersion in virtual environments.
Proceedings of the Human Factors Society, 38th Annual Meeting.
1154-1157.

Witmer, B. G., & Singer, M. S. (1994). Measuring presence in
virtual environments (ARI Technical Report 1014). Alexandria,
VA: U.S. Army Research Institute for the Behavioral and
Social Sciences. (AD A286 183)

31



Research Report 1704

Small Team Portal Into the 21st Century - STP21

Margaret S. Salter, Bruce W. Knerr, Donald R. Lampton,
Gene W. Fober, and J. Douglas Dressel

U.S. Army Research Institute

Infantry Forces Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

Office, Deputy Chief of Staff for Personnel

Department of the Army
December 1996
Nene ne EEE EEE EEEEeoemee!
Army Project Number Education and Training
20263007A794

Approved for public release; distribution is unlimited.



obtain, due primarily to difficulties with the prototype
equipment. Also of interest was team survivability -- whether
they could engage before they were themselves engaged. Teams
were occasionally overrun, perhaps due to the tendency to become
focused on the device they were using rather than the situation.

Another concern was the amount of terrain over which any one
team could be expected to maintain observation -- the maximum
limits of awareness. Generally, a 360 degree area of
responsibility proved too great. A balanced team has roles and
responsibilities allocated to maximize combat effectiveness.
While individual situational awareness should be enhanced, combat
power is enhanced through team work.

The third EEA focused on the simulation as a concept
exploration tool. This included the transparency of the
simulation to the players -- their ability to accept it and the
prototype equipment, and finally, the capability of the
simulation to provide meaningful data. The simulation provided
useful information. Some training issues (prototype equipment,
practice effects) and simulator issues (sickness, artificiality)
were potential distracters. Difficulty with equipment was
undeniable (the COVER joystick), and practice effects were
apparent (MOF); they could be overcome with better equipment and
more training time. For the duration of these exercises,
simulator sickness was not an issue; the artificiality of the
environment was of little impact, and may have lessened any ill
effects.

The simulation fidelity was adequate for representing the
critical role that the small team can play in dealing with
incomplete, imprecise, or erroneous information from electronic
and other remote sensors. However, limitations in visual
fidelity made it difficult to distinguish military from civilian
entities. Therefore, the simulation could not demonstrate a
benefit of having observers on the ground: avoiding collateral
damage to civilians and civilian structures.

Lessons Learned: Equipment

It is fully acknowledged that the specific equipment used in
the simulation is different from that which would be found on the
battlefield of the year 2010. However, certain lessons learned
merit discussion.

Treadport. The scenarios, as run, provided no opportunities
to take advantage of the unique capabilities of the treadport.
No observation points were accessible only on foot, and
observation from a prone position was not possible. When

24

soitaetar



AAR had two segments. The first focused on the tactical events,
the second on aspects of the simulation.

Group interview. The ARI researchers led a group interview
on the morning of the last day of the second week. The intent
was to solicit information from the participants about manpower,
personnel, and training issues. Questions asked were:

e What equipment or procedures worked best? What
worked least well?

e What was the hardest task for you to perform? What
was the easiest?

e What equipment or procedures, if any, did you need
more training on?

e What equipment or procedures did you have the most
confidence in? What equipment or procedures did you
have the least confidence in?

e What distracted you the most?

e If you participated in the same exercises again,
would you do anything differently? If so, what?

e What prior experience or training (if any) helped
you in these exercises?

e How many people should be in an observer group?
Why?

e What was the most difficult duty position? What was
the easiest? Why?

* Do you have any ideas about how to share the
workload?

e What do you think a man can do here that a computer
can’t?

e What did you have to do that you think should be
done by computer instead of a human?

10



First, the MOF was a conventional laptop that used a
graphical user interface with icons and pull-down menus similar
to Windows or Macintosh operating systems. Given the familiarity
of the participants with laptop computers, they may have found it
easy to learn the basic concepts. Second, since the MOF was used
in at least two and usually three duty stations during each
excursion, the participants had sufficient practice with it to
become proficient in its use. Finally, the MOF was easily
modifiable and the software was modified on several occasions
during the excursions, based on comments from the participants.

Treadport

With respect to the treadport (see Table 5), the task of
doing something else while walking was the only task that was
considered difficult. However, it was not performed frequently,
so participants had little practice.

Table 5.

Ratings of Treadport Tasks (Means)

Task Difficulty Frequency Learning
Difficulty
Doing something else 4.00 1.33 NA
while walking
Maintaining orientation 2.33 3.33 NA
Navigate in virtual 2.33 2.50 1.63
environment
Maintaining balance 2.33 2.33 2.25
Walking on the Treadport 2.00 2.67 NA
Change orientation NA NA 2.88
N= 4

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-very Difficult

Grunt

Adjusting fires with the Grunt Personal Data Assistant (see
Table 6) was difficult to do and the most difficult Grunt task to
learn to do. However, this task was not performed frequently.
Overall, the Grunt was considered easy to use. Some problems
using the Grunt did surface in the group interviews. The primary
problem was a lack of any indication that a message had been

16



Participant Immersive Tendencies

Because of the distractions inherent in the simulation
facility, participants were asked about their ability to become
involved in activities and to ignore distractions. They were
also asked about experience with computer and video games.
(Responses are shown in Appendix H.) The group showed
considerable variability in responses to the questions that asked
about involvement in video games, sports, and other activities.

However, most reported that they were good at blocking out
distractions, concentrated well on enjoyable activities, switched
attention among activities well, and were not easily disturbed
when working on a task. While only one reported more than
occasional playing of arcade or video games, three reported more
than occasional playing of computer games.

Equipment Use

The Human Factors Equipment and Functions Questionnaire
provided information about task frequency and difficulty. The
Learning Difficulty Questionnaire showed how hard it was to learn
to perform certain tasks. While most tasks were equipment
specific, a few were general. Some tasks could also be performed
with multiple items of equipment. Participants completed the
questionnaires, responding only for pieces of equipment that they
had used in the immediately preceding excursion. Equipment usage
varied considerably, and therefore the number of responses on
which these ratings are based does as well. For example, the
Treadport was used a total of four times by three different
participants, while the Map of the Future was used a total of 24
times by six different participants. Questionnaire means were
calculated by first obtaining the mean response for each
participant across all six administrations of the questionnaires,
and then obtaining the mean across all participants. This had
the effect of weighting each participant’s response equally,
regardless of the number of times each used a particular item of
equipment . -

COVER

For COVER (see Table 2), distinguishing friendly and neutral
targets was rated the most difficult task. As will be presented
later, it was also the most difficult for the Melios, the other
piece of equipment that required direct visual observation to
make this determination. However, this reflects not the
equipment, but the simulation, which presented no neutral
entities and therefore did not test the capability.

12



movement was required to obtain or maintain line of sight to a
target, it had to be accomplished more rapidly than could be done
by foot. Questions remain about the value added of this concept
in this domain.

Dial-A-Tank. This surrogate was rarely used for mobility.
When it was used, it was apparent that the soldiers were having
difficulty in maintaining orientation. Additionally, there was a
tendency to drive using the MOF display rather than the terrain
display.

Map of the Future. The original concept was for the MOF to
be used only as a map; in practice it was used to call for and
receive fire missions, and for E-mail. The MOF was clearly not
designed for these missions. The map scale was not suitable for
the size of the area available; preformatted messages were not
well thought out. Provisions were not made for feedback on
requests for fire and targeting effects, but need for feedback
was clearly demonstrated.

Melios. The current configuration of Melios may have
precluded seeing the advantages of this device. Besides the
previously noted weight problems, and the difficulty of using
Melios while on a treadmill, Melios would be most useful in flat
and desert-like terrain, rarely available in this scenario.

COVER. While the concept appears viable, the specific
implementation (joystick control) does not. The joystick was
often unresponsive, erratic, and could not be held steady. It
impacted on the operators’ abilities to use the device
effectively. A fielded, future version would respond better, and
could have locations (grids, direction, azimuth) displayed
directly on the MOF.

Grunt. The consensus was that this device was in many
respects redundant to, and less useful than, the MOF. Having two
systems, with different conventions and key strokes, is
inefficient. A single device would reduce training time and
COVER and Melios could be linked directly to the MOF rather than
to the Grunt.

Lessons Learned: Personnel, Organization, Workload

Three person groups seemed most useful, with the team leader
assisting by filtering information (nice to know vs. need to
know). The groups that appeared to do well assisted each other
and developed SOPs. Within-group communication and cooperation
are critical -- groups need to work together to service targets,
and frequently can share in the identification process. They

25

seam



The sensor used at Portal 2 was known as COVER. A simulated
tethered aerial sensor, COVER could be elevated from tree top
level up to 300 feet in the air above the team’s position, and
turned for a 360 degree view. The COVER, like the Melios, hada
laser range finder that gave range and azimuth to any target
detected. It was designed to provide both optical and thermal
views, although only the day capability was demonstrated. COVER
was operated by a joystick; the input and view from the sensor
were depicted on a small screen monitor at the soldier’s station.

Each portal had two data entry/receipt devices for text and
graphic data communication and display. The Grunt was linked
either to the Melios or the COVER device. Grunt was a small
computer keypad with data entry by means of a stylus. It
displayed the range and azimuth from the Melios or COVER,
together with global positioning system (GPS) information, and
enabled an operator to call for fire. Preformatted messages
could be accessed and sent after the operator inserted grid
location and other selected information. Free text messages
could be sent, but Grunt had no receiving capability.

The other data entry device was the Map of the Future (MOF).
The MOF, conceptually a pocket-sized fold up electronic map but
represented in the simulation by a laptop computer, displayed an
electronic digital map. The map was the same terrain database
depicted in the WISE at Portal 1 and on the wide screen at Portal
2. Any section of a standard 1:50,000 map sheet could be viewed,
and expanded or minimized through a zoom feature. The MOF screen
presented a small top-down view of the entire database and a
larger display of a map section of a particular slice of terrain.
The smaller display contained a “pan” box that the user could
move to determine which map section was displayed. Friendly and
enemy positions, man-placed sensors, remotely monitored
battlefield sensor systems (REMBASS), and minefields could also
be positioned and represented on the map. Messages appeared on
the MOF as E-mail, and calls for fire could be sent through E-
mail back to the FDC. The MOF message function was used for
sending and receiving sensor and target information, and for
updates on target status.

Excursion Scenarios

In the general scenario, the U.S. supported a European
nation. After the threat was turned back, U.S. personnel were
inserted from a precision strike battalion to maintain regained
ground. This follow-on force, the small teams, was positioned
and equipped with devices for detecting and identifying targets,
processing fire requests, and communicating these requests to a
fire direction center at another location.



14. How proficient in moving and interacting with the virtual
environment did you feel at the end of the experience?

| 1 | |__XXXX__|_XXXKX__|_XXX_|_
NOT REASONABLY VERY
PROFICIENT PROFICIENT

PROFICIENT

15. Were you involved in the scenarios to the extent that you
lost track of time?

| x | XXX__|__ XX | XX. | XX. | XX |

NOT AT ALL ~ SOMEWHAT COMPLETELY

16. Were there moments during the scenarios when you felt
completely focused on the task or environment?

! l [__ XX |__XXXX__|__XXXX__| XX. I

NONE OCCASIONALLY FREQUENTLY

17. How easily did you adjust to the control devices used to
interact with the virtual environment?

| |__XXX |XX XXXX__f_XXX__| |

DIFFICULT MODERATE EASILY

Dl. How much did the visual display quality interfere or
distract you from performing assigned tasks or required
activities?

{xx X___|__XXX__|_XXXxx_| x__t | I
NOT AT ALL INTERFERED PREVENTED
SOMEWHAT TASK

PERFORMANCE

D2. How well could you concentrate on the assigned tasks or
required activities rather than on the mechanisms used to
perform those tasks or activities?

\ \ |__Xx |__ XXX |__XXXXX_|__XX \ |

NOT AT ALL SOMEWHAT COMPLETELY

D3. To what extent did events occurring outside the virtual
environment distract from your experience in the virtual
environment?

|__XXX__|__XXXXX_| xX__ xX t_ XX | |
NOT AT ALL MODERATELY VERY MUCH

I-5



CONTENTS (Continued)

DISCUSSION .......-+--- bc cece eee eee eee ee eee eee eee teens 23

Overview ........-66- ec cec cece cece eee asceee
Lessons Learned: Equipment
Lessons Learned: Personnel, Organization, Workload... 25
Lessons Learned: Observer and Participant Training .. 26

Lessons Learned: Simulations ........ ras eee QT
Recommendations for Future Excursions ...... wee e eens 28
CONCLUSIONS ......- eee eee eee cece eee ewes ery eee ewe eee 29
REFERENCES ......+-66 eee c er eecees bee eee eee eee e eee eee eeee 31
LIST OF ACRONYMS 2... eee cece ee eee eee reece reece eee eee eeee 33
APPENDIX A. Observer Worksheet .........--.---- see wee eeee eee AL
B. STP21 Questionnaire ...... eee eee eee ee eee tenes B-1
C. Health Questionnaire ............ eee c ecw e cece neces c-1
D. Symptom Checklist ........... sce e ee ee eee weeeeeee. Dol

E. Human Factors Equipment and Functions
Questionnaire wo. cece eee ee ee ee eee eee eee ee eee E-1
F. Learning to Use the Equipment ............--- weeee Fol
G. Simulator Questionnaire ..... eee eee eee eee eee eee G-1

H. Results of the Participant Immersive Tendencies
Questionnaire ............. ewe ecw eee e reser eeeees H-1

I. Immersion Questionnaire Results ............ weeee. I-l

LIST OF TABLES

Table 1. Participant Experience with Relevant Technology...... 11
2. Ratings of COVER Tasks (Means) ..........- a eee eee eee .13
3. Ratings of Melios Tasks (Means) .......-.e.eeeee ‘veeeee 14
4, Ratings of Map of the Future Tasks (Means) ..........-- 15



baseline (pre) and the post-ratings. Initially only the symptom
blurred vision was rated as high as “moderate;” in all other
instances symptoms were rated “none” or “slight.” In the post-
simulation ratings, of the 16 symptoms, 8 were rated at least
once as slight or moderate. Slight fatigue was the most
frequently reported symptom, followed by eyestrain, fullness of
the head, and blurred vision. The one instance of moderately
blurred vision was reported by the same individual who had
reported that problem on the pre-simulation questionnaire for
that day.

Participants reported in the group interview that they felt
they would be unable to perform this type of activity for 8-10
hours. Participation in the simulation produced some symptoms of
discomfort among the participants. Fatigue, eyestrain, and a
feeling of pressure in their head were common symptoms. No one
considered symptoms to be severe.

Table 9

Pre- and Post-Simulation Self-Ratings of Severity of Simulator
Sickness Symptoms (Percent)

Symptom % None % Slight |% Moderate
Pre Post | Pre Post | Pre Post

Fatigue 719 37 21 62

Eye strain 96 54 4 37 0 8
Fullness of the head 92 67 8 33

Blurred vision 92 88 4 8 4 4
General discomfort 96 87 4 12

Headache 96 100 4 0

Dizzy (eyes open) 100 96 0 4

Difficulty concentrating | 92 96 8 4

Difficulty focusing 92 96 8 4

Personnel, Skill, and Training Requirements

Participants were asked what prior experience or training
helped in these exercises. One factor noted was computer
experience. As discussed earlier, all participants reported at
least some experience with laptop computers, and four of the six
reported experience sending and receiving E-mail. Although they
were not asked about experience with computers in general, it was
obvious that they were quite familiar with the basics of
interacting with computers: cursor control, icons, pull-down
menus, and display windows. A related factor was experience with

22



Table 8.

Ratings of Non-Equipment-Specific Tasks (Means)

Task N Frequency |Difficulty
Keeping track of 33 3.84 3.21
multiple targets

Doing multiple tasks 26 3.61 2.88
simultaneously

Remembering to make 29 2.29 2.77
corrections

Switching among displays 23 3.08 2.49
Maintaining situational 30 4.06 2.47
awareness

Communicate (Team & 24 3.48 2.19
Higher Headquarters)

Coordinate with team 24 4.45 2.17
Imember

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty Key: 1-Very Easy; 2-Easy; 3-Neither Easy nor
Difficult; 4-Difficult; 5-Very Difficult

System Integration

Since the simulation was created largely by building on
existing equipment, it is not surprising that some instances of
incompatibility and incomplete integration of components were
observed. There was no way to hand off a target from the MOF to
the Grunt. Early in the week, the MOF showed direction to
targets in degrees, while the COVER and Melios reported viewing
direction in mils. Changing the MOF readout to mils was simple
and could be done using the MOF user menus. However, until
someone who knew how to make the change was made aware of the
need for it, the participants used plastic protractors, marked in
both mils and degrees, as a job aid to make the conversion.

Another problem that participants noted was the difficulty
of correlating the various terrain representations. The paper
map, MOF, ModSAF, and the visual displays (out the window, COVER,
and Melios) each used similar but slightly different
representations.

Team Organization and Workload

One of the critical issues related to team organization is
the role of the team leader. While he had overall responsibility
for the functioning of the targeting elements, he was often out

19



SMALL TEAM PORTAL INTO THE 21ST CENTURY—SPT21

EXECUTIVE SUMMARY

Research Requirement:

As a part of the 1996 Defense Science Board (DSB)
Summer Study “Tactics and Technology for 21st Century
Military Superiority,” the Institute for Defense Analyses
Simulation Center, assisted by behavioral scientists from
the U.S. Army Research Institute for the Behavioral and
Social Sciences, conducted a set of excursions designed to
provide analytical insights about concepts and technologies
being considered for small team operations on the DSB’s
conceptual 21st Century battlefield.

The DSB focus was on the concept of using technology to
enable small, rapidly deployable forces to accomplish
missions previously only available to larger forces. An
additional area of interest was the viability of a virtual
simulation for concept exploration.

Procedure:

This report documents a series of exercises conducted
in a virtual simulation environment using man-in-the-loop
virtual excursions into the 21st Century battlefield. U.S.
Army and Marine Corps personnel used specially designed
devices in a virtual simulation facility to test concepts
about the capabilities of small (3- to 12-man) teams
operating in a sensor rich environment. In addition to
computer recorded performance data, behavioral and tactical
observers documented soldier performance and interactions
with specific equipment.

Findings:

Combat effectiveness, although hampered by constraints
from present day and prototype equipment, was enhanced
through sophisticated communication devices and computers.
Personnel were able in the virtual environment to perform
tasks similar to those that might occur in a future
battlefield scenario. A benefit of the simulation was the
ability to portray future missions with prototype equipment
to determine potential for success. An additional benefit
came from the ability to provide man-in-the-loop human
performance data in the initial stages of concept
exploration.

vii



FOREWORD

This report presents behavioral observations and human
performance analyses obtained by the U.S. Army Research
Institute for the Behavioral and Social Sciences (ARI) in
support of a concept exploration conducted by the Institute
for Defense Analyses (IDA) Simulation Center. As a part of
the 1996 Defense Science Board Summer Study “Tactics and
Technology for 21st Century Military Superiority,” IDA
sponsored a set of excursions into the 21st Century virtual
battlefield. ARI provided technical advisory service (TAS).
This assistance, related to but not a part of the ARI work
program, is an important part of the Army and Department of
Defense’s training and personnel performance research and
development mission.

Personnel from three ARI research units (the Infantry
Forces Research Unit at Fort Benning, the Simulator Systems
Research Unit at Orlando, and the Advanced Training Methods
Research Unit in Alexandria) worked together with IDA
personnel on the virtual simulation-based excursion entitled
“The Small Team Portal Into the 21st Century (STP21).” Data
collected from the virtual environment’s conceptual
battlefield provided insight into future small team
capabilities and demonstrated the value of virtual
simulation environments for concept testing.

ZITA M. SIMUTIS EDGAR M. JOHNSON
Technical Director Director



The overall intent was to specify targets and their priority
for destruction, using the varying sensor capabilities available
to the U.S. team. Measures of effectiveness included target
detection, identification and request time, numbers of targets
attacked, area of control, workload and task management.

The players were given overall rules of engagement with
instruction on positive identification of targets to preclude
civilian casualties. Players were to observe the battlefield,
call for and adjust fires on valid targets, and designate targets
as appropriate. They were to report battle damage, coordinating
with each other and the intermediate leader. The intermediate
leader’s role was to exercise command and control over the
targeting element, to plan the operation, and to service targets
if appropriate. The focus was on the small team portals.

The primary terrain data base was the wooded mixed terrain
of Hohenfels, Germany, shown in the Synthetic Theater of War,
Europe (STOW-E) database. Some excursions employed the Hunter-
Ligget desert database of 29 Palms, California, or the simulated
urban terrain of the McKenna MOUT Site at Fort Benning, Georgia.
All exercises were in daylight mode. Potential threat targets
were tank and/or BMP platoons, trucks, and dismounted infantry.

Data Collection

Automated data collection provided by the computer system
provided information on target presentation, exposure times, time
to hit and kill, calls for fire, and battle damage assessment.
Data were collected for both portals, and for all sensor suites
and weapon systems. These data, transparent to the players, were
collected by IDA’s distributed interactive simulation network to
supplement the observational data collected by tactical and
behavioral observers.

The operational and behavioral observers maintained
worksheets (Appendix A) to record elapsed times and significant
events during the excursions. Verbatim comments from the players
were also noted, as well as the observers’ informal
interpretations of what was happening. Observer data sheets were
collected daily, and their content linked to the computer
generated data. Video coverage was provided by a film crew that
was making a demonstration tape to document the simulation
exercises and some after action reviews for DSB personnel.

Questionnaires
Several questionnaires were developed to elicit information

from the participants before and after the excursions. The STP21
Questionnaire (Appendix B) obtained background data on previous



SMALL TEAM PORTAL INTO THE 21ST CENTURY--SPT21

CONTENTS

INTRODUCTION ........--- Srey .

Excursion Concept .
Army Research Institute Role. .
ARI Perspective on the Value of Simulation ............- wee 2

APPROACH AND PROCEDURES ....- cece eee e eee r reece eee reece eeeeccseee 3

Simulation Center Facility
Participants ..... Serra
General Concept ..... scene
Facilities and Equipment ..
Excursion Scenarios
Data Collection....

QuestionnaiLes ... cece eee eee ee eee ee ee eee reece eee eeee

After Action Reviews and Interviews ............+2-eee
RESULTS .....eeeeeeeeee i iy 11
ParticipantS .... ccc cece e eee eee eeeeeene 11

Participant Immersive Tendencies ..

Equipment Use .......
COVER ....-.--
Melios ..........-.
Map of the Future
Treadport ........
Grunt 2... cece rece cece reereee
Dial-A-Tank Surrogate Vehicle

System ISSUES .... cece eee eee eee
Non-equipment-Specific Tasks.
System Integration ...........
Team Organization and Workload...
Suggested System Improvements ....

Situational Awareness ..... se eeeee
Fatigue and Side Effects ............
Personnel, Skill, and Training Requirements ............6- 22

ix



or more targets, even if azimuth and range were known. They did
not slew to the approximate target location under normal
magnification, then increase the magnification; they later
expressed a desire for a zoom capability on the COVER.

Melios

For Melios (see Table 3), as with the COVER, and as
discussed above, the most difficult task was distinguishing
friendly and neutrals. Seven tasks were performed with both the

COVER and the Melios. Five were rated as more difficult on the
Melios, and two rated more difficult with the COVER.

Table 3.

Ratings of Melios Tasks (Means)

Task Difficulty Frequency Learning
Difficulty
Distinguish friendly and 4.50 2.75 NA
neutral
Designate targets 3.67 3.00 2.67
Lazing 3.25 4.00 NA
Perform BDA 3.00 2.75 2.00
Detect targets 2.75 4.00 2.08
Identify targets (name 2.50 3.75 2.50
them)
Prioritize targets 2.00 3.00 NA
N=5

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-Very Difficult

This may represent chance variation, but it may also reflect
the following. COVER was employed while the participant was
seated in a chair at a desk but the Melios was employed while the
participant was walking, standing or sitting on the Treadport,
and was frequently devoting some portion of his attention to
maintaining or changing his position. This may have made the
Melios tasks appear more difficult. Second, the Melios is a
fairly heavy hand-held device (about 6 pounds) and holding it at
eye level for an extended period caused fatigue and instability;
at Portal 2 the participant’s elbows could be braced on the table
for support.

14



RU Ref NSU

U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1704

Small Team Portal Into the
21st Century--STP21

Margaret S. Salter, Bruce W. Knerr, Donald R. Lampton,
Gene W. Fober, and J. Douglas Dressel
U.S. Army Research Institute

December 1996

Approved for public release; distribution is unlimited.



Appendix D

Symptom Checklist

Instructions: Please indicate the severity of symptoms that apply
to you right _now by circling the appropriate word.

1. General discomfort

2. Fatigue

3. Headache

4. Eye Strain

5. Difficulty focusing

6. Salivation increased

7. Sweating

8. Nausea

9. Difficulty concentrating
10. *"Fullness of the Head"
11. Blurred Vision

12. a. Dizziness with eyes open

b. Dizziness with eyes closed

13. Vertigo
14, **Stomach awareness
15. Burping

16. Other:

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

-None

Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight
Slight

Slight

Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate

Moderate

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

Severe

* Fullness of head is a sensation of pressure in the head, just short of

a headache.

** Stomach awareness is usually used to indicate a feeling of discomfort

which is just short of nausea.



Small Team Portal Into the 21st Century—STP21

Introduction

In response to a request from the Under Secretary of Defense
for Acquisition and Technology, the Defense Science Board (DSB)
initiated a 1996 Summer Study “Tactics and Technology for 21st
Century Military Superiority.” The overall DSB focus was on the
concept of enabling relatively small rapidly deployable forces to
accomplish missions heretofore only possible with much larger and
massed forces. Small (3-12 man) teams, specially equipped and
trained, would achieve enhanced combat capability from remote
sensors and weapons. Questions revolved around mission
expectations, operational and technical capabilities to enhance
team effectiveness, flexibility, and survivability.

The DSB requested assistance from, among others, the
Institute for Defense Analyses (IDA) Simulation Center in
Alexandria, VA. IDA was to provide analytical insights about
concepts and technologies being considered for small team
operations on the DSB’s conceptual 21st Century Battlefield
(2010-2015 timeframe) by developing a virtual simulation system
to support man-in-the-loop excursions. The system had a
simulated intermediate leader portal, two small team portals, and
control cells representing higher headquarters, support elements,
and computer generated forces.

Excursion Concept

The study of rapidly deployable, specially equipped teams
supported by remote sensors has several target audiences and
proponents. The IDA simulation was named Small Team Portal into
the 21st Century (Virtual) Battlefield, or STP21. Related
programs include the U.S. Marine Corps Sea Dragon, the Training
and Doctrine Command (TRADOC) Army After Next, the U.S. Army
Infantry School Rapid Force Projection Initiative (RFPI), and the
Defense Advanced Research Projects Agency (DARPA) Small Unit
Operations (SUO).

IDA’s goal was to assess small unit battlefield operations,
and at the same time, to evaluate the STP21 virtual simulation as
a tool for studying future operations. More specifically, the
STP21 exercise was designed to investigate the ability of small
independently operating units to remotely place precision fires
on long range targets, using sophisticated sensor suites and
digitized equipment. In addition, the intent was to assess the
ability of small groups of soldiers, in a semi-autonomous mode,
to perform functions formerly done by a larger force with more



CONTENTS

Table 5.

Figure 1.
2.
3.

4,

(Continued)
Page
LIST OF TABLES (Continued)
Ratings of Treadport Tasks (Means) .....-+++++--eee wee. 16
Ratings of Grunt Tasks (Means) .....--.-seeeeeeeee wees 17

Ratings of Dial-A-Tank Tasks (Means) ..... esse eee eee ee 18
Ratings of Non-Equipment-Specific Tasks (Means) ...... 19
Pre- and Post-Simulation Self-ratings of Severity of

Simulator Sickness Symptoms (Percent) ..........e+2eee 22

LIST OF FIGURES

STP21 Simulation Center concept exploration facility. 4
STP21 simulation facility layout
Portal Lviv cece eee cece eee eee eee eee

Portal 1.......

xi



7. How completely were you able to actively survey or search
the environment using vision?

I [xx | xX XXX __XXX__ aa
NOT AT ALL SOMEWHAT COMPLETELY

8. How compelling was your sense of moving around inside the
virtual environment?

to XKXX__I __XXXXX_| XX, { x I l
NOT MODERATELY VERY
COMPELLING COMPELLING COMPELLING

9. How closely were you able to examine objects?

1 |__XxXxx__|__ XX Xl XX XXX_| I
NOT AT ALL PRETTY VERY
CLOSELY CLOSELY

10. How well could you examine objects from multiple
viewpoints?

| x | Xx |__XXX__|_XXXX_ |XX Xd |
NOT AT ALL SOMEWHAT EXTENSIVELY

11. How involved were you in the virtual environment
experience?

| | | X__|__XXxXX_|_XXXxXX__]__XX__|___d
NOT MILDLY COMPLETELY
INVOLVED INVOLVED ENGROSSED

12. How much delay did you experience between your actions and
expected outcomes?

} | I X___|__XXXXX_|__XXXX__|__XK__| |
NO DELAYS MODERATE LONG

DELAYS DELAYS

13. How quickly did you adjust to the virtual environment
experience?

| | | | X__|__XXX___| XXXXXXX_] x
NOT AT ALL SLOWLY LESS THAN
ONE MINUTE

I-4



Appendix A

Observer Worksheet

. OBSERVER DATA COLLECTION SHEET - STP21 Excursions

Observer Name Date/Time

Location: STP1 (treadport) STP2 (dial-a-tank) Leader Station
“jogger” “biker” “walker”

Excursion Number Trial Number Probe Number

Data Base: mixed___ desert__ urban __ day/night
Equipment: Dial-a-tank / Foot Pedal Treadport
MOF Grunt II PDA
Cover w joystick GATH w melios

STARTEX EXDEX . .
TIME _| EVENT NOTES/OBSERVATIONS



Situational Awareness

It appears that the use of the various technological items
both increased and decreased participant situational awareness.
By providing an integrated picture of the friendly and the enemy
situation (as could be determined from various intelligence
assets), they provided good “global” situational awareness, i.e.,
the big picture. However, by focusing their attention on a few
display screens during periods of high workload and stress they
decreased “local” situational awareness. Several instances were
observed in which enemy were in clear view on the “out the
window” displays, but were not detected. In these high workload
situations, participants tended to focus attention on their
device displays, and paid little attention to the simulated “out
the window” view. This is a common phenomenon called
“attentional narrowing” (Endsley, 1995).

There were numerous reasons for this that might not apply in
combat. First, given the tactical situation, the out-the-window
view was a low priority information source. The soldiers needed
to attack the targets beyond unaided visual range. Second,
close-in targets were rare, so there was little reinforcement for
using the out-the window view. Third, the COVER could be used to
provide the same information with less disruption of ongoing
activities. Fourth, the low resolution of the simulation made it
difficult to detect targets with unaided vision. Finally, by the
time a target was detected with unaided vision, it was too late
to call in fire. No munitions could arrive in time and the team
had no organic firepower.

Fatigue and Side Effects

The results of the post-immersion questionnaire are shown in
Appendix I. There was a slight tendency for the participants to
report more immersion as they became more familiar with the
system. Scores for the second administration (day 4) were
generally higher than those for the first (day 2). There was
also a wide range of responses to most of the items, reflecting
substantial individual differences in how the participants
responded to the simulation. The general tone of the responses
was that they were able to become involved in the virtual
environment experience, adjusted quickly to it, became at least
reasonably proficient in moving and interacting with it, and
most, at least occasionally, felt completely focused on the task
or environment.

The pre- and post-simulation self-ratings on simulator
sickness are presented in Table 9. (Symptoms for which there
were only ratings of “none” are omitted from the table, as is the
unused “severe” rating.) This table provides a comparison of the

21

oe smtacnsssiatiniéats



Appendix E
Human Factors Equipment and Functions Questionnaire

For each of the following areas, please use the following three scales.
Indicate your answer in the table below.

How frequently did you do this during the last scenario? (Frequency)
Never

Occasionally

Moderately Often

Frequently

Almost constantly

OeRWNe

How difficult was it to use the equipment or perform the task? (Do)
Very Easy

Easy

Neither Easy nor Difficult

Difficult

Very Difficult

UaWNE

FREQUENCY DO

Maintaining situational awareness

Switching among displays - from MOF to
PDA to big screen, etc.

Keeping track of multiple targets

Remembering to make last minute
corrections per FSO instructions

Doing multiple tasks simultaneously

Dial a Tank Surrogate Vehicle

Accelerating vehicle

Braking vehicle

Stopping vehicle

Steering vehicle

Maintaining orientation

Navigate in virtual environment

Treadport

Walking on the treadport

Maintaining orientation +

Navigate in virtual environment

Maintaining balance

Doing something else while walking

Map of the Future
Maintaining map orientation
Seeing the map
Understanding map graphics
Keeping track of things on map
Reading the map

Using the map



of the information loop. He did not prioritize targets or fire
requests and did not know when subordinates requested fires using
the Grunt. Despite this, the participants felt that the leader
had too much to do to also be a part of a subordinate element.
(These excursions did not address other team members responsible
for security, logistics, etc.) The most efficient structure of
the targeting element of a team was found to be three persons -~
Leader/MOF Operator, a Vehicle Driver/COVER Operator and Grunt
Operator.

Workload during the excursions varied from long periods
during which no targets were visible to periods during which
multiple weapons strikes were in flight against enemy targets.
Participants reported difficulty managing the latter situation.
It was difficult to remember which mission was being fired
against what target, and where the target had been and was likely
to be when the munitions arrived and designation (for a smart
weapon) or battle damage assessment was likely to be required.
Even with the viewing elevation provided by the COVER, a target
was unlikely to remain in continuous view for the duration of a
fire mission. It tended to disappear behind a woodline or
terrain elevation. If multiple targets were thought to be in the
area, the COVER would search for additional targets after a fire
mission had been requested, and then, at the designated time, try
to relocate the first target.

As the number of targets increased, so did the complexity of
this operation. One group used paper and pencil to record target
number, type, and last known location. The only automated
support in this operation was an algorithm that predicted target
location based on a straight line, constant speed, extrapolation
from two consecutive lasings. Unfortunately, land combat
vehicles rarely travel in a straight line at a constant speed for
the duration of a 10 to 20 minute engagement cycle.

Suggested System Improvements

The system improvements that the participants suggested
focused primarily on ways to help them deal with periods of high
workload. The first was the development and use of “fire and
forget” weapons or ones with automatic tracking sensors. A less
sophisticated, partial solution would be to provide a capability
on COVER to memorize and return to a previous location, much like
a “last channel” button on a TV remote. Another solution would
be to provide a way to indicate, on the fire request, that you do
not expect to be able to maintain line of sight to a target.

This would alert the FDC not to fire a weapon that required
designation. A final proposed system improvement was the
addition of thermal and infrared capability to the COVER.

20



Utilization of Findings:

The results of these excursions were briefed back to
the DSB in August 1966. Intermediate briefings and
demonstrations in the virtual simulation facility were
provided as in-progress reviews and to sponsors throughout
the course of the research. It is apparent that the
potential for future experimentation and concept exploration
using the techniques of virtual simulation offers many
benefits.

viii



Results

Participants

Serving in the portals were six male Marine Corps second
lieutenants. All were in the active component, and were in
student status. All had BA or BS degrees. In terms of
experience, there were two distinct groups of three each. The
members of the first group had recently completed Officer
Candidate School (OCS) and had a mean of seven months of military
service. The members of the second group, who were awaiting the
start of OCS, all had prior enlisted service (mean = 7 years, 10
months). Four of the six had had some forward observer training.
None were experienced in navigating in simulated terrain
databases. None had been to Desert Shield/Desert Storm or to a
Combat Training Center.

Their experience with the technology used or simulated in
these excursions is shown in Table 1. The generally high level
of experience with laptop computers, and to a lesser extent,
sending and receiving E-mail, is worthy of note.

Table 1

Participant’ Experience with Relevant Technology

Experience
Technology None Limited | Moderate | Considerable
Global Positioning 2 3 1 0
System (GPS)
Laser range finders 5 1 0 e)
Target designators 5 1 0 0
Helmet mounted 5 1 0 0
displays
SIMNET-type 6 0 ~ 0 ie)
environments
JANUS 6 ) 0 0
Sending/receiving E- 2 1 2 1
mail
Laptop computers 0 2 2 2
Thermal sights 3 3 0 0

Note. The numbers in the cells are the number of participants
reporting that level of experience.

11



15. Were you involved in the
scenarios to the extent that you lost
track of time?

3.58

16. Were there moments during the
scenarios when you felt completely
focused on the task or environment?

17. How easily did you adjust to the
control devices used to interact with
the virtual environment?

4.58

Dl. How much did the visual display
quality interfere or distract you from
performing assigned tasks or required
activities?

D2. How well could you concentrate on
the assigned tasks or required
activities rather than on the
mechanisms used to perform those tasks
or activities?

D3. To what extent did events
occurring outside the virtual
environment distract from your
experience in the virtual environment?

D4. Overall, how much did you focus
on using the display and control
devices instead of the virtual
experience and experimental tasks?

2.67

Note. Scale values range from 1 to 7, with 1 indicating low
immersion and 7 indicating high immersion.

I-2



it is not possible to obtain detailed quantitative data on
operator performance. The simulation afforded the opportunity to
make changes to items of hardware and software during the
exercises.

Demonstrating concepts. Simulation can be used to
demonstrate concepts, as long as participants know the intent and
limitations of the simulation. Participant training is required
to ensure that the simulation will be able to show the value of
each piece of equipment.

Placing humans in the loop. It also showed the value of
putting the human back in the loop. Modeling can quickly be
invalidated by individual differences in performance. Operators
take shortcuts and make mistakes.

Increasing innovation. Soldiers in the field environment
are often hampered by what they already know (“we always do it
this way”) because there is a frame of reference for everything.
In a simulation, reality is suspended, and participants may be
more open to new ideas, often on some very basic things.

Simulator sickness was not a serious problem. The SSQ
scores were lower than levels observed after initial training
with a tank driver simulator (Lampton et al., 1995) and in
experiments in virtual environments using helmet mounted displays
(Lampton et al., 1994). The symptoms observed here may have
resulted from several factors. Any simulator that portrays self-
motion may produce simulator sickness. In both the WISE and the
vehicle screens, motion was depicted periodically and was
observable by all the team members at each particular station,
not just the person walking or running on the WISE, or driving
the vehicle. Also, because of the low level of illumination of
the displays and ambient light, displays were frequently
difficult to view. This may have contributed to eye strain or
fatigue.

Recommendations for Future Excursions

Part of the value of this simulation is in learning how to
do it better another time. If these excursions were repeated,
with different equipment characteristics, the results might be
very different.

From a system design standpoint, the equipment that supports
STP21 must be an integrated system. The human should not be in
the role of the conduit passing information from one component to
another. Effective information management requires feedback.
Units need reporting SOPs. Update reports at random intervals
overload even experienced personnel.

28



Despite attempts to use digital communication throughout,
voice communication was faster, more efficient, and probably
caused fewer errors. The soldiers relied on the familiar system,
the one from which they received immediate and reliable feedback.
Feedback needs to be designed into the digital communications as
well.

Conventions are needed for tracking messages. To assist in
information management, E-mail must indicate senders, priorities,
and the time. Changes needed are better ways to accept, send,
save and delete messages, as well as a better way to display
them. Some preformatted messages are mandatory.

There must be a way to designate or identify a potential
target and follow it, assessing speed, direction, etc., without
having to go through a laborious device-based process. Each time
attention is diverted from the field of view, potential
information is lost.

Equipment changes need to be directed toward saving time and
making processes more efficient, to reduce errors, leave fewer
things to memory, improve the operator’s ability to use the
devices, and maintain situational awareness. One device should
be able to combine the functions of the Grunt, the MOF, and any
kind of PDA, with only one set of protocols to learn. This would
also ease training.

Conclusions

Overall, this research project reinforced the view that
virtual simulations can greatly enhance the concept development
process by providing the capability to assess mission performance
before a system is fielded. Modeling cannot incorporate the
dynamics of a small team the way a simulation does.

The STP21 simulation was hampered by present day equipment,
and perhaps by preconceived ideas. Borrowed equipment did not
link well in the simulation environment, but simulations are
ideal for first steps. Prototype equipment, despite issues
related to hardiness, interconnectivity, and linking, is
characterized by the inherent ability to change it.

Other limitations of these excursions as STP21 predictors
should be noted. First, equipment components were not designed
to function as an integrated system, and their future design
might be very different. Procedures for employment were not
fully developed and tested, and personnel were not thoroughly
trained. They had not adequately learned and practiced the basic
procedures, e.g., scanning and target identification. As a

29



15. Were you involved in the scenarios to the extent that you lost
track of time?

| 1 | I i ! l
NOT AT ALL SOMEWHAT COMPLETELY

16. Were there moments during the scenarios when you felt completely
focused on the task or environment?

I | 1 | | i ! |
NONE OCCASIONALLY FREQUENTLY

17. How easily did you adjust to the control devices used to interact
with the virtual environment?

| | I | | I | |
DIFFICULT MODERATE EASILY

Section 2 - DISTRACTORS

1. How much did the visual display quality interfere or distract you
from performing assigned tasks or required activities?

| | ! | | | | |
NOT AT ALL INTERFERED PREVENTED
SOMEWHAT TASK PERFORMANCE

2. How well could you concentrate on the assigned tasks or required
activities rather than on the mechanisms used to perform those tasks or
activities?

| | i i I | |
NOT AT ALL SOMEWHAT COMPLETELY

3. To what extent did events occurring outside the virtual environment
distract from your experience in the virtual environment?

! | | | I i | |
NOT AT ALL MODERATELY VERY MUCH

4. Overall, how much did you focus on using the display and control
devices instead of the virtual experience and experimental tasks?

| | | | | | | |
NOT AT ALL SOMEWHAT VERY MUCH

G-3



Table 6.

Ratings of Grunt Tasks (Means)

Task Difficulty Frequency Learning
Difficulty

Adjust fires with PDA 3.25 1.37 2.93

Composing text_messages 2.63 2.06 2.72

Making PDA do what you 2.30 3.87 2.28

wanted

Understanding PDA menu 2.11 4.06 1.89

Remembering PDA 2.08 3.67 2.17

functions

Understanding PDA 2.06 3.53 NA

symbols

Seeing PDA display 2.03 4.17 NA

Realizing a message is 1.72 3.14 2.17

waiting

N= 14

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-Very Difficult

sent, received by another participant, or that a request for fire
had been acted upon. The last was not a Grunt problem per se,
but reflected the initial lack of a detailed SOP (the FDC should
indicate when a request for fire has been filled), and the
workload placed on the role players in the simulation.

Dial-A-Tank Surrogate Vehicle -

The Dial-A-Tank (see Table 7) was considered easy to use and
to learn to use.

System Issues

System issues are related to the general characteristics of
the system, simulation, or setting, or to multiple items of
equipment. They include non-equipment specific tasks, system
integration, suggested system improvements, and team organization
and workload. Data on systems issues come from the group
interview, observer notes, and to a lesser extent, the few
questionnaire items that were not related to items of equipment.

17



APPENDIX I

Immersion Questionaire Results

Table I-1

Immersion Question Means For The First And Second

Administrations

Stem

Admin
1

Admin
2

1. How natural did your interactions
with the environment seem?

3.00

3.83

2. How much did the visual aspects of
the environment involve you?

4.50

4.17

3. How natural was the mechanism
which controlled movement through the
environment?

3.00

3.83

4. How compelling was your sense of
objects moving through space?

3.67

4.42

5. How much did your experiences in
the virtual environment seem
consistent with your real world
experiences?

2.83

6. Were you able to anticipate what
would happen next in response to the
actions that you performed?

7. How completely were you able to
actively survey or search the
environment using vision?

8. How compelling was your sense of
moving around inside the virtual
environment?

4.00

4.00

9. How closely were you able to
examine objects?

3.33

10. How well could you examine
objects from multiple viewpoints?

3.33

11. How involved were you in the
virtual environment experience?

12. How much delay did you experience
between your actions and expected
outcomes?

13. How quickly did you adjust to the
virtual environment experience?

14. How proficient in moving and
interacting with the virtual
environment did you feel at the end of
the experience?

I-1



ACKNOWLEDGMENTS

The authors wish to extend thanks to Neale Cosby, Gary
Coe, Jim Madden, Larry Mengel, Rick Wright, Bennett Dickson,
and Chris Turrell for their help and support, and to the
STP21 Marines who will be the soldiers of the twenty-first
century.

di



Appendix C
Health Questionnaire

1. Are you in your usual state of alertness? YES NO
2. Do you have any illness? YES NO
If yes, please state the nature of your illness (flu, cold,

headache, sinus problems, etc.).
3. Please indicate medications you have used in the past 24 hours:

(You need not include prescription medication routinely taken on a

daily basis.)

6.

(a)
(b)
(c)
(d)
(e)
(£)

NONE

Sedatives or tranquilizers
Aspirin, Tylenol, other analgesics
Anti-histamines

Decongestants

Other (specify):

How many hours of sleep did you get last night?

Was this amount sufficient? YES

Did you do any PT in the last 24 hours? YES

___ (Hours)

NO

NO



Appendix B
STP21 Questionnaire

SECTION I (background information)

1. Specialty Code/Branch/MOS

2. Rank

3. Time in service Years Months

4. What is your current duty position?

5. Which military schools have you attended? (e.g., basic and specialty
courses, etc. List all.)

6. Civilian education: degree major

7. Do you have difficulty distinguishing colors (color blind, color
weak, etc.)? Yes No

8. Which of the following duty positions have you held? Check all that
apply, and as appropriate, whether at battalion or brigade level.

TOW/Tank/BFV Gunner Tank/BFV/LAV Commander
Platoon Leader Specialty Platoon Leader (specify)
Company XO Company Commander

Staff Officer (specify)

Engineer officer NBC/Chemical officer ADAO
CESO/SIGO LNO/ANGLICO Other (specify)
FO FSO FSCOORD

If you have had any experience or training as a forward observer, or in
requesting/calling for fires, please describe this experience -- duty
position, location, etc.

9. Have you participated in rotations at any CTC? How many — where?

10. Did you participate in Desert Shield/Storm? Yes No
Describe. ~~



General Concept

Two weeks of data collection were planned. According to the
original plan, an overview and training on the equipment would be
followed by computer-based simulation exercises during the
remainder of the week. Each day’s trials would end with after
action reviews (AARS) to elicit lessons learned. The second week
was to repeat the first with different test subjects. Each day
was to be increasingly more complex, to test the limits of the
simulation and to develop engagement timelines to measure
situational awareness and combat effectiveness. The opposing
force (OPFOR), terrain, and sensors were to be varied, as well as
team size and mobility. ARI was to address the human factors and
behavioral interface issues. Specific items of interest were
operator workload, equipment problems that impacted on test
subject ability to conduct the mission, and simulator sickness or
fatigue.

Difficulties interconnecting the prototype equipment forced
changes to data collection and the overall plan. The first week
became a pilot week to resolve problems and make equipment
changes. Standing operating procedures (SOPs) for equipment use
and radio communications were developed. The second week
provided data and performance measures related to the EEAs.

Facilities and Equipment

The Simulation Center consisted of an exercise control area
(ECA), two small team portals, and an intermediate team leader
portal/station, plus tactical (military) and behavioral observer
stations. The small team and leader stations were networked to
the ECA that supported the team portals. (See layout at Figures
1 and 2.) ECA equipment included the simulated FDC/Fire Support
Station, the OPFOR Control Center, and the Remote Sensor
Center/Intelligence Station. A briefing area was available.

‘Small Team
Small Team
Portal 1 Portal 2
Intermediate
Leader Portal

‘Observer Team

i Fire
Remote Sensor Excursion Control Remote
erStation Station ‘Station
Executive Seminar
Facility
(OA Steath Room

Figure 1. S'P21 Simulation Center concept exploration facility.



8. How easily can you switch attention from the activity in
which you are currently involved to a new and completely
different activity?

| | 1x | KX XKX_I \
NOT SO FAIRLY QUITE
EASILY EASILY EASILY

9. How often do you play games on computers?

| xX__I__ Xx | |_ xx x | |

NOT AT ALL OCCASIONALLY FREQUENTLY

10. How many different video, computer, or arcade games have
you become reasonably good at playing?

| xX |__ xx x x__i I__X
NONE ONE TWO THREE FOUR FIVE SIX OR
MORE

11. Have you ever felt completely caught up in an experience,
aware of everything going on and completely open to all of it?

| {___XxXx__| | |__ xx xl |
NEVER OCCASIONALLY FREQUENTLY

12. Are you easily disturbed when involved in an activity or
working on a task?

| |__Xxx__|_ XX xX__d | |

NEVER OCCASIONALLY OFTEN



D4. Overall, how much did you focus on using the display and
control devices instead of the virtual experience and
experimental tasks?

| | I X___|_XXXXXXX|__XXX__| xl I
NOT AT ALL SOMEWHAT VERY MUCH

1-6



Reflector Coa

7
“Treadoot Motion Piatforn

Figure 3. Portal 1.

Portal 2 (Figure 4) was a two-person, then eventually a
three-person “mobile” station, where one person could drive, in
addition to assisting in target location. The surrogate vehicle,
the Dial-A-Tank, had a steering yoke shift lever, and foot pedals
which the operator could use to control forward and reverse
motion, acceleration and braking, and direction. The terrain
data base was depicted on two large screens to the front of the
vehicle. In actuality the team was seated at a table.

LtVehicle “Walking
Accelerator Pedal

Figure 4. Portal 2.

The Melios (Mini Eyesafe Laser Infrared Observation Set) was
the sensor device used at Portal 1. A variation on current
equipment, Melios was a ten power binocular-like hand held
device. It had an embedded laser range finder that permitted the
operator to laze on a target, with target range and azimuth
depicted on a small screen display monitor. Target information
was relayed back to the FDC. The system featured a second laze
capability designed to predict the location of the target over
time.



REPORT DOCUMENTATION PAGE

4. REPORT DATE 2. REPORT TYPE 3. DATES COVERED (from. . . to)
1996, December Final June 1996-August 1996
4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER
Small Team Portal Into the 21st Century—STP21 5b. PROGRAM ELEMENT NUMBER
06023007A.
6. AUTHOR(S) 5c. PROJECT NUMBER
A794
Margaret S. Salter, Bruce W. Knerr, Donald R. Lampton, Gene W. 5d. TASK NUMBER
Fober, and J. Douglas Dressel 2127
Se. WORK UNIT NUMBER:
HOl
7, PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
U.S. Army Research Institute for the Behavioral and Social Sciences
ATTN: PERI-IJ
5001 Eisenhower Avenue
Alexandria, VA 22333-5600
9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM
U.S. Army Research Institute for the Behavioral and Social Sciences ARI
5001 Eisenhower Avenue
Alexandria, VA 22333-5600 11, MONITOR REPORT NUMBER
Research Report 1704

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14, ABSTRACT (Maximum 200 words):

Behavioral scientists from the U.S. Anny Research Institute for the Behavioral and Social Sciences assisted the Institute for Defense Analyses
(DA) Simulation Center in conduct of excursions into the virtual 21st Century battlefield. The 1996 Defense Science Board (DSB) Summer Study
requested analytical insights about concepts and technologies being considered for small team operations on the DSB’s conceptual 21st Century
Battlefield. The DSB focused on the concept of using technology to enable small, rapidly deployable forces to accomplish missions previously only
available to larger forces.

Exercises were conducted in a virtual simulation environment. U.S. Army and Marine Corps personnel used specially designed devices in a virtual
simulation facility to test concepts about the capabilities of small (3- to 12-man) teams operating in a sensor-rich environment. In addition to computer
reported data, behavioral and tactical observers documented man-in-the-loop soldier performance and interactions with specific equipment. Combat
effectiveness was enhanced through sophisticated communication devices and computers. Personnel were able in the virtual environment to perform
tasks similar to those that might occur in a fiuture battlefield scenario. A benefit of the simulation was the ability to portray future missions with

prototype equipment.

15. SUBJECT TERMS

Small Team Portal (STP) 21st Century Battlefield
Virtual Environment
: SECURITY CLASSIFICATION OF _

19. LIMITATION OF } 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number)
16. REPORT 17. ABSTRACT | 18. THIS PAGE

Unclassified | Unclassified | Unclassified ) Unlimited 63



restriking keys. For example, maps required a finite time to
recreate when they were moved -- impatient keystrokes did not
speed up the system. Variability in operator performance and
efficiency might also be attributed to, for example, differing
typing ability levels. Additional training would alleviate some
of these problems and improve system performance.

Scanning and searching for soft-skinned or personnel targets
at short ranges in a limited sector, and preparing for direct
fire weapon engagements, do not prepare the soldier for the wide
and long range area search required in this simulation. They
lacked scanning and searching techniques, and did not know about
being systematic, using patterns, or varying power to get better
or different perspectives. Training such as is available for
Scouts or long range surveillance units (LRSU) may be required.

Small team members need cross training on all of the
devices. This is mandatory as a casualty backup plan, but also
helps with work and rest plans, boredom, and stress reduction. A
single device (like the MOF) which can accomplish multiple
functions and can be distributed to each of the players, may be a
way of sharing workload while maintaining efficiency. SOPs are
critical; job aids may help in keeping track of information.

Lessons Learned: Simulations

Some of the most valuable lessons from simulation this far
in advance of development do not have to do with the man-machine
interface of specific items of equipment (keyboard layouts or
menus), but with interfaces among the various systems and the
load they put on the humans in the loop (manually transcribing
information from one electronic system to another). Fifteen
years ago, only a few computer scientists would have been
familiar with the windows-based interface used in these systems.
Today, it is widely understood. Whatever the advances in
computer interface technology over the next fifteen years, it
seems certain that they will render obsolete any interface
changes made today. However, basic human capabilities and
limitations will remain unchanged, and systems designs will have
to allocate functions between humans and computers in a way that
best takes advantage of those capabilities and limitations.

The exercises demonstrated that simulations can contribute
to the development of new concepts in the following ways:

Improving equipment design. Using simulation for
development helps improve the design of equipment and systems.
The simulation identified operator interface capabilities and
problems with each piece of equipment, and the impact on operator
workload. As in the STP21 demonstration, this can occur even if

27



These issues were raised in the group interview, with
disagreement among the participants on whether the COVER or the
Melios was better. COVER provided a more stable viewing platform
and was easier to use with eyeglasses. The use of Melios was
more intuitive than COVER. The physical movements made with
Melios made it easier to return to a previous position. In this
sense, Melios contributed to better situational awareness. The
weight of Melios was criticized. The green color used to display
range and azimuth on the display tended to blend with some
backgrounds, making it difficult to read.

Map of the Future

Despite its complexity, the MOF (see Table 4) was not
perceived as difficult to use or to learn to use. Only the Call
for Fire task received a high difficulty rating. Several factors
may account for this.

Table 4.

Ratings of Map of the Future Tasks (Means)

Task Difficulty Frequency Learning
Difficulty

iCall for fire 3.21 2.73 2.71

Understanding map 2.68 4.36 NA

graphics

Make a report, using 2.65 2.74 NA

device

Keeping track of things 2.63 4.53 2.45

lon_map

Reading the map 2.49 4.47 NA

Receive a map, using 2.40 2.56 NA

device

Seeing the map 2.13 4.64 NA

Using the map 2.09 4.46 NA

Maintaining map 1.77 4.50 1.80

orientation

Learn the menus NA NA 2.34

Respond to a message NA NA 2.16

N = 24

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-Very Difficult

15



U.S. ARMY RESEARCH INSTITUTE
FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency Under the Jurisdiction
of the Deputy Chief of Staff for Personnel

EDGAR M. JOHNSON
Director

al
Technical review by

Jean L. Dyer
Larry W. Mengel
Richard K. Wright

ade by ARI. Please wp cospir
¢ the Behavioral afid Social Sciences, ~

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not return it to
the U.S. Army Research Institute for the Behavioral and Social Sciences.

NOTICES

NOTE: The findings in this report are not to be construed as an official Department of the Army position,
unless so designated by other authorized documents.



Receive a report, using device

Call for fire

Make a report, using device

Grunt II PDA

Seeing PDA display

Understanding PDA menu

Realizing a message is waiting

Remembering PDA functions

Understanding PDA symbols

Composing text messages

Making PDA do what you wanted

Adjust fires with PDA

GATH/Melios

Detect targets

Identify targets (name them)

Prioritize targets

Designate targets

Distinguish friendly & neutral

Perform BDA

Lazing

COVER

Detect targets

Identify targets (name them)

Prioritize targets

Designate targets

Distinguish friendly & neutral

Perform BDA

Using the joystick controller

Making joystick work right

Maintaining orientation

Lazing

COMMO

Communicate (Team & Higher HQ)

Coordinate with team member



computer games. Participants saw this as highly relevant. Five
reported some computer game playing, and three indicated more
than occasional game playing. For actual deployment of an STP21
team, more computer expertise would be required. Soldiers would
not only have to know how to use the systems, but also to
diagnose and correct hardware and software problems that might
arise.

Participants also reported that prior leadership experience
and tactical experience were beneficial. Forward observer
training was considered largely irrelevant because current FO
procedures are not applicable to the future digital environment’s
long range engagement cycles for remotely delivered weapons,
although it did provide familiarity with basic concepts.

Participants reported that they began to feel comfortable
with the equipment and their roles by the end of the second or
the beginning of the third day. However, it was clear from their
comments and questions during AARs and from observer notes that
they were still learning after this time. Some training was
actually conducted on the morning of the fourth day. They
expressed a need for additional and more structured training,
with specially designed training scenarios and a “coach” to
provide advice. They noted that most computer games provide
training scenarios.

Discussion
Overview

Within certain limits the concept, and the simulation of
that concept, worked quite well as the flexibility inherent in
the simulation permitted modification both to scenarios and
equipment. The EEAs were addressed, if informally.

The first objective or EEA dealt with small team situational
awareness, whether devices assisted in utilization of sensor
information, and the added value for each kind of sensor. The
second, related, EEA explored enhanced combat effectiveness,
through bringing accurate and timely fires in support of
neighboring teams and a higher headquarters.

The overall concern was whether the teams could utilize
information received from sensors and the various PDAs. Although
effectiveness varied, the teams were able to perform the required
tasks. Both information receipt and use were demonstrated.

Measures of performance included the ability to service

targets, and the timeline from presentation through calls for
fire and adjustment if needed. Timelines were difficult to

23



AAR
ARI
BDA
DARPA
DSB
ECA
EEA
FDC
FOV
GPS
IDA
ITQ
LRSU
MELIOS
ModSAF
MOF
ocs
OPFOR
PDA

PQ
REMBASS
RFPI
SIMNET
SOP
ssQ
STOW-E
STP
suo
TRADOC
WISE

LIST OF ACRONYMS

After Action Review

Army Research Institute

Battle Damage Assessment

Defense Advanced Research Projects Agency
Defense Sciences Board

Exercise Control Area

Essential Elements of Analysis

Fire Direction Center

Field of View

Global Positioning System

Institute for Defense Analyses

Immersive Tendencies Questionnaire

Long Range Surveillance Unit

Mini Eyesafe Laser Infrared Observation Set
Modular Semi-Automated Forces

Map of the Future

Officer Candidate School

Opposing Forces

Personal Data Assistant

Presence Questionnaire

Remotely Monitored Battlefield Sensor Systems

Rapid Force Projection Initiative
Simulation Network

Standing Operating Procedures
Simulator Sickness Questionnaire
Synthetic Theater of War - Europe
Small Team Portal

Small Unit Operations

Training and Doctrine Command
Walk-In Synthetic Environment

33



Table 7.

Ratings of Dial-A-Tank Tasks (Means)

Tasks Difficulty Frequency Learning
Difficulty
Steering vehicle 2.13 2.38 NA
Maintaining orientation 1.75 3.00 1.93
Navigate in virtual 1.63 2.00 2.07
environment
Accelerating vehicle 1.50 2.00 NA
Braking vehicle 1.50 2.00 NA
Stopping Vehicle 1.50 2.00 NA
Drive the Vehicle NA NA 2.07
N=7

Frequency Key: 1-Never; 2-Occasionally; 3-Moderately Often; 4-
Frequently; 5-Almost Constantly

Difficulty and Learning Difficulty Key: 1-Very Easy; 2-Easy; 3-
Neither Easy nor Difficult; 4-Difficult; 5-Very Difficult

Non-equipment-Specific Tasks

The most difficult of the non-equipment-specific tasks (see
Table 8) was keeping track of multiple targets, a result that was
also confirmed in the group interview. Closely related was the
task of remembering to make corrections. Despite the low
difficulty of communications, participants reported in the group
interview that the simulation of radio traffic was distracting.
The voice-actuated microphones, intended to carry traffic among
the elements (station and team leader, station and fire direction
center, etc.) also tended to pick up communications intended for
other, physically co-located members at the station. Some
participants found this distracting. Despite the need of the
operator of the simulated radio to listen to both radio and local
communications, the earpieces on the headphones completely
covered both ears. Thus participants tended to wear them at an
angle, leaving one ear uncovered. This eventually became
uncomfortable. Either a single-ear headphone or smaller
earpieces that do not cover the ear would have alleviated the
problem.

18



equipment. (See Coe, Madden, Mengel, & Wright, 1996, for greater
detail.)

Data collection addressed three essential elements of
analysis (EEA):

EEA 1 explored enhancement of small team situational
awareness through sensors and two computer devices: palmtop
personal data assistants (PDA), and the laptop Map of the Future
(MOF) .

EEA 2 explored enhancement of combat effectiveness by
measuring remote fires and use of specialized devices.

EEA 3 examined the utility of individual and small team
virtual simulation in concept exploration.

Army Research Institute Role

IDA requested that behavioral scientists from the Army
Research Institute (ARI) participate in the simulation exercise
as observers during the tactical scenarios. ARI was to look at
several broad areas. Issues included the small team’s ability to
manage large amounts of data, training on surrogate equipment,
man-machine interface, and identification of the skills needed to
operate on the DSB’s conceptual 21st Century Battlefield.

Five research psychologists from ARI (two from the Infantry
Forces Research Unit at Fort Benning, GA, two from the Simulator
Systems Research Unit at Orlando, FL, and one from the Advanced
Training Methods Research Unit at Alexandria, VA) participated in
the evaluation from June through August, 1996. ARI developed and
administered questionnaires to measure participant attitudes
toward and difficulties with use of the specialized equipment,
conducted interviews, and observed the scenarios.

This report documents the observations made and research
performed in support of the IDA DSB concept explorations. It
comments on four specific areas related to the concept of small
unit operations: equipment, personnel, training, and simulation.

ARI Perspective on the Value of Simulation

Virtual simulations can enhance the concept development
process by providing an opportunity to assess mission performance
with humans “in the loop,” performing tasks as they are expected
to once the system is fielded. Information about their
performance can be collected and analyzed to evaluate the concept
and identify ways to improve it. The information can vary from
empirical data about response times and performance accuracy to



SECTION II (experiences)

1. For each item in the table below, please indicate your experience
level using the following scale:

1 = none
2 limited
3 = moderate
4 = considerable
Equipment/Simulated Environments Experience Level

Global Positioning System (GPS)

Laser Range Finders

Target Designators

Helmet Mounted Displays

SIMNET-Type Environments

JANUS

Sending/Receiving E-mail

Laptop Computers

Thermal Sights

Other Simulations (name & describe)

2. Have you ever navigated in a terrain data base? Yes __ No

If yes, describe where, when, mounted, dismounted, etc.).

SECTION III (focusing)

Answer by marking an "X" in the appropriate box of the seven point
scale. Consider the whole scale when responding, as intermediate levels
may apply. For example, if your response is once or twice, mark the 2nd
box from the left. If your response is many times but not extremely
often, then the 6th (or 2nd box from the right) should be marked.

1. Do you ever become so involved in a video game that it is as if you
are inside the game rather than moving a joystick and watching the
screen?

| | ! | | I | |
NEVER OCCASIONALLY OFTEN

2. How good are you at blocking out external distractions when you are
involved in something?

| | I | | | |

NOT VERY SOMEWHAT VERY GOOD
GOOD GOOD
E-3



Appendix G

Simulator Questionnaire

Section 1 SIMULATOR REALISM

Characterize your experience in the simulation, by marking an "X" in the
appropriate box of the 7-point scale. Consider the entire scale when
making your responses, as intermediate levels may apply. Answer the
questions in the order that they appear. Do not skip questions or
return to a previous question to change your answer.

1. How natural did your interactions with the environment seem?

| | | I I | | I
EXTREMELY BORDERLINE COMPLETELY
ARTIFICIAL NATURAL

2. How much did the visual aspects of the environment involve you?

t | | I I | | |
NOT AT ALL SOMEWHAT COMPLETELY

3. How natural was the mechanism which controlled movement through the
environment?

{ | 1 | | | |
EXTREMELY BORDERLINE COMPLETELY
ARTIFICIAL NATURAL

4, How compelling was your sense of objects moving through space?

| | | | | i | |
NOT AT ALL MODERATELY VERY
COMPELLING COMPELLING

5. How much did your experiences in the virtual environment seem
consistent with your real world experiences?

Hl | | | | | | |
NOT MODERATELY VERY
CONSISTENT CONSISTENT CONSISTENT

6. Were you able to anticipate what would happen next in response to
the actions that you performed?

| | | | | | |
NOT AT ALL SOMEWHAT COMPLETELY



[ess]
cont Sain
Contro! Stations

Fire Station

Entrance

.
YX Area
i

i

orepeuuen

Figure 2. STP21 simulation facility layout.

The Fire Support Station received and responded to calls
from the team portals. The remote fire station delivered
artillery, missile, naval gunfire, and close air support in
response to preplanned fire requests or requests for fire froma
team or intermediate leader. After player calls for fire,
weapons were selected and status reported. This station was
operated by the Marine captains, assisted by an IDA civilian.
IDA also staffed the Sensor Station and the Modular Semi-
Automated Forces (ModSAF) OPFOR control center.

The intermediate leader station was a coordination cell
located in an area adjacent to one of the teams. Teams had voice
radio communication with the leader, and with each other, as well
as with the FDC. Each player had earphones for aural replication
of battlefield noises and engine sounds in addition to normal
radio traffic; only one key player at each station had radio
communication. The team portals and the intermediate leader
station equipment could be observed by the operational observers
and behavioral analysts. Observer monitors slaved to the team
equipment replicated the soldiers’ views; the virtual terrain was
easily visible from all locations.

The team stations had both disparate and common elements of
equipment. Portal 1 (Figure 3), was a two person station with a
treadmill/treadport motion platform device. The motion of the
treadport simulated the soldier walking over terrain. The
terrain, a digitized data base, was depicted in the WISE (Walk-In
Synthetic Environment). The WISE had three large screens which,
via a rear projection device, depicted the terrain and thus
appeared to surround the soldier. The soldier on the treadport
could stand still, walk or run. (The device would not support a
view of the battlefield from the crawl position.) A surrogate
vehicle was also available at this portal.