Instruments

Biographical Questionnaire. The Biographical Information Questionnaire (Appendix A)
was a multiple-choice/short answer paper-and-pencil instrument designed to document the prior
military training and experience of each soldier as well as their experience with computers and
simulations.

Real World and Virtual Environment Post Training Questionnaires. After completing
the mission rehearsal training, soldiers completed one version of this multiple choice/short
answer questionnaire (Appendix C or D). The questionnaire consisted of six items that required
participants to rate the effectiveness of the training received, rate the adequacy of time allocated
for rehearsing, state what they liked most/least about the rehearsal setting, and state how they
would modify the rehearsals to make them more effective.

Mission Rehearsal Comparison Questionnaire. After soldiers had the opportunity to
execute missions in both settings (virtual environment and real world), they completed the
Mission Rehearsal Comparison Questionnaire (Appendix E). The questionnaire consisted of six
multiple choice/short answer items on the relative effectiveness of the rehearsal settings, the
advantages and disadvantages to conducting mission rehearsals in a virtual environment, and
how to incorporate virtual environments in unit training.

Evaluation Checklist. The Evaluation Checklist (Appendix F) was completed after the
execution of each mission by two O/Cs. The checklist consisted of 17 measures (only 15
measures were used for assessing performance) of squad and individual performance for clearing
a building, e.g., number of rooms not properly cleared, failure to provide supporting fires, failure
to maintain proper dispersion at danger areas, and personnel “flagging” of squad members or
their weapons in windows or around corners. The O/Cs used the checklist to record the number
of times these events were observed.

For each mission that involved a force-on-force encounter with the OPFOR, the O/Cs
completed a formal evaluation. Each squad received four sets of ratings from each O/C, two
ratings based on real-world mission performance and two ratings based on mission performance
in the virtual environment. Later, the O/Cs reviewed replays and data files from each mission to
verify their assessments and correct any discrepancies.

Frequency counts were obtained for each task/measure (i.e., the number of instances the
squad or squad member did not correctly perform a specific task, number of friendly and OPFOR
casualties, and number of fratricides). For all measures except the number of OPFOR casualties,
higher frequency counts represented poorer tactical performance. Inspection of the collective
tasks indicated that they clearly varied in importance or criticality (see Appendix F for further
details). As a result, a scoring and weighting scheme was developed by military subject matter
experts to reflect these differences and allow all measures to be combined into a single
performance score.

Of the tasks that were identified, certain tasks were recognized as having a more
significant effect in determining the final outcome of a mission and in assessing overall unit



difficulty using the menu to execute various actions, and the narrow field of view (FOV).
Soldiers felt that these deficiencies decreased the realism of the training.

Suggestions for improving the effectiveness of mission rehearsals. Soldiers were asked
what they would do to improve the effectiveness of mission rehearsals. Comments are
summarized in Table 6. The majority of the soldiers in the RW rehearsal condition indicated that
they would not modify anything.

Soldiers with specific comments suggested that trainers alternate leadership positions
between missions to increase the squad members’ understanding of how each position affects the
mission. They also recommended increasing the number of members in each squad (from 7 to
9), increasing the training time and number of rehearsals, and conducting additional basic urban
operation skill training prior to rehearsal.

Table 6
Suggestions for Improving Mission Rehearsal Training in Real-World and Virtual-Environment

Settings

Real-World Alternate squad member leadership positions

Improve unit level urban operation skills prior to mission rehearsal
Provide more time for additional training and rehearsal

Increase the size of each squad (9-man squad)

Use more instructors :

Virtual
Environment

Remedy virtual figures sticking in walls

Separate each control button from the menu

Improve mobility of motion through the environment

Add additional urban operation skills training

Enlarge the field of view (FOV)

Create a virtual view based on head movement, not a toggle
Add maps and target references on the virtual screen
Improve grenade usability

Soldiers in the VE rehearsal condition provided a number of suggestions for improving
the rehearsal process. The biggest issue was correcting the “sticking” problem in the SVS
system. Soldiers felt that the training effectiveness of the SVS system could be greatly enhanced
if this problem was resolved. They viewed the current unstick option on the screen menu as
inadequate.

Soldiers recommended the development of an “unstick” button that is separate from the
weapons selection menu so it could be used more quickly and easily. Currently, to unstick you
must select (using the thumb switch on the weapon) the first item among a list of weapon choices
appearing on the screen. Because it is the first item, it is sometimes easy to overlook causing the
user to have to scroll back to the beginning. This provides time for the enemy to locate and
engage the soldiers. This is compounded if the soldier’s virtual limbs or weapon are sticking
through the wall revealing the soldier’s position while he is attempting to free himself.

30



Group performance on individual level I tasks. Analysis of the incident rates for the
virtual environment missions showed that more Level I task incidents were observed for the RW
rehearsal group than for the VE rehearsal group. The performance of the VE rehearsal group
showed some improvement over time. However, the RW rehearsal group’s performance got
worse over the two missions (see Figure 11).

Virtual Environment Level | Incidents

60
50 ae

40 ee
No. of 30 | IVE Reh
Incidents RW Reh
20 +
10 +

ti)

Mission Number

Figure 11. Total number of level I task tactical incidents by rehearsal
group at the virtual-training site.

Further analysis showed that Task 10, personnel “ flagging” (body exposed in windows
or head exposed around corners) accounted for the majority of Level I task incidents that were
observed. Overall, 142 flagging incidents were noted (60 incidents for the VE rehearsal group
and 82 incidents for the RW rehearsal group). Incident rate across all tasks were more frequent
in the RW rehearsal condition (94) than in the VE rehearsal condition (73).

Two tasks, task 11, weapon “flagging” (extending M16 barrel beyond cover) and task 15
(succession of command not established) had the next most frequently observed incident rates o
and 8 incidents respectively). For weapon flagging, the incident rate was roughly the same for
each rehearsal group (5 for the VE rehearsal group and 4 for the RW rehearsal group). For both
groups, all incidents were observed during Mission 1. For task 15, the overall incident rate for
failing to establish succession of command was identical for both rehearsal conditions (4).
However, all VE rehearsal group incidents occurred during mission 1 (see Appendix I, Table 1-6
for individual incident rates and point totals for all Level I tasks).

Comparison of Field Versus Virtual-Training Site Performance
Breaking down performance by task level and training site did reveal some interesting
patterns. For Level III task performance, friendly casualties were high (50 percent or higher) for

both groups across both settings. Differences in the total number of OPFOR casualties inflicted
across rehearsal condition and setting were negligible. Of the sixteen missions that were

26



Table 11

Soldier Responses to Question “How Would You Incorporate Virtual Environments in Your
Unit’s Training?”

As pre-training before training in the real world

To familiarize unit members with environment

For mission planning

For leadership development

To introduce urban operation skills before training

For teaching mission tactics

For building teamwork within a squad

To explore different mission environments and scenarios

To retain proficiency in urban operation skills

When urban operation training sites are not available and time/resources are limited
For alternating training environments with different training objectives
To improve decision making-skills

To develop unit SOP

For rehearsing non-urban operation mission scenarios

To improve marksmanship

For reconnaissance training

Discussion

The objective of this research was to compare the effectiveness of rehearsing in a virtual
environment versus rehearsing in a more traditional field setting. The two questions addressed
by this research focused on whether skills acquired in the virtual environment transfer to the real
world and how real-world performance differs as a function of the training site (i.e., rehearsal
setting).

Transfer of Skills Acquired in the Virtual Environment to the Real World

One could argue that since the VE rehearsal group was able to rehearse in the identical
building sections they would encounter in the real-world missions with a live OPFOR, real-world
performance should be superior to that of the RW rehearsal group. This was not the case.
Despite fewer rehearsal trials, no OPFOR, and rehearsing in a building that only approximated
the building sections they would clear in the field (real world), the performance of the RW
rehearsal group did not differ from that of the VE rehearsal group. The data show that VE

rehearsal did not hurt real-world performance, but is equivocal as to how much transfer did take
place.

Although no data was collected during the talk/walk and run rehearsal phases for both
groups, data was collected for the two force-on-force missions conducted in the virtual-training
environment. These two trials served as dress rehearsals for the VE rehearsal group.
Performance during these two trials was comparable to the performance during the two missions

35



Science and Technology Objective (STO) Virtual Environment Research

In 1998, the U.S. Army Research Institute (ARI) established a four-year Science and
Technology Objective (STO) entitled Virtual Environments for Dismounted Soldier Simulation
Training and Mission Rehearsal. The purpose of the STO was to examine selected technological
and training issues that currently limit high fidelity dismounted simulation (see Pleban et al.,
2000). A collaborative STO effort was established between the Infantry Forces and Simulation
Systems Research Units of the ARI, the U.S. Army Simulation, Training, and Instrumentation
Command, and the Human Research and Engineering and Information Sciences and Technology
Directorates of the U.S. Army Research Laboratory to address these issues. The ARI portion of
the STO was covered under the work package Virtual Environment Research for Infantry
Training and Simulation (VERITAS). Key VERITAS work objectives included the following:

© Identify potential high payoff tasks for small unit leader virtual environment
(VE) training

Evaluate small unit training vignettes for use in infantry urban operations training
Develop training strategies and performance measures

Evaluate the training effectiveness of simulation systems

Evaluate the use of VE for night operations training

The research addressing each of these objectives is thoroughly described in Pleban et al.
(2000), Pleban et al. (2001), and Pleban and Beal (2002). Two key findings that emerged from
this research were the utility of virtual simulation systems for conducting decision-skills training
(Pleban et al., 2001) and mission rehearsal activities (Pleban et al., 2000). Empirical data in
support of the mission rehearsal function of virtual environments were severely limited,
however. In the Pleban et al. (2000) experiment, soldiers rehearsed a building-clearing mission
in an exact virtual representation of the McKenna urban operations training site and subsequently
executed multiple versions of this mission at the actual McKenna site. Funding restrictions
precluded the use of a baseline or control condition that involved rehearsing in a more traditional
field setting. The feedback provided from the soldiers (though entirely subjective), indicated that
the virtual environment rehearsals had improved their performance on many of the same tasks
executed at the McKenna training site. Soldiers also indicated that these virtual rehearsals
minimized on-site planning time since they were already familiar with the interior layout of the
buildings and helped mentally prepare squad members for the upcoming mission. The research
objective during the final year of the STO work effort was to conduct a more comprehensive
evaluation of the use of virtual environments as a viable dismounted infantry mission rehearsal
tool.

Virtual Environments for Mission Rehearsal

Simulations, particularly flight simulator systems, have been used extensively in aviation
settings to familiarize and train pilots in the operation of a particular aircraft. However, with the
advent of powerful digital terrain database generation capabilities, accurate representations of
specific, real-world areas could be created. The pairing of high-end simulation with high fidelity
terrain database generation capability represented an evolutionary change in flight simulation
technology. Now, crews could not only be trained on specific aviation task functions, but they



Appendix G
Operation Order No.1
Your unit is always the 2d Squad of the 2d Platoon.

Enemy Situation: Agular is considered a Black Sabbath stronghold with both the
terrorist political and military infrastructure based in the town. The terrorists are
considered dedicated to their cause and will most likely fight to the death. Armed
presence is considered moderate to heavy in sector with squad-size cells fighting
from building to building. There is no mechanized or motorized threat. However,
indirect fire, in the form of mortars is available to the enemy. They are heavily
armed with automatic weapons and rocket-propelled grenades. Remnants of the
Del Rio Dignity Brigade are also located in the town. They are ill equipped and
not capable of sustained operations. They are not trusted by their Black Sabbath
counterparts and will most likely be restricted to defending isolated individual

positions. Their morale is low. As a result, they are expected to provide only
token resistance.

Mission of Higher Units: The 22d IN BN continues its ground assault operation
to clear the town of Agular. The town is divided along the two long axes formed
by the east-west roads within the town (see map). Companies A and C
conducted the initial assault. Company A is securing BLDGs G and H.
Company C is in BLDGs A1 and J2. Companies A and C are preparing to
assault BLDGs J1 and B1, respectively. Your company, Company B, is in the
south. 1st PLT has cleared BLDG E. 3d Platoon has cleared BLDG N. Your
platoon, 2d Platoon has secured BLDG P3. Your platoon leader has just
returned from an update briefing. Your platoon has been ordered to seize the
BLDG C4 complex. Your platoon leader has elected to gain entry through BLDG
C4a. 1st Squad will conduct the initial assault from BLDG P3. Its mission is to
establish a breech into C4a. 2d Squad, your squad, is second in order of
movement. It will move from BLDG P3 and pass through the 1st Squad and
clear both floors of the building. 3d Squad is the platoon reserve. It is to be
prepared to assume all clearing responsibilities on order. 1st Squad will be the
platoon reserve once they have established a breech into Section A.

Your Mission: On order, 2d Squad, your squad, will pass through the 1st Squad
and clear both floors of BLDG C4a.

Rules of Engagement: Although the town populace of Agular is extremely
sympathetic to the rebel cause, the town population is not completely hostile.
Unarmed civilians were caught in the town by the rapid U.S. attack. Given that
the town population is not completely hostile, the rules of engagement remain
restrictive. The ROE is restrictive - only return fire after fired upon. There will be
no weapon firing except in self-defense. Self-defense is defined as a serious
threat to life or limb. Armed civilians and Del Rio soldiers actively engaging U.S.

G-1



Williams, H. P., Wickens, C. D., & Hutchinson, S. (1994). Fidelity and interactivity in
navigational training: A comparison of three methods. Proceedings of the Annual
Meeting of the Human Factors and Ergonomics Society, Santa Monica, CA, 38, 1163-
1167.

Witmer, B. G., Bailey, J. H., Knerr, B. W., & Parsons, K. C. (1996). Virtual spaces and real

world places: Transfer of route knowledge. International Journal of Human-Computer
Studies, 45, 413-428.

45



References

Annett, J. (1989). Training skilled performance. In A. M. Colley & J. R. Beech (Eds.),
Acquisition and performance of cognitive skills (pp. 61-84). New York: Wiley.

Campbell, J. P. (1988). Training design for performance improvement. In J. P. Campbell, R. J.
Campbell, & Associates, (Eds.), Productivity in organizations (pp. 177-215).
San Francisco: Jossey-Bass.

Driskell, J. E., Copper, C. & Moran, A. (1994). Does mental practice enhance performance?
Journal of Applied Psychology, 79, 481-492.

Driskell, J. E., Willis, R. P., & Copper, C. (1992). Effect of overlearning on retention. Journal
of Applied Psychology, 77, 615-622.

Headquarters, Department of the Army. (1999). An infantryman’s guide to combat in built-up
areas (FM 90-10-1). Washington DC: Author.

Klein, G. (1997). An overview of naturalistic decision making applications. In C. E. Zsambok
& G. Klein (Eds.), Naturalistic decision making (pp. 49-59). Mahwah, NJ: Erlbaum.

Klein, G. (1993). A recognition-primed decision (RPD) model of rapid decision-making. In G.
A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in
action: Models and methods (pp.138-147). Norwood, NJ: Ablex.

Knerr, B. W., Lampton, D. R., Thomas, M. A., Comer, B. D., Grosse, J. R., Blankenbeckler, P.,
Centric, J. H., Dlubac, M., Wampler, R. L., Garfield, K. A., Martin, G. A., & Washburn,
D. A. (2003). Virtual environments for dismounted solider simulation: Results of the
2002 culminating event. Manuscript in preparation.

Krebs, W. K., McCarley, J. S., & Bryant, E. V. (1999). Effects of mission rehearsal simulation
on air-to-ground target acquisition. Human Factors, 41, 553-558.

Lintern, G., Sheppard, D. J., Parker, D. L., Yates, K. E., & Nolan, M. D. (1989). Simulator
design and instructional features for air-to-ground attack: A transfer study. Human
Factors, 31, 87-99.

Nullmeyer, R. T., Bruce, P. D., Conquest, M. T., & Reed, E. T. (1992). Using the MH-53J
weapon system trainer/mission rehearsal system: Initial assessments and lessons learned.

Proceedings of the 14" Interservice/Industry T; raining Systems Conference, San Antonio,
TX, 47-60.

Nullmeyer, R. T., & Spiker, V. A. (2000). Simulation-based mission rehearsal and human

performance. In H. F. O'Neil, Jr. & D. H. Andrews (Eds.), Aircrew training and
assessment (Chapter 7). Mahwah, NJ: Erlbaum.

43



Appendix B

Demographic and Training Background for IOBC, MP, and Drill Sergeant Samples

Table B-1

Demographic and Relevant Training Experience for IOBC Student Sample
Min Max Mean sD

Age (yrs) 21.9 34.2 26.9 3.9

Time in service (yrs) 2 14.5 5.5 4.2

Hours/wk playing 0 10 1.0 2.2

virtual reality games

Times trained at 0 3.0 58 1.05

McKenna

Note, n= 24.

Table B-2

Relevant Training Experience for IOBC Student Sample by Course and Simulation System
(percent completed course or used simulation system)

PLDC | BNCOC | ANCOC | Airborne | Air Other | Indiv. Cbt. | Other Mil.

Assault | Courses*| Simulator | Simulations?
37.5 20.8 4.2 70.8 33.3 62.5 16.7 45.8

Note. n= 24. PLDC — Primary Leadership Development Course; BNCOC — Basic Non-Commissioned Officer
Course; ANCOC — Advanced Non-Commissioned Officer Course.

* Included Ranger School, Mountain/Jungle Warfare School, Sapper, Jumpmaster.

* Included Janus, Close Combat Tactical Trainer, Avenger/Stinger missile system simulators, tank simulators.

Table B-3

Demographic and Relevant Training Experience for MP Sample
Min Max Mean sD

Age (yrs) 24.8 46.9 34.5 74

Time in service (yrs) 4.0 26.8 12.4 7.0

‘Hours/wk playing 0 3.0 54 11

virtual reality games

Times trained at ft) 1.0 1S 37

McKenna

Note. n= 13.

B-1



D2



All soldiers felt that additional urban operations training time was needed prior to
conducting the mission rehearsals. Time scheduling permitted only one hour of training in the
basic skills, which was clearly not enough. This could account, in part, for the lowered total
weighted scores achieved by each group.

In terms of improving rehearsals in general, some soldiers felt that it would be useful to
rotate squad members through different leadership positions to develop an increased awareness
of the various leadership demands placed on the squad and team leaders. This could be quite
useful but would involve more rehearsal trials to rotate squad members through the different

leader positions. This could be a problem, from both a time and cost perspective, particularly for
real-world rehearsals.

Mission Rehearsal Effectiveness Ratings

The effectiveness ratings showed a clear bias that was dependent on the rehearsal setting.
When asked if virtual environments could be used to conduct effective mission rehearsals,
almost every soldier in the VE rehearsal group said yes. In contrast, less than half of the RW
rehearsal group felt the same way. This was most likely due to the fact that the RW rehearsal
group conducted no rehearsals in the virtual environment. Their exposure was limited to a
familiarization period and the two missions. The major reason for bringing the RW rehearsal
group back to the Virtual Simulation Lab was to provide them some experience in the virtual
environment so they could make some comparisons between the two environments. For all
intents and purposes, the critical baseline data was collected the first day of the experiment and
soldiers knew they were not using the virtual-training site to rehearse an upcoming mission. The
reduced exposure time to the virtual environment, and possibly lower incentive generated by this
“supplemental training phase” may be responsible for the lower ratings provided by the RW
rehearsal group. These factors may have also negatively affected their virtual performance on
the Level I task, personnel flagging (increased number of incidents) during Mission 2.

Comparing the effectiveness ratings of the two settings, approximately one half of the
soldiers in the VE rehearsal group felt that virtual environments were less effective than real-
world settings for conducting mission rehearsals. In contrast, all soldiers in the RW rehearsal
group said that virtual environments were less effective than real-world settings for conducting
mission rehearsals. The “hands-on” factor played a major role in the shaping the ratings of the
RW rehearsal group. When these soldiers moved to the virtual world on Day 2, the reality
difference was clearly accentuated. This was a major consideration for soldiers, and most people

were in agreement that certain simulated capabilities needed to be more realistic or at least
function more smoothly.

Training in Virtual Environments

Virtual environments provide a vehicle for conducting training in a number of areas.
While the capabilities of the VE hardware/software can affect both training utility and trainee
perceptions, other factors may have more impact than pure fidelity. The overall training value of
these environments ultimately depends on the quality of the scenarios, instructor effectiveness,
and the soundness of the instructional approach. As Pleban et al. (2001) showed, well-crafted

41



Table 3
Technical Specifications of the Immersive SVS and Desktop SVS Simulation Systems

e Pentium III - 450 MHz microprocessor
System Hardware |* 128 Mb RAM
(immersive and ® Obsidian 200 — 8440 3D Graphics Card
Desktop) © SoundBlaster AWE 64 Gold Audio Card
e Removable 4.55 GB SCSI Hard Drive
© Weapon-mounted thumb switch
Movement Control ¢ Desktop SVS — Microsoft joystick control
Motion Capture/ | ¢ InterSense Mark2 X-Bar Tracking System
Weapon Tracking |e Weapon tracking accurate to within % of 1°
© 90° x 60° FOV at center of enclosure (varies with position change)
Visual Display e Rear screen projection resolution 1024 x 768
e Desktop SVS resolution 800 x 600
Enclosures e Aluminum frame over black sound-dampening fabric. (10 ft. x 10 ft.
x 12 ft.)
Software e Advanced Interactive Systems

After Action Review (AAR) System. A prototype AAR system, located in the OPFOR
building, was employed that included two 53-inch screens and two personal computers. The
system, which was linked to both the immersive SVS and desktop systems, was able to mark and
store key events during the mission, e.g., fratricides, casualties, rounds fired, for subsequent
analysis. Marking could be performed automatically or manually by the O/C. These data could
not be summarized and displayed on the monitors at this time. However, casualty rates were
available for discussion purposes during the AAR. Missions could be replayed from several
vantage points. The top-down mode provided a view of the database looking straight down from
above. Other viewing modes used included a two dimensional view (top-down) without depth
perspective, an entity view (displays what a elected entity sees), and a fly mode used by the O/C
to “fly” through the data base using the mouse for control. The O/C selected parts of the mission
to replay during the AAR. The O/C also controlled the visual replay of the mission. Replay
controls included such actions as pause, stop, record, play, fast forward, fast reverse and rewind.
Synchronized audio replay capability was not available for this experiment.

Field-Training Site

Building Instrumentation. The building complex where the missions were executed was
fully instrumented. Rooms, stairwells and hallways were monitored with multiple cameras set in
fixed locations. Video and audio streams were fed back to the control center that allowed the
O/C to track the mission from a distance.

AAR System. AARs were conducted at the control center building. Here, the mission

was replayed on multiple large screen monitors. Top-down, two-dimensional animated replays
were shown along with ground level video of the squad moving from room to room. The video

13



Soldiers preferred a larger FOV in the virtual environment, suggesting that the current
screen be enlarged to allow for 360 degree viewing of the mission environment. At the very
least the soldiers recommended a screen that would allow the user to look up, down, and side to
side. For changing views, soldiers indicated that instead of toggle manipulation, the view should
be based on the head movements of the user.

A number of soldiers found the virtual grenades difficult to use. If the grenade toss
(using the weapon) was not timed correctly according to a calibration figure in the center of the
screen, the grenade would drop at the soldier’s feet and explode, producing a fratricide. Clearly,
amore realistic, user-friendly means of throwing grenades is needed.

Other suggestions for improving the effectiveness of virtual environment mission
rehearsals focused on enhanced maneuverability, a revised menu consisting of separate control
buttons for each choice of weapons, a target reference for shooting, and a reference map showing
the position and the direction of travel of all squad members during the exercise. This last
feature is not available in the real world. In addition, soldiers recommended including more
realistic battlefield sounds such as footsteps, weapons being loaded and fired, people talking in
whispered English and other foreign languages depending on the enemy’s origin, and grenades
being activated and dropped. Soldiers also felt they needed additional training time on the
virtual system.

Soldier Comparison of Real-World Versus Virtual-Environment Mission Rehearsals

Effectiveness of rehearsal setting. At the end of the experiment, soldiers completed the
Mission Rehearsal Comparison Questionnaire. This questionnaire was used to give soldiers a
chance to compare the effectiveness of each environment for conducting rehearsals. Table 7
summarizes soldiers’ responses to the question “Based on your experiences this week, could
virtual environments be used to conduct effective mission rehearsals?”

Table 7
Percent Response to Question “Could Virtual Environments be Used
to Conduct Effective Mission Rehearsals?”

VE Rehearsal RW Rehearsal
Squads Squads
Yes 94 44
No 6 19
Yes and No 0 31
No answer 0 6

Note. n = 16 for each condition.

Ninety-four percent of the soldiers in the VE rehearsal condition felt that virtual environments
could be effectively used to conduct mission rehearsals. In contrast, only 44 % of the soldiers in
the RW rehearsal condition felt the same way. Half of the RW rehearsal soldiers expressed
either mixed feelings (yes and no) toward virtual environments or felt these VE systems were not
an effective means for conducting mission rehearsals.

31



VERITAS Evatuation Check

Mission: 48 SQD: 4 = =DAY: 1 Mission:
STARTER: 14:12:00 ENDEX: 1419:36 Duration:
Reasonfor ENDEX: Mission Failure = Al EXFOR KIA

wr
t



FOREWORD

This report describes selected aspects of the fourth year work effort under the Science
and Technology Objective (STO) entitled Virtual Environments for Dismounted Soldier
Simulation, Training, and Mission Rehearsal. The U.S. Army Research Institute for the
Behavioral and Social Sciences (ARI) Infantry Forces Research Unit performed this research in
collaboration with the ARI Simulation Systems Research Unit, the U.S. Army Simulation,
Training, and Instrumentation Command, and the U.S. Army Research Laboratory. The primary
objective of the STO was to address selected technological and training issues related to high
fidelity dismounted soldier simulation.

This report describes a research effort that examined the utility of virtual environments
for conducting small unit dismounted mission rehearsals in simulated urban operations. The
training was evaluated at the Dismounted Battlespace Battle Lab (DBBL) Virtual Simulation
Lab, and the McKenna urban operations facility at Fort Benning, Georgia. The effectiveness of
this approach was compared to real-world rehearsals. The results showed that while virtual
environment systems show promise, there are still a number of interface and technology
problems to overcome. Currently, virtual environments do not appear to be as effective as real-
world tactical training for improving skills underlying specific small unit tasks or battle drills.
However, these environments may be used effectively at selected stages of training to enhance
cognitive skill development in such areas as decision-making and situation awareness. Critical
aspects of the research were briefed to key DBBL staff in August 2002.

KATHLEEN A. QUINKERT
Acting Technical Director





soldiers will be fired on. Civilians found to be armed, but not firing on U.S. troops
are to be disarmed, detained, and processed through company channels.
Weapons contro! status is yellow.

Your immediate task is to brief your squad members and organize the squad to
complete the mission. Call signs remain constant.

What are your questions?
The scenario will begin once you have briefed your subordinates.
SCENARIO:

Force-on-Force: The squad leader will be informed that the 1st Squad is
notional while conducting force-on-force training. 2d Squad will begin the
exercise from BLDG P3. The start point will be 2d SQD LDR receiving a radio
call from the platoon leader, notifying him that 1st Squad is in place and the
breech (the door) is blown. Start point for 2d SQD LDR will be the left window of
BLDG P3 facing BLDG 4Ca. Orange or red tape will be used to block off and
separate the second stories between 4Ca and 4Cb. The SQD LDR will be told
not to break the tape and enter 4Cb.

Virtua! Environment: DBBL will program the 1st Squad conducting the initial
assault from BLDG P3 to BLDG C4a. They will show the squad move and blow
the doorway. Once the doorway is blown, the platoon leader will call 2d Squad
and order their movement. Start point for 2d SQD LDR will be the left window of
BLDG P3 facing BLDG 4Ca. The SQD LDR will be told not to enter 4Cb.

OPFOR locations - See map

G-2


no weapon firing except in self-defense. Self-defense is defined as a serious
threat to life or limb. Armed civilians and Del Rio soldiers actively engaging U.S.
soldiers will be fired on. Civilians found to be armed, but not firing on U.S. troops
are to be disarmed, detained, and processed through company channels.
Weapons control status is yellow.

Your immediate task is to brief your squad members and organize the squad to
complete the mission. Call signs remain constant.

What are your questions?
The scenario will begin once you have briefed your subordinates.
SCENARIO:

Force-on-Force: The squad leader will be informed that the 1st and 3d Squads
are notional while conducting force-on-force training. 2d Squad will begin the
exercise from BLDG P3. The start point will be 2d SQD LDR receiving a radio
call from the platoon leader, notifying him that 1st Squad has established the
breech into BLDG 4Cb by blowing the doorway. He will also be informed that 3d
Squad has cleared the 1st floor of BLDG C4b. He will be informed that his
mission is to clear both top floors of BLDG C4b, then C4a. The squad leader will
be informed that the steps leading to the 2d floor of 4Cb have been cleared, but
that the entire second floor has not been cleared. He will be reminded that his
mission is to first clear the top floor of 4Cb, then clear the top floor of 4Ca.

Virtual Environment: DBBL will program the 1st Squad conducting its initial
assault from BLDG P3 to BLDG 4Cb and blowing the doorway. 3d Squad will be
shown passing through 1st Squad. Once the doorway is blown, and 3d squad
has had a few minutes to clear the 1st floor, the platoon leader will call 2d Squad
and order them to clear both top floors of BLDG 4Cb and then 4Ca, respectively.
As the 2d Squad enters 4Cb, they will see individuals from the 1st Squad
securing the doorway. They will also see individuals from 3d Squad positioned in
each room and one soldier at the top of the stairs. Start point for 2d SQD LDR
will be the left window of BLDG P3 facing BLDG 4Cb.

OPFOR locations - See map

H-2



Virtual-Training Site Casualties

7
6
54
Mean No. of 44 IVE Reh Gp
Friendly 34 EIRW Reh Gp
Casualties 5 |
1
0
4 2
Mission Number

Figure 8. Mean number of friendly squad casualties by rehearsal
group at the virtual-training site.

All but one of the 16 (94 percent) observed fratricides occurred in the virtual
environment. As can be seen from Figure 9, the fratricide rate increased across missions
performed in the virtual environment by the VE rehearsal group. The increased rate of
fratricides can most likely be attributed to the emphasis placed on the use of grenades for
Mission 2 for one of the squads and inadequate training in the use of the grenade system.
(Twelve of the 15 fratricides observed in the virtual environment were the result of grenades.
All VE rehearsal group fratricides were from grenades.)

Virtual-Training Site Fratricides

7
6
5
No.of 4 IVE Reh Gp
Fratricides 3 | EBRW Reh Gp

Mission Number

Figure 9. Total number of fratricides by rehearsal group at the virtual-
training site.

24



performance. Therefore, these tasks were assigned a heavier weight than other tasks.
Ultimately, three weight categories were developed.

A weight factor of three (3) was assigned to three measures deemed critical for
determining the squad's performance at end state. These were: the squad's manpower strength at
end of the mission, the OPFOR threat to the clearing operation - all OPFOR had to be eliminated
in order for the squad to successfully complete the mission and, the number of fratricides.

A weight factor of two (2) was given to tasks that, if not correctly performed, could result
in casualties and/or compromise mission success. Assessment areas included: the number of
rooms not properly cleared; dispersion at danger areas; soldier placement, to include individual
location in relation to the rest of his unit, and his ability to provide supporting fires from that
location; and the soldier’s understanding of his leader's directions and intent.

A weight factor of one (1) was assigned to tasks that dealt more with correct individual
performance rather than situations entailing the possibility of producing casualties. These areas
included successful urban operation techniques, safety actions, and command and control issues,
e.g., Did the soldier needlessly expose himself to possible sniper fire? Were weapon muzzles
pointed away from friendly soldiers? Was a succession of command established and how well
did the squad react when a leader became a casualty? For descriptive purposes, tasks or
measures receiving weights of 3, 2, or 1 were classified as Level Ill, II, and I tasks, respectively.

For each task, frequency bands were calculated. Depending on the raw frequency count,
a specific score was assigned to the task based on the band that the frequency tally fell in. For
example, if 10 counts of personnel flagging were observed, the squad would receive a score of 3.
This task would then be multiplied by its weight, 1. The final transformed score for this task
would be 3. Each task was scored in a similar fashion. These individual weighted scores were
then summed to provide a total score. The maximum total weighted score that could be obtained
was 135. Table 2 lists each of the collective tasks and behaviors, their corresponding weights
and frequency band scores.

10


Supplemental practice at virtual-training site. On Day 2, soldiers reported to the Virtual
Simulation Lab. The purpose for having soldiers in the real-world rehearsal condition come to
the lab was to allow them to execute the same missions that they performed in a real-world
setting (on Day 1) ina virtual environment. This would provide them an opportunity to compare
the strengths and weaknesses of each environment for conducting mission rehearsals.

Soldiers first received approximately 90 minutes of familiarization training on the SVS
systems. They used this time to practice moving inside and outside of buildings using the
weapon mounted thumb switch, weapon firing, and generally familiarizing themselves with the
various computer generated entities including friendly and enemy soldiers, vehicles, aircraft, and
furniture. In addition, the soldiers conducted a practice scenario that emphasized moving as a
group in and around a building similar to the one they rehearsed in the day before. Prior
experience had shown that soldiers needed additional time to adapt to the constraints of the SVS
system such as moving tactically, in close contact with others, inside rooms and around corners,
sticking in walls, and learning how to back out of walls. The OPFOR was also given time to
become familiarized with the desktop system. This system was easier to use and did not take
nearly as long for soldiers to master.

Following the familiarization training, the squad received their operation order for the
first mission. The OPFOR was briefed separately. Both groups were given 30 minutes for
planning. No additional rehearsals were allowed. The squad then executed the mission (VE
missions ranged from approximately 10 to 21 minutes). The building section for Mission 1, as
well for Mission 2, were virtual representations of the same building sections that the squad had
cleared during Day 1 at the McKenna field-training site. Both O/Cs observed the mission on
desktop systems in the same room occupied by the battlemaster. This was followed by an AAR.
During the transition period prior to the AAR, the O/Cs each completed the Evaluation
Checklist.

After the AAR, the squad received their next operation order and the same procedure was
followed as described earlier. After the second AAR, soldiers then completed the Mission
Rehearsal Comparison Questionnaire.

Virtual Environment Rehearsal Condition

Preparation. Soldiers in the virtual environment rehearsal condition arrived at the
Virtual Simulation Lab on Day 1. They were briefed on the objectives of the experiment and
also given the opportunity to ask any questions concerning their roles in the experiment. All
soldiers then completed the Biographical Information Questionnaire. Soldiers were then
assigned to either the squad or OPFOR element. Both groups received the same SVS system
familiarization training as did the real-world rehearsal condition soldiers on Day 2.

Following the familiarization training, squad members received approximately one hour
of training on urban operation tactics, techniques, and procedures from one of the O/Cs. The
second O/C provided some very basic instruction to the OPFOR members on proper defensive
procedures from inside a building (rooms). Instruction for the two groups was provided at

16


Mission Scenarios

All scenarios were set in a small European town. The town was, in fact, the McKenna
urban operations training site. A virtual representation of the town was created for mission
rehearsal and execution at the Virtual Simulation Lab. Military subject matter experts developed
two scenarios that required clearing different sections of a two-story building complex
(Appendixes G and H). Scenarios assessed squad tactical proficiency across a predetermined set
of doctrinally approved tasks. The scenarios were also scripted to last approximately 30 minutes
for scheduling purposes and to maintain group focus. The OPFOR consisted of three soldiers
from the participating unit. Live OPFOR elements were used in both training settings.

Equipment
Virtual-Training Site

Soldier Visualization Station (SVS). Nine full-immersion SVS systems (helmet mounted
display [HMD], weapon, screen) were employed along with four joystick controlled, desktop
systems. The full-immersion systems were housed in their own enclosures but linked with each
other and the desktops. Technical specifications of the two different versions are shown in Table
3. Squad members could communicate with each other and the squad leader. The squad leader
could communicate with the platoon leader, role played by one of the O/Cs. Communication
procedures were similar, but not identical to what soldiers would be accustomed to in a real-
world environment.

Each squad member operated an SVS, while the platoon leader operated a desktop
system. The squad members were located in one building while the platoon leader and
battlemaster (simulation system/scenario coordinator) were adjacent to each other in another
room, away from the SVS systems. The OPFOR element operated the remaining desktops from
another building.

12



The physical act of throwing a grenade in a virtual environment was very different
(awkward and unrealistic) from throwing a grenade in the real world. Overall, it was clear that
more practice should have been allocated for soldiers to familiarize themselves with how to
effectively “throw” grenades in the virtual environment (see Appendix I, Table I-4 for individual
incident rates and point totals for all Level Ill tasks).

As was the case at the field-training site, OPFOR casualty rates were essentially the same
for each group. There was no change in the number of OPFOR casualties inflicted from Mission
1 to Mission 2.

Group performance on individual level II tasks. Approximately 4 times as many Level II
task tactical incidents were observed for the VE rehearsal group compared to the RW rehearsal
group. Incidents decreased over missions for the VE rehearsal group but remained stable and
low for the RW rehearsal group (see Figure 10).

Virtual-Training Site Level Il Incidents

14
12
40
Total No. of 8 } |-——--—J IBVE Roh Gp |
Incidents 6 EIRW Reh Gp
4
2
0

1 2
Mission Number

Figure 10. Total number of level II task tactical incidents by rehearsal
group at the virtual-training site.

Further analysis also showed that three tasks accounted for the majority of errors that
were observed, task 4 (rooms not properly cleared); task 5 (failure to maintain proper position

for providing protective cover); and task 7 (failure to place soldiers to provide supporting fires
for other soldiers conducting movement).

For task 4, all room clearing incidents that were observed (7) came from the VE rehearsal
group. For task 5, all but one incident of soldiers failing to maintain relative proper position with
each other to provide protective cover was observed for the VE rehearsal group (6 versus 1). A
similar, but slight trend was observed for task 7. All incidents (3) of improperly placing soldiers
to provide supporting fires for soldiers conducting movement came from the VE rehearsal group
(see Appendix I, Table I-5 for individual incident rates and point totals for all Level II tasks).

25



floor plans for each section of the building were similar (see Appendix G). A small opposing
force (OPFOR) element was positioned in specific rooms within the building.

Soldiers in the VE rehearsal group first rehearsed the missions in a virtual representation
of the exact building sections they would clear at a real-world field-training site. These
rehearsals were conducted without an OPFOR. The VE rehearsal group then rehearsed the
missions in the virtual environment with an OPFOR. More specifically, the VE group rehearsed
(talk-walk and run) without an OPFOR in one building section then rehearsed again with an
OPFOR in the same section. They then rehearsed (talk-walk and run) the second mission
without an OPFOR in a different building section and then rehearsed again with an OPFOR in
the very same section. The RW rehearsal group rehearsed in an actual two-story building that
was similar to the one they would have to clear at the field-training site. Rehearsals for this
group were conducted without an OPFOR.

Two retired Army personnel served as observer/controllers (O/Cs) and conducted the
After Action Reviews (AAR) following each mission. They also provided assessments of the
squads’ performance for each mission.

The full experimental design is presented in Table 1. The VE rehearsal group received six
rehearsal trials. Four rehearsals [talk-walk (2) and run (2)] were conducted without OPFOR.
Two rehearsals were performed with an OPFOR. The RW rehearsal group received two
rehearsal trials without an OPFOR. This disparity in rehearsal opportunities between the two
groups was the result of several factors. First, from a logistical standpoint, it was not possible to
rehearse the VE group at the Virtual Simulation Lab and transport everyone to the urban
operations site to conduct the real-world missions in the same day. The set up and preparation
required for the McKenna phase dictated that the VE training session be divided over two days.
While it was possible to limit the VE rehearsals to the same two (talk-walk and run) trials
without troops that the RW rehearsal group received, this option was rejected. This approach
underutilizes the virtual environment capabilities. Previous experience with the SVS systems
had indicated that soldiers quickly get bored moving around inside a virtual building without any
type of force-on-force engagements with either a live or computer-generated OPFOR.

Given the increased training incentive from using live OPFOR (Pleban et al., 2000), and
the ease of setting up and running virtual scenarios, it was determined that a fairer test of VE
capabilities would be to allow soldiers to rehearse without soldiers (talk-walk and run) in virtual
environment settings identical to those they would encounter the following day (one talk-walk
and run rehearsal in each setting). In addition, soldiers would also receive one full rehearsal with
a live OPFOR in a virtual environment identical to each setting they would encounter the
following day (two full rehearsals). This also exploited another advantage of virtual rehearsals,

ie., the ability to generate a low cost, realistic simulation of the operational setting that will be
encountered by the unit.

As can seen from Table 1, the RW rehearsal group was brought to the Virtual Simulation
Lab on Day 2 for “supplemental” training. This was done to give this group experience in the
virtual environment and to allow them to make comparisons between the two rehearsal
environments.



Appendix H
Operation Order No. 2

Your unit is always the 2d Squad of the 2d Platoon.

Enemy Situation: Agular is considered a Black Sabbath stronghold with both the
terrorist political and military infrastructure based in the town. The terrorists are
considered dedicated to their cause and will most likely fight to the death. Armed
presence is considered moderate to heavy in sector with squad-size cells fighting
from building to building. There is no mechanized or motorized threat. However,
indirect fire, in the form of mortars is available to the enemy. They are heavily
armed with automatic weapons and rocket-propelied grenades. Remnants of the
Del Rio Dignity Brigade are also located in the town. They are ill equipped and
not capable of sustained operations. They are not trusted by their Black Sabbath
counterparts and will most likely be restricted to defending isolated individual

positions. Their morale is low. As a result, they are expected to provide only
token resistance.

Mission of Higher Units: The 22d IN BN continues its ground assault operation
to clear the town of Agular. The town is divided along the two long axes formed
by the east-west roads within the town (see map). Companies A and C
conducted the initial assault. Company A is securing BLDGs G and H.
Company C is in BLDGs A1 and J2. Companies A and C are preparing to
assault BLDGs J1 and B1, respectively. Your company, Company B, is in the
south. 1st PLT has cleared BLDG E. 3d Platoon has cleared BLDG N. Your
platoon, 2d Platoon has secured BLDG P3. Your platoon leader has just
returned from an update briefing. Your platoon has been ordered to seize the
BLDG C4 complex. Your platoon leader has elected to gain entry through BLDG
C4b. 1st Squad will conduct the initial assault from BLDG P3. Its mission is to
establish a breech into BLDG 4Cb. On order, 3d Squad will move from BLDG
P3, pass through 1st Squad, and clear the 1st floor of 4Cb. 2d Squad is the
platoon reserve. It is last in order of movement. On order, 2d Squad will assume
clearing responsibilities. 1st Squad will be the platoon reserve once they have
established a breech into BLDG 4Cb and 3d and 2d Squads have passed
through.

Your Mission: 2d Squad, your squad, is the platoon reserve located in BLDG
P3. On order, your squad will move to BLDG 4Cb, pass through the 1st Squad,
and assume clearing responsibilities.

Rules of Engagement: Although the town populace of Agular is extremely
sympathetic to the rebel cause, the town population is not completely hostile.
Unarmed civilians were caught in the town by the rapid U.S. attack. Given that
the town population is not completely hostile, the rules of engagement remain
restrictive. The ROE is restrictive - only returm fire after fired upon. There will be

H-1



Appendix C
Real World Post Training Questionnaire

Name. Date Condition

1. Mission role:

BLUFOR OPFOR

2. How effective was the rehearsal training you received in preparing you for the clear building
missions?

Very Ineffective Ineffective Fairly Effective Very Effective

3. Was the rehearsal time adequate?

4. What did you like most about rehearsing in a realistic field setting?

5. What did you like least about rehearsing in a realistic field setting?

6. What, if anything, would you do to modify the rehearsals to make them more effective?

C-1



CONTENTS (continued)

Page

CONCLUSIONG......scscssssssssssessessssonssnnsssssecnssssnsesssossnnnnsssssssessscecsscceseseessessesssssssntnsessssonts 42

REFERENCES... 43
APPENDIX A__ Biographical Information Questionnaire...u...ssssssssssssssssssssssesssusse A-l

APPENDIX B Demographic and Training Background for IOBC, MP, and Drill

Sergeant Samples . B-1
APPENDIX C_ Real-World Post Training Questionnaire ...........ssssessssessssssersessseeees C-1
APPENDIX D_ Virtual Environment Post Training Questionnaire ............sssssssessssese D-1
APPENDIX E_ Mission Rehearsal Comparison Questionnaire ..........csssesssesccneeneeeeeee E-1
APPENDIX F Evaluation Checklist ........sssssssssssssssssssesseescsssssessessessssneseessssveceessssssseeees F-1
APPENDIX G_ Operation Order No. 1 .. G-1
APPENDIX H Operation Order No. 2.....scssssssssssssesssssssesessessssssssssssaecsssssssenseesssvseaveess H-1

APPENDIX I Evaluation Checklist Incident Rates and Weighted Scores by Task... I-1

Table 1
Table 2
Table 3
| Table 4
Table 5
Table 6

Table 7

LIST OF TABLES
Sequence of Events for Virtual Environment (VE) and Real-World (RW)
Rehearsal Conditions.........sssssssssssesssessecsssseessseecsscssssessssessasesessseseesssnesssessee 8
Importance Ratings and Associated Frequency Band Scores by Task
(from Evaluation Checklist).........csssssssssssssnssssssseesssssessssnseesssseessnsneessonests 11
Technical Specifications of the Immersive SVS and Desktop SVS
Simulation Systems ..........ccsssessessessssesssessseessnesnessesnsesnsssssssseesesaseerseessasessee 13
What Soldiers Liked Most About Rehearsing in Real-World and Virtual
Environment Settings ... 28
What Soldiers Liked Least
Environment Settings 29
Suggestions for Improving Mission Rehearsal Training in Real-World
and Virtual Environment SettingS............ssssssessssssssscsssssssssesnsssssssnsssseseeese 30
Percent Response to Question “Could Virtual Environments be Used
to Conduct Effective Mission Rehearsals?” .......ssssssssssesssssessssesesssssessesseees 31



Field-Training Site Level | Incidents

25

20

No.of 154
Incidents 45 |

IVE Reh Gp
ERW Reh Gp

54

0

Mission Number

Figure 5. Total number of level I task tactical incidents by rehearsal
group at the field-training site.

Virtual-Training Site Performance

The weighted group performance scores for the two virtual environment missions were
also computed to address performance differences between rehearsal environments and to assess
possible simulation system shortcomings (see Figures 6 and 7). Figure 6 shows that overall
performance improved from Mission 1 to Mission 2 for the VE rehearsal group, increasing from
70.5 to 82. In contrast, the RW rehearsal group’s performance showed relatively little change
over missions, 87.5 to 91.5 for Missions 1 and 2 respectively (see Figure 7). Overall, the RW
rehearsal group’s performance at the virtual-training site was better than that of the VE rehearsal
group for each mission.

Further inspection of Figures 6 and 7 shows that the RW rehearsal group’s performance
was better on Level III as well as Level II tasks. VE rehearsal group scores for Level I tasks
during Mission 2 were better than those obtained for the RW rehearsal group.

The VE rehearsal group showed improvement over missions for both Level II and Level I
tasks. Improvement on Level II tasks was most apparent. Level III task performance was
relatively poor and decreased slightly over missions. For the RW rehearsal group, performance
improved for Level II tasks but remained essentially unchanged for both Level III and Level I
tasks.

22



Selection and weighting of evaluation criteria

The development of an evaluation assessment tool was evolutionary. First, a
subject matter expert in urban operations reviewed different doctrinal manuals and
evaluation checklists used in previous exercises. A panel of three retired officers and a
non-commissioned officer then applied their military knowledge and expertise gained
from years of active duty service in the Infantry, which included training and evaluating
urban operation skills, to refine the list of critical tasks required to "clear a building".
This expertise factor also included recently acquired experience in observing different
urban operation exercises and training events, both in the real world and in virtual
environments. The analysis resulted in 22 critical questions.

. What was the total number of rounds expended by each soldier?

. Were there any weapon malfunctions that impacted the tactical scenario?

. What was the number of BLUFOR casualties and was the squad combat effective at
ENDEX?

4. What was the number of OPFOR casualties?

5. Were any of the BLUFOR casualties caused by fratricide?

6. What was the number of personnel disoriented/lost in the building?

7, Were all rooms properly cleared (closets, behind doors, and behind/under furniture)?

wn

8. Did squad leader maintain positive control of the squad?

9. Did team leaders maintain positive control of their respective teams?

10. Did soldiers "flag" (expose their head around corners or their bodies in windows) for
greater than 5 seconds?

11. Did soldiers maintain their relative positions with each other and provide protective
cover?

12. Did soldiers "flag" (extend barrel beyond cover) their weapon barrels at corners,
possibly alerting enemy soldiers of their presence?

13. Did soldiers maintain dispersion at danger areas (areas with more than one angle of
attack)?

14. Did the team leader report the status of the room being cleared back to his squad
leader?

15. Were soldiers placed to provide supporting fires for another soldier conducting a
movement?

16. Did soldiers accurately report location of OPFOR?

17. Did soldiers and team leaders provide all of the information requested by the squad
leader?

18. Were weapons held correctly when entering and inside a room (muzzles pointed away
from friendly forces)?

19. Did each team member understand his respective sector of fire and provide adequate
fire support?

20. Once room was cleared, did team members yell, "Clear" to inform the support
element?

21. Were unit tactics doctrinally sound?

22. Was a succession of command established or did the squad adequately adjust to the
situation?



Task

Rationale

12. Personnel “flagging”
(exposed in windows or head
exposed around corner) for > 5
seconds,

Personnel flagging was selected for evaluation from both inside the
building (doorways) and from outside the building (windows) because
this provides your position and announces your intentions to the enemy.
Additionally, flagging provides the enemy with a target while not
allowing you to return effective fire. An environmental comparison of
flagging was necessary to track the possibility of creating bad habits in a
virtual environment where a clear sensing of immediate surroundings is
not as easy as in the real world.

13. Soldiers" flagged"
(extended M16 barrel beyond
cover) at corners.

Weapon flagging was selected for evaluation from both inside the
building (doorways) and from outside the building (windows) because
this activity announces the soldier's position and intentions to the enemy;
eliminating any opportunity for surprise. An environmental comparison
of flagging was necessary to track the possibility of creating bad habits in
a virtual environment where sensing of immediate surroundings is not as
easy as in the real world.

14. Weapons NOT held
correctly entering aroom
(muzzles pointed at friendly
forces).

Maintaining individual sectors of fires (muzzle control) allows for
immediate suppression of enemy forces in multiple directions while
reducing fratricide in the event of an accidental weapon discharge.

15. Once room was cleared,
team did NOT yell, "Clear" to
inform the support element.

During MOUT operations the squad leader’s location will normally be
one or two rooms removed from the fight. He must have a clear
understanding of the status of rooms cleared vs. not cleared in order to
effectively direct the fight. This enhances positive control of both teams
while simultaneously reducing possible fratricides.

16. Unit tactics were NOT
doctrinally sound.

While the Army encourages initiative and rewards heroism with medals,
the goal is teamwork following approved tactics. Applying correct
doctrine increases the chances of mission success.

17. Succession of command
NOT established.

Casualties will happen! Having a succession of command, that soldiers
know, reduces the battlefield confusion and allows the momentum of
battle to continue under new leadership.

Of the criteria that were developed, certain criteria were recognized as having a
more significant effect in determining the final outcome of a mission and in assessing
overall unit performance. Therefore, these criteria were assigned a heavier weight than
other criteria. Ultimately, three weight categories were developed.

A weight factor of three (3) addressed critical mission essential evaluation
criteria. These tasks required the squad to meet very selective assessment standards. For
example, three criteria were deemed critical at determining the squad's performance at
end state. These were: the squad's manpower strength at end of mission to determine if
the squad was still mission capable; the OPFOR threat to the clearing operation had to be
eliminated in order for the squad to successfully complete the mission and; the negative
impact of fratricides on unit morale and physical mission performance had to be

addressed.

A weight factor of two (2) included activities that, if not correctly performed,
could result in casualties and/or compromise mission success. Assessment areas
included: the number of rooms not properly cleared; dispersion at danger areas; soldier
placement, to include individual location in relation to the rest of his unit, and his ability

F-5



to provide supporting fires from that location; and the soldier’s understanding of his
leader's directions and intent. These areas constituted the bulk of the assessment criteria.

The final category included a weight factor of one (1). These activities were still
important, but dealt more with correct individual performance measures rather than
situations entailing the possibility of producing casualties. These areas included
successful MOUT operation techniques, safety actions, and command and control issues,
e.g., Did the soldier needlessly expose himself to possible sniper fire? Were weapon
muzzles pointed away from friendly soldiers? Was a succession of command established
and how well did the squad react when a leader became a casualty?

The final portion of the evaluation checklist was the development of the
assessment scoring standards. An initial set of standards were developed, but proved to
be only an “initial best guess" as to what constituted "good" or "bad" squad performance.
It was later determined that the best option was to evaluate each squad's performance
during the evaluation, then based on the performance of all the squads, develop a
performance scale. In some instances, tasks were assessed using a graduated scale. This
was true of most tasks. In other instances, performance was evaluated on a simple
pass/fail standard, either the squad performed correctly or it did it wrong.

It was also determined that two other areas needed to be included in the checklist.
First, the assessment had to provide a method for identifying the location where incidents
occurred. Second, there had to be a notes section that allowed the evaluators an area
where some narrative description could be added to explain some of the incidents, when
needed.

A standardized map was developed that numbered each room and hallway.
Incidents and their sequence were then marked on the map via "stubby pencil" during the
course of the evaluation of each squad mission. The location and number of incidents at
that location were then transposed to a Room Column on the evaluation form. For
example, a fratricide may have occurred in the first hallway on the second floor of the
C4B building. During the evaluation, a note was made on the map at the hallway labeled
Room 16. That note was then transposed to the evaluation checklist as "Rm 16: 1" using
the room column. This indicated that one incident occurred in Room 16 on the map. In
the notes column on the evaluation checklist, a corresponding entry may have been
included that stated, "2B1 killed 2A2", adding necessary details to the event. The
pertinent portions of the final event map and corresponding evaluation checklist for one
of the missions is shown below.

F-6


Field-Training Site Performance
Real World Rehearsal Squads

100

80 |
Mean Total 69 E.Level Ill Tasks |
Weighted ElLevel tl Tasks |
Score 40 Devel | Tasks |
20 ;
24.5
0
1 2

Mission Number

Figure 2. Mean performance scores by task level for RW rehearsal
squads at the field-training site

Comparing both groups’ performance on their last mission showed no differences (VE rehearsal
group = 92 versus RW rehearsal group = 91).

The VE rehearsal group’s performance on Level III tasks was better than that of the RW
rehearsal group, particularly during Mission 2. The RW rehearsal group performed noticeably
better on the Level II tasks. For Level I tasks, the VE rehearsal group’s performance was
slightly better than that of the RW rehearsal group.

In terms of within group change over missions, the VE rehearsal group showed
improvement across all task levels. However the changes observed for Level III and Level I
tasks were very slight. For the RW rehearsal group, improvement was most apparent for Level II
tasks. Performance showed only slight improvement for Level I tasks. Performance decreased
over missions for Level III tasks.

To determine which tasks were most impacted by the rehearsal manipulation, individual
tasks within each task level were analyzed. The results are discussed in the following sections.

Group performance on individual level III tasks. Individual task analysis focused on the
raw data frequency counts. Figure 3 shows the mean number of friendly squad casualties
suffered by each group by mission. Squad casualty rates were generally high (50 percent or
more). Mean casualty rates dropped slightly for the VE rehearsal group across missions (6 to
4.5) and increased slightly for the RW rehearsal group for the same missions (5 to 6.0).

Fratricide incidents were seldom, if ever, observed during the execution of the missions.
Only one fratricide was observed for the RW rehearsal group. This occurred during the second
mission from weapons fire (M16).

19



Table 1-3

Performance Results at the Field-Training Site: Level I Tasks and Behaviors

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2
Condition
# Personnel Flagging Incidents (10) Personnel Flagging Points
RW Mean 7.0 3.5 4.0 45
VE Mean 4.5 2.5 45 5.0
# Weapon Flagging Incidents (11) Weapon Flagging Points
RW Mean 1.0 1.5 5.0 5.0
VE Mean 1.0 2.0 5.0 5.0
# Incidents Where Weapons Not ‘Weapon Points
Held Correctly (12)
RW Mean 0.0 0.5 5.0 4.5
VE Mean 1.5 0.0 3.5 5.0
# Incidents Soldiers Where Team “Clear” Points
Did Not Yell “Clear” (13)
RW Mean 0.0 0.0 5.0 5.0
VE Mean 0.5 0.0 4.5 5.0
# Incidents Where Unit Tactics Tactics Points
Were Not Sound (14)
RW Mean 1.0 0.5 2.5 4.0
VE Mean 0.0 1.5 5.0 2.5
# Incidents Where Succession of Succession of Command Points
Command Not Established (15)
RW Mean 2.5 2.0 1.5 1.5
‘VE Mean 1.0 0.5 2.5 4.0.
23.0 24.5
23.0 24.5
25.0 26.5
25.0 26.5
Note. Level I Total

number/behavior in parenthesis.

1-3

Scores were obtained yy mul plying the Summed Points by 1. Task



USING VIRTUAL ENVIRONMENTS FOR CONDUCTING SMALL UNIT DISMOUNTED
MISSION REHEARSALS

EXECUTIVE SUMMARY

Research Requirements:

Soldiers and small unit leaders (platoon, squad, team) must be capable of taking effective
independent actions across an increasingly diverse range of military missions. The time and
resources required for conducting the appropriate training in realistic, live exercise settings can
be quite costly and also inefficient for certain skill areas (e.g., cognitive skills training). One
solution is to conduct a portion of this training in virtual environments using individual
combatant simulators. The objective of this research was to assess the use of virtual

environments as a viable dismounted infantry mission rehearsal tool. Two research questions
were addressed:

e Do skills acquired in the virtual environment transfer to the real world?

¢ How does real world performance differ as a function of rehearsal training site?

Procedure:

Thirty-two soldiers (four squads) received instruction on urban operation tactics,
techniques, and procedures. Each squad was then given two missions that involved clearing a
two-story building located at an urban operations training site. Two squads rehearsed (and
executed) the mission in a virtual representation of the exact building they would clear at the
urban training site. The remaining squads rehearsed in an actual two-story building that was
similar to the one they had to clear at the urban training site. Squad and individual performance
was objectively assessed by two observer/controllers for each mission. Soldier responses to the
training and evaluations comparing the effectiveness of the two rehearsal settings were obtained.

Findings:

Performance differences between the rehearsal groups across the two ‘real-world’
missions for both squad and individual tasks were small to negligible. Group performance
differences for casualties, i.e., fratricides, and personnel flagging (silhouetting oneself in
doorways and windows) were strongly affected by simulator constraints that made it very
difficult to execute precision movements in confined areas and effectively throw hand grenades.
The results showed that virtual environment rehearsal did not hurt real-world performance, but
were equivocal as to how much transfer did take place. Soldier effectiveness ratings for the two
rehearsal modes showed a clear bias that was dependent on the setting where they rehearsed.
Half of the soldiers who rehearsed in a virtual setting felt that virtual environments were just as
effective for conducting mission rehearsals as rehearsing in the real-world. In contrast, all

soldiers who rehearsed in the real-world felt that virtual environments were inferior to real-world
rehearsals.

vii



ACKNOWLEDGEMENT.

The authors would like to thank Mr. Richard Wampler of the TRW Systems and
Information Technology Group for his overall support of this research and his comments on a an
earlier version of this report. Mr. James Centric of TRW provided critical input into the structure
and development of the scenarios, the squad assessment instrument, and subsequent data analysis
and interpretation. He was invaluable in his role as an observer/controller. Mr. Michael Dlubac
assisted in the scenario development process, instrument refinement, and data analysis. He also
provided instruction in urban operations, conducted both the mission rehearsals and After Action
Reviews, and served as a second observer/controller. His support in these areas was outstanding.

We would also like to thank Second Lieutenant Charles Schwab for his overall support of
this research. Lieutenant Schwab served as one of the squad members in the experiment. He
was also responsible for conducting the majority of the statistical analyses involving the squad
assessment data. His assistance in these areas was greatly appreciated.

vi



Table 1

Sequence of Events for Virtual-Environment (VE) and Real-World (RW) Rehearsal Conditions

Event Rehearsal Condition
Real-World (RW) Virtual Environment (VE)

Day 1 Room clearing talk-walk and run Room clearing talk-walk and
Talk-Walk and Run rehearsals | rehearsals at field-training site. run rehearsals at virtual-
w/o OPFOR Building section similar to building | training site. Building

sections for RW missions sections identical to building

sections for RW missions

Day 1 Performance assessment at field- | Rehearsal at virtual-training
Mission 1 w/ OPFOR training site site
Day 1 Performance assessment at field- | Rehearsal at virtual-training
Mission 2 w/ OPFOR training site site
Day 2 Supplemental practice at virtual- Performance assessment at
Mission 1 w/ OPFOR training site field-training site
Day 2 Supplemental practice at virtual- Performance assessment at
Mission 2 w/ OPFOR training site field-training site

Note. Two squads per rehearsal condition.
Participants

Participants were 43 soldiers stationed at Fort Benning, Georgia. Soldiers were from
three groups, Infantry Officer Basic Course (IOBC) students, a Military Police (MP) National
Guard Reserve Unit, and drill sergeants (which also included command and support personnel).
Demographic and relevant training experience for the three groups are presented in Appendix B.

The IOBC sample was the youngest (26.9 years) of the three groups followed by the MPs
(34.5 years) and the drill sergeants (39.3 years). On the average the three groups spent very little
time each week playing virtual reality games (20-60 minutes per week) and had rarely, if ever,
trained at the McKenna urban operations training site. Overall, the IOBC sample had a wider
experience base in terms of courses completed and simulation exposure.

Soldiers were assigned to four squads for the experiment. Squad members had little, if
any, experience working together as a collective element. Rank structure within the squads did
not represent the structure of a typical infantry squad. The four squads were randomly assigned
to the two rehearsal conditions. One squad was composed of MPs and drill sergeants, one squad
consisted entirely of MPs, and two squads were composed of IOBC students. The composite
MP/drill sergeant squad and one IOBC squad were assigned to the RW rehearsal condition. The
remaining squads were assigned to the VE rehearsal condition. Each rehearsal condition
contained one 7 and one 9-man squad. The remaining soldiers served as OPFOR.



USING VIRTUAL ENVIRONMENTS FOR CONDUCTING SMALL UNIT DISMOUNTED
MISSION REHEARSALS

CONTENTS

INTRODUCTION .wsesssssssssssssessecccsssssssssescascsssnvecseesessssnnmuussssseegennumuesseescnasseeasesnneeesoanessese 1

Science and Technology Objective (STO) Virtual Environment Research... 3
Virtual Environments for Mission Rehearsal 3
Strategies for Effective Rehearsal 4
Research Questions 6
METHOD. 6
Overview... 6
Participants .. 8
Instruments 9
Mission Scenario 12
Equipment ... 12
Virtual-Training Site 12
Field-Training Site. 13
Procedure............ 14
Real-World Rehearsal Condition. 14
Virtual Environment Rehearsal Condition. 16
RESULTS .. 18
Field-Training Site Performance... 18
Virtual-Training Site Performance. 22
Comparison of Field Versus Virtual-Training Site Performance 26
Soldier Assessment of Rehearsal Training 27
Soldier Comparison of Real-World Versus Virtual Environment Mission
Rehearsals ......sssssscsssssessessessssssssesensnsesecssneesesssnecensusssscegennecsssnuenssssurecaneesacsaveeanecenseess 31
DISCUSSION .....sscscsssssssesssseesssnscecssnssnesessnsessesssesssensscesssnasecsssnasesssnuesstinessucesseeanessaeensees 35
Transfer of Skills Acquired in the Virtual Environment to the Real-World... 35
Real-World Performance Differences as a Function of Training Site 37
Soldier Comments on Using Virtual Environments for Mission Rehears: 39
Mission Rehearsal Effectiveness Ratings.. 41
Training in Virtual Environments 41

ix



Table I-5
Performance Results at the Virtual-Training Site: Level II Tasks and Behaviors

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2
Condition
# Rooms not Properly Cleared (4) Room Points
RW Mean 0.0 0.0 5.0 5.0
VE Mean 2.0 1.5 2.5 2.5
# Incidents Soldiers did not Position Points
Maintain Position (5)
RW Mean 0.5 0.0 2.5 5.0
VE Mean 1.5 1.5 2.5 2.5
# Incidents Soldiers did not Dispersion Points
Maintain Dispersion (6)
RW Mean 0.0 0.0 5.0 5.0
‘VE Mean 0.0 0.0 5.0 5.0
# Incidents Soldiers did not Supporting Fire Points
Provide Supporting Fires (7)
RW Mean 0.0 0.0 5.0 5.0
‘VE Mean 1.5 0.0 2.5 5.0
# Incidents Soldiers did not OPFOR Location Points
Accurately Report OPFOR
Location (8)
RW Mean 0.0 1.0 5.0 2.5
‘VE Mean 0.0 0.0 5.0 5.0
# Incidents Tm. Member did not Sector of Fire Points
Understand his Sector of Fire (9)
RW Mean 1.0 0.0 2.5 5.0
VE Mean 1.0 0.0 3.0 5.0
: 25.0 27.5
50.0 55.0
20.5 25.0
41.0 50.0

Note. Level Il Total Wt. Scores were obtained by multiplying the Summed Points by 2. Task
number/behavior in parenthesis.

1-5



replay was complemented with synchronized audio playback of verbal comments from the
soldiers during the mission. Specific data concerning the number of casualties and rounds fired
were captured but were not able to be summarized and displayed on the video screens. Casualty
counts were, however, available for discussion purposes during the AAR. McKenna support
personnel controlled the presentation of the audio and video feeds during the AAR. This allowed
the O/C to easily structure the focus of the AAR, e.g., what part of the mission to replay, viewing

angle.

. Simunitions and weapon instrumentation. Soldiers were issued Simunitions, rubber
bullets that contain red (OPFOR) or blue (BLUFOR) colored detergent, to help determine if a
casualty was the result of enemy or friendly fire. In addition, a laser system was used to
specifically determine who inflicted a particular casualty. This system consisted of a laser
device mounted on the weapon and a set of sensors that were attached to the soldier’s upper
body.

Procedure

Real-World Rehearsal Condition

Preparation. Soldiers in the real-world rehearsal condition reported to the McKenna
field-training site on Day 1 of the two-day experiment. They were briefed on the objectives of
the experiment, and were given the chance to ask any questions concerning their roles in the
experiment. They then completed the Biographical Information Questionnaire.

After completing the questionnaire, soldiers were divided into squad and OPFOR
elements. Squad and OPFOR members were briefed on the use of Simunitions and the laser
monitoring system. All soldiers then zeroed there weapons using the Simunitions. This took
approximately 30 to 45 minutes. Soldiers were issued two 20-round magazines (per mission).
Squads were issued 3 smoke grenades per squad per mission), and 6 flashbang grenades per
squad per mission not including rehearsals).

‘Next, the squad members received approximately one hour of training, provided by an
OIC, that included tactics, techniques, and procedures selected from FM 90-10-1 (Department of
the Army, An Infantryman’s Guide to Combat in Built-Up Areas - Chapter 5 and Appendix K,
1999). Training was designed to fit individual squad training needs, e. g., instruction on the
fundamentals of urban operations or refresher training. The second O/C provided some very
basic instruction to the OPFOR members on proper defensive procedures from inside a building
(rooms).

The OPFOR received the same guidance from the O/C for all scenarios, virtual and real
world. Due to manpower limitations and the need to maintain force ratios of approximately 3:1
(three squad members for every OPFOR soldier), the stairwell was “given” to the squad. That is,
the squad was given unhindered access off the stairwells. Moreover, the OPFOR was instructed
not to fire on the exercise squad until the third man had deployed on that particular floor of the
building. This restriction was established to enhance the squad’s survivability at an obvious kill
zone.

14


U.S. Army Research Institute
for the Behavioral and Social Sciences

Research Report 1806

Using Virtual Environments for Conducting
Small Unit Dismounted Mission Rehearsals

Robert J. Pleban
U.S. Army Research Institute

Jena Salvetti
Consortium Research Fellows Program

June 2003

Approved for public release; distribution is unlimited.

20030701 123 |



limited to one trial, that performance (spatial learning) was generally poorer for subjects who
rehearsed in a virtual environment setting as opposed to those who rehearsed in a real-world
environment. However, Ruddle, Payne, and Jones (1997) reported similar levels of performance
(spatial learning) were obtained for subjects who rehearsed multiple times (10) in a virtual
environment compared to those who rehearsed twice using a map. This research suggests that
rehearsing in a virtual environment may require more trials to reach the same level of proficiency
than is required for real-world rehearsals.

Real-World Performance Differences as a Function of Training Site

While the item weighting and scoring approach described earlier was useful in providing
an overall picture of group performance, it did not allow for in depth comparisons of specific
collective and individual tasks across groups and training sites. Some of the key task differences
that were identified are discussed in the following sections.

Level III tasks. The number of casualties suffered by the four squads was high in both
environments (real world and virtual). Analysis by casualty type (number OPFOR casualties,
number of friendly casualties from OPFOR fire and fratricide) showed that fratricides were most
responsible for group performance differences observed in these environments. All but one of
the fratricides occurred during execution of the virtual environment missions. The majority of
these fratricides were system related. The SVS system’s way of “throwing” grenades was
extremely awkward. The current way of throwing the grenade by timing the toss of the grenade
using the weapon with a calibration figure on the screen was difficult to master in the time
allotted to train this task. Also, when a soldier gets stuck in a wall, for example, if his grenade is
armed, and the soldier executes the unstick command (using his weapon mounted thumb switch),
the grenade drops and explodes, killing everyone in the immediate area.

Because of the manner in which grenades were thrown in the two environments was so
dissimilar, there was no carryover to the real-world missions for the VE rehearsal group. The
fratricide rate dropped to zero during the real-world missions for the VE rehearsal group. In
contrast, when the RW rehearsal group executed the same missions in the virtual environment
following their real-world missions, the fratricide rate jumped noticeably (one half of these
fratricides were due to grenades). If this same grenade system is used in the future, more time
will have to be allotted to training this task. The problems experienced by soldiers on this task
clearly diminished the realistic feel of these virtual mission rehearsals.

From a more general standpoint, there are several possible reasons for the high number of
casualties suffered by the squads in both environments. The scenarios involved a platoon attack
which typically involves three squads. The manpower for this exercise was limited to either a 7
or 9-man exercise squad. For practical and experimental reasons, the decision was made to
continue the mission well beyond the standard criterion of combat ineffectiveness (loss of one
fire team's worth of manpower), often to the extent of total attrition.

Another problem with the exercise was the force ratios (combat power including combat

multipliers) between the OPFOR and the exercise squad. Typically, force ratios for an offensive
operation in open terrain (terrain permitting unhindered movement) is 3:1 (three friendly soldiers

37



USING VIRTUAL ENVIRONMENTS FOR CONDUCTING SMALL UNIT DISMOUNTED
MISSION REHEARSALS

Introduction

Current Army transformation initiatives will require increasingly innovative
methods for training soldiers and small unit leaders (platoon, squad, team). Leaders, for
example, must be able to effectively lead multifunctional units, use digital systems, and
conduct decentralized and dispersed operations across the full-spectrum of possible
missions. Preparing soldiers for the diversity of missions and skills required to
effectively operate in these situations will present many challenges to trainers.

The time and resources required for conducting the appropriate training in realistic, live
exercise settings can be quite costly and inefficient, particularly for the specific leader skills
needed for such operations (see Pleban, Eakin, Salter, & Matthews, 2001). To develop and
sustain the tactical and technical proficiency required in these varied situations, tools are needed
that can provide (in a cost effective manner) frequent and repetitious complex task training
across diverse scenarios.

One potentially useful training tool that has received considerable attention involves the
use of virtual simulations, specifically, individual combatant simulators. A virtual training
system can be particularly useful for “walk-level” and sustainment trairiing (Pleban et al., 2001).
Virtual training not only provides the capability to immerse soldiers in a realistic operational
environment, but this training can be accomplished without risking the safety of soldiers,
breaking expensive equipment, or potentially polluting or destroying the environment. Virtual
training also allows soldiers to perform tasks too dangerous for the live environment such as
conducting close quarters combat, operating in NBC environments, and adjusting artillery or
close air support fires on or near an occupied friendly position. The virtual environment (VE)
provides a primary means for units to frequently and repetitively train realistic fire and maneuver
techniques (TRADOC System Manager Combined Arms Tactical Trainer, 2002). Virtual
environments can be effectively applied to enhance small unit leader skill training in decision-
making (Pleban et al., 2001), and may have training value in other areas such as operational
planning, command and control of small units, direct fire coordination, fire support coordination,
and integration of obstacles and logistical support.

At the Virtual Simulation Lab at Fort Benning, Georgia, the individual soldier or small
unit leader can explore innovative approaches for conducting specific tactical operations in
virtual settings. Through the use of individual combatant simulators, soldiers can immerse
themselves in virtual representations (using terrain data bases) of various training sites such as
the McKenna urban operations site at Fort Benning and conduct limited missions (e.g., clear a
building, conduct area reconnaissance). Virtual environments, in theory, offer soldiers the
opportunity to thoroughly familiarize themselves with procedural aspects of specific tasks as
well as a chance to examine new tactics and techniques. These simulators allow the soldiers to
execute scenarios and determine the impact of various courses of action on the likely success of a
mission (Pleban, Eakin, & Salter, 2000).



CONTENTS (continued)
Page
Table 8 Percent Response to Question “Compare the Effectiveness of Virtual
Environments Versus the Real-World for Conducting Mission
Rehearsals” .. 32
Table 9 Advantages to Conducting Mission Rehearsals in a Virtual
Environment .........scscssssssssessscsnecsnseesssnssceneessusesessssessauecesssnecssssesenssnesessenues 33
Table 10 Disadvantages to Conducting Mission Rehearsals in a Virtual
Environment .........cssscssessosesssessssensssessenseesnsessssnssceesanecsnsseessenssessssessesnsveseensne 33
Table 11 Soldier Responses to Question “How Would You Incorporate Virtual
Environments in Your Unit's Training?” ........csscssssssssssseessssssassesecessssseeees 35
LIST OF FIGURES
Figure 1 Mean performance scores by task level for VE rehearsal squads at the
field-training site 18
Figure 2 Mean performance scores by
field-training site ...........sssssssssssessssssesssssssseecssssnsnsneeeseenueseeeseessssnnseseesessessnsees 19
Figure 3 Mean number of friendly squad casualties by rehearsal group at the field-
training site - 20
Figure 4 Total number of level II task tactical incidents by
field-training site 21
Figure 5 Total number of level I task tactical incidents by rehearsal group at the
field-training site ............ssscseccseccsesecesseeesseeessnscessssnecsssescesssssucesssssseessnnsseesess 22
Figure 6 Mean performance scores by task level for VE rehearsal squads at the
Virtual-training Site .........ssseccssesssecssssssecssussssssecscerssseessesessessneesessnecesessssnsesess 23
Figure 7 Mean performance scores by task level for RW rehearsal squads at the
virtual-training Site ............sssssesssecsssssessensecsssessesssssessssseecsssssessnssuscesssensseessees 23
Figure 8 Mean number of friendly squad casualties by rehearsal group at the
virtual-training site we 24
Figure 9 Total number of fratricides by rehearsal group at the virtual-training site... 24
Figure 10 Total number of level II task tactical incidents by rehearsal group at the
virtual-training site 25
Figure 11 Total number of level I task tactical incidents by rehearsal group a!
virtual-training site 26



Table B-4
Relevant Training Experience for MP Student Sample by Course and Simulation System

(percent completed course or used simulation system)

PLDC | BNCOC | ANCOC | Airborne | Air Other | Indiv. Cbt. | Other Mil.
Assault | Courses*} Simulator | Simulations?

38.5 15.4 0 V7 0 30.8 53.8 38.5

Note. n= 13.

* Included MP school, NBC Defense.

» Included Simulation Networking, Firearms Training System, Engagement Skills Trainer, Virtual MP training,
Soldier Visualization System.

Table B-5
Demographic and Relevant Training Experience for Drill Sergeant Sample

Min Max Mean SD
Age (yrs) 29.6 43.3 39.3 49
Time in service (yrs) 9.9 22.1 18.2 4.5
Hours/wk playing 0 1.0 33 52
virtual reality games
Times trained at 0 0 0 0
McKenna
Note. n= 6.
Table B-6

Relevant Training Experience for Drill Sergeant Sample by Course and Simulation System
(percent completed course or used simulation system)

PLDC | BNCOC | ANCOC | Airborne | Air Other | Indiv. Cbt. | Other Mil.
Assault | Courses*| Simulator | Simulations”

83.3 83.3 66.7 33.3 0 100 0 16.7

Note. n= 6.
* Included Drill Sergeant School, Combat Lifesaver Course, Engineer Officer Advanced Course, Small Arms

School.
© Other military simulations were not listed.

B-2


Table 9
Advantages to Conducting Mission Rehearsals in a Virtual Environment

Provides a realistic preview of mission environment

Provides a comfortable training environment

Provides a safe training environment

Is effective use of training time and cost

Requires less physical demands, therefore allowing for more rehearsals
Allows leaders to analyze tactical issues/scenarios prior to mission execution
Can rehearse a mission many times if necessary

Is available when urban training sites are reserved

Builds unit cohesion and teamwork

Does not waste ammunition

Is good for teaching urban operation principles and skills

Allows for planning and rehearsing missions quickly

Disadvantages to rehearsing in a virtual environment. Soldiers were also asked to list
the disadvantages to rehearsing in a virtual environment setting. Table 10 summarizes their
comments.

Table 10 :
Disadvantages to Conducting Mission Rehearsals in a Virtual Environment

Restricts movement

Provides environment that is too comfortable

Lacks realism

Is difficult to use the controls and menu

Has a narrow FOV

Is less hands-on than the real-world training

Contains too many glitches in the technology (e.g., sticking)
Is too much like video game

Lack of mobility represented a major disadvantage according to many of the soldiers. The
‘weapon-mounted thumb switch did not allow soldiers to move fluidly. This thumb switch or
toggle approach to movement did not allow for effective rehearsal of such key urban operation
skills as moving through doors, moving through a building as a squad, and peeking around
corners, throwing grenades, or stacking.

As noted previously, soldiers clearly liked the comfortable environmental conditions
associated with virtual training. However, they felt that they still needed the more physical real-
world training to fully appreciate the conditions they would face in a real-world urban operation
setting such as heat, rain, cold, fatigue, stress, carrying heavy equipment and weapon recoil.

Soldiers felt that the controls were difficult to master to the level necessary to effectively
perform the missions in the virtual environment. Calibrating grenades and choosing weapons

33



Table I-6

Performance Results at the Virtual-Training Site: Level I Tasks and Behaviors

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2
Condition
# Personnel Flagging Incidents (10) Personnel Flagging Points
RW Mean 17.0 24.0 1.0 0.0
‘VE Mean 15.5 14.5 1.0 1.0
# Weapon Flagging Incidents (11) ‘Weapon Flagging Points
RW Mean 2.0 0.0 5.0 5.0
‘VE Mean 2.5 0.0 45 5.0
# Incidents Where Weapons Not Weapon Points
Held Correctly (12)
RW Mean 0.0 0.0 5.0 5.0
VE Mean 0.0 0.0 5.0 5.0
# Incidents Soldiers Where Team “Clear” Points
Did Not Yell “Clear” (13)
RW Mean 0.0 0.0 5.0 5.0
VE Mean 0.0 0.5 5.0 45
# Incidents Where Unit Tactics Tactics Points
Were Not Sound (14)
RW Mean 1.0 1.0 2.5 2.5
VE Mean 1.0 0.5 2.5 4.0
# Incidents Where Succession of Succession of Command Points
Command Not Established (15)
RW Mean 1.0 1.0 2.5 2.5
VE Mean 2.0 0.0 2.5 5.0
“21.0 20.0
21.0 20.0
20.5 24.5
20.5 24.5

number/behavior in parenthesis.

1-6

Note. Level I Total Wt. Scores were obtained by multiplying the Summed Points by 1. Task



Field-Training Site Level Il Incidents

No. of
Incidents

MVE Reh Gp
EIRW Reh Gp

Mission

Figure 4. Total number of level II task tactical incidents by rehearsal group
at the field-training site.

Group performance on individual level I tasks. Figure 5 shows the frequency of Level I
task incidents by rehearsal group across missions. More Level I incidents were observed for the
RW rehearsal group than the VE rehearsal group. Performance improved from the first to the
second mission for both groups. The majority of incidents observed centered around three tasks.
Task 10 - personnel “flagging” (body exposed in windows or head exposed around corners) had
the most incidents. A total of 35 flagging incidents were observed. The RW rehearsal group
flagged 21 times during the missions compared to 14 for the VE rehearsal group. Flagging
incidents decreased over missions for both groups.

For the remaining two tasks, task 11, weapon “flagging” (extending M16 barrel beyond
cover) and task 15, succession of command not established, the number of incidents observed
was 11 and 12 respectively. For weapon flagging, there was no difference between the two
groups, in terms of total incidents observed. With regard to not establishing succession of
command, the RW rehearsal was cited more often (9 incidents) than the VE rehearsal group (3
incidents). For both groups, performance changes from Mission 1 to Mission 2 were negligible
for these two tasks (see Appendix I, Table I-3 for individual incident rates and point totals for all
Level I tasks). .

21



or effects of combat multipliers to one enemy soldier). Due to the nature of combat in built-up
areas, more troops are normally needed than in other combat situations. In a built-up area, the
requirement can be as much as "three to five times greater than for an attack in open terrain"
[U. S. Department of the Army, 1999 (FM 90-10-1, pg. 9)].

As mentioned earlier, to preserve this ratio throughout as much of the mission as
possible, the squad was allowed unhindered access off the stairwells, an obvious kill zone, and
restrictions were placed on when the OPFOR could engage the exercise squad, i.e., not to fire on
the exercise squad until the third man had deployed on that particular floor of the building. This
restriction helped to some extent to enhance squad survivability and to counterbalance the
limiting force ratio, but could not completely counter the lack of manpower. In summary,
rehearsal site did not appear to have any real effect on the number of casualties suffered by each

group.

Level II tasks. When only the real-world performance of the two rehearsal groups is
considered, differences were very small. These differences were observed for only a subset of
Level II tasks. Incident rates were highest for failure to properly clear rooms, maintain proper
position for providing protective cover, place soldiers to provide supporting fires, and accurately
report OPFOR location.

Performance differences (total number of incidents across tasks) at the field-training site
were particularly noticeable at Mission 2. The incident rate for the VE rehearsal group was
higher than that of the RW rehearsal group. Performance at the virtual-training site revealed the
same incident pattern for both missions (more incidents observed for the VE rehearsal group than
the RW rehearsal group. The overall incident pattern across all tasks appears to show that the
RW rehearsal group’s performance in the virtual environment benefited from their prior real-
world experience. Prior virtual environment experience did not, however, lead to improved real-
world performance, i.e., fewer negative incidents detected.

Why the virtual environment mission performance of the VE rehearsal group was so
much poorer than the RW rehearsal group for the Level II tasks, particularly for room clearing,
providing protective cover, and to a lesser extent providing supporting fire, is not clear. Given
the small sample size, and the unexpected pattern of results, any explanation should be viewed
cautiously.

Level I tasks. Of the six Level I tasks, personnel flagging accounted for the large
majority of total incidents observed. Most of these personnel flaggings occurred in the virtual
environment. This problem is primarily a function of SVS system constraints. Close, precise
movements, such as moving around corners, crouching under windows while moving, and wall
hugging are difficult to accomplish smoothly in the virtual environment. In order to move
around corners, the soldier has to move further away from the wall than he would like, which
causes him to silhouette himself in doorways. Also, the challenge of moving in a physically
restricted area and then crouching and moving as one approaches a window is frequently
overlooked as the soldier is too busy focusing on trying to move from one spot to another with
his team. Asa result, he silhouettes himself in the window. With regard to hugging walls,

38



With regard to OPFOR casualties, casualty rates were essentially the same for each
group. There was no change in the number of OPFOR casualties inflicted from mission 1 to
Mission 2 (see Appendix I, Table I-1 for individual incident rates and point totals for all Level III
tasks). :

Field-Training Site Casualties
7
6
54
Mean No. of 4 |
Friendly 3 MVE Reh Gp
Casualties ~ | EIRW Reh Gp
24
44
o4
41 2
Mission Number

Figure 3. Mean number of friendly squad casualties by rehearsal group
at the field-training site.

Group performance on individual level II tasks. Figure 4 shows the frequency of Level
II task incidents observed over the two missions. As can be seen from Figure 4, more Level II
incidents were observed, overall, for the VE rehearsal group (17) than the RW rehearsal group
(11) over missions. Incidents for this group increased from Mission 1 to Mission 2. In contrast,
Level Il incidents decreased by approximately 50 percent across missions for the RW rehearsal

group.

Further analysis showed that three tasks accounted for most of the incidents that were
observed (task 4 - rooms not properly cleared); task 7 - failure to place soldiers to provide
supporting fires; and task 8 - failure to accurately report OPFOR location). The rehearsal
groups did not differ in the total number of room clearing incidents observed. Interestingly, all
incidents of improper room clearing were committed on the first mission for the RW rehearsal
group. In contrast, the incident rate stayed about the same for the VE rehearsal group from
Mission 1 to Mission 2.

For task 7, failure to provide supporting fires, all incidents observed (4) were committed
by the VE rehearsal group. Performance for this group degraded from Mission 1 to Mission 2.
For task 8, slightly more incidents of not reporting the OPFOR location were noted for the VE
rehearsal group (4 versus 2 for the RW rehearsal group). Performance for both groups degraded
slightly over missions. The overall incident rate, however, for both tasks 7 and 8 was low (see
Appendix I, Table I-2 for individual incident rates and point totals for all Level II tasks).

20



9. Soldiers were NOT placed to
provide supporting fires for
another soldier conducting a
movement

Room 1

Stairwell.

10. Soldiers did NOT accurately
report location of OPFOR

Room 1

Stairwell

TI. Team member did NOT
understand his respective sector
of fire and provide adequate fire

support.

Room 1.

Stairwell

12. Personnel "flagging" (exposed
in windows) for greater than 5
seconds

Room 1

Stairwell.

13. Soldiers “flagged” (extend
barrel beyond cover) their
weapon barrels at corners

Room |

Stairwell

14. Weapons were NOT held
correctly when entering and
inside aroom. Muzzles were
NOT pointed away from friendly
forces.

Room 1

Stairwell

15. Once room has been cleared,
team did NOT yell, "Clear" to
inform the support party.

Room 1

Stairwell.

16. Unit tactics were NOT
doctrinally sound

Room 1

Stairwell

17. Succession of command was
NOT established

Room 1

Stairwell

F-2



Utilization of Findings:

This research showed that the effectiveness of virtual environments for conducting small
unit dismounted mission rehearsals is limited by a number of interface and technology problems.
These shortcomings make it difficult to train specific squad drills and tasks, like building
clearing, which emphasize rapid and precise positioning and movement and use of weapons.
While virtual environments show promise for this type of training, they do not appear to be as
effective as real-world tactical training at the present time. Virtual environments could,
however, be effectively used during the “walk” phase of training for improving decision-making,
situation awareness, communication, and coordination skills. These environments, if used in
conjunction with realistic field exercises, could play a major role in enhancing the training of
soldiers and small unit leaders.

viii



Table 2
Importance Ratings and Associated Frequency Band Scores by Task (from Evaluation Checklist)
Task/Behavior Weight Frequency Band and
Associated Score
1. Number of BLUFOR casualties 3 0=5 pts 3=Ipt
1=4 pts >4=0pts
2=2pts
2. Number of OPFOR casualties 3 3=5 pts
<2=0 pts
3. Number of BLUFOR fratricides 3 0=5 pts
>1=0 pts
4. Number of rooms not properly cleared 2 0=5 pts .
>1=0 pts
5. Soldiers did not maintain relative position with each 2 0=5 pts
other providing protective cover >1=0 pts
6. Soldiers did not maintain dispersion at danger areas 2
(areas with more than one angle of attack)
7. Soldiers were not placed to provide supporting fires 2
for other soldier conducting movement
8. Soldiers did not accurately report OPFOR location 2
9. Team member did not understand his respective 2
sector of fire and provide adequate support
10. Personnel “flagging” (exposed in windows or head 1
exposed around corner) for > 5 seconds
11. Soldiers “flagged” (extended M16 barrel beyond 1
cover) at corners
12. Weapons not held correctly (entering a room 1
muzzles pointed at friendly forces)
13. Once room was cleared, team did not yell, “Clear” 1
to inform the support element 1=4 pts 4=I1pt
2=3 pts >5 = 0 pts
14. Unit tactics were not doctrinally sound 1 0=5 pts
1=3 pts
22 = 0 pts
15. Succession of command not established 1 0=5 pts
1=3 pts
>2=0 pts

Note. 3 = high priority task; 2 = moderate priority task; 1 = low priority task

11



13. How often have you trained at the McKenna MOUT site (not including demos)?

times
14. Have you ever been in a Virtual Individual Combatant (VIC) simulator at the Land Warrior
Test Bed before?

Yes No

If YES, which one(s)? (Describe if you cannot remember the name)

15. Have you had any other experience with military computer simulations?

If yes, please describe briefly or give the names of the simulators.

A-2


Table I-4

Performance Results at the Virtual-Training Site: Level III Tasks and Behaviors

number/behavior in parenthesis.

1-4

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2
Cendition
# BLUFOR Casualties (1) BLUFOR Casualty Points
RW Mean 5.0 5.0 0.5 0.5
.| VE Mean 6.0 5.5 0.5 0.0
# OPFOR Casualties (2) OPFOR Casualty Points
RW Mean _ | 2.0 2.5 2.5 2.5
VE Mean 2.0 2.0 2.5 2.5
# Fratricides (3) Fratricide Points
RW Mean 15 15 2.5 2.5
VE Mean 1.5 3.0 0.0 0.0
RW Summed Points eee 5.5 5.5
RW Total Wt. Score 16.5 16.5
‘VE Summed Points 3.0 2.5
9.0 75

Note. Level Hl Total We Scores were obtained by multiplying the Summed Points by 3. Task


E-2



Soldiers remarked that the real-world training environment was “hands-on” and allowed
them to experience the environment as they would during a real-world mission. The most
frequently cited characteristics that contributed to this “hands-on” training environment included
the ability to climb stairs, lean against walls, throw grenades, use hand signals, peek around
corners, use peripheral vision, and move obstacles. All of these tasks or behaviors were viewed
as essential components that must be addressed in training soldiers how to effectively clear
buildings. Other characteristics that soldiers mentioned that increased the realism of this type of
rehearsal training included climate and battle gear requirements.

Additional comments indicated that the soldiers clearly enjoyed using simunitions during
the execution of the missions. Soldiers felt that these paint filled bullets created the perception
of live fire battle, and motivated them to take cover and to move more tactically. Soldiers added
that getting shot by a simunition round provided them with instant feedback that they had made a
mistake. One soldier commented that training in the real world with simunitions helped in
“., eliminating anxiety about what would happen when rounds fly.” Soldiers also noted that the
training allowed them to identify team weaknesses and taught them to respond more quickly and
efficiently.

Table 4
What Soldiers Liked Most About Rehearsing in Real-World and Virtual-Environment Settings
Real World Provided a realistic training setting
Improved understanding of the mission and related tasks
Enhanced the learning process
Addressed team weaknesses quickly
Provided effective urban operations training
Made mistakes more visible.
Virtual Environment Provided a comfortable environment

Was similar to real-world training

Improved preparation for urban operations
Provided a safe training environment

Aided mission planning

Introduced mission related tactics effectively
Increased squad integrity or completeness
Provided entertaining, attention grabbing training

eoeveeevevieeeveee

The general theme concerning virtual mission rehearsal training was that this
environment provided soldiers with a comfortable training setting. The training was not
physically taxing, they were not required to wear full battle gear, and the climate was ideal (cool
with low humidity). Soldiers indicated that the absence of these physical distractions allowed
them to be able to conduct more rehearsals and train for longer periods of time.

A second common theme was safety. Soldiers believed that they were safe from injury
while training in the virtual environment.

28





What soldiers liked least about rehearsing in a real-world or virtual-environment setting.
Aspects of rehearsing in a real-world or virtual-environment setting that soldiers liked the least
are summarized in Table 5. For soldiers in the RW rehearsal condition, the primary training
comments focused on the slow pace of training and limited rehearsal opportunities (two). A few
soldiers mentioned that they would have liked more basic urban operation skill training (to
include clearing buildings) at the unit level, prior to rehearsing at the McKenna site.

The remaining comments focused on the weather conditions at the McKenna site.
Soldiers disliked the heat that they were exposed to during the training. This research was
conducted in mid July where temperatures hovered in the mid 90s with very high humidity. The
discomfort was magnified by the additional protective gear (body vests, face shields) the soldiers
wore to protect them from the simunition rounds. The extreme heat was one reason for the slow
training tempo. The impact that climatic conditions may have had on training was not discussed.

Table 5
What Soldiers Liked Least About Rehearsing in Real-World and Virtual Environment Settings
Real World The heat

Not enough time for many rehearsals

The slow pace

Virtual Environment Difficult to maneuver through the environment
Less realistic than the real-world training

Time required to learn to use the virtual system
The menu was difficult to use

Getting stuck in walls affects the mission
Grenades are difficult to use

Field of view is limited

The microphone was too loud

Soldiers in the VE rehearsal condition commented on several aspects of the environment
that they felt had negatively affected their mission performance. The major theme focused on
maneuverability. Soldiers indicated that they were not able to move as fluidly or as naturally
(using the weapon mounted thumb switch) as they would have in the real-world environment.
They felt that the constraints on mobility increased the difficulty for them to execute the
movements necessary to complete mission related tasks.

Soldiers also felt that the “sticking” aspect of this virtual system further affected the
fluidity of motion. If the soldier got too close to the sides of a building, parts of his body would
get stuck in the wall. Sometimes the soldier’s limbs or weapon would protrude through the wall
into the enemy’s view. When the sticking resulted in exposed limbs or weapons, the element of
surprise was lost, and soldiers shot at the limbs/weapons, resulting in casualties early in the
mission.

Soldiers mentioned several other aspects of the system that affected its efficiency as a
training tool. These included: difficulty in performing simple tasks (i.e., throwing grenades),

29



Orasanu, J. & Salas, E. (1993). Team decision making in complex environments. In G. A.
Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action:
Models and methods (pp. 327- 345). Norwood, NJ: Ablex.

Pearson, C. (1991). An assessment of extrinsic feedback on participation, role perceptions,
motivation, and job satisfaction in a self-managed system for monitoring group
achievement. Human Relations, 44, 517-537.

Pleban, R. J., & Beal, S. A. (2002). Simulating night vision goggle effects in a virtual
environment: A preliminary evaluation (Research Report 1789). Alexandria, VA: U. S.
Army Research Institute for the Behavioral and Social Sciences. (ADA402194)

Pleban, R. J., Eakin, D. E., & Salter, M. S. (2000). Analysis of mission-based scenarios for
training soldiers and small unit leaders in virtual environments (Research Report 1754).
Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
(ADA373762)

Pleban, R. J., Eakin, D. E., Salter, M. S., & Matthews, M. D. (2001). Training and assessment of
decision-making skills in virtual environments (Research Report 1767). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social Sciences. (ADA389677)

Richardson, A. E., Montrello, D. R., & Hegarty, M. (1999). Spatial knowledge acquisition from
maps and navigation in real and virtual environments. Memory and Cognition, 27,741-
750.

Ruddle, R. A., Payne, S. J., & Jones, D. M. (1997). Navigating buildings in “desk-top” virtual
environments: Experimental investigations using extended navigational experience.
Journal of Experimental Psychology: Applied, 3, 143-159.

Salter, M. S., Eakin, D. E., & Knerr, B. W. (1999). Dismounted warrior network enhanced
restricted terrain (DWN ERT): An independent assessment (Research Report 1742).
Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
(ADA364607)

Salvendy, G. (1997). Handbook of human factors and ergonomics (2" ed), New York: Wiley.

Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles
in three paradigms suggest new concepts for training. Psychological Science, 3, 207-217.

Swezey, R. W., & Llanerus, R. E. (1997). Models in training and instruction. In G. Salvendy
(Ed), Handbook of human factors and ergonomics (2TM ed., pp. 514-577). New York:
Wiley.

TRADOC System Manager Combined Arms Tactical Trainer (2002). Initial system training
plan for the virtual leader effects trainer (Appendix F). Fort Leavenworth, KS: Author.



separate locations at the Virtual Simulation Lab. Squad members then conducted a practice
scenario (15 minutes) in the simulators to fine-tune their collective movement skills.

This practice scenario was conducted in a virtual building similar to the one they would rehearse
in later.

Virtual-environment rehearsal at virtual-training site. The squad then received their
operation order and given approximately 30 minutes for planning. The OPFOR was briefed
separately. Squad members then rehearsed the mission twice in the simulators without an
OPFOR (talk-walk and run) with both O/Cs observing from desktop monitors in the battlemaster
room. This took approximately 45 minutes (30 minutes for the talk-walk, and 15 minutes for the
tun rehearsals) . These rehearsals were conducted in a virtual representation of the same section
of building that they would clear the next day at McKenna. Corrective feedback was provided
by the O/Cs following each rehearsal. Unlike the talk-walk and run rehearsals conducted in the
real world, the O/C could not walk with the soldiers in the various rooms, hallways, or stairwell
to physically point out danger areas, and demonstrate correct tactical movements. The O/C
could, however, talk, demonstrate, etc. once all soldiers came out of their SVS systems, but this
was conducted in an open area without direct physical access to the building.

Once this rehearsal phase was completed, the squad then executed the same mission
against the OPFOR element. After the mission, the O/Cs completed the Evaluation Checklist
and then conducted the AAR. The same procedure was repeated a second time. For Mission 2,
the rehearsal was conducted first without (talk-walk and run) and then with OPFOR in a virtual
representation of the same building section (different from the section used in Mission 1) they
would clear the following day.

Following the mission, the O/Cs completed the Evaluation Checklist and conducted the
AAR. The soldiers then completed the Post Training Questionnaire. This completed the first
day of testing.

Performance assessment at field-training site. On Day 2, soldiers reported to the
McKenna urban operations site. Preparation was the same as for the RW rehearsal group.
Soldiers were instructed in the use of Simunitions and the laser weapon monitoring system.

After zeroing their weapons, soldiers received their first operation order. The OPFOR was
briefed separately. As before, the squad and OPFOR were given 30 minutes for planning. The
squad then executed the mission, with the two O/Cs monitoring the events at the mission site and
the control center. After the mission, the soldiers proceeded to the control center for the AAR
while the O/Cs completed the Evaluation Checklist.

The procedure was repeated one more time starting with receipt of the second operation
order. After the final AAR was conducted, soldiers then completed the Mission Rehearsal
Comparison Questionnaire.

17



Appendix A

Biographical Information Questionnaire

Name. Unit. Date.

Please fill in the blank or mark or circle the appropriate response.

1, What is yourage? __—s—sYears _—— Months
2. MOS
3. Rank
4. Time inservice Years =» Months__

What is your current (or most recent) duty position?
How long in this position?

6. What Army training courses have you completed? Check all that apply.

PLDC BNCOC ANCOC Airborne Air Assault

Other (please specify)

7. How susceptible to motion or car sickness do you feel you are?

1 2 3 4 5 6 7
not moderately highly
susceptible susceptible susceptible

8. Do you have normal 20/20 vision without glasses? Yes __No

9. Do you have 20/20 vision with contact lenses or glasses? Yes No NA

10. Are you color blind? Yes No

11. Are you right handed? left handed?

12. How many hours per week do you play ‘virtual reality’ type games? hours per week



Appendix F

Evaluation Check List
(Circle correct responses)

. SQD: 1 2 3 4 DAY: 1 2 Mission: Rehearsal 1 2
Mission 1 Mission2 Environment: LWTB McKenna
Start Time ENDEX.
Location at ENDEX.
Reason for ENDEX.
|
Task Evaluated wt. NOTES (locations of incidents) ADDITIONAL
NOTES
1. Rounds expended
(data not included in 1 | 5.56 mm Issued:
analysis) 5.56 mm Expended:
2. Weapon malfunctions* Information only:
(data not included in analysis)
3. Number of BLUFOR
casualties Room 1 2. 3. 4 5.
3
Stairwell 6. 7 8 9

4. Number of OPFOR casualties
Room | 2. 3 4 5

3
Stairwell. 6 7. 8 9
5. Number of BLUFOR
fratricides Room 1. 2 3 4 5
3
Stairwell 6. 7. 8 9
6. Number of rooms NOT
properly cleared. (Closets, behind Room 1 2. 3. 4 5.
doors, & behind and under 2
furniture) Stairwell 6 7. 8 9

7. Soldiers did NOT maintain
their relative positions with each Room 1 2 3 4 5
other providing protective cover 2

Stairwell. 6 7 8 9

. 8. Soldiers did NOT maintain
dispersion at danger areas (areas 2 | Room 1 2 3 4 5
with more than one angle of
attack) Stairwell 6 7 8 9

F-1



Research Report 1806

Using Virtual Environments for Conducting Small Unit
Dismounted Mission Rehearsals

Robert J. Pleban
U.S. Army Research Institute

Jena Salvetti
Consortium Research Fellows Program

Infantry Forces Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

June 2003
Army Project Number Personnel Performance and
20262785A790 Training Technology

Approved for public release; distribution is unlimited.

iii



at the field-training site, particularly Mission 2 at the virtual-training site and the two field-
training site missions.

However, the data seem to show that optimal performance was achieved somewhat
quicker in the RW rehearsal condition. The primary reason the VE rehearsal group appeared to
take longer to attain the same level of performance proficiency as the RW rehearsal group could
be attributed to the lack of physical fidelity of the SVS systems, particularly in the areas of
mobility, weapon system operation, and the lack of interaction with the virtual environment
(inability to lean against walls, climb stairs, move stealthfully around corners, move obstacles,
use hand and arm signals) most notably for Level II tasks.

The majority of the soldiers felt that a large component of urban operation training was
the “hands-on” time required to physically rehearse such tasks as entering and clearing buildings,
and the associated tactical/high precision movements. This perception was particularly strong
with soldiers in this experiment since, as a collective group, they had little if any urban operation
training and had never worked together as a cohesive small unit element.

It is interesting to note, however, that the talk/walk and run rehearsal phases for the RW
rehearsal group were relatively brief (one hour for both phases) but “hands-on”. This may have
been enough time to familiarize them with the realistic task demands of the upcoming missions
since both missions were fairly straightforward and did not require extensive planning.

In summary, using the criterion discussed earlier, VE rehearsal did not produce clear,
positive transfer. Bctween group differences were relatively small for the most part. Group
performance, as indexed by the weighted total scores, shows that even with some of the
limitations listed above, the virtual rehearsals did not negatively impact performance.

Rehearsal time. As mentioned earlier, resource constraints required certain compromises
in the design of the experiment. The number of rehearsals was limited to two for the RW
rehearsal group and six for the VE rehearsal group (two talk/walk, two run, and two force-on-
force rehearsals (virtual-training site Missions 1 and 2). Earlier research (see Nullmeyer &
Spiker, 2000) indicated that increasing the amount of practice should result in improved
performance. The maximum weighted total score attainable from the Evaluation Checklist was
135, so there was clearly room for improvement for both groups. Based on the final mission
performance scores during the last mission, both groups had attained approximately 68 percent
of the total points possible. Performance appeared fairly stable across missions (trials).
Additional research is needed to determine how many trials are required to substantially improve
performance from current levels and the cost/benefit of conducting extra trials at both the Virtual

_Simulation Lab and the McKenna training site. It would also be instructive to compare the
performance patterns of the inexperienced squads used in this experiment with more experienced
intact squads who have worked together with respect to learning rates and final proficiency
scores,

It could be argued that the four additional rehearsals provided the RW rehearsal group

was responsible, in part, for the similar levels of performance between the two rehearsal groups.
Richardson, Montello and Hegarty (1999) found that when learning or rehearsal trials were

36



caused some frustration during mission rehearsal. Soldiers commented that it took too much
time to scroll through the menu options and they often missed the weapon they had intended to
select and had to start from the beginning of the menu. This delay in action often provided the
enemy with needed time to detect and engage the soldier, which resulted in additional casualties
for the squad.

Soldiers indicated that the limited FOV in the virtual environment did not permit them to
use their peripheral vision. In order to see peripherally, they had to move their own avatar
(simulated soldier image) by manipulating the toggle which was awkward and imprecise.
Soldiers stated that peripheral vision is a sense that is needed for conducting critical operational
tasks (precision movements through doors and rooms).

Another disadvantage mentioned by soldiers was the lack of physical interaction with the
virtual environment. Specific examples cited included the inability to lean against walls, feel the
presence of team members who are in close proximity, and use hand signals and eye contact for
communication. Soldiers also felt that because they could not feel or hear rounds flying through
the air that they were missing valuable environmental cues to take protective cover.

Incorporating virtual environments in unit training. Soldiers thought virtual
environments could be effectively used at the unit level for familiarizing unit members with the
mission environment prior to arrival at the real-world site (see Table 11). This included
familiarization of the layout of the buildings, rooms and surrounding areas (e.g., possible target
locations, enemy movement routes, observation points). Virtual environments could also be
incorporated in unit training for mission planning and for refining tactics.

Soldiers mentioned that virtual environments could provide the unit with alternate, cost
effective training that could be exploited when time/resource constraints limit the availability of
real-world training sites. For example, fundamental building clearing procedures, tactics, and
leadership training could be conducted before transitioning to the to the real-world site. The
actual urban training site could then be more efficiently used to focus on the physical execution
of the mission. In addition, soldiers indicated that the virtual environments could be used to
conduct refresher training. Virtual environment training was also viewed as an effective means
to teach leaders and squad members how to work together, develop unit standing operating
procedures (SOP), and improve decision-making skills.

34



The 22 questions were once again reviewed, this time for application. Those
deemed subjective in nature or too difficult to observe in the virtual environment (for
comparative analysis) were eliminated. This reduced the assessment criteria to 17. The
remaining assessment criteria and supporting rationale are listed in the following table.

Task

Rationale

1, Rounds expended.

Because of the potentially high ammunition expenditure normally
associated with a MOUT mission, it was necessary to compare
expenditures in the two environments in semi-controlled scenarios.
There is a possibility that the virtual environment might create bad habits
(unlimited bullets/grenades).

2. Weapon malfunctions.

‘Weapon malfunctions are a fact of life during training and real life
operations. Based on personal military experience there are more
malfunctions with live ammunition than with blank ammunition (most
caused by old magazines). Virtual and Simunition weapon malfunctions
were tracked, as these magazines are not a true representation of actual
magazines.

3. Number of BLUFOR
casualties.

Tracking casualties is one of the most frequently used methods of
“keeping score”/evaluating performance to determine mission
success/failure.

4. Number of OPFOR.
casualties,

‘Tracking casualties is one of the most frequently used methods of
“keeping score”/evaluating performance to determine mission
success/failure.

5, Number of BLUFOR
fratricides.

Fratricides are unacceptable in any military operation. They not only
hinder unit capabilities by reducing available friendly strength, but also
tend to degrade unit morale, thus hindering performance even further.

‘6. Number of rooms NOT
properly cleared.

Complete room clearing (closets and under/behind furniture) eliminates
the chance of a surprise attack from a “previously cleared” room. This
also eliminates the need to re-clear rooms.

7. Soldiers did NOT maintain

Maintaining an individual position with its proper sector of fire and

relative position with each interlocking that sector with fellow team members’ positions and their
other providing protective sectors of fire is essential in room clearing to fully cover the entire area,
cover. to ensure all enemy are eliminated, and to prevent fratricide.

8. Soldiers did NOT maintain Maintaining individual dispersion reduces the chance that a single
dispersion at danger areas bullet/grenade will kill or wound several personnel simultaneously,
(areas with more than one angle | thereby rendering the unit combat ineffective. It also allows the unit to
of attack). protect and secure a larger area.

9. Soldiers were NOT placed to | Controlled supporting fires during room clearing operations greatly
provide supporting fires for reduces the risk of fratricide and simultaneously increases the

other soldiers conducting effectiveness of those fires in engaging the enemy.

movement.

10. Soldiers did NOT The importance of communicating the location of the enemy to fellow
accurately report OPFOR, squad members allows the leader to maneuver soldiers and mass fires
location. against the enemy, while reducing the likelihood of fratricide.

11. Team member did NOT The team leaders ability to clearly communicate instructions to team
understand his respective sector | members reduces the battlefield confusion inherent in MOUT operations.
of fire and provide adequate This allows positive contro] of his team and enhances the team’s ability
support. to control the flow of the fight.

F-4



could also rehearse an upcoming mission in a tactical environment similar or identical to the one
they would encounter in the real world (Nullmeyer & Spiker, 2000). Through the use of
networked simulators, an entire operational team (ground forces, additional aviation systems,
command and control group) can interact with each other and rehearse as a coordinated element.

There are advantages to this type of rehearsal approach. First, the mission plans can be
validated using live players. Second, tactics can be refined and improved prior to the actual
mission. Third, unit members, having executed the virtual mission, should be more prepared and
confident of their abilities. Finally, command and contro] elements should have a clearer
understanding of potential battlefield dynamics and risks (Nullmeyer & Spiker, 2000).

The concept of simulation-based mission rehearsal represents a potentially powerful
training tool. Due to the recency of this application, there is little direct research documenting
the effectiveness of virtual environment mission rehearsal in either aviation or dismounted
infantry settings.

The available research has shown that virtual environment mission rehearsal has been
effectively used, for example, to improve pilots’ understanding of mission plans [Nullmeyer,
Bruce, Conquest, & Reed, 1992); enhance subsequent use of available flight time (Lintern,
Sheppard, Parker, Yates, & Nolan, 1989); improve target detection of objects at greater ranges
(Krebs, McCarley, & Bryant, 1999) and; enhance route knowledge within buildings (Witmer,
Bailey, Knerr, & Parsons. 1996). Virtual environments were also found to be as effective as
more traditional rehearsal strategies, i.e., map-study, with regard to training navigational
accuracy (Williams, Wickens, & Hutchinson, 1994).

Strategies for Effective Rehearsal

Direct evidence that details how to best use these simulation systems to conduct effective
mission rehearsals is also limited. To maximize the effectiveness of virtual environment systems
for conducting mission rehearsals, Nullmeyer and Spiker (2000) present a number of learning
and instructional principles that trainers should consider when developing specific virtual
environment rehearsal strategies. Some of these principles are discussed in the following
sections.

Encoding. For rehearsals to be effective, the individual must internally encode the key
behaviors or tasks to be trained. Research has shown that visually replaying the task or mentally
rehearsing the steps involved in doing a task, e.g., karate moves, shooting free throws, rotary
pursuit tracking, typing, guitar and vocal performance etc. has a positive but (sometimes) modest
effect on subsequent performance (see Nullmeyer & Spiker, 2000). Driskell, Copper, and Moran
(1994) conducted a meta analysis of 35 studies on mental practice. They found that mental
practice is an effective but weaker means for enhancing performance than overt, physical
practice. This type (mental) of practice is particularly effective for enhancing performance on
cognitive tasks but the positive effects of mental practice declines over time. Finally,
experienced subjects benefit equally well from mental practice regardless of task type, while
novices benefit more from mental practice on cognitive tasks.





Appendix D

Virtual Environment Post Training Questionnaire

Name Date Condition.

1. Mission role:
BLUFOR OPFOR

2. How effectively do you think this rehearsal training has prepared you for the clear building
missions you will conduct tomorrow?

Very Ineffective Ineffective Fairly Effective Very Effective

3. Was the rehearsal time adequate?

Yes No

4. What did you like most about rehearsing in a virtual environment?

5. What did you like least about rehearsing in a virtual environment??

6. What, if anything, would you do to improve the effectiveness of virtual environments for
conducting mission rehearsals?

D-1



should focus on relevant task functions that are critical to mission success and which can be
rehearsed in a 30 to 60 minute block of time (Nullmeyer & Spiker, 2000). The scenarios should
include appropriate mission specific information that will enhance the cognitive processing
capabilities of the participants in such areas as self/situation awareness, problem solving, and
planning (Nullmeyer & Spiker, 2000).

Structure mission rehearsals to promote decision-making and shared understanding.
The execution of any tactical mission is a team effort. Accordingly, the mission rehearsal setting
should incorporate techniques that enhance group performance. Particular attention should be
directed to improving decision-making skills since mission outcome is often the function of a
series of complex decisions. Klein (1993) stresses the importance of situational assessment and
the experience of the decision-maker in evaluating the shortcomings of a course of action. These
factors can be systematically addressed in a series of well-crafted scenarios. Pre-executing the
mission in a realistic rehearsal environment gives the leader and team members an opportunity to
address relevant decision issues in the context of the specific mission to be accomplished.
Together, with the appropriate feedback, simulation-based rehearsals should help participants
establish more accurate expectations concerning the planned course of action, identify key
decision points, and better prepare them to operate under time limited and stressful conditions.

Research Questions

The primary objective of this research was to compare the effectiveness of rehearsing for
a mission in a virtual environment versus rehearsing in a real-world (RW), i.e., field setting.
There were two key research questions: 1) Do skills acquired in the virtual environment transfer
to the real world? and 2) How does real-world performance differ as a function of training
(rehearsal) site?

Resource constraints required certain compromises. Frequent deployments and
increasing commitments made it particularly difficult to find soldiers in sufficient numbers. This
‘was compounded by the fact that performance assessment was based on the squad’s actions
which severely reduced the sample size. (Individual performance was assessed in a squad
context.) The number of rehearsals were also truncated due to cost limitations. Wherever
possible, the structure of the rehearsals were crafted to maximize learning and to enhance
transfer of training effects. More specifically, rehearsal sessions were followed by timely,
constructive feedback. Rehearsals were generally short (less than 30 minutes) and designed to
promote leader and squad member decision-making and situational understanding.

Method
Overview

Soldiers received a brief block of instruction on urban operations tactics, techniques and
procedures. Training was conducted by a retired non-commissioned officer with extensive
experience in urban operations. Soldiers, acting as a squad element, executed two missions.
Both missions involved clearing a two-story building. For this experiment, a large section of one
building was divided in half. The missions involved clearing each half of the building. The



REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE. 3. DATES COVERED (from. . . to)
June 2003 Final

January 02 — January 03

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER,
Using Virtual Environments for Conducting Small Unit Dismounted

Mission Rehearsals 5b. PROGRAM ELEMENT NUMBER

622785
6. AUTHOR(S) Sc. PROJECT NUMBER
Robert J. Pleban (ARI), and Jena Salvetti (Embry-Riddle Aeronautical A790
University) Sd. TASK NUMBER
202B

Se. WORK UNIT NUMBER,

7, PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
U. S. Army Research Institute for the Behavioral and Social Sciences
ATTN: TAPC-ARI-IJ

P.O. Box 52086

Fort Benning, GA 31995-2086

9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM.

U.S. Army Research Institute for the Behavioral and Social Sciences ARI
5001 Eisenhower Avenue
Alexandria, VA 22333-5600 U1. MONITOR REPORT NUMBER,

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words):

This research examined the use of virtual environments as a viable dismounted infantry mission rehearsal tool. Four squads of
soldiers individually conducted two missions that involved clearing a two-story building located at an urban operations training
site. Two squads rehearsed the mission in a virtual representation of the exact building they would clear at the urban training site.
The remaining squads rehearsed in an actual two-story building that was similar to the one they would clear at the urban training
site. Squads executed both missions in each environment. Performance differences between the rehearsal groups across the two
‘real-world’ missions were small to negligible. Group performance differences for fratricides and personnel flagging were
negatively affected by simulator constraints. Effectiveness ratings for the two rehearsal modes were clearly dependent on the
setting where soldiers rehearsed. The research showed that while virtual environments show promise for this type of training, a
number of interface and technology problems must be overcome. Currently, virtual environments do not appear to be as effective
as real-world tactical training for improving skills underlying specific smal] unit tasks or battle drills. However, these
environments may be used effectively at selected stages of training to enhance cognitive skills development.

15. SUBJECT TERMS

Mission rehearsal Virtual environment mission rehearsal Small unit training Individual combatant simulator
Small unit dismounted mission rehearsal in virtual environments Dismounted infantry simulation

20. NUMBER 21. RESPONSIBLE PERSON
ABSTRACT OF PAGES (Name and Telephone Number)
Dr. Robert Pleban

Unlimited (706) 545-3617

16. REPORT
‘Unclassified

17. ABSTRACT
Unclassified

18. THIS PAGE
Unclassified



conducted (4 squads X 2 virtual-training site missions X 2 field-training site missions), the
squads suffered one hundred percent casualties on seven of the missions.

The one Level III task performance measure that showed any significant variability was
fratricides. Fratricides decreased noticeably for the VE rehearsal group as they moved from the
virtual-training site (9) to the field-training site (0). As mentioned earlier, this improvement was
due primarily to SVS system constraints. When the RW rehearsal group moved from the field-
training site to the virtual-training site, fratricides increased noticeably (1 fratricide versus 6
fratricides for field and virtual-training sites, respectively). [See Appendix I, Tables I-1 and I-4).
The rather large point discrepancy (9.5 points) between the two rehearsal groups for mission 2
observed at the field-training site, can be attributed primarily to the one fratricide committed by
the real-world rehearsal group.

For Level II tasks, analysis of the total number of incidents at both the field and virtual-
training sites revealed that over twice as many incidents were recorded for the VE rehearsal
group (35) than for the RW rehearsal group (16). Tasks 4, 5, 7 and 8 accounted for most of the
incidents that were observed. Interestingly, Figures 1 and 2 show that Level II performance
improved for both groups at the field-training site, but the incident rates did not support this
pattern for the VE rehearsal group. In this instance, incident rates increased over missions for
the VE rehearsal group but decreased for the RW rehearsal group (see Figure 4). This is most
likely a scoring artifact. For 5 of the six Level II tasks, the occurrence of one incident results in
the squad receiving a score of zero. Additional incidents, therefore, will not be reflected in the
weighted score. So, it is possible for a squad to have an increasing incident rate over missions,
yet their weighted scores continue to show improvement. This could happen, for example, if the
incidents are concentrated in a few tasks. This would also explain why large differences in

incident frequencies between groups do not necessarily correspond to large differences in
weighted scores.

Level I tasks clearly had the most recorded incidents. For the six Level I tasks, 236
negative incidents were observed across both environments. Task 10, personnel “flagging”
(body exposed in windows or head exposed around corners) accounted for 75 percent (177
incidents) of the total incidents observed. The majority of flagging incidents occurred at the
virtual-training site (142 or 80 percent).

Soldier Assessment of Rehearsal Training

After the rehearsal training, soldiers completed either the Real World or Virtual
Environment Post Training Questionnaire. Open-ended questions were content analyzed and
summarized in Tables 4-6. The most common responses are listed first.

What soldiers liked most about rehearsing in a real-world or virtual-environment setting.
Table 4 lists the most positive aspects noted for rehearsing in each setting. The majority of
soldiers in the RW rehearsal condition felt that these rehearsals were comparable to performing a
similar mission in the real world, and provided them with a better understanding of the essential
tasks and skills that were required to complete the mission.

27



Appendix I

Evaluation Checklist Incident Rates and Weighted Scores by Task

Table I-1
Performance Results at the Field-Training Site: Level III Tasks and Behaviors

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2

Condition

# BLUFOR Casualties (1) BLUFOR Casualty Points
RW Mean 5.0 6.0 0.0 0.5
VE Mean 6.0 4.5 0.5 1.0
# OPFOR Casualties (2) OPFOR Casualty Points
RW Mean 2.0 2.0 2.5 2.5
VE Mean 2.5 2.0 25 2.5
# Fratricides (3) Fratricide Points
RW Mean 0.0 0.5 5.0 2.5
‘VE Mean 0.0 0.0 5.0 5.0
[RW Summed Points : Le 75 5.5

RW Total Wt. Score S : 22.5 16.5
VE Summed Points 8.0 8.5
VE Total Wt. Score 2 24.0 25.5
Note. Level III Total Wt. Scores were obtained by multiplying the Summed Points by 3. Task
number/behavior in parenthesis.

I-1



Instruction for the two groups was provided at separate locations on the McKenna field-
training site. Once the instruction was completed the squad members received their operation
order. OPFOR members were briefed on their mission objectives in a separate location. Each
group was briefed by one of the O/Cs. Each O/C was assigned primary responsibility to either
the squad or the OPFOR.

Real-world rehearsal at field-training site. Both the squad and OPFOR were given
approximately 30 minutes to plan for the upcoming mission. Once the planning phase was
completed, the squad accompanied the O/C to a building site that was similar to the building they
would clear during the actual missions. The O/C and the squad then proceeded to “talk and
walk” the mission. The O/C accompanied them to the entrance of the building and discussed
how they would enter and position themselves as they entered, and the procedures they would
use to systematically clear designated rooms or areas in the building. The entire squad
accompanied by the O/C then entered the building. The O/C walked through each of the rooms,
hallways, and the stairwell with the soldiers and proceeded to point out danger areas and
discussed the appropriate tactical response for each of these situations.

The “talk/walk” phase constituted the first rehearsal session and lasted approximately 30
minutes. This was followed by a 30-minute break. The second rehearsal session was the “run”
phase. Here, the squad executed the mission at close to full speed, but without an OPFOR. This
was followed by on-the-spot corrective feedback from the O/C. The time allotted for this phase
was 30 minutes. At the same time the OPFOR was preparing for the mission with the second
ore.

Performance assessment at field-training site. Following the “run” rehearsal, the squad
then executed the mission against the OPFOR. The first O/C stayed with the squad at the
building site and monitored the mission as it unfolded in the building. The second O/C, who was
assigned to the OPFOR, observed the mission from the control center building and served as the
platoon leader. After the mission (RW missions ranged from approximately 8 to 20 minutes in
duration), both squad and OPFOR members participated in an AAR at the control center
building. Prior to the start of the AAR, the O/Cs both completed the Evaluation Checklist. This
checklist was used as a general guide for identifying areas of discussion during the AAR.
During the AAR, the first O/C reviewed the mission, highlighted key segments of the mission
and identified appropriate and inappropriate tactical actions. In addition, the O/C encouraged
squad and OPFOR members to elaborate on key events and discuss specific lessons learned.

After the AAR, the squad received their next operation order. The OPFOR, as before,
was briefed on the mission at a separate location. The mission required the squad to clear a
second section in the same large building complex used for the first mission. The major
difference was that the interior floor plan of the second section differed from the first floor plan
and presented a slightly different set of challenges to the squad.

No further rehearsals were allowed. Squad and OPFOR elements were allowed 30
minutes for planning which was followed by the actual execution of the mission. After the
mission, the same procedure was followed as described earlier. After the AAR, all soldiers
completed the Post Training Questionnaire.

15



U.S. Army Research Institute
for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

ZITA M. SIMUTIS
Director

Technical review by

Bruce W. Knerr, U.S. Army Research Institute
Jean L. Dyer, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.
Please address correspondence conceming distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001
Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral
and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.


insufficient numbers of body sensors on the soldier allows him to pass through a wall, frequently
exposing his arms or legs on the other side of the wall. This makes him visible to the enemy.

The lower incident rate for the VE rehearsal group during the real-world missions could
be due to their prior experience at the virtual-training site. However, at Mission 2, any residual
experience effects were quite weak, since group performance differences were negligible.

Two Level I tasks accounted for the bulk of the remaining negative incidents that were
observed, weapon flagging and succession of command not established. The only clear trend
that emerged was for succession of command. Soldiers in the RW rehearsal group were more
likely to not establish succession of command during the real-world missions as the VE rehearsal
group. Group differences could be the result of an “adrenaline” factor of the impending real-
world mission, where the squad/team leaders were so focused that they forgot to tell others of the
succession of command. Apparently, for the VE rehearsal group, soldiers grasped the
importance of this task after their first virtual-training mission. After that, errors dropped to zero
with the next mission and stayed low during the field-training missions.

Soldier Comments on Using Virtual Environments for Mission Rehearsals

Soldiers who rehearsed in a real-world mission setting clearly liked the “hands-on” aspect
of this training strategy. The real-world rehearsal allowed soldiers the chance to climb stairs,
lean against walls, throw grenades, use hand signals, peek around comers, and move obstacles.
The execution of these physical tasks, they felt, was critical for effective urban operation
training. Soldiers also liked the added realism created by the simunition rounds. Getting shot
with simunition rounds provided immediate feedback that the soldier had made a mistake and
created a real incentive for him to take cover and move more tactically. In general, it enhanced
the realistic feel of the field-training missions.

Although the physical execution of these tasks is an important part of the training/
rehearsal process, the data nevertheless show that some learning can take place in a virtual
environment under far less realistic conditions. As noted by the soldiers, these environments can
serve as a valuable training option for when the unit does not have access to real-world training
sites. These environments provide a potentially effective means to conduct the “walk” phase of
training. The real-world training sites can then be more efficiently used for the “run” phase of
training where the unit gets the chance to physically execute the missions with a solid “blue
print” for action.

While virtual environments were never meant to replace real-world training sites, efforts
are needed to enhance the realism of at least the core aspects of these environments. The biggest
complaint from soldiers concerned maneuverability. The inability to perform fine, precise
movements near or within buildings in close proximity with others was a major source of
frustration. While movement (in the SVS) will never duplicate real-world movement patterns,
virtual movements can be made smoother and more precise. The source of much of this problem
in movement has to do with the design of the thumb switch (Pleban et al., 2001; Pleban et al.,
2000). As discussed in earlier work, the current thumb switch used in the SVS system is stiff
and consequently it is very difficult to synchronize the amount and direction of pressure that is

39



Virtual-Training Site Performance
VE Rehearsal Squads

100
80 |
Mean Total 60 |ElLevel Il Tasks
Weighted ELevel Il Tasks
Score 40 (Level | Tasks
20
24.5
0)
1 2

Mission Number

Figure 6. Mean performance scores by task level for VE rehearsal
squads at the virtual-training site.

Virtual-Training Site Performance
RW Rehearsal Squads

100
80
Mean Total 60 EiLevel ll! Tasks |
Weighted EdLevel ll! Tasks
Score 40 Level | Tasks
20
0
1 2
Mission Number

Figure 7. Mean performance scores by task level for RW rehearsal
squads at the virtual-training site.

To determine which tasks were most impacted by the rehearsal manipulation, individual
tasks within each task level were analyzed. The results are discussed in the following sections.

Group performance on individual level IIT tasks. Individual task analysis once again
focused on the raw data frequency counts. Figure 8 shows the mean number of friendly squad
casualties suffered by each group by mission. Virtual-training site casualty rates remained high
(50 percent or higher) across missions for both groups [see Figure 8]. Overall, the casualty rates
differed slightly as a function of rehearsal condition (RW rehearsal group = 5.0 and VE rehearsal
group = 5.7)

23



Soldiers were asked to specifically compare the effectiveness of virtual environments
versus the real world for conducting mission rehearsals. Table 8 shows the percentage
breakdown by rehearsal condition. As shown, 50 % of the soldiers in the VE rehearsal condition
felt that virtual environments were as effective as the real world for conducting mission
rehearsals. Forty-four percent said that virtual environments were less effective than the real
world. In contrast, all soldiers in the RW rehearsal condition felt that the virtual environment
was less effective than rehearsing in a real-world setting.

Table 8
Percent Response to Question “Compare the Effectiveness of Virtual
Environments Versus the Real World for Conducting Mission Rehearsals”

VE Mission RW Mission
Rehearsal Condition | Rehearsal Condition
VE less effective 44 100

than RW

VE just as effective 50 0
as RW

VE more effective 6 0

than RW

Note. n= 16 for each condition.

Soldiers who felt that virtual environments were less effective than real-world settings for
conducting mission rehearsals emphasized the realistic “hands-on” nature of real-world training
Movement in the virtual environment via the thumb switch, getting stuck in walls, and menu
layout at various times frustrated soldiers and negatively affected the realism and overall training
value of the virtual environment rehearsal process. Some soldiers felt that virtual environment
rehearsal could be as effective as a real-world rehearsal, but not with the current level of
technology.

Soldiers who felt virtual environments were as effective as real-world rehearsals
mentioned the safety aspects of this type of training, the optimal training conditions (climate, not.
physically strenuous), and cost effectiveness (can quickly and cheaply construct mock-ups of
different terrain and buildings, and the ability to conduct more repetitions per unit of time).

Advantages of rehearsing in a virtual environment. Soldiers were asked to list the major
advantages to rehearsing in a virtual environment. Their comments are summarized in Table 9.
Soldiers felt that virtual environment systems could provide an effective method for familiarizing
units (e.g., squads) with the mission environment and for teaching (basic) urban operation
principles and skills. These systems could also be used for refining tactics, mission planning,
and building unit cohesion and teamwork. Virtual rehearsal training was viewed as more time
and cost effective for a number of different reasons detailed in Table 9.

32



Results

Since the focus was on squad performance, sample size was reduced to 4 (two squads per
rehearsal condition). As a result, only descriptive statistics (j.e., means, standard deviations,
frequencies) are presented since standard inferential statistics would not be appropriate with such
asmall sample. For clarity purposes, the performance data is presented by setting (field-training
site versus virtual-training site).

Field-Training Site Performance
Figures 1 and 2 depict the transformed field-training site performance scores for each

rehearsal group. Figure 1 shows that overall performance improved from Mission 1 to Mission 2
for the VE rehearsal group, increasing from 84 to 92.

Field-Training Site Performance
VE Rehearsal Squads

100

80
Mean Total go GiLevel Ill Tasks
Weighted Bevel Il Tasks
Score 404 Devel | Tasks

20 25 26.5
1)
1 2
Mission Number

Figure 1. Mean performance scores by task level for VE rehearsal
squads at the field-training site. .

In contrast, the RW rehearsal group’s performance showed little change over missions, 90.5 to
91 for Missions 1 and 2 respectively (see Figure 2).

18



One of the best performing of the currently existing individual combatant simulation
systems is the Soldier Visualization Station - SVS (see Salter, Eakin, & Knerr, 1999) developed
by Advanced Interactive Systems. This system represents the currently most viable overall
technical approach for enabling soldiers to shoot, move, and communicate in virtual
environments. In this system, the soldier stands in front of a large screen holding a rifle. The
images depicted on the screen, including buildings, vehicles, and people, are reasonably life like
in size and actions. The combination of images and action creates a very immersive (virtual)
environment for the soldier (see Figure 1).

The SVS is a personal computer-based system with an inertial/acoustic tracker for
simulated body position and weapon pointing. It includes a helmet-mounted display - HMD
(helmet mounted monocular eyepiece linked to a camera on the rifle) that can be used to assist in
aiming and looking around corners of buildings. The SVS has one flat screen on which images
are presented by a rear projection device. Movement is accomplished by applying pressure to a
weapon-mounted thumb switch. This allows the individual to move rather effortlessly
throughout the virtual battlefield to include open terrain and urban environments. The thumb
switch also controls access to an on-screen menu that allows the individual to select among
various weapons, e.g., M-4 rifle, squad automatic weapon, shotgun, pistol, grenades, smoke, and
includes a program for weapon zeroing.

Figure 1. Soldier in an SVS system.



Appendix E
Mission Rehearsal Comparison Questionnaire

Name. Date Condition

1. Mission role:
BLUFOR OPFOR

2. Based on your experiences this week, could virtual environments be used to conduct effective
mission rehearsals? Yes No

3. How would you compare the effectiveness of virtual environments versus the real world
for conducting mission rehearsals?

Less effective than rehearsing in the real world.
Just as effective as rehearsing in the real world.
More effective than rehearsing in the real world.

Briefly explain your rating.

4. What do you think are the major advantages to conducting mission rehearsals in a virtual
environment?

5. What do you think are the major disadvantages to conducting mission rehearsals in a virtual
environment?

6. How would you incorporate virtual environments in your unit’s training?

7. Other comments concerning virtual and real world mission rehearsals.



Table I-2
Performance Results at the Field-Training Site: Level II Tasks and Behaviors

Rehearsal Mission 1 Mission 2 Mission 1 Mission 2
Condition
# Rooms not Properly Cleared (4) Room Points
RW Mean 3.0 0.0 0.0 5.0
‘VE Mean 1.5 2.0 2.5 0.0
# Incidents Soldiers did not Position Points
Maintain Position (5)
RW Mean 0 0 5.0 5.0
VE Mean 0.5 0 2.5 5.0
# Incidents Soldiers did not Dispersion Points
Maintain Dispersion (6)
RW Mean 1.0 0.5 2.5 2.5
VE Mean 0.5 0.0 2.5 5.0
# Incidents Soldiers did not Supporting Fire Points
Provide Supporting Fires (7)
RW Mean 0.0 0.0 5.0 5.0
VE Mean 0.5 1.5 2.5 2.5
# Incidents Soldiers did not OPFOR Location Points
Accurately Report OPFOR
Location (8)
RW Mean 0.0 1.0 5.0 2.5
VE Mean 0.5 1.5 2.5 2.5
# Incidents Tm. Member did not Sector of Fire Points
Understand his Sector of Fire (9)
RW Mean 0.0 0.0 5.0 5.0
VE Mean 0.0 0.0 5.0 5.0
| ; ; ices 22.5 25.0
45.0 50.0
17.5 20.0
35.0 40.0

Note. Level I Total Wt. Scores were obtained by multiplying the Summed Points by 2. Task
number/behavior in parenthesis.


applied to guide movement. The lack of play in the switch makes it very difficult to control
precise movement patterns.

The “sticking” problem described earlier became a major issue in this experiment since
the scenarios required so much movement on the soldiers’ part. Sticking is primarily due to
current system constraints that limit the number of body sensors that can be used on the soldier.
Like the thumb switch problem, this is also correctable but will require additional computing
power.

The building clearing scenarios required the use of grenades. However the use of
grenades resorted in numerous fratricides during the execution of the virtual environment
missions. The problems associated with “throwing” grenades in the SVS system have already
been discussed. Ideally, a more realistic way of throwing grenades is needed. If this cannot be
accomplished, then the procedures should be simplified, so the task can be mastered with
minimal training.

Soldiers also reported problems using the on-screen menu. Up until this experiment,
soldiers did not view menu design and function as a problem. Apparently, the design of the
menu resulted in some soldiers missing the unstick option. This increased their exposure time to
the enemy as they cycled through the menu a second time to locate the correct option. The
scenarios may have accentuated this problem since they required extensive movement within
buildings. This, in turn, increased the number of “sticking” incidents and highlighted possible
menu design issues.

The limited FOV provided by the SVS system appeared to be more problematic in this
experiment. Again, the scenarios may have exacerbated this shortcoming. Clearing rooms
requires that the soldier’s head be able to swivel, look up, down, side-to-side, etc. The limited
peripheral vision and the manner required for soldiers to look to their side (using the thumb
switch) negatively impacted their performance in these scenarios (from the soldiers’ standpoint).
For rapid, realistic changes in views, soldiers preferred a system that was based on head
movements rather than the thumb switch.

It should be noted that the scenarios developed for this experiment required that squad
members execute basic tactical behaviors such as moving inside buildings and throwing hand
grenades. The problems observed were not so much a function of the scenarios as they were
weaknesses in this simulation (SVS) system.

Both groups felt that training should be expanded but for different reasons. For the RW
rehearsal group, soldiers felt that the time scheduled for rehearsals was too short. For the VE
rehearsal group, rehearsal time appeared adequate, but the time allotted for familiarization of the
SVS system was viewed as insufficient. The vast majority of soldiers in this experiment had
little if any training in urban operations, so it is not surprising that they felt the rehearsal time
was too limited. This is particularly the case for the RW rehearsal group which had only two
rehearsal trials.

40


scenarios coupled with an effective instructional strategy can so thoroughly immerse the soldier
in the mission that many realism issues are overlooked, and significant learning can still be
achieved. Lintern et al. (1989) further showed that physical fidelity is not a requirement for
positive transfer. They argue that the more important challenge is enhancing the psychological
fidelity or functional relevance of the simulation training environment.

There is a broad range of tactical skills that could be trained in the virtual environment.
At one end of the continuum are the types of cognitive skills (decision-making) trained in the
Pleban et al. (2001) experiment. This type of training does not require high fidelity, fast, or
precise interface with the virtual world. Success in this instance is more likely to depend on the
factors described above. At the other end of the continuum are the specific squad drills and
tasks, like building clearing, which involve less decision-making, a lot of communication and
coordination, but above all require rapid and precise positioning, movement and use of weapons
(Knerr et al., 2003). This experiment addressed one of the most difficult virtual environment
training problems. As Knerr et al. (2003) point out, virtual environment training of tasks such as
“Clear a building” and “Clear a room” have shown the least improvement in terms of enhanced
skill proficiency. While virtual environments show promise for this type of training, there are a
number of interface and technology problems to overcome. Currently, virtual environments do
not appear to be as effective as real-world tactical training for improving skills underlying
specific small unit tasks or battle drills.

Conclusions

The results from this research, while preliminary, indicate that certain small unit (squad)
dismounted infantry tasks can be rehearsed with some degree of success in a virtual
environment. Field-training site performance of the virtual environment and the real-world
rehearsal groups were comparable. For fratricides and personnel flagging, a disproportionate
number of incidents were a direct function of the training site, i.e., simulator system constraints.
The research presented showed that while virtual environment systems may be lacking realistic
qualities in certain areas, they could be used effectively for certain types of training. Virtual
environments could be used during the “walk” phase of training, for improving decision-making,
situation awareness, communication and coordination skills. These environments, if used
appropriately in conjunction with realistic field- exercises, could play a major role in enhancing
the training of soldiers and small unit leaders.

42



Practice. The rehearsal of specific behavioral or cognitive responses over time provides
individuals with an opportunity to hone the skills and behaviors needed in the operational or
criterion environment. In general, the longer the duration of practice, the better the performance.
This finding has been documented over such varied tasks as visuo-motor behavior, word
retention, theater, and chess (see Nullmeyer & Spiker, 2000). Driskell, Willis, and Copper
(1992) found that overlearning is an effective means of enhancing subsequent performance, but
only if the task that is overlearned is the task called for in the criterion performance setting.
Overlearning also appears to be an effective training strategy for both physical and cognitive
tasks. However, its impact is somewhat stronger for cognitive tasks.

While overlearning is generally associated with enhanced performance, there is the
possibility that extended practice can lead to a loss of focus or concentration (Driskell et al.,
1994) and stereotypic, rigid responding (see Nullmeyer & Spiker, 2000). This can be
particularly problematical when conditions in the mission environment are fluid and uncertain.
This suggests that trainers should have individuals practice the same mission but under varying
scenarios that require them to discern key differences in the situation(s) and respond accordingly.
In these instances, extended practice will help refine core, mission critical skills and also help
develop mental schemas (or blueprints for how to respond) that will allow them to rapidly
interpret the situation and respond in an appropriate manner (see Klein, 1997).

The type of practice can also have a major impact on performance, particularly in a
typical mission rehearsal context where the unit(s) must master multiple tasks. In this context,
distributed practice tends to be superior to massed practice (Schmidt & Bjork, 1992) and self-
paced practice is more effective than fixed-paced (Swezey & Llanerus, 1997).

Feedback. Feedback can either be external or internal. External knowledge of results has
been shown to improve performance in virtually every domain studied (Salvendy, 1997).
Knowledge of results is more effective when it provides some form of directional information
(fast/slow, left/right) than purely evaluative (good/bad) [Annett, 1989]. For individuals, internal
(ie., kinesthetic) feedback can significantly enhance the acquisition of motor skills beyond that
offered by external knowledge of results (Swezey & Llaneras, 1997).

In general, for feedback to be most effective, it must be accurate, timely, credible, and
constructive (Campbell, 1988). Feedback that is inconsistent, erroneous, and delayed can
significantly decrease response accuracy and increase response time (Swezey & Llaneras, 1997).

As noted by Nullmeyer and Spiker (2000), mission rehearsal in military applications is an
observable, group event in which information sharing and joint decision-making are vital. Some
research has shown that external, collective feedback is optimal in such situations (Pearson,
1991). A major advantage of this feedback approach is that it can facilitate the development of
shared mental models that allows all individuals to share a common internal knowledge base,
increasing the likelihood that team members can anticipate problems and respond ina
coordinated manner (Orasanu & Salas, 1993).

Direct focus of mission rehearsal sessions. The mere presence of an expensive, high
fidelity simulator, does not mean that these systems will be used effectively. Mission rehearsals