Technical Report 1184 

Instructional Features for Training 

in Virtual Environments 

Michael J. Singer 

U. S. Army Research Institute 

Jason P. Kring 

University of Central Florida 

Consortium Research Fellows Program 

Roger M. Hamilton 

University of Central Florida 

Consortium Research Fellows Program 

July 2006 

 United States Army Research Institute 

 for the Behavioral and Social Sciences 

Approved for public release; distribution is unlimited. 

 
U.S. Army Research Institute 

for the Behavioral And Social Sciences 

A Directorate for the Department of the Army 

Deputy Chief of Staff, G1 

Authorized and approved for distribution: 

 

 STEPHEN GOLDBERG MICHELLE SAMS 

 Acting Technical Director Acting Director 

Technical Review by 

Dr. Scott Beal, Infantry Forces Research Unit, ARI 

Dr. Bruce Knerr, Simulation Systems Research Unit, ARI 

NOTICES 

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, ATTN: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, VA 22202-3926. 

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Technical Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents.

 
REPORT DOCUMENTATION PAGE 

1. REPORT DATE (dd-mm-yy) 

July 2006 

2. REPORT TYPE 

Final 

3. DATES COVERED (from. . . to) 

July 2002 - December 2005 

4. TITLE AND SUBTITLE 

Instructional Features for Training in Virtual Environments 

5a. CONTRACT OR GRANT NUMBER 

5b. PROGRAM ELEMENT NUMBER 

622785 

6. AUTHOR(S) 

Michael J. Singer (U.S. Army Research Institute for the 

Behavioral and Social Sciences); Jason P. Kring and 

5c. PROJECT NUMBER 

A790 

Roger M. Hamilton (University of Central Florida) 5d. TASK NUMBER 

233 

 5e. WORK UNIT NUMBER 

H01 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social 

Sciences

ATTN: DAPE-ARI-IF 

12350 Research Parkway 

Orlando, FL 32826-3276

8. PERFORMING ORGANIZATION REPORT NUMBER 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social 

Sciences

10. MONITOR ACRONYM 

ARI 

2511 Jefferson Davis Highway, Arlington, VA 22202-3926 11. MONITOR REPORT NUMBER 

Technical Report 1184 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES 

Subject Matter POC: Michael J. Singer 

14. ABSTRACT (Maximum 200 words): 

The U.S. Army is committed to using interactive simulations to provide training for Soldiers. Virtual Environment 

(VE) and gaming technology may allow the U.S. Army to cost-effectively conduct planning, training, and rehearsal 

activities for both individual and collective dismounted Soldier tasks. The simulation technology also supports or 

provides stimuli that could enhance learning through instructional strategies, tactics, and instructional features. 

Research on effects of specific VE system characteristics and instructional applications must be performed to 

establish the benefits, problems, and guidelines for training and rehearsing complex activities and tasks using VE 

technology. This experiment investigated the training effect of instructional interventions in VE for training 

representative Soldier tasks. The research addressed Interrogative Coaching and an Attention-Direction 

Instructional Feature on initial skill acquisition of dismounted Soldier tasks incorporating basic recognition and 

decision skills. The results indicate that there is no initial learning advantage to the type of Attention Direction 

feature that was used, and that the Interrogative Coaching did seem to aid the skill acquisition of the more complex 

Bounding Overwatch task. The results are also being used to shape continued investigations into the use of 

instructional strategies, tactics, and features in VE simulations for dismounted Soldier tasks. 

15. SUBJECT TERMS 

Virtual Environments, Simulation, Instructional Features, Instruction, Training

 SECURITY CLASSIFICATION OF 19. LIMITATION OF 

ABSTRACT 

20. NUMBER 

OF PAGES 

21. RESPONSIBLE PERSON 

16. REPORT 

Unclassified

17. ABSTRACT 

Unclassified

18. THIS PAGE

Unclassified

 

Unlimited 

 Ellen Kinzer

Technical Publication Specialist 

703-602-8047

i 
 

ii 
Technical Report 1184 

Instructional Features for Training 

in Virtual Environments 

Michael J. Singer 

U. S. Army Research Institute 

Jason P. Kring 

University of Central Florida 

Consortium Research Fellows Program 

Roger M. Hamilton 

University of Central Florida 

Consortium Research Fellows Program 

Simulator Systems Research Unit, Orlando FL 

Stephen L. Goldberg, Chief 

U.S. Army Research Institute for the Behavioral & Social Sciences 

2511 Jefferson Davis Highway, Arlington, VA 22202-3926 

 

July 2006 

622785A790 Personnel Performance and 

 Training Technology

Approved for public release; distribution is unlimited. 

iii 
iv 
FOREWORD

The U.S. Army has made a substantial commitment to the use of simulations for training 

and readiness. Many current simulators are networked and designed to provide realistic training 

and rehearsal for large combined arms groups of vehicles and major weapon systems. These 

simulators represent dismounted Soldier activities, but are not intended to directly train or 

rehearse individual dismounted Soldiers. Virtual Environment (VE) technology, which typically 

includes head-mounted visual displays with tracking devices for limbs and individual weapons, 

provides increasing capabilities that enable a more immersive, person-centered simulation and 

training capability for dismounted Soldiers. These systems are being investigated in order to 

include individual dismounted Soldiers in the larger networked simulation systems, and to 

support training and rehearsal for dismounted Soldiers. One research challenge arising from

these efforts is identifying and quantifying the effects of VE system characteristics and use on 

learning, retention, and transfer of skills required for Army tasks. 

This report describes one experiment in an ongoing program of research conducted by the 

U.S. Army Research Institute for the Behavioral and Social Sciences (ARI), Simulation Systems 

Research Unit (SSRU) that addresses the use of VE technology for training dismounted Soldiers 

in simulations. This experiment investigated the effects of different supplemental stimuli for 

directing attention and providing task guidance during repeated performance in virtual 

simulation exercises. The findings from this research will be used to recommend VE 

characteristics and instructional methods for incorporation in distributed VE simulation systems 

for training. 

SSRU conducts research with the goal of providing information that will improve the 

effectiveness of training simulators and simulations. The work described here is a part of ARI 

Research Task 233, VICTOR - Virtual Individual and Collective Training for Objective Force 

Warrior. The initial results from this work were presented at the Interservice/Industry Training, 

Simulation, and Education Conference 2004. 

 STEPHEN L. GOLDBERG

 Acting Technical Director

v 
ACKNOWLEDGMENT

This research could not have been conducted without the excellent programming and technical 

support provided by the Institute for Simulation and Training (IST) at the University of Central 

Florida. We especially thank Glenn Martin and Jason Daly of IST for their creative efforts in 

producing the virtual environment models and data collection software used in the experiment. 

vi 
INSTRUCTIONAL FEATURES FOR TRAINING IN VIRTUAL ENVIRONMENTS 

EXECUTIVE SUMMARY 

Research Requirement: 

The U.S. Army is committed to using interactive simulations for mission planning, 

rehearsal, training, concept development, and test and evaluation. Many current simulators are 

designed to provide training for Soldiers fighting from vehicles, but are not designed to provide 

realistic training on multiple tasks for dismounted Infantry. Virtual Environment (VE) and 

gaming technology is providing new capabilities for simulating real world activities for 

individual dismounted Soldiers. This technology may allow the U.S. Army to cost-effectively 

conduct planning, training, and rehearsal activities for both individual and collective dismounted 

Soldier tasks. Basic to these simulations is the common context for individual combatants who 

need to move, observe, shoot, and communicate. Accepting as a given the baseline of adequate 

functional and physical fidelity of equipment and environment for the performance of Soldier 

tasks, the capabilities of VE technology can go much further. The technology can support or 

provide stimuli to the learner that could enhance learning through the use of training or 

instructional strategies, tactics, and the addition of instructional features. Research on the effects

of specific VE system characteristics and instructional applications must be performed in order 

to establish the benefits, problems, and guidelines for training and rehearsing complex activities 

and tasks using VE technology. 

Procedure: 

This experiment was developed to investigate the training effect of instructional 

interventions in the use of VE technology during training of representative Soldier tasks. We

used previous analyses to select the representative tasks, and a defining structure is suggested for 

a hierarchy of instructional approaches, leading to instructional interventions that should apply to 

the representative tasks. The experimental training protocol is designed to examine the effects of 

the presence or absence of two factors: Interrogative Coaching and an Attention-Direction 

Instructional Feature, on the initial skill acquisition performance of dismounted Soldier tasks that

incorporate basic recognition and decision skills. The tasks addressed were Bounding 

Overwatch movement, Helicopter Landing Zone identification and marking, and 

Observation/Fire Post positioning (around the landing zone). The instructional factors were 

investigated using a relatively simple 2x2 experimental design, allowing repeated exercises to be 

used for monitoring initial skill acquisition. Participants were recruited from a local university, 

and compensated for their time in the experiment. Participants trained on the tasks using an 

instruction booklet, and conducted an exercise incorporating basic instruction and practice on 

operations in the VE before beginning the experiment. During the experiment participants 

worked through three training exercises with appropriate experimental instructional 

interventions, and participated in After Action Reviews (AARs) following each exercise. A 

fourth “test” mission without intervention or AAR was then conducted.

vii 
Findings: 

Data were collected on the rated performance of the three tasks, the time to perform the 

tasks, the number of buildings inspected during the tasks, and the amount of time taken to inspect 

buildings during the tasks. The performance data was sufficiently non-normal that the ratings 

were transformed before analysis. Analyses revealed significant improvement over the repeated 

exercises in both time and performance on the Bounding Overwatch and Landing Zone tasks, 

indicating that participants learned (improved performance in less time) during the training 

exercises. Both the Interrogative Coaching and Attention-Direction Arrow Instructional Features 

significantly interacted with the repeated missions in the Bounding Overwatch task. The data 

show that the Coaching intervention improved performance relative to the non-coached 

condition, but the Attention-Direction aided group performed worse over trials than those not 

receiving the intervention. Analysis of the Landing Zone performance times found the Coaching 

intervention significantly interacting over missions, with the intervention leading to slower 

performance times. Analysis of the fourth mission only found significant improvements in 

performance on the Bounding Overwatch task from the Coaching intervention. 

Utilization and Dissemination of Findings: 

The results of this initial investigation indicate that there is no initial learning advantage to 

the type of Attention Direction feature that was used, and that the application may have 

interfered with initial skill acquisition. The Interrogative Coaching did seem to aid the skill 

acquisition of the more complex Bounding Overwatch task, although it also extended the time to 

perform the Landing Zone task. These results indicate that instructional interventions are 

difficult to successfully implement in an experimental setting. The results are also being used to 

shape continued investigations into the use of instructional strategies, tactics, and features in VE 

simulations for dismounted Soldier tasks. The next experiment will address different 

characteristics in both the delivery and format of coaching during practice performances, and 

alterations in visual stimuli for Attention-Direction information presentation (non-normal, taskspecific stimuli that enhance or call attention to normal environmental task cues). The results of

this and succeeding experiments can and should inform the design and development of 

simulation environments and training/rehearsal applications for U. S. Army Soldiers. 

viii 
INSTRUCTIONAL FEATURES FOR TRAINING IN VIRTUAL ENVIRONMENTS 

CONTENTS

Page 

Introduction .............................................................................................. 1

Simulation Research and Application .......................................................... 1

Increased Requirements for Improved Training .............................................. 2

Common Soldier Tasks ................................................................................ 4

Task Selection ....................................................................................... 6

Instructional Framework .......................................................................... 7

Conceptual Integration: Instructional Goals, Strategies, Tactics, and Features ................. 8

Definitions .......................................................................................... 9

Instructional Feature Categories ................................................................. 10

Instructional Feature Selection .................................................................. 16

Instructional Feature Experiment .............................................................................. 18

Method ................................................................................................. 19

Results ................................................................................................ 23

Discussion ................................................................................................. 29

Further Research ..................................................................................... 32

References ............................................................................................... 35

Appendix A ............................................................................................ A-1

Appendix B ............................................................................................ B-1

Appendix C ............................................................................................ C-1

Appendix D ............................................................................................ D-1

LIST OF TABLES 

Table 1: Top Activities for Soldier Tasks and Supportability by VE .......................... 5

Table 2. Categorization of a Directional Attention Arrow Instructional Feature ............. 17

Table 3. Actual and Transformed Mean Bounding Overwatch Performance Ratings & 

Times over Missions & Conditions ............................................................. 28

ix 
CONTENTS (continued)

Page 

Table 4. Actual and Transformed Mean Landing Zone Performance Ratings & Times over 

Missions & Conditions........................................................................ 29

LIST OF FIGURES

Figure 1. Relationship among instructional goal, strategies, tactics, and features ............ 9

Figure 2. Example dynamic diagram ............................................................... 12

Figure 3. Example text-based message ............................................................. 14

Figure 4. Example highlighting of door sequence ................................................ 15

Figure 5. Position-linked attention-direction arrow instructional feature ..................... 16

Figure 6. Experimental design ....................................................................... 19

x 
INSTRUCTIONAL FEATURES FOR TRAINING IN VIRTUAL ENVIRONMENTS 

Introduction 

U.S. Army Soldiers are tasked with a growing array of complex and challenging tasks and 

missions. Dismounted Infantry Soldiers, in particular, must possess knowledge and skills that 

are instantly accessible to survive and excel in warfare operations today. One aspect of the U.S. 

Army Research Institute for the Behavioral and Social Sciences (ARI) mission is to enhance 

individual Soldier and group performance along with group and individual decision making 

(ARI, 2004, 2005). Information supporting improvements in training for Soldiers through the 

use of simulation technology is a main thrust of the Simulation System Research Unit (SSRU), in 

supporting the ARI mission. 

Simulation Research and Application 

Simulation has long been proven effective for training new skills and allowing Soldiers to 

practice these skills in an interactive, dynamic fashion. Older examples include flight simulators 

(Williges, Roscoe, & Williges, 1973) and tank simulators (Alluisi, 1991). Newer examples 

include the Virtual Sand Table - a computer-generated version of the traditional sand table 

exercise which gives personnel the opportunity to practice military doctrine in a hands-on 

manner (Wisher, Macpherson, Abramson, Thornton, & Dees, 2001), and more immersive 

systems that allow Soldiers to navigate through virtual settings with helmet mounted displays 

(HMDs) and body-movement tracking (Knerr, et al., 2003). 

The continual increase in use of Virtual Environment (VE) simulation can be attributed to 

the numerous benefits that simulation provides in comparison to traditional, real world 

experience training programs (Hays & Singer, 1989; Swezey & Andrews, 2001). First, as with 

the earlier vehicle-based simulation systems, VEs afford greater flexibility to precisely alter 

learning environments and mission conditions. An example of this flexibility is the ability to 

implement instructional strategies and features that are difficult or impossible to implement in 

real world settings. Second, they can provide superior performance assessment capabilities, 

although additional processing may be needed for interpretation of data (Banta, Troillet, 

Hefferman, Plamondon, & Beal, 2004). For example, VE systems are well suited to capture and 

store detailed performance data (e.g. Knerr, et al., 2002). Third, evidence shows that the 

effectiveness of VE training for some tasks is equivalent to, or in some cases better than, real 

world training (Rose et al., 2000; Todorov, Shadmehr, & Bizzi, 1997). Finally, it is obvious that 

simulation-based training is safer, as emergencies can be presented and emergency procedures 

practiced in a simulator that would never be allowed during the operation of real equipment 

(Williges, Roscoe, & Williges, 1973). In summary, VE-based simulation will continue to 

increase in use and usefulness for training many tasks and skills. This has already been 

demonstrated in training a variety of tasks or jobs, for example with pilots (e.g. Bell & Waag, 

1998; Lintern, Roscoe, Koonce, & Segal, 1990); ship navigators (Hays & Vincenzi, 2000; 

Magee, 1997); emergency personnel (Bliss, Tidwell, & Guest, 1997); first responders to bioterrorism (Stansfield, Shawver, Sobel, Prasad, & Tapia, 2000); and space mission ground control 

staff (Loftin, Wang, Baffes, & Hua, 1992). 

1 
ARI’s SRU), contractually supported by the University of Central Florida’s Institute for 

Simulation and Training (IST), established a research program in VE technology in order to 

investigate a wide range of potential VE training issues and factors for training individuals and 

teams. One research and development goal is to “develop and evaluate procedures and 

techniques for using embedded training, low-cost simulation, and augmented reality to enhance 

the fighting capabilities of digitally equipped dismounted individual Soldiers and leaders” (pg. 6, 

U.S. Army Research Institute, 2004). Earlier research and analysis focused on the VE

requirements and application guidelines for leader and individual performance in unit tasks 

(Jacobs, et al., 1994), the determination of necessary characteristics for VE-based individual 

combatant training (Knerr, 1998), and the evaluation of potential transfer of training to military 

operations (Knerr, et al., 2003). More recent work evaluated the feasibility of training

individuals in a shared virtual space while geographically distributed (Singer, Grant, 

Commarford, Kring, & Zavod, 2001) and communication modality and After Action Review 

(AAR) performance in a distributed immersive VE (Kring, 2004). The latter work follows a 

more general approach, that of conducting research with non-military (untrained on military 

tasks) participant populations including: research aids, student interns, and university students. 

This approach has the benefit of easing the acquisition of participants for general training issues

research, and the drawback of not using a more representative population sample (e.g. Army

Soldiers). 

The next step in this program of research is to find specific ways to enhance the training 

effectiveness and efficiency of VE systems. Results from that research can then be applied to 

training dismounted Soldiers, small unit teams, and small unit leaders. The results should also 

apply to other, closely related simulation environments such as embedded systems and 

augmented reality systems (discussed below). The research approach must address VE-specific 

capabilities that can enhance learning and skill levels for critical or important dismounted Soldier 

tasks. This line of research has become more important because the Army is increasing the 

emphasis on adaptive and flexible training in response to new equipment (TRADOC Pamphlet 

525-66, 2005) and the current operating environment. 

Increased Requirements for Improved Training 

The changes in equipment and training needs are resulting in increasing cognitive loads 

being imposed on the squad and platoon leaders. Early changes were the result of strategic plans 

developed for Soldiers and Soldiers equipment through the Land Warrior and Future Force 

Warrior programs (National Research Council, 1997, TRADOC Capability Development 

Document for Ground Soldier System, 2004). Other changes in training are being caused by the 

current operating environment from the operations in Afghanistan and Iraq. According to the 

National Research Council (1997), learning to execute common and repetitive skilled activities 

within the three most significant Infantry missions may be necessary for the success of those 

missions. As the information load of small unit leaders and Soldiers increases, rapid acquisition 

of skill in dealing with that information load will be even more important. One way to keep 

technology-based increases in information flow within the cognitive resource bounds of the 

small unit leader may be to reduce the load required for the successful performance of other 

ubiquitous activities through improved training. 

2 
Serendipitously, one way to conduct research on the instructional tactics, techniques, and 

features for improved training within this technological context is through the use of VE 

systems, conceptually referred to as constructive approaches. Fielding fully immersive VEbased training is currently constrained by many technological and cost factors. Fully immersive 

VE-based instruction (e.g. using helmet mounted displays to totally control the visual field) will 

continue to be relatively expensive and difficult to deploy for individual training in the 

immediate future. However many applications can and will use less than fully immersive 

systems (for example game systems, Mayo, Singer, & Kusumoto, 2005). It is also clear that 

there are multiple levels of virtuality within which instructional strategies can be implemented. 

For example, there is a continuum of artificiality that can support task activities (Milgram, et al., 

1994). At one end of this spectrum are VE systems in which everything is artificial, and at the 

other end is everyday, normal reality. However there are displays that can add stimuli to normal 

reality, creating an augmented reality (AR), as well as physically real objects (e.g. weapons or 

tools) that can be added to VEs. This range has been labeled Mixed Reality (Milgram, et al., 

1994) which includes fully immersive VEs and the lower fidelity applications (for example those 

based on gaming technology and presented on desktop screens). 

AR technology adds artificial stimuli (e.g. sounds, visual displays) to the stimuli from the 

real world. As an example, a heads-up display could overlay heat signatures to the real world 

view. VE-based gaming technology can also introduce artificial stimuli to media or graphics 

representing the real world, and use the entire stimulus set for interactive training (McCollum, et 

al., 2004; Singer, Comer, & Gross, 2004). In these lower fidelity, less immersive systems, 

simulated cues can also be added to the natural-looking stimulus set (the artificially presented 

environment). For example, within a graphically presented virtual world, the instructional 

system may insert arrows to point out possible locations for improvised explosive devices, in 

order to train environmental inspection techniques. 

So the argument is that in addition to knowing the real world stimuli (e.g. weapons visual 

representations) and functionality (e.g. sound of firing, effects downrange) that have to be 

represented for training in a simulation, the instructor may wish to insert special cues for 

instructional purposes. Therefore, instructional design research into instructional tactics, 

techniques and features may provide results that can be used in fully immersive systems, ARbased training systems (Kirkley, et al., 2003. 2004), game-based simulation systems (McCollum,

et al., 2004; Singer, Comer, & Gross, 2004), or in any class of mixed reality training system. 

In order to support as wide a range of applications as possible, the focus in this effort is on 

initial skill training. The goal of skill-based training is to efficiently move someone from initial 

declarative knowledge (being able to answer questions about rules or concepts) through slow and 

error-ridden performance to faster, more competent, and less-effortful performance. In 

psychological terms, this means moving the learner from the declarative state knowledge of a 

skill into some level of procedural knowledge, and within the procedural area from more 

effortful and error-ridden executive-controlled processes toward more rapid and correct 

performance with less effort. As discussed below, procedural knowledge is about how to do 

things, and is often associated with performance automaticity. Automaticity refers to processes 

that are easily initiated by selected classes of stimuli, that proceed with little probability of error, 

and require minimal cognitive effort. Initial skill training sets the learner on the path toward 

3 
automaticity, and instructional strategies with appropriately used instructional features should 

make those first steps as efficient and effective as possible. 

The approach used in this effort was to identify an instructional strategy and appropriate 

features used in non-VE domains in academia, industry, and certain military training programs 

(e.g. see Loube, 2001), and then examine their potential for use in dismounted Soldier VE 

training systems. The first step was to identify a group of common small group leader tasks, 

based on an assessment of activities necessary for US Army operations, as candidates for fully 

immersive VE-based training with instructional features. Second, informed by empirical 

findings and theoretical work, we defined and conceptually integrated instructional goals, 

features, strategies, and tactics to provide a reasonable degree of order and clarity to our 

approach. Third, we identified several instructional features within an instructional strategy that

seemed likely to improve the effectiveness and efficiency of VE-based training. Finally, we 

conducted an experiment that tested the effect of two instructional features, within the constraints 

of a commonly used instructional strategy, as applied to small group leader task accomplishment. 

The results of this research are reviewed in order to suggest guidelines for using instructional 

features in future VE and AR training programs. These results and the information from related 

literature are also used to suggest further research on the complicated and interrelated aspects of 

Instructional Strategies, Tactics, and Features. 

Common Soldier Tasks 

Soldiers use a large number of specific types of knowledge, skills, and abilities (KSAs) in 

the course of operations. Jacobs et al. (1994) analyzed tasks and activities of individual 

combatants for ARI in order to recommend behavioral and technological requirements for VE 

training (for examples see Table 1). The analysis revealed 230 unique activities associated with 

67 Soldier tasks, with most tasks supported by multiple activities. For instance, the task of

clearing a building involves giving verbal orders, calling in preplanned fire requests, hearing 

orders, identifying safe and danger areas, perceiving the relative position of other units, and at 

least 34 other unique activities. In addition, Jacobs et al. rated each activity according to its 

primary and secondary sensory modality (auditory, visual, tactile, and force feedback), as well as 

the required effector —the primary method by which the response of the Soldier is monitored 

and injected into the VE simulation (hand, finger, head, body, speech, and instrumented objects). 

Jacobs et al. also rank ordered activities according to the frequency of the activity in various 

tasks, and how well VE technology could support the activity in training systems. Table 1 lists 

the top 15 activities and ratings for sensory modalities and effectors from the Jacobs et al. work. 

The activities identified by Jacobs et al. (1994) are categorized by sensory and 

psychomotor functions, yet many computer-based training scenarios used by the U. S. Army to 

teach command decision making clearly emphasize higher-level cognitive functions. For 

instance, a scenario for training platoon leaders on how to handle Operations Other than War 

(OOTW) events requires trainees to assess a situation, select a course of action, and properly 

implement the action. Training methods are used to provide information and point out stimuli, 

and in general help guide the trainee through a series of decision points. These, and other current 

U. S. Army training programs, support the concept that skills such as problem identification, 

4 
decision making, problem solving, and action implementation deserve more attention in VEbased training. 

Table 1: 

Top Activities for Soldier Tasks and Supportability by VE 

Activity Sensory Modality Effector

Primary Secondary

1. Give verbal orders Sound Speech 

2. Use password Sound Speech 

3. Blow whistle for signal Sound Hand/Head 

4. Call in preplanned fire requests Sound Speech 

5. Inspect for correct “Soldier’s load” Visual Force Feedback Hand/Head

6. Hear orders Sound Speech 

7. Operate radio or telephone Sound Visual/Tactile Hand/Head/Instr. Object

8. Identify safe or danger area Visual Head/Body/Hand 

9. Perceive relative position of other units Visual Sound Head/Body/Speech 

10. Give hand and arm signals Visual Hand/Body

11. Move in accordance with directions Visual Tactile/Force 

Feedback/Sound 

Hand/Head/Body

12. Visually search for enemy Visual Head/Speech

13. Identify hand and arm signals Visual Head/Speech

14. Aim and fire individual weapon Visual Tactile/Sound Head/Body/Hand 

15. Aim and fire crew served weapon Visual Tactile/Sound Head/Body/Hand

Note. Adapted from “Training Dismounted Soldiers in Virtual Environments: Task and 

Research Requirements” by Jacobs et al., 1994. 

We are limited, however, in the number of tasks we can evaluate in order to show how the 

optimal use of instructional features might improve VE-based training. For our research 

purposes, we abstracted many of the tasks and activities from Jacobs et al. (1994), as well as 

several cognitive tasks, into three, higher-order groups of tasks. These three groups are 

collectively referred to as Position/Maneuver expertise (PME) and are integral to the critical

missions identified in the National Research Council report (1997). The three groups incorporate 

both perceptual-psychomotor and cognitive dimensions: 

1. Enemy Position: refers to knowledge and awareness of enemy locations and firepower 

capabilities (type, range, threat zone, etc.). Example activities from Jacobs et al.’s list include: 

♦ Identify safe or danger area 

♦ Visually search for enemy 

♦ Identify areas that mask supporting enemy fires 

♦ Perceive relative position of weapon fire (own forces and opposing forces) 

♦ Identify obstacles 

♦ Identify light reflected from shiny objects 

♦ Identify glow from cigarette 

♦ Identify flashes from enemy weapons 

♦ Discern direction enemy is moving 

♦ Identify enemy Soldiers

5 
2. Own Position: refers to knowledge and awareness of trainee’s fellow Soldiers, troops, 

and other assets (e.g. artillery, tanks). Examples include: 

♦ Call in preplanned fire requests 

♦ Identify safe and danger area 

♦ Perceive relative positions of other units 

♦ Move in accordance with directions 

♦ Identify hand and arm signals 

♦ Identify support position to fire on enemy 

♦ Maintain position relative to other personnel 

♦ Identify overwatch position 

♦ Discern location within an area 

♦ Identify firing positions in urban area

3. Access Identification: refers to knowledge and awareness of avenues for logistics 

support (Medical Evacuation, supply vehicles, other troops). For example, evaluating if a 

helicopter could land in the area and identifying events that could prevent future access (building 

collapse due to bombing, enemy movement into open field, etc.). Examples include: 

♦ Identify safe and danger area 

♦ Identify obstacles 

♦ Discern location within an area 

♦ Count/inventory expendable supplies 

♦ Identify approach to a landing zone (LZ)/drop zone (DZ) free of tall trees, etc. 

♦ Identify distribution points for supplies 

♦ Identify LZ/DZ 

♦ Identify slopes which must be climbed 

♦ Inspect boat landing 

♦ Mark LZ/DZ 

For the most part, these tasks are complex, require visual and auditory stimuli based on 

terrain and environmental effects, and involve dynamic interaction with other personnel. The 

development of expertise in these areas could be fostered by performing the tasks within varying 

and efficiently-repeatable scenarios that require the execution of these complex procedures. VEbased training appears to be well suited for meeting these requirements. 

Task Selection 

The minimal task analysis conducted by Jacobs et al. (1994) focused on the use of 

projected VE technology that could be used in the relatively near future. We have used that 

information and judgments as a basis for selecting relatively typical activities that are sufficiently 

and quickly trained that they can be used as the basis for simulation-based training research. By 

reviewing Field Manual 7-8, Infantry Rifle Platoon and Squad (2001) the specific information 

needed for instruction and the development of a VE for application and practice of the task 

activities was derived. Relatively standard task analytic approaches for simulation-based 

training were used in this process (Hays & Singer, 1989). 

6 
Two knowledge types were utilized in each of the three task clusters listed above, and in 

the tasks selected as a basis for research: declarative and procedural knowledge. In brief, 

declarative knowledge refers to knowledge of facts, schemata and propositions, the what of the 

task, whereas procedural knowledge concerns how to do something. This requires converting 

presented information into particular cognitive instantiations – a mental model. For example, a 

Soldier learns the basic procedures of securing a building, such as search patterns and movement 

protocols, through documents, presentations, and instruction. This declarative knowledge is then 

called on when the Soldier gains procedural knowledge by actually practicing the application of 

structured rules for searching and securing a building, for example during standard field training 

exercises. In this manner, the declarative knowledge (the describable "what") is elaborated into 

procedural knowledge (the observable "how"), and the application of the knowledge is 

demonstrated for performance measurement. 

Successful Soldiers need to make the transition from declarative to procedural knowledge 

almost immediately, rapidly decreasing the amount of thought or diversion of attentional 

resources needed for correct performance. The situationally correct instantaneous application of 

procedural knowledge, or procedural automaticity, should therefore be the ultimate goal for any 

procedural training program. Soldiers in particular must be able to collect information from the 

environment, assess the meaning and importance of the information, and immediately apply their 

knowledge to choose the optimal course of action. 

Unfortunately, reaching automaticity is a challenge. Although extended practice is 

necessary to produce automaticity for most tasks (Logan & Klapp, 1991; Palmeri, 1997), simply 

increasing practice duration is not sufficient (Fisk, Ackerman, & Schneider, 1987). For practice 

to be effective, stimuli from the environment, and rules governing responses, must be 

consistently mapped to a response (Fisk, Hodge, Lee, & Rogers, 1990; Logan, 1988). In cases 

when stimuli or the application of procedures vary from one practice session to the next, as 

occurs in many simulations, automaticity will develop more slowly, if at all. Still, through 

repeated application, performance accuracy and/or speed should show improvement. 

Differences in performance improvement over trials should reflect differences in the 

instructional approach, which is the basic assumption behind classic transfer of training

investigations (see Hays & Singer, 1989 for a review of approaches). 

Instructional Framework 

The tasks require acquiring information, applying rules, and convergent problem solving. 

The development of expertise with these outcomes could be fostered by some combination of the 

following instructional tactics (Jonassen & Tessmer, 1996, except where noted): 

1. Describe required performance, conditions, and criteria 

2. Determine purposes or relevance of content 

3. Provide authentic, functional tasks 

4. Present facts, concept/rule/principle definitions and attributes 

5. Provide prototypical examples/demonstrations 

6. Provide worked examples 

7. Provide graphic cues; lines, colors, boxes, arrows, highlighting (Jonassen, Grabinger, 

& Harris, 1990) 

7 
8. Provide oral cues: oral direction (Jonassen, Grabinger, & Harris, 1990) 

9. Provide special information in windows (Jonassen, Grabinger, & Harris, 1990) 

10. Describe key aspects of information 

11. Apply in real world or simulated situations (near transfer) 

12. Change contexts or circumstances (far transfer) 

13. Provide confirmatory, knowledge of correct response 

14. Provide corrective and remedial feedback 

15. Provide expert solution models 

16. Provide hierarchical prerequisite sequence 

17. Provide procedural, job sequence 

18. Demonstrate strategy corrections 

19. Review prerequisite skills or knowledge 

20. Create images 

21. Create story lines: narrative descriptions of information 

We believe mission rehearsal in a relatively complex VE fits the task requirements 

outlined above, and can help train Soldiers to this procedural automaticity. However, given the 

obstacles to achieving automaticity, doing so will require the use of strategies and techniques 

that optimize the learning environment to ensure that procedural knowledge is consistently 

mapped, or related to, appropriate actions. We therefore examined relevant literature related to 

instructional strategies and features used to support such strategies, in order to identify several 

promising approaches that could be used in future VE-based training programs. 

Conceptual Integration: Instructional Goals, Strategies, Tactics, and Features 

Finding the best way to impart declarative and procedural knowledge in an explicit and 

directed manner has been a major research focus in a number of scientific and applied domains, 

particularly in the cognitive psychology, educational psychology, industrial and organizational 

(I/O) psychology, and military disciplines. Educators, for instance, have continually sought 

better ways to teach their students, from improved textbooks and lecture formats to computerbased simulations of basic concepts (e.g., Graham, Alloway, & Krames, 1994; Horton, 1994; 

Savelsbergh, de Jong, & Ferguson-Hessler, 2000). Instructional Systems researchers, in the end, 

are striving to identify “the extent to which various arrangements and characteristics of stimuli 

promote learning” (Boldovici, 1992, p.2). 

One purpose of this report is to bring sufficient order to the instructional literature, as it 

applies to VE-based training, to enable a taxonomic approach to be developed that can support 

research planning and eventual optimization of future VE-based training systems. To this end, 

we have attempted to define, contrast, and place in a hierarchical relationship the terms 

instructional goal, strategy, tactic, and feature. Unfortunately, the abundance of research on 

directed learning has made it difficult to define and differentiate between these concepts. For 

example, multiple terms are often used to describe similar ideas or concepts. Nevertheless, we 

argue that for most directed learning approaches, there are generally four main components: 

goals, strategies, tactics, and features. 

8 
Definitions 

Instructional goals, approaches, strategies, tactics, and features relate to one another in 

structured and supportive relationships. First, if we accept that directed learning is the 

purposeful transfer of information, knowledge, skills, abilities, and/or attitudes from one source 

(e.g., instructor, computer software, simulation, or other system) to an individual or group (Hays, 

2001), then the purpose of a directed learning program can be termed the Instructional Goal. 

This purpose has also been referred to as the instructional objective, outcome, or task. Second, 

directed learning programs must have one or more (usually multiple) explicit approaches, or 

Instructional Strategies— a “plan, method, or series of activities aimed at obtaining a specific 

goal” (Jonassen & Grabowski, 1993, p.20). This leads to Instructional Tactics which are 

“specific actions which are well-rehearsed and are used to enable the strategy” (Jonassen & 

Grabowski, 1993, p.20). Instructional tactics are maneuvers or manipulations embedded within a 

strategy that can be used to change a learner’s knowledge state so that it enables the learner to 

reach the instructional goal. The tactics should enable a learner to integrate or absorb or

assimilate new knowledge into their repertoire. Finally, Instructional Features are “elements of 

training devices that can improve training efficiency on individual tasks” (Sticha, et al., 1990, 

p.17). Instructional Features thus refers to a wide variety of tools and/or techniques which 

instructors can use to support and implement the instructional tactics and strategies. Based on 

these definitions and our presented relational structure, it should be clear that these concepts can 

be organized hierarchically, as depicted for the example training goal in Figure 1. 

Instructional Strategy: Use Simulation to Improve 

Application of Landing Zone Area Rules

Instructional Strategy: 

Provide Adjunct Cue

Instructional Strategy:

Support Cognitive

Elaboration

Instructional Tactic: Use 

Graphics to Indicate

Important Areas

Instructional Tactic: Use 

Auditory (Oral) Prompts

Instructional Feature: 

Instructor Selectable Area

Addressed by Graphic

Instructional Feature: 

Channel enabling

Instructor/Trainee

Communication 

Instructional Goal: Identify & Select Landing Zone

Figure 1. Relationship among instructional goal, strategies, tactics, and features. 

9 
In Figure 1, the box on the bottom left describes an instructional feature: a tool enabling 

the instructor to select an area that is important for the application of Landing Zone rules. For 

example, the instructor may want to indicate that a nearby tree is too tall for a helicopter glide 

path. The instructional feature enables the instructional strategy of providing cues, implemented 

through the instructional tactic of using graphics cues (an arrow in our research, see the example, 

and discussion below) to point out critical environmental features. This enables the instructional 

strategy of providing an adjunct cue (something not normally in the environment, see discussion 

below) in order to aid in the correct application of known rules. 

The second Instructional Strategy (see Figure 1) was to provide support for cognitive 

elaboration in the application of Landing Zone rules. The enabling tactic was to provide oral 

cues (oral questions) that elicited the most applicable rule given the environmental and task 

situation. The tactic of providing oral cues during rehearsal in a simulation is in turn supported 

and enabled by an instructional feature – a communications channel between the trainee and 

instructor. 

Seldom do strategies, tactics, or features appear in sufficiently clear isolation within an

instructional program that would enable analysis of their effects. In this experiment we are 

providing clear and isolated cue combinations of an Instructional Feature Graphic (directional 

arrow) and oral question tactic as cues and feedback (coaching), in order to measure the benefit 

to the instructional strategy. It should be noted that a feature can support more than one strategy 

or tactic. For example, the graphic, when used to highlight an error, may also implement a 

feedback strategy through an informational feedback tactic (Bailey, 1982). 

Instructional Feature Categories 

As noted above, a number of Instructional Features (IF) can be used to support and 

implement specific instructional strategies. These features can be categorized along four 

dimensions based on characteristics of the feature and how it is applied: Temporal, 

Adjunct/Augment, Naturalness, and Control. As with strategies, these categories are constructed 

to be of most relevance and use when addressing simulations, whether for training or skill 

maintenance. The categories will apply to all training approaches, in general form, but make 

most sense for simulation-based training. 

The Temporal category concerns when the IF is implemented; before, during, or after a 

training task or event. For example, many demonstration features occur prior to participants’ 

practicing tasks or skills. Review features, on the other hand, may occur at any time surrounding 

an event. There are also features that can slow or speed portions of practiced tasks or 

presentations, skip over segments, and repeat them as necessary or desired. And of course, as 

shown in Figure 1, there are many IF applications that provide support during training strategies 

(e.g. coaching). 

The Adjunct/Augment category concerns how the feature is implemented, and addresses the 

stimuli inherent in the information presentation (Boldovici, 1992). Practice with actual 

equipment, simulations, media presentations, etc. will (or should) present the necessary

initiating, guiding, and correct-performance reinforcing stimuli for every required activity. Even 

10 
text or lecture will identify these stimuli so that the performer can correctly attend to and use 

them. Augmenting an initiating, guiding, or reinforcing cue means manipulating its normal 

sensory characteristics to make it more noticeable or salient (Boldovici, 1992). Adjunct cues or 

stimuli are those that do not normally belong in the target task environment, and the IF consists 

of inserting something new which calls attention to or makes more salient the important 

initiating, guiding, or reinforcing stimuli. In general, this category relates to features 

implemented during performance of a training task. 

The Control category concerns who is responsible for administering or directing the 

feature. In most cases, features are either trainee controlled, instructor controlled, or are 

automated such that the VE or computer-based system recognizes events and then presents the 

feature. 

Finally, the Naturalistic category concerns the degree to which the feature imitates or has 

the appearance of stimuli in the real world task. Most features are classified as unnatural 

because, by their very nature, the features add to or alter the natural environment in order to 

train. 

As minimally discussed above, practice alone is unlikely to provide or impart automaticity 

in knowledge application, for example in performing a task. Several researchers have argued 

(Fisk et al., 1990; Logan, 1988) that in a training session, stimuli must be consistently paired 

with appropriate responses if automaticity in the form of accurate and correct performance is to 

develop at a relatively rapid pace. Unfortunately, this leads to a dilemma because training 

Soldiers repeatedly with the exact same stimuli—the approach that most quickly builds 

automaticity—could lead to inflexibility. In other words, Soldiers might learn how to rapidly 

and automatically respond to identical conditions in real situations, but may not be able to 

rapidly adapt and transfer this performance from one situation to another, as is the general goal 

in Army training. 

The Army does not need to train Soldiers all the way to automaticity, but Soldiers do need 

to reach a well-practiced, easily performed and highly skilled level, with skills that are readily 

adaptable and transferable. The appropriate knowledge should be rapidly and easily retrieved, 

and the important decisions should be quickly and easily made with a minimum of cognitive 

load. The application of selected IFs within VE simulations may be a promising solution. A 

major advantage of VE-based training is the flexibility of inserting stimuli. Environments can be 

automatically constructed from geographical databases and satellite imagery. With minimal 

alterations, instructors can add new attributes or dimensions to VE simulations to significantly 

change the trainees’ experience. For example, the ability to shift from daytime to nighttime 

operations, add night-vision displays, or to increase the number of opposing forces or change 

their strategies and actions, can be preprogrammed into the system to give instructors greater 

selection and control of environments and conditions. This flexibility also means that instructors 

can employ a number of IF, when necessary, to enhance training. 

The first step to providing recommendations about IFs for use within Army VE 

simulations is to identify features that have shown promise in improving training in similar 

simulations for similar tasks. We searched the literature from academia, industry, the military, 

11 
and similar areas for studies that either evaluated or employed IFs in an instructional simulation. 

Although a search using reasonable keywords yielded a large amount of material, most articles 

offered little concrete information about the efficacy of a specific IF, primarily because the goals 

were not centered on features (i.e., a feature was used, but no evaluation of specific 

effectiveness). Primarily based on older reviews of the area (Ayres et al. 1984; Knerr et al., 

1998; Sticha et al., 1990), in addition to insights from discussions with our developers at the 

Institute for Simulation and Training, we developed a set of three IFs that appear to have the 

greatest potential for success, and therefore deserve empirical evaluation. 

To describe the selected features, we used one of the three common task groups previously 

mentioned, those related to Enemy Position, as a framing example. In addition, each feature is 

related to one or more instructional strategies. 

Dynamic Diagrams 

Computer-generated diagrams, graphics, or maps that demonstrate tasks or procedures 

through the use of time-linked elements that indicate movement and sequence are referred to as 

dynamic. For example, Park and Gittleman (1995) compared Dynamic Visual Displays with 

Static Visual Displays in their effect on the formation of mental models of electronic circuits. 

Although the authors found no difference on mental model formation, students who worked with 

Dynamic Visual Displays showed better learning early on and faster overall problem solving. 

Figure 2. Example dynamic diagram. 

12 
Figure 2 illustrates a dynamic diagram (as well as can be done given print media) based on 

a previous ARI developed VE research simulation database (Singer et al., 2001). The line 

starting in the small vestibule at the bottom of the figure extending through the first room, in and 

out of the small room on the lower-right, and ending in the second room on the right illustrates 

the correct order of room search (i.e., the “right hand rule”). Trainees would see this line appear 

and then extend from room to room in a dynamic path illustrating the rule. (Note that this 

dynamic diagram was not actually used in that research.) 

Strategies. Dynamic diagrams could implement a provide examples instructional strategy 

through a demonstration instructional tactic to teach or further explain concepts. The 

instructional tactic would be to use the dynamic diagram instructional feature to show the 

performance of critical situational procedures. For Enemy Position tasks, a dynamic diagram

instructional feature could be used to show known enemy movements, artillery ranges, etc. 

Dynamic diagrams could also be used to implement a provide feedback strategy through an 

information feedback tactic, allowing trainees to see a representation of themselves performing 

the procedures in question. Dynamic diagrams like this have been implemented in an 

experimental AAR system for VE simulations (Knerr, et al., 2002), in which participants can 

observe their own and others movements represented as icons or avatars, leaving colored trails or 

lines indicating the traversed path. 

Potential Problems. This kind of feature presents two developmental difficulties; display 

work load and application dynamics. While the example presented above is relatively simple 

and direct, implementing a dynamic visual display is not trivial and can add significantly to the 

cost of display development if the illustration becomes complex. For example, the dynamics 

require not just visually modeling all components of a piece of equipment that has to be

disassembled, but also showing the dynamic motions required for that disassembly. The second 

issue is application dynamics, in which the determination of important aspects of a situation, how 

much detail, and when to show those details are just the grossest considerations. A rational and 

effective approach to evaluating this class of IF would be to address a single generic form of this 

feature, a prototypical example of the feature developed for a clearly defined category of skills, 

knowledges, or abilities. 

Text-based Messages 

Text-based messages are usually text windows that pop-up into the field of view (FOV) at 

an instructionally appropriate time (see Figure 3). Common behavioral tenets indicate that error 

messages should be presented immediately after the trainee makes an error, although they can 

appear during an activity. Similar approaches have proven successful in training pilots in flight 

simulators (Skitka, Mosier, & Burdick, 1999), mission control personnel on satellite deployment 

procedures (Loftin, Wang, Baffes, & Hua, 1992), and showing users how to operate the Navy’s 

consolidated Area Telephone System (Cowen, 1993). 

Strategies. Error messages could afford a provide feedback strategy through a provide 

corrective and remedial feedback tactic (Jonassen & Tessmer, 1996) as in Figure 3. For the 

Enemy Position class of tasks and activities, these pop-up messages might provide warnings, 

13 
emphasize inspection or movement protocols, or provide reminders for the types of cues 

indicating enemy movement or positions. 

Figure 3. Example text-based message. 

Potential problems. The crutch effect (Boldovici, 1992) occurs when an inserted or added 

feature or function in a training simulation produces a reliance on that feature as a cue for 

performance. In this case, the effect might be based on an over-reliance on the text error 

feedback that transfers to real world situations and degrades initial transfer performance. As 

always, implementation issues (e.g., form. location, timing, duration) can present problems for 

the addition of non-normal cues or information in a simulation. In this case, message position in 

the FOV remains a question. If appearance is dynamic, based on performance monitoring, the 

text-window potentially could overlay important cues for correct performance when it appears. 

At this point, placing the message toward the bottom of the trainee’s FOV, such that around 75% 

of normal view remains, seems to avoid most of the apparent location problems (see Figure 3). 

Augmenting and Adjuncting Cues 

These categories are derived from a theoretical approach developed by Boldovici (1992) in 

an attempt to clarify actions that can be taken to increase the meaningfulness (salience, to use his 

term) of stimuli during training. Augmenting means emphasizing or de-emphasizing the 

naturally occurring attributes or characteristics of the task-important initiating, guiding, or 

performance feedback cues in a simulation environment in order to alert or orient trainees to 

critical stimuli. This can be accomplished by making one or more characteristics of existing 

critical stimuli more salient to the trainee. For example, changing the form or duration of a 

natural stimulus by enhancing the color or sound would be augmenting that dimension of the 

stimulus. Alternatively, one could diminish or de-emphasize the surrounding visual stimuli or 

sounds so as to make the key characteristic more noticeable (the decrease in distracters is 

described as fading by Boldovici). Adjuncting refers to adding a cue or stimulus to the 

14 
environment that does not normally occur (referred to as supplementing by Boldovici). 

Employing an unnatural stimulus (e.g., a colored outline, flashing dots, pointing arrows, etc) will 

draw attention to those environmental stimuli (Boldovici). This approach has shown limited 

promise in teaching firefighters how to recognize the direction and spread of brushfires 

(Lewandowsky, Dunn, Kirsner, & Randell, 1997). 

As noted by Boldovici (1992), salience or meaningfulness is not a property of the 

environmental stimuli – it is a property assigned by the organism during perception. Boldovici 

presents a somewhat behavioristic approach to stimuli and responses in a task environment, 

although it seems more reasonable to include those behavioral tenets within a more cognitive 

framework. To that end we can restate his point about salience in the following way, that the 

trainee/learner assigned meaningfulness of the stimuli is the key to a cue’s effectiveness in the 

learning environment. While this meaningfulness can be derived without symbolic 

representation or cognitive modeling by an organism through behavioral conditioning 

techniques, transmission of information through language can also lead to information being 

incorporated in a mental model. Therefore, explaining the use of an augmenting cue or adjunct 

cue to the learner provides the basis for the meaningfulness of important stimuli in a simulated 

task environment. If a normally occurring visual or auditory characteristic is altered, and the 

trainee does not perceive the change, or does not connect the perceived change to important 

information about task performance, then the manipulation is without value, and may even be 

detrimental to performance. 

For example, to train proper building search movement, “flashing” doors in a VE building 

interior could be used to indicate the room that team members should enter next, as illustrated by 

the surrounding lines around a door in Figure 4. There has to be precursor training in which 

“flashing” has been clearly explained as an aiding stimulus for where to go next, and the team

members have to remember that explanation. 

Figure 4. Example highlighting of door sequence. 

Strategies. A highlighting tactic would implement a provide cueing system strategy. This 

tactic would be appropriate for acquiring bodies of information, structuring mental models, 

15 
forming concepts, using procedures, and convergent problem solving (Jonassen & Tessmer, 

1996; Park & Gittleman, 1996). 

Potential problems. There does not appear to be any empirical support for the 

effectiveness of this approach in conjunction with movement or search tasks. However, to the 

extent that movement and search are based on reasoning or problem solving the previous 

research may indicate potential for increases in training effectiveness. 

Instructional Feature Selection 

The overriding goal of the present research was to identify and then test an IF for use in 

future US Army VE training systems. The first step in our program was the development of IFs 

for evaluation. As the features were developed within a framework of training task and 

instructional strategies, the opportunity to document and evaluate the development easily arises. 

Based on the preceding discussion, we chose to first evaluate an adjunct-type IF, illustrated in 

Figure 5, whereby a position-linked (both with the trainee and the location of importance) arrow 

is used to point out the location of a set of stimuli that must be re-evaluated for improved task 

performance. For example, the participant may have placed or marked a position (see discussion 

of tasks, below) incorrectly, rather than correctly applying the rules for placing or marking a 

position (the task at this point is irrelevant). When the instructor detects and confirms the error, 

the Attention-Direction (AD) Arrow IF can be used to point to the incorrect position, or to 

indicate the stimuli that should be used to determine correct placement or marking, or can simply 

be used to indicate where the mark should have been placed. The arrow is designed to move as 

Figure 5. Position-linked attention-direction arrow instructional feature.

16 
the user’s FOV changes so that it always points at the location specified by the trainer/instructor. 

(Note that this will sometimes mean that the arrow will be pointing directly at the trainee, as the 

indicated location is directly behind the trainee.) 

Using the classification structure introduced above, this feature is categorized as an 

unnatural IF, one that trainers/instructors manipulate to point out critical areas the participant 

may have missed in the visual scan of the environment. Temporally, the arrow feature is 

employed after a trainee makes an error, such as not looking at potential OpFor positions, and is 

obviously time-linked. Categorization of the arrow feature, in the context of IF and strategies, is 

presented in Table 2. 

The second IF chosen for this research was Coaching. The coaching was presented in the 

form of an oral interrogative spoken by the experimenter (heard by the trainee in the VE system

headset) after commission of an error. The purpose of the coaching question was to encourage 

participants to reconsider previous actions for appropriateness based upon rules that had been 

learned during the introductory training. The coaching questions used were designed to elicit the 

correct task-situational application of one of the rules that were trained before the mission 

planning sessions began. This was accomplished by literally asking for the appropriate rule, 

implicitly providing feedback that some aspect of the immediately preceding performance or 

decision was incorrect, and explicitly identifying which aspect by the content of the question 

stem. If participants responded incorrectly, the correct rule of information was provided. 

Table 2. 

Categorization of a Directional Attention Arrow Instructional Feature 

Instructional 

Strategies 

Temporal Adjunct/Augment Control Naturalistic

Demonstration During 

Presentations

Adjunct Cue Pre-Planned by 

Instructor 

Unnatural 

Practice NA NA NA NA

Error 

Identify/Avoid 

After failure to 

look at area 

Adjunct Cue Instructor 

Controlled 

Unnatural 

Review During 

Presentation

Adjunct Cue Instructor 

Controlled 

Unnatural 

The timing of the two IF interventions was an important consideration for us, in order to be 

able to cleanly categorize the intervention characteristics. One of the most ubiquitous 

instructional strategies used in military training is to coach during the performance of early skill 

acquisition trials. Given the increasing use of artificial intelligence methods for supporting 

training strategies, we decided to mimic Artificial Intelligence routines by designing the AD 

Arrow and Coaching IFs to be performance-error linked and guided-discovery based. 

Performance-error linked AD Arrow or Coaching intervention means that no adjunct stimuli or 

coaching are provided until a decision point or performance step is reached and the student has 

performed some sub-task segment incorrectly. During the training, the coach then immediately 

uses a context sensitive question probe to elicit from the trainee the known rule or information 

17 
needed for a correct decision or performance step. Only when rules are not correctly retrieved or 

expressed by the trainee does the “coach” provide direct information, in the form of pertinent 

information or an important. The AD Arrow is applied in the same fashion, appearing and 

indicating a location of importance in the immediately preceding task or performance step. 

Instructional Feature Experiment 

This experiment was designed to investigate the training effect of two instructional factors: 

Interrogative Coaching (IC) intervention and an Attention-Directing (AD) Arrow. In the 

experiment, during instruction, a participant would receive an oral question as an intervention, an 

AD Arrow IF, both, or neither. It also produced one awkward, but necessary, instructional 

situation and one more normal situation: the awkward situation had an instructional feature that 

appeared only when performance errors were committed and had no accompanying explanation, 

the second, more normal situation, provided coaching when an error had been made but no 

indication was given of the location of important cues. As initial acquisition of skills over 

repeated trials is the framing context, the experimental instruction was conducted over three 

repeated sessions, during which participants practiced and received feedback, and tested during a 

final (fourth) mission without instructional interventions. 

The application of these conditions results in a standard 2 (presence or absence of coaching 

intervention) x 2 (use or non-use of Attention-Direction Arrow) x 4 trial repeated measures 

experimental design (see Figure 6). The general hypotheses were that some coaching is better 

than none, and that attention-direction information will have positive effects over no attentiondirection information. These effects were measured in terms of accuracy and time of 

performance during the several tasks addressed (the dependent variables are presented in greater 

detail in the procedures section, below). 

The general hypothesized differences were anticipated to be the case both over the 

repeated training missions (during learning) and during the final or test mission. We also were 

interested in the possibility of interactions being found between coaching and attention-direction 

IFs, in that the application of these tactics together logically should be better than either applied 

separately. However, there was no research that we were aware of that would support prediction 

of that result. It was also expected that participants would improve over repeated trials, but the 

most interesting outcomes involved the effects of the instructional manipulations over the 

repeated training trials and the final outcome mission. 

18 
1st Mission

No IC IC Applied

No

AD Arrow

AD Arrow

Applied

2nd Mission

3rd Mission

4th or Test

Mission

Figure 6. Experimental design.

Method 

Participants 

Thirty-two participants were recruited from the University of Central Florida student 

population, and paid or given class credit for completing participation. Nine of the participants 

were female and twenty-three were male. The participants’ average age was 20.59 years. 

Thirty-one of the thirty-two participants rated themselves as average to highly confident on 

computers. Thirty of the thirty-two rated themselves as average to highly skilled on video 

games. While ten claimed to play video games only an hour or less per week, nineteen played 

from one to eighteen hours per week, with three claiming twenty plus hours per week (one 

claimed the maximum of thirty-five). Fifteen of the participants had never experienced virtual 

reality, six had experienced it once, and only one claimed more than ten experiences. All 

claimed normal vision, with only one reporting any color blindness. No relationships between 

these variables and the experimental factors were found. 

Materials and Apparatus 

The VE scenarios were rendered on a computer-based system consisting of linked PCs. 

MotionStartm sensors were used to track participants’ physical movements, and Virtual Research 

19 
V8tm head mounted displays (HMD) presented head-slaved, computer-generated, stereoscopic 

color imagery to the participants. Stereo sound for object collision, gunshot noises (for a paint 

gun), and oral coaching were provided through earphones attached to the HMD. The software 

was written by IST using Performer, C++, and Java. In all, the experiment required the use of

six networked personal computers and one stand-alone personal computer for questionnaire 

administration. All activities took place within a safety frame referred to as a safety pod. The 

safety pod was a raised platform with approximately waist-high encircling railings to prevent 

falls and limit movement. The HMD with head tracking sensor and body position sensors 

(including shoulder position, hand unit, elbow, and ankle sensors) were connected to a backpack 

frame, with cables running up to a boom positioned overhead. This arrangement prevented 

excessive movement, precluded falls, and allowed relatively free movement through several 360 

degree turns. 

Virtual Environments. Two real-world environments were digitally modeled: the 

Shughart-Gordon Military Operations in Urban Terrain (MOUT) site at Ft. Polk, Louisiana, and 

the McKenna MOUT site at Ft. Benning, Georgia. Each of these environments was slightly 

altered for experimental use, and had two different starting points established. This allowed each 

environment to be used twice in the four-scenario experimental series. 

The AD Arrow IF, designed by IST, was controlled by the experimenter. After 

participants committed an error, the experimenter would select the location by using a mousecontrol key combination, causing the AD Arrow IF to appear in the bottom center of the 

participant’s FOV, with a changing orientation that always pointed toward the Experimenter 

designated target location. The arrow remained in the FOV until the participant shifted the 

center of their gaze to within a 20-degree deviation from the target location, whereupon it then 

disappeared from view. This center of view determination was based on calculations on the 

FOV in the HMD, and implemented based on the presumption that the participant would actually 

observe the indicated area when viewing the area within the 20-degree deviation. The 

interrogative coaching statements were scripted for critical types of errors and read over the VE 

sound system by the experimenter at the appropriate time, based upon the error detected by the 

experimenter. 

Questionnaires. Several questionnaires were administered before, during, and after the 

sequence of VE scenarios. All questionnaires were implemented in an Accesstm database for 

ease of administration and analysis. The Biographical questionnaire asked questions about 

participant characteristics and experience, primarily with video games and VEs (see Appendix 

A). The Simulator Sickness Questionnaire (SSQ; Kennedy, Lane, Berbaum, & Lilienthal, 1993) 

was used to address the level of symptoms before, during, and after the VE episodes (1 training 

experience and 4 mission sessions). This allowed continuous monitoring of the participant’s 

physical health (see Procedures, below). Other questionnaires included the Immersive 

Tendencies Questionnaire (ITQ; Witmer & Singer, 1998), administered at the beginning of the 

experiment, and the Presence Questionnaire (PQ Vs. 3.0; Witmer & Singer, 1998), administered 

three times (see Procedures, below). The ITQ addresses the participant’s tendencies toward 

immersion and involvement in general, while the PQ addresses the participant’s level of selfperceived immersion and involvement in the immediately preceding virtual reality-based 

20 
experience or episode. These data are collected as a part of ongoing ARI SSRU research into 

perceptions and attention during VE experiences. 

Procedures 

As is required for the ethical treatment of experimental participants, the purposes of the 

research were explained to each participant, their questions were answered to their satisfaction, 

and their agreement to participate in the research was recorded (Ethical Principles of 

Psychologists and Code of Conduct, American Psychologist, 1992, as reprinted in Appendix C of 

the Publication Manual of the American Psychological Association, 2001; Army Research 

Institute for the Behavioral & Social Sciences, 2003). The experimental protocol sequence was 

as follows: 

1. Upon arrival, each participant was informed about compensation, right to withdraw, 

and basic procedures to be used in the research. 

2. A demonstration of a typical planning exercise was provided on a large flat screen 

monitor. A different VE terrain from those used for the experimental sessions was 

used for this demonstration, to prevent environment knowledge transfer. 

3. The sequence of experimental activities was reviewed with the participant, and 

agreement to participate was documented. 

4. A biographical questionnaire was administered, followed by the ITQ, and the first 

administration of the SSQ. 

5. The participant was introduced to the VE experience by learning to move, walk, and 

pace distances in the environment, as well as using the marking unit (a virtual paint 

gun) to mark positions for the different tasks using different colors. Participants were 

coached and practiced until minimal performance standards were met. The training 

session was limited to twelve minutes, with all participants successfully completing 

training in that time. (It should be noted that distance examples were provided 

visually, and with movement time examples. No accuracy in judging distances was 

taught or tested.) 

6. Following the initial VE training a post-experience SSQ was administered. Criteria for 

suspension of the experiment based on sickness are discussed below. After completion 

of the SSQ, the first PQ was administered. 

7. A manual was given to the participants (see Appendix B), containing instructions and 

rules for the dismounted Soldier synthetic tasks: a) Bounding Overwatch (BO) 

movement, b) Landing Zone (LZ) selection, and c) Observation/Fire Post (OFP) 

emplacement. A questionnaire and map test (Appendix C) was used to insure that the 

participant understood all task rules and requirements. Participants were tested on 

correct knowledge of features and application of rules using a multiple choice test 

(requiring 13 of 15 responses correct) and a short map planning exercise (no more than 

2 errors, verbally corrected by the experimenter), before being allowed to participate in 

the VE exercises. 

8. Following successful training, participants began the 4 mission experiment cycle. (The 

order of presentation of environments was counterbalanced.) Each mission cycle was 

conducted as follows: 

21 
• Participants were first reminded of the mission goals and criteria. They were then 

given a mission map (see Appendix D) and asked to plan their movement using 

BO patterns and select a goal location for a helicopter LZ and attendant OFP 

emplacement. 

• After completing a pre-exercise SSQ, and a review of their completed map, each 

participant entered the appropriate environment and conducted a mission 

reconnaissance exercise in which they marked their planned route, LZ, and OFPs. 

Participants were reminded before beginning that they could change the mapplanned route and locations based on environmental appraisal using rules from the 

training manual. 

• Following the VE mission reconnaissance exercises (limited to twelve minutes in 

the VE), participants removed the VE equipment, exited the safety pod, and 

moved to a computer to complete a post-exercise SSQ. Immediately after the 

SSQ, an After Action Review (AAR) of five minutes duration was administered. 

During the AAR only previously uncorrected errors were called out, and as with 

the IC Strategy Condition rules, the participant was asked a question about the 

applicable rule, and required to provide an explanation of the decisions or actions 

performed. That is, after the replay of the mission segment containing the error, 

in accordance with Bailey (1982), the participant was prompted to: (a) state what 

the error was; (b) why it occurred, and; (c) what could be done to correct it. Only 

the most critical single error from each of the three mission tasks (BO, LZ, and 

OFP) was presented for discussion during the AAR. The AAR of critical errors 

with coaching was the only “instruction” that the No IC/No AD Arrow IF

(Control) group experienced. No AAR was administered after the fourth exercise. 

• Participants were required to remain out of the VE for a minimum twenty minute 

intervention period between sessions to minimize sickness and enable adaptation. 

As noted above, a PQ was also administered after the first and fourth VE session. 

This intervening period was typically occupied by the SSQ, the AAR, and 

preparation for the following exercise. 

9. After the four mission cycle was completed, the purpose of the experiment was again 

reviewed with the participant and compensation for participation in the experiment was 

provided. Before being released at twenty minutes after exiting the last VE experience, 

the participant completed a final SSQ to ensure complete recovery from their VE 

experiences.

Participants were not constrained to time limits for the three individual tasks during the 

exercises; although they were given a one minute warning before the twelve minute session limit 

was reached. All participants were stopped at the twelve minute limit, sometimes without 

completing all of the tasks (see Results, below). 

To recapitulate, each participant proceeded through four map and VE exercises during the 

experiment, with differing VE terrain databases (two different MOUT sites, each with two 

widely spaced starting points) being presented in counter-balanced order while following the 

same process and protocol for each reconnaissance exercise. The counter-balancing structure 

presented each of the four environments in each sequential position an equal number of times. 

The counter-balancing also insured that participants never experienced the same MOUT 

22 
environment sequentially. This counter-balancing produced eight unique sequences, which were 

then used as the basis for the number of participants in each condition. For all participants, the 

fourth mission was the no-intervention test mission, and as noted above no AAR was 

administered after the fourth (test) mission. 

Results 

Questionnaire Data Analyses 

The most basic use of the biographical data is to discover any possible intervening issues 

that the participants bring to the research that would affect the analysis of the independent 

variables in the experiment. General Linear Model analyses (i.e. 2x2 ANOVA) of the

participant’s basic characteristics and prior experiences found no significant differences from

any of the reported biographical information between the major factors (Interrogative Coaching 

or Attention Direction) in this experiment. The important descriptive information about the 

participant sample was presented in the Participants section, above. 

As introduced above, the ITQ is intended to provide information about qualities and 

tendencies of participants in the evaluation of their environmental experience that may influence 

or explain performance in the simulated environment. The first analysis is the same as 

performed on the biographical data, a General Linear Model analysis examining differences in 

the ITQ scales between the major factors (Interrogative Coaching and Attention Direction) in the 

experiment. No significant differences in the ITQ scores or scales were found between the major 

factors and interactions used in this experiment. 

The PQ is intended to directly address the subjective experience in the simulation 

environment, and unlike the biographical or ITQ, was administered at several different points in 

the experimental sequence (see Procedures, above). In order to establish whether there was any 

change in the PQ that could be directly attributed to the factors used in the experiment a 

repeated-measures General Linear Model was performed in similar fashion to the ITQ and 

biographical analyses discussed above, with the addition of the repeated administrations of the 

PQ. While the PQ did change significantly over the three repeated administrations (F(2, 54) = 

14.172, p<.001), no significant differences were found between the factors or factor interactions. 

The PQ decreased from the first administration following post-movement training (M=98.53; 

averaged over all conditions), to the post-mission one administration (M=95.65), and increased 

following the fourth or test mission (M=106.84). These results do not appear to provide any 

additional explanation of the experiment factor results presented below, and therefore no further 

analyses or discussion of the PQ results are presented. 

As explained in the procedures section, above, the SSQ was administered before and after 

each VE session, including the movement and equipment operations training that began the 

experiment sequence. The data of interest for the performance analysis is whether there was a 

pattern in the changes (post-test minus pretest) over the experimental missions. A repeatedmeasures General Linear Model analysis was performed, which found no significant differences 

over the four mission differences, between the factors (Interrogative Coaching and Attention 

Direction) or the factor interactions. This is not totally unexpected, as anyone experiencing an 

23 
abnormally high SSQ score before any VE session was not allowed to participate (rated as being 

in the upper ten percent based on Kennedy, et al., 1993). In addition, any participant 

experiencing a large increase in their SSQ score post-mission (the rule of thumb used was a 

doubling in their score) either had to recover rapidly and not experience a repeated similar 

episode or withdraw from the experiment (receiving credit for participation to that point). A 

total of five participants were withdrawn from the experiment based on these conditions. 

Performance Measurement 

As these are complicated activities, there were several process and outcome measures that 

were used for analysis of the VE instructional interventions. Each of the three tasks (Bounding 

Overwatch [BO], Landing Zone placement [LZ], and Observation/Fire Post placement [OFP]) 

was measured separately, and in several ways. The task measures included; task performance 

times, time spent inspecting buildings during the task, number of buildings inspected, and rated 

performance scores. The task performance times and rated performance scores are outcome 

measures, while the building inspection times and numbers should reflect rule application 

processes associated with the tasks. The rationale for using these measures is that they should 

change based on the efficacy of the instructional approach and IF being used. The following 

situation is provided as an example of the instructional approach in applying both of the 

instructional features. The Experimenter would use the Directional Arrow to point at a building 

that should be inspected (based on rules provided during instruction, see Appendix B), while 

asking a question (Interrogative Coaching application) about the appropriate rule. This should 

increase the application of the rule by the participant, and improve the correct performance of 

that aspect of the task. 

Performance Time. The VE system was set up to make a record of all important events 

that occurred during each second. Total mission time recording began (controlled by the 

Experimenter) when the participant was told to begin the mission and ended when the participant 

indicated that they were finished, or the Experimenter enforced the twelve minute time limit. 

Because there was unavoidable lag time at both the beginning and end of every session, we 

measured performance time data from the time that the first mark was made (usually a BO mark) 

until the time that the last mark made (almost always an OFP mark). Data reduction routines 

were used to track and compile time spent on each task based upon the marks used and the 

recorded times. Post-processing software accumulated the time between making the marks, and 

summed those intervals based upon the preceding type of mark, so that total performance times 

for each separate task was calculated even if the task performances overlapped. 

Building Inspection Time. As environment inspection was an important part of every task 

(based on the task instructions, see Appendix B), and the environments were urban, building 

inspection by the participant during each task was also used as a measure for the task. Buildings 

were considered to be under inspection during any time interval if fifty percent of any face of the 

building was in the central visual area of the HMD (calculated by vector analyzing (invisible) 

points on each face of a building during post-processing of the participant visual field from each 

mission). In this manner, at distance, several buildings could be counted as being inspected 

simultaneously during a one second interval. In order to better measure inspection, rather than 

the number of times that a building was viewed for even a part of a second, the decision was 

24 
made to only include the building if it was in the FOV for three consecutive seconds. The 

amount of time spent inspecting buildings was the accumulation of seconds with one or more 

buildings in view (for at least three consecutive seconds) during performance of a particular task. 

Number of Buildings Inspected. As with building inspection time, the total number of 

buildings inspected was determined by calculating intervals of consecutive 3-second gazes at an 

individual building and counting that as a single building inspection. In addition, a building 

would have to be within the central visual area of the HMD during another three second interval 

to be counted as another building being inspected. This is based on the idea that one should 

count repeated inspections of the same building, as they may have been inspected from different 

positions or for different reasons. Again, multiple buildings could be under inspection at any 

time, and as noted above buildings could be repeatedly inspected. The number of building 

inspections meeting these criteria during a particular task was then the sum of times that each 

building was inspected based upon time and FOV. 

Task Performance. Task performance was derived by rating the correct application of

applicable rules for the different aspects of each task (see Appendix B for the trained rules). BO 

positions were scored on spacing distance between positions, cover, and concealment available at

those positions, and visual overlapping fields of fire between the current and previously marked 

position. The BO score was then summed (over each category) and averaged (over categories) to 

generate a rating for each participant. LZ positions (for a helicopter) were scored on clearance 

from obstacles, proximity to village center, and the correctness of size and shape. These ratings 

were then summed and averaged to generate a rating for each participant. OFP positions were 

scored based on cover and concealment, coverage of the LZ, and visual overlapping fields of fire 

between the positions. The scores were summed and averaged over positions to generate a rating 

for each participant on the OFP task. Final performance scores were determined by averaging 

three Experimenter/Subject Matter Experts (SMEs) ratings on how well the rules for task 

performance on these described characteristics were followed for each task. Inter-rater reliability 

(α) for the ratings on the test (fourth) mission ranged from a high of .96 to a low of .63 for these 

ratings, which was deemed acceptable. 

Missing Data. Because the performance of the tasks within the missions was not held to a 

performance timeline (although the mission itself was limited to twelve minutes), there were a 

few instances in which LZ tasks were not completed, and a larger number in which the OFP task 

(typically performed last in the sequence) was not performed. Two participants in the first 

mission and two in the third mission did not sufficiently complete the LZ phase for judging. 

These missing data were replaced using the mean of the condition for each of those participants 

for those missions. The condition mean was used as replacement data in order to minimize the 

possible distortion of the results for the experimental conditions, while maintaining requirements 

for the analysis of variance. Condition or cell means were not considered to bias the analysis of 

the individual factors, maintained adequate representation in each group, and were considered to 

be more representative than factor means or the mean for the mission (over all factors). 

For the OFP task, five participants did not perform the task at all on the first mission, seven 

did not perform that task on the second mission, three did not perform the task on the third 

mission, and two did not perform the task in the fourth mission. One participant did not perform

25 
the task during any of the four missions. Because this was a large amount of missing data, and 

replacement using overall experimental means would have precluded reasonable analyses, no 

analyses were performed on the OFP task data. 

Data Normality. Explorations of the BO and LZ rating data were conducted to check for 

normality and possible confounding. The BO and LZ performance ratings were found to be nonnormal over almost all missions in that all of the data were negatively skewed and a majority of 

the mission data also had large kurtosis (using the Kolmogorov-Smirnov test in SPSS vs.12, as 

well as the standard z transformation of the skewness and kurtosis values for the data). The 

recommendation by Tabachnick and Fidell (1996) for data sets with these characteristics is to 

either adjust outliers that are driving the non-normality (deemed an unacceptable approach with 

these data), transform the entire data set to reduce or eliminate the non-normal characteristics, or 

both. The recommended sequence for adjustment for transformation of such data sets is to first 

inspect the data for potential intervening factors that could be contributing to the outlying data 

points, then transform the data based upon the type and amount of non-normal characteristics 

(skewness and kurtosis), then inspect for and correct as necessary any remaining outliers 

(Tabachnick & Fidell, 1996). The process followed with this data set is described in the 

following paragraphs. 

As noted earlier in connection with the questionnaire data analysis, there were no 

significant relationships found between that data and the experiment performance data that 

would account for the skewed and kurtosed distributions. Given the relatively consistent shape 

of the distributions of the performance data (negatively skewed and positively kurtosed), and the 

fact that outliers did not seem to be “driving” the distribution of the conditions, the performance 

rating data for BO and LZ missions were transformed using the sequence recommended by 

Tabachnick and Fidell (1996). The data were reflected (each value was subtracted from the 

maximum possible value plus 1.0) in order to make the data positively skewed (with a minimum 

value of 1.0). The reflected data were then logarithmically transformed, to reduce both the 

skewness and kurtosis. This reduced the non-normal characteristics for the BO and LZ 

performance data, leaving only the data for mission 4 with any significant kurtosis in any 

condition. The transformation did leave some outliers remaining in the LZ data, but it seemed 

more reasonable to leave these cases untouched as there were no data-driven rationales for 

adjusting them, and the distributions were no longer significantly different from normal at the 

overall mission level (with the exception mentioned above). 

Following the reflection and transformation of the data, another outlier inspection was 

performed. The questionnaire data was used to check for possible intervening factors that might 

be contributing to any remaining outliers. An inspection of likely candidate factors found no 

correlations between presence or immersive tendencies and the performance ratings for the two 

tasks. Many of the other factors of interest were not suitable for examination through 

correlation, as high percentages of the data were anchored with zero occurrences or ratings. A 

visual examination of that data found that for two relatively consistent outlier cases in the LZ 

ratings, consistent but high simulator sickness ratings had been collected, although the levels 

never reached the criteria used for dismissal from the experiment. These outliers were adjusted 

on the basis of possible negative effects from simulator sickness on their performance in the 

experiment. The adjustment technique adopted was as follows: using the original data, the 

26 
highest outlier value for these cases was changed to one interval less than the next score (closer 

to the mean for that cell). For example, a case with an outlier value of 8 was changed to 12.5, 

with the next higher value in the distribution being 13.5. Following this adjustment, the data 

were again reflected and logarithmically transformed for analysis. This technique is suggested in 

order to maintain the greatest amount of information in the data while decreasing the 

confounding influence of that case on the distribution of the data (Tabachnick & Fidell, 1996). 

Experiment Data Analyses 

In order to test for differences between the experimental factors over multiple missions, a 

repeated measures General Linear Model (i.e. MANOVA, using SPSS vs.12) analyses was 

conducted for each task over the four repeated missions with IC (2 levels) and AD Arrow (2 

levels) factors. Separate analyses were performed for each task using the performance ratings 

(transformed as described above), performance times, time spent inspecting buildings, and 

number of buildings inspected. Analyses were performed separately on the tasks and measures 

because there was no reason to combine the BO and LZ tasks for analysis purposes, nor was 

there a relationship between performance ratings, performance time, time spent inspecting 

buildings, or number of buildings inspected. In fact, combining some of these dependent 

variables would have violated independence, as the time measures are related. The significant 

results of these several analyses are therefore presented individually, and non-significant results 

are not reported in detail at all. 

Bounding Overwatch. The BO Performance transformed data still evidenced homogeneity 

problems between conditions (indicated by a significant Mauchly’s W (W(5)=.520; p=.004), and 

therefore the Huynh-Feldt adjusted degrees of freedom (provided by SPSS vs.13) were used to 

compensate for the non-homogeneity. This analysis for BO performance found significant 

differences over Missions (F(2.444, 68.431)=13.273; p<.001; Power = .999), Missions X Coaching 

(F(2.444, 68.431)=3.806 p=.020, Power = .735) and Missions X AD Arrow (F(2.444, 68.431)=2.923; 

p=.050, Power = .610). The means for these data over missions are presented in Table 3. 

The univariate analysis of the between subjects conditions for Coaching and AD Arrow 

were not significant using the transformed data averaged over the four missions. The means over 

missions and conditions for both the original and the transformed BO performance ratings are 

presented in Table 3. (Note that for the transformed data, lower values reflect better 

performance. It should also be noted that outlier corrections were not used in calculating the 

untransformed means. Finally, although all four missions were used in the analyses, and are 

presented, the reader should remember that no treatment was applied to any group during the 

fourth mission.) 

As noted above, a separate analysis was performed using the BO Performance Time data 

(not transformed, see the bottom entries in Table 3), with a significant result for Missions (F(3, 

84)=5.017 p=.003, Power = .904). The instructional factors did not significantly interact with 

overall Mission times, nor were they significant for BO Performance Time when averaged over 

all four missions. Separate but similar analyses of the Number of Buildings Inspection and Time

Spent Inspecting Buildings measures revealed no significant effects, and these analyses and data 

are therefore not presented. 

27 
Table 3. 

Actual and Transformed Mean Bounding Overwatch Performance Ratings & Times over 

Missions & Conditions

Missions One Two Three Four

BO Performance Ratings * 

Transformed Data 

30.6507

.6712

32.3640

.5864

33.8477 

.4421 

34.2201

.3764

Coaching Condition* 

Transformed Data

30.2651

.7208

33.0874

.5014

33.7949 

.4307 

34.9659

.2684

No Coaching Condition*

Transformed Data

31.0363

.6217

31.6407

.6714

33.9005 

.4535 

33.4744

.4844

Attention-Direction Condition* 

Transformed Data

30.1361

.6776

32.7985

.5355

33.0342 

.5435 

34.0231

.4033

No Attention-Direction Condition* 

Transformed Data

31.1653

.6649

31.9296

.6373

34.6613 

.3407 

34.4172

.3495

BO Performance Times (seconds) 294.6250 272.9688 259.0000 224.8125

*Non-transformed data without outlier adjustment are presented for clarity of presentation. 

Landing Zone. A repeated measures analysis of the transformed LZ performance data 

found significant differences over the repeated Missions (F(3,84)=5.479, p=.002, Power=.929). 

No other tests were significant for the within subjects analysis. The actual and transformed 

means for the significant factors are presented in Table 4. 

The between subjects test, with transformed data averaged over missions, found the AD

Arrow significant (F(1,28)=5.794, p=.023, Power=.642). In this situation, AD Arrow mean was 

.979 (SE=.033) and No-Arrow Mean was .868 (SE=.033), indicating that not using the AD 

Arrow provided generally better performance on the LZ task. LZ performance times were also 

significantly different over Missions (F(3, 84)=8.774, P<.001, Power=.993), and significantly 

different over Mission X Coaching (F(3,84)=2.826, p=.044, Power=.660; see Table 4). 

In the (averaged over missions) univariate analysis of performance times between subjects 

effects Interrogative Coaching was again significant (F(1,26)=4.654, p=.04, Power=.549; see Table 

5, overall times). Separate analyses of the observation and time-based measures did not find 

significant effects, and these data are therefore not presented.

Test Mission Analyses. Separate ANOVAs were also conducted on the Performance Time, 

Building Inspection Time, Number of Buildings Inspected, and transformed Performance scores 

for both the BO and LZ tasks for only the fourth (test) mission. The preliminary tests for the 

univariate analysis found that the group variances were unequal for the transformed BO 

Performance data (Levene’s F(3,28)=4.39; p<.012) but not for any other data. Only the 

28 
transformed BO Performance for Interrogative Coaching was found to be significant in the 

General Linear Model univariate analysis (F(1,28)=7.099, p=.013, power=.730). A non-parametric 

analysis of the transformed performance score for the final mission BO phase also found that it 

differed significantly over the Coaching condition (Mann-Whitney U=66.0, p=.019; 

Interrogative Coaching Mrank=12.63, No-Coaching Mrank=20.38; note that as the data were 

reflected and log transformed the lower rank represents better performance). 

Table 4. 

Actual and Transformed Mean Landing Zone Performance Ratings & Times over Missions & 

Conditions 

Missions One Two Three Four

LZ Performance Ratings 25.3307 26.4888 28.50 28.6016

Transformed Data* 1.0005 .9580 .8863 .8483

LZ Times (overall) 268.8393 216.0089 223.6250 179.0938

Coach Condition 

Overall Mission Mean=246.862

310.6786 229.3304 266.9375 180.5000

No Coach Condition 

Overall Mission Mean=196.922

227.0000 202.6875 180.3125 177.6875

*Note: Outliers were adjusted before data transformation. 

Discussion 

The primary goal of this research was to assess the benefits of two VE-based instructional 

features: oral interrogative coaching and a simple, direction-indicating arrow (which pointed at 

cues critically related to performed errors). Coaching, whether in interrogative or directive form, 

serves to provide feedback about performance as well as information and/or cues about improved 

or correct performance. In a similar fashion, if an AD Arrow points at some stimulus in the 

environment, the presumption is that some aspect of performance is less than optimal and the 

indicated stimulus is related to better performance (as noted in the protocol instructions). 

Obviously, when Coaching is combined with some AD device, the presumption is that more 

information has been provided about how to improve performance. In this way, the coaching 

intervention frames the Attention-Direction indicated stimulus and an interaction effect 

(improvement over the presentation of either intervention alone) might be expected. 

The significant improvements in both performance and the time to perform found for both 

the BO and LZ phases over the repeated Missions provide adequate evidence of learning. The 

basic patterns for the time to perform the tasks is relatively clear, with steadily decreasing 

amounts of time required for steadily improving (overall) performance. The conclusion that 

participants learned over the repeated missions is not startling, but does provide a solid basis for 

interpreting the results of interest. 

29 
Increased support for the efficacy of an Instructional Feature would be provided if that 

treatment demonstrated a more rapid learning than the “control” condition of no instructional 

intervention, as would be indicated by a Mission by IF interaction. The significant interaction 

with BO data (transformed) for Missions with Coaching seems to indicate that questions leading 

to or reminding participants about the rules that guide performance, administered during the 

training exercises supports improved performance (see Table 3), which is inferred as 

representing improved learning for those receiving that intervention. The data pattern indicates 

that Coached performance started off slightly worse than non-Coached, but steadily improved 

(decreasing transformed values). This pattern seems to provide an understandable explanation 

for the results of this analysis (note that no post hoc analysis of the repeated measures and 

between-subjects factor can be performed). 

There was also a significant interaction in the BO data for Missions with the AD Arrow, 

and these transformed data also have an understandable pattern of results (see Table 3). There is 

general improvement over the missions, but in this situation NOT providing an AD Arrow 

intervention provides better improvement. So for some reason, pointing out stimuli relevant to 

the application of the task performance rules did not improve performance. Given the 

experimental structure, half of the group shown an AD Arrow did receive coaching, as did half 

of those not shown an AD Arrow. The conclusion must be that the AD intervention did not 

provide any extra clarifying information that could support skill acquisition, and actually delayed 

performance improvements over the missions. 

Significant interactions were not found in the transformed LZ performance data, but were 

found with the time to perform the LZ task. The transformed LZ performance data did find 

performance improvements over Missions (see Table 4), indicating that participants did learn to 

perform the task better over trials. The analysis of Performance Time data found significant 

improvement over Missions, and a significant interaction with the Interrogative Coaching 

intervention (see Table 4). The non-Coached group performed more quickly overall than the 

Coached participants, although the Interrogatively Coached participants performed almost as 

rapidly in the fourth or Test Mission. 

The Mission 4 (the test mission) analysis investigated any outcomes from the different 

conditions found during a non-interventional (no IF applied) final trial. This trial was expected 

to show the final level of skill acquired from the preceding mission interventions and AAR 

sessions. The only finding in this data was a significant effect on BO performance from the 

Coaching intervention (both with the ANOVA and the confirmatory Non-parametric test on that 

specific condition). 

Overall, these results seem to be consistent with Bailey’s (1982, p. 326) conclusion that 

“speech may be preferred when the message calls for immediate action, vision is already 

overburdened, or the job requires the user to move about continually.” The speech in this case, 

while interrupting the task activity, was also providing some learning benefit for subsequent 

performances. The visual system was already overloaded with task activities, hence the lack of 

significant improvement from the addition of an attention direction mechanism. However, this 

conclusion is weakened since these factors were characteristics of the LZ task as well. Not 

finding this difference in the LZ task, prevents the generation of clearly supported guidance. 

30 
Further analysis of the tasks might provide insight into the learning pattern found in this 

research. However, that analysis is probably not an efficient approach because training tasks 

through the use of coaching and attention direction mechanisms like these is seldom restricted to 

the experimental controls necessarily used in this research. Normal BO and LZ training would 

encompass more classroom instruction and a wider variety of field practice, which would not be 

limited to a single coaching or attention-direction intervention during each exercise. 

An inspection of the data (see Tables 3 & 4, above) reveals that more time was expended 

during the training missions on the BO task, relative to the LZ task (see above comments about 

non-performance of the OFP task, which also provides support for the following argument). The 

difference in time to perform could support the argument that improvement in the BO task was 

not possible in the time available for the LZ task. This would also mean that results similar to 

those found with the BO task might be expected for the LZ task if comparable time for rule 

applications and cognitive effort were expended in the training exercises. 

The possibility also exists that the application of the AD Arrow IF was not well understood 

by some participants, regardless of the repeated instructions (participants were reminded of the 

experimental interventions and intent prior to each trial for that specific condition). In the AD 

Arrow only condition, the directional stimulus would point toward the environmental 

configuration that should have changed the decision-based performance (as the stimulus was 

only implemented following an error). It may have been that although the participants looked in 

the indicated direction as instructed, that the additional visual inspection during their tasks may 

not have supported the desired change in understanding (an implicit feedback situation). In this 

situation, better performance could have been based on the un-interrupted application of those 

rules that were understood. So the AD Arrow, alone or combined with Interrogative Coaching, 

could have interfered just enough with LZ performance that the non-AD Arrow condition (which 

combined both non-coached and coached without presentation of an AD arrow) evidenced 

enough improved performance to preclude finding any significant difference. (This is not an 

argument that the conditions were equivalent based upon finding no significant differences, but a 

possible set of conditions that lead to not finding significant differences.) 

Finally, and in retrospect, it seemed that the time required for the question-based coaching 

or AD Arrow interventions during the missions could have actively taken time and cognitive 

resources away from those trainees, relative to others in the other conditions. This would have 

left those with the highest level of intervention (both IFs) with less time and cognitive resources 

for actually learning to perform the task, although they presumably could have had improved 

mental models of the required task performance parameters. Some minimal support for this 

interpretation may be derived from the BO phase analysis results, in that overall the individual 

factors supported significant improvement, while the combined factors did not. 

These initial results from this line of research indicate that injecting more information into 

the virtual training environment does not necessarily translate into better learning performance 

for all tasks. The results do indicate an interrogative-style coaching instructional feature during 

the VE mission can be beneficial in the initial acquisition of an important environmentevaluation based decision activity. What is not clear is whether that form of coaching 

31 
intervention is best for initial skill acquisition on different types of tasks. Obviously, care must 

be taken in using any interventional instructional feature or performance decrements may result. 

The results of this and future research (see below) will be used to provide guidelines for the 

implementation and use of instructional features in simulation-based training. 

Further Research 

A second instructional feature experiment has been conducted to explore and extend the 

findings of this first experiment. The design and variables were changed as little as possible 

from the original experiment in order to leverage insights and resources across experiments. 

However, the following presentation and methodological changes have been made. 

1. An additional independent variable has been added to the experimental design - a text

window with a coaching feedback message that matches the auditory coaching intervention. 

Like the other instructional features, the text window appears (under control of the experimenter) 

in the FOV immediately after the first error of each mission phase. Similar approaches have 

proven successful in simulator-based flight training (Skitka, Mosier, & Burdick, 1999; Loftin, 

Wang, Baffes, & Hua, 1992). 

2. The form of the auditory intervention to participants has been changed from an 

interrogative question-based to a specific directive-based intervention. The coaching has the 

following format: (a) you have done something wrong; (b) here are the parameters on how to do 

it right, and; (c) do the task activity again. 

3. Strict time limits were enforced for each of the three mission phases or tasks. This was 

done to ensure that time pressure was applied equivalently for each of the three tasks, and that 

each of the three tasks was addressed by every participant. 

4. The experimenter-standardized auditory directive coaching was recorded and has been 

used in a more automated play-back manner (again, under control of the experimenter) in order 

to mitigate any experimenter time or error-based confounding during the instructional 

interventions. The same directives are presented in the text window coaching condition (see 

above) and during the AAR, in response to similar critical errors. 

5. The length and format of the AD Arrow IF stimulus was changed from a short arrow in 

front of the participant to three converging lines that extend all the way to the error location or

important stimuli, as marked by the experimenter. 

6. On the experimenter’s scenario control monitor, unseen by the participant, during the 

VE mission, 5- and 50-meter circles surround participant-made marks for BO, and OFP 

positions. LZ positions will be surrounded by 7- and 50-meter circles indicating the unique 

distances for the proper LZ shape. These marks were shown to the participant during the postmission AAR’s. Using these circles was intended to provide several advantages. First, it 

enabled the experimenter to make better judgments of distance-related errors and therefore 

support better intervention timing during the mission exercise. Second, it provided more precise 

visual feedback to participants during the post-mission exercise AAR’s, which was intended to 

32 
enable better understanding of both the feedback and the distances involved. Third, it enabled 

more precise mission scoring for distance-related errors following the data collection efforts. 

33 
34 
References 

Alluisi, E. A. (1991). The Development of Technology for Collective Training: SIMNET, a Case 

History. Human Factors, 33, 343-362. 

American Psychological Association. (2001). Publication manual of the American psychological 

association (5th Edition). Washington, DC: American Psychological Association. 

Appendix C: 387-396. 

Ayres, A., Hays, R. T., Singer, M. J., & Heinicke, M. (1984). An Annotated Bibliography of 

Abstracts on the Use of Simulators in Technical Training (Rep. No. Research Product 

8421, ADA 156 792). Alexandria, VA: U. S. Army Research Institute for the Behavioral 

and Social Sciences. 

Bailey, R.W., (1982). Human performance engineering. Englewood Cliffs, NJ: Prentice-Hall. 

Banta, H. G., Troillet, D. B., Heffernan, N. T., Plamondon, B., & Beal, S. A. (December, 2004). 

The Virtual Observer/Controller (VOC): Automated Intelligent Coaching in Dismounted 

Warrior Simulations. (Research Note 2005-01, ADA 429445). Alexandria, VA: U. S. 

Army Research Institute for the Behavioral and Social Sciences. 

Bell, H. H., & Waag, W. L. (1998). Evaluating the effectiveness of flight simulators for training 

combat skills: A review. International Journal of Aviation Psychology, 8(3), 223-242. 

Bliss, J. P., Tidwell, P. D., & Guest, M. A., (1997). The effectiveness of virtual reality for 

administering spatial navigation training to firefighters. Presence, 6, 73-86. 

Boldovici, J. A. (1992). Toward a theory of adaptive training (Technical Report 959, ADA 

254903). Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social 

Sciences. 

Capability Development Document for Ground Soldier Systems (Draft). (July 2004). Training 

and Doctrine Command, U.S. Army. 

Cowen, M. B. (1993). Designing an instructional simulation for a program entry panel. 

Simulation & Gaming, 24(4), 500-506. 

Field Manual 7-8: Infantry Rifle Platoon and Squad. (2001). Washington, D.C.: Headquarters, 

U. S. Army.

TRADOC Pamphlet 525-66. Force Operating Capability. (July 2005). Training and Doctrine 

Command, U. S. Army. 

Fisk, A. D., Ackerman, P. L., & Schneider, W. (1987). Automatic and controlled processing 

theory and it s applications to human factors problems. In P. A. Hancock (Ed.), Human 

factors psychology (pp. 159-197. North-Holland: Elsevier Science Publishers B. V. 

Fisk, A. D., Hodge, K. A., Lee, M. D., & Rogers, W. A. (1990). Automatic information 

processing and high performance skills: Acquisition, transfer, and retention. US AFHRL 

Technical Report AL/HRPP, US (Tech Rpt 89-69). 

Hays, R. T. (2001). Theoretical foundation for advanced distributed learning research (TR2001-006). Orlando, FL: Naval Air Warfare Center Training Systems Division. 

Hays, R. T., & Singer, M. J. (1989). Simulation fidelity in Training System Design: Bridging the 

Gap Between Reality and Training. New York: Springer-Verlag. 

Hays, R. T. & Vincenzi, D. A. (2000). Fleet assessments of a virtual reality training system. 

Military Psychology, 12(3), 161-186. 

Horton, R. A. Jr. (1994, November). X-ray diffraction as an instructional tool at all levels of the 

geology curriculum. Journal of Geological Education, 42, 452-454.

35 
Jacobs, R. S., Crooks, W. H., Crooks, J. R., Colburn, E., Fraser, II, R. E., Gorman, P. F., 

Madden, J. L., Furness, III, T. A., & Tice, S. E., (October, 1994). Training Dismounted 

Soldiers in Virtual Environments: Task and Research Requirements. (TR 1011, ADA 

286311). Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social 

Sciences. 

Jonassen, D. H. Grabinger, R. S., & Harris, N. D. C. (1990). Analyzing instructional strategies 

and tactics. Performance and Instruction Quarterly, 3(2), 29-45. 

Jonassen, D. H., & Grabowski, B. L. (1993). Handbook of individual differences, learning, & 

instruction. Hillsdale, NJ, Lawrence Erlbaum Associates. 

Jonassen, D. H. & Tessmer, M. (1996). An outcomes-based taxonomy for instructional systems 

design, evaluation, and research. Training Research Journal, 2, 11-46. 

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). A simulator sickness 

questionnaire (SSQ): A new method for quantifying simulator sickness. International 

Journal of Aviation Psychology, 3(3), 203-220. 

Kirkley, J. R., Kirkley, S. E., Myers, T. E., Lindsay, N., & Singer, M. J. (December, 2003). 

Problem-based embedded training: An instructional methodology for embedded training 

using mixed and virtual reality technologies. Proceedings of the 2003 

Interservice/Industry Training Systems and Education Conference. Arlington, VA: 

National Training Systems Association. 

Knerr, B. W., Lampton, D. R., Martin, G. A., Washburn, D. A., & Cope, D. (December, 2002). 

Developing an After Action Review System for Virtual Dismounted Infantry Simulations.

Proceedings of the 2002 Interservice/Industry Training Systems and Education 

Conference. Arlington, VA: National Training Systems Association. Pp. 1112-1121. 

Knerr, B. W., Lampton, D. R., Singer, M. J., Witmer, B. G., Goldberg, S. L., Parsons, K. J., & 

Parsons, J. (November, 1998). Virtual Environments for Dismounted Soldier Training 

and Performance: Results, Recommendations, and Issues. (TR 1089, ADA 360109). 

Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social Sciences. 

Knerr, B. W., Lampton, D. R., Thomas, M., Comer, B. D., Grosse, J. R., Centric, J. H., 

Blankenbeckler, P., Dlubac, M., Wampler, R. L., Siddon, D., Garfield, K. A., Martin, G. 

A., & Washburn, D. A. (September, 2003). Virtual Environments for Dismounted Soldier 

Simulation, Training, and Mission Rehearsal: Results of the FY 2002 Culminating Event.

(TR 1038, ADA 417360). Alexandria, VA: U. S. Army Research Institute for the 

Behavioral and Social Sciences. 

Lewandowsky, S., Dunn, J. C., Kirsner, K., & Randell, M. (1997). Expertise in the management 

of bushfires: Training and decision support. Australian Psychologist, 32(3), 172-177. 

Lintern, G., Roscoe, S. N., Koonce, J. M., & Segal, L. D. (1990). Transfer of landing skills in 

beginning flight simulation. Human Factors, 32, 319-327. 

Loftin, R. B., Wang, L., Baffes, P., & Hua, G. (1992). An intelligent system for training space 

shuttle flight controllers in satellite deployment procedures. In M. J. Farr & J. Psotka 

(Eds.), Intelligent instruction by computer: Theory and practice (pp. 251-261). 

Washington, DC: Taylor & Francis. 

Logan, G. D. (1988). Automaticity, resources, and memory: Theoretical controversies and 

practical implications. Human Factors, 30, 583-598. 

Logan, G. D., & Klapp, S. T. (1991). Automatizing alphabet arithmetic: I. Is extended practice 

necessary to produce automaticity? Journal of Experimental Psychology: Learning, 

Memory, & Cognition, 17(2), 179-195.

36 
Loube, J. (2001). Embedded training: Fact or fantasy. MS&T (Military Simulation & 

Technology: The International Defence Training Journal), Issue 2, 24-27. 

National Research Council, Committee on Human Factors, Panel on Human Factors in the 

Design of Tactical Display Systems, (1997). Tactical display for soldiers: Human factors 

considerations. Washington, DC: National Academy Press. 

Magee, L. E. (1997). Virtual reality simulator (VRS) for training ship handling skills. In R. J. 

Seidel & P. Chatelier (Eds.), Virtual reality. Training’s future? Perspectives on virtual 

reality and related emerging technologies (19-29). New York: Plenum Press. 

Mayo, M., Singer, M. J., & Kusumoto, L. (December, 2005). Massively Multi-Player (MMP) 

Environments for Asymmetric Warfare. Journal of Defense Modeling and Simulation. 

Arlington, VA: National Training Systems Association. 

McCollum, C., Deaton, J., Barba, C., Santerelli, T., Singer, M. J., & Knerr, B. W. (December, 

2004). Developing an immersive, cultural training system. Proceedings of the 2004 

Interservice/Industry Training Systems and Education Conference. Arlington, VA: 

National Training Systems Association. 

Milgram, P., Takemura, H., Utsumi, A., Kishino, F. (1994). Augmented reality: A class of 

displays on the reality-virtuality continuum. SPIE Vol. 2351, Telemanipulator and 

Telepresence Technologies, 1994. 

Palmeri, T. J. (1997). Exemplar similarity and the development of automaticity. Journal of 

Experimental Psychology: Learning, Memory, & Cognition, 23(2), 324-354. 

Park, O-C., & Gittelman, S. S. (1995). Dynamic characteristics of mental models and dynamic 

visual displays. Instructional Science, 23(5-6), 303-320. 

Rose, F. D., Attree, E. A., Brooks, B. M., Parslow, D. M., Penn, P. R., & Ambihaipahan, N. 

(2000). Training in virtual environments: Transfer to real world tasks and equivalence to 

real task training. Ergonomics, 43, 494-511. 

Savelsbergh, E. R., de Jong, T., & Ferguson-Hessler, M.G.M. (2000). Learning physics with a 

computer algebra system. Journal of Computer Assisted Learning, 16(3), 229-242. 

Skitka, L., Mosier, K., & Burdick, M. D. (2000). Accountability and automation bias. 

International Journal of Human-Computer Studies, 52(4), 701-717. 

Singer, M. J., Comer, B., & Grosse, J., (October, 2004). Asymmetric Warfare Virtual Training 

Technology: Formative Evaluation Shaped Development. NATO Special Workshop on 

Exploiting Commercial Games for Military Use, TNO, The Hague, Netherlands. 

Singer, M. J., Grant, S., Commarford, P. M., Kring, & Zavod, M. (2001). Team performance in 

distributed virtual environments (ARI Technical Report 1118, ADA 396489). 

Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social Sciences. 

Stansfield, S., Shawver, D., Sobel, A., Prasad, M., & Tapia, L. (2000). Design and 

implementation of a virtual reality system and its application to training medical first 

responders. Presence, 9(6), 524-556. 

Sticha, P. J., Singer, M. J., Blacksten, H. R., Morrison, J. E., & Cross, K. D. (1990). Research 

and methods for simulation design: State of the art (Rep. No. Technical Report 914, 

ADA 230 076). Alexandria, VA: U. S. Army Research Institute for the Behavioral and 

Social Sciences. 

Swezey, R. W., & Andrews, D. H. (Eds.). (2001). Readings in Training and Simulation: A 30-

Year Perspective. Santa Monica, CA: Human Factors & Ergonomics Society.

37 
Tabachnick, B. G., & Fidell, L. S. (1996). Using Multivariate Statistics (4th ed.). Boston, MA: 

Allyn & Bacon. 

Todorov, E., Shadmehr, R., & Bizzi, E. (1997). Augmented feedback presented in a virtual 

environment accelerates learning of a difficult motor task. Journal of Motor Behavior, 

29(2), 147-158. 

U. S. Army Research Institute for the Behavioral and Social Sciences. (n.d.). ARI Mission 

Statement. Retrieved December 22, 2005, from

http://www.hqda.army.mil/ari/about/mission.shtml

U. S. Army Research Institute for the Behavioral & Social Sciences. (2003). Policies and 

Procedures: Use of human subjects in research. Alexandria, VA: U. S. Army Research 

Institute for the Behavioral and Social Sciences.

U. S. Army Research Institute for the Behavioral & Social Sciences. (2004). Fiscal Year 2004 

Program. Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social 

Sciences. 

Williges, B. H., Roscoe, S. N., & Williges, R. C. (1973). Synthetic Flight Training Revisited. 

Human Factors. 15, 543-560. 

Wisher, R. A., Macpherson, D. H., Abramson, L. J., Thornton, D. M., & Dees, J. J. (2001). The 

Virtual Sand Table: Intelligent tutoring for field artillery training (ARI Research Report 

1768, ADA 388158). Alexandria, VA: U. S. Army Research Institute for the Behavioral 

and Social Sciences. 

Witmer, B. G., & Singer, M. J. (June, 1998). Measuring presence in virtual environments: A 

presence questionnaire, Presence. 7(3), 225-240. 

38 
Appendix A 

Biographical Questionnaire 
Research Participant Information Questionnaire ID ____________ 

Please fill in the blank or circle the appropriate response.

1. What is your age? _____ years 

2. What is your gender? female male 

3. Are you currently in your usual state of good fitness? yes no 

4. How many hours sleep did you get last night? _____ hours 

4a. Was it sufficient? yes no 

5. Indicate all medications/substances you have used in the past 24 hours: 

 CIRCLE ALL THAT APPLY

0 - none 

1 - sedatives or tranquilzers 

2 - aspirin, tylenol, other analgesics 

3 - anti-histamines 

4 - decongestants 

5 - other prescription drugs 

6 - other medications (please list: ___________________________________) 

6. Have you ever experienced motion or car sickness? yes no 

7. How susceptible to motion or car sickness do you feel you are?

 0 1 2 3 4 5 6 7 

 not very average very 

susceptible mildly highly 

8. Do you have a good sense of direction? yes no 

9. How many hours per week do you use computers? _____ hours per week 

10. My level of confidence in using computers is 

 1 2 3 4 5 

low average high 

11. I enjoy playing video games (home or arcade). 

 1 2 3 4 5 

disagree unsure agree 

A-1 
12. I am _____ at playing video games. 

 1 2 3 4 5 

bad average good 

13. How many hours per week do you play video games? _____ 

14. How many times in the last year have you experienced a virtual reality game or 

entertainment? 

 0 1 2 3 4 5 6 7 8 9 10 11 12+ 

15. Do you have a history of epilepsy or seizures? yes no 

16. Do you have normal or corrected to normal 20/20 vision? yes no 

17. Are you color blind? yes no 

A-2 
Appendix B 

Task Training Manual 
INTRODUCTION 

As described in the experiment briefing and demonstration, you will complete a 

series of planning exercises modeled after US Army planning operations in urban (citylike) environments. During each exercise, you will mark a plan on a map and then move 

through a virtual enviornment (VE) representing the different towns. The exercise 

requires marking the location for a helicopter landing zone (LZ), planning movement of a 

platoon to that LZ, and identifying locations for protecting the landing zone, called 

observation/fire posts (OFP). In essence, you are planning as if you would lead a platoon 

to execute the mission of establishing an LZ. 

This manual presents the basics of military operations in urban terrain (MOUT) 

necessary to correctly plan for LZ placement, use a map in planning, and the three types 

of marks you will use in the exercises. In the first section, movement through MOUT and 

LZ requirements are explained, including how to mark locations in this research. In the 

second section, you will learn the basics of how to read military maps, including contour 

lines and color codes. After training you will be given a multiple choice test on this 

information. Then you will practice planning movements and marking locations on a 

terrain map. You must learn and demonstrate the basic information in this manual in 

order to participate in the experiment. 

Military Operations in Urban Terrain 

Built-up or urban areas consist mainly of man-made features such as buildings and 

roads. Buildings provide cover and concealment, limit fields of observation and fire, and 

block the movement of troops. Streets are usually avenues of approach. However, forces 

moving along streets are threatened by the buildings and have little space for cover. 

Land navigation is important. There are four steps to land navigation. Having an 

objective and the requirement to move there, you must know where you are, how to plan

the route, how to stay on the route, and how to recognize the objective. 

Step 1: Know Where You Are. You must know where you are on the map and on 

the ground at all times and in every possible way. This includes knowing where you are 

relative to: 

• Your orientation 

•

•

•

•

•

The direction and distances to your objective(s) 

Other landmarks and features 

Any impassable terrain, the enemy, civilians, and danger areas 

Both the advantages and disadvantages presented by the terrain between you and 

your objective. 

These steps are accomplished by knowing how to read a map, recognize and 

identify landmarks; determine direction; and measure or estimate distances. 

Step 2: Plan the Route. Depending on the length and type of movement to be 

conducted, several factors should be considered in selecting a good route. These include: 

Time & Distance 

B - 1 
• Tactical aspects of terrain (called OCOKA: described below) 

•

•

Potential for encountering the enemy 

Availability of good checkpoints or identifying landmarks 

The plan must be the result of careful map study and should address the 

requirements of the mission and time available. It must also provide for ease of 

movement and navigation. 

Step 3: Stay on the Route. In order to know that you are still on the correct route, 

you must be able to compare and recognize the environmental landmarks (roads, 

buildings, etc.) you encounter in the VE, as you move according to the plan you 

developed on the map. There are usually two ways to navigate an environment: dead 

reckoning (using a compass) or terrain association (recognizing various landmarks from

the map in their anticipated positions and sequences as you pass them). Because you will 

not have a compass in the VE, you must use landmark or terrain association to navigate 

in these VE exercises. 

Step 4: Recognize the Objective. Your objective in these VE missions is locating 

an LZ for a helicopter. Therefore, recognizing an appropriate location for the LZ is the 

first step in map planning. You will then need to employ terrain association in order to 

recognize landmarks that identify the LZ area when the plan is evaluated in the VE 

exercise. 

In MOUT areas, the ranges of observation and fields of fire are reduced by 

structures. Targets are usually briefly exposed at ranges of 100 meters or less. For this 

reason, you should evaluate all potential enemy threats within 100 meters of yourself at 

all times and make your movement and emplacement plans accordingly. 

Bounding Overwatch 

When moving down a street, the dismounted platoon should move in at least two 

separate squads, with one squad moving forward while the other squad provides cover. 

This process of “leap-frogging” ensures safer movement through an urban terrain. 

Locations where one team should stop while the other moves forward are called 

Bounding Overwatch (BO) positions. Figure 1 provides a diagram of the movement 

pattern. 

B - 2 
Figure 1. “Leapfrog” Movement 

Each BO position should offer cover and concealment for the occupying squad as 

well as open lines of fire to cover the bounding or next forward moving squad. Example 

BO positions include the corners of buildings or beside thick brush or trees. BO positions 

should not be more than 30 to 50 meters apart, as a greater separation would not provide 

the cover necessary for safe movement. 

In some cases, you will need to plan for crossing through relatively open areas, 

such as streets, alleys, and parks. These should be avoided whenever possible as they are 

natural kill zones. If such movements are necessary, however, take extra precautions to 

protect the platoon by planning for the most efficient path (i.e., the fastest) between 

relatively safe areas, such as near buildings. This means selecting good overwatch 

positions that cover short distance movements by the bounding squad. Note that this may 

mean taking a longer route or setting multiple BO positions to an objective in order to 

obtain good overwatch or a safer path. 

Choosing the best path depends on multiple factors related to the natural 

surroundings, obstacles, and potential enemy threats. To assist in this process, you should 

analyze the terrain using the following military-based criteria, referred to by the acronym

OCOKA. 

• Observation and fields of fire, 

•

•

•

•

Cover and concealment, 

Obstacles, 

Key terrain 

Avenues of approach. 

Observation and Fields of Fire. Observation means seeing the enemy but not 

being seen. Anything that can be seen can be hit. Therefore, a field of fire is an area that a 

weapon or a group of weapons can cover effectively with fire from a given position. For 

purposes of this experiment, 100 meters is set as the operational range for weapons. 

B - 3 
Cover and Concealment. Cover is either natural or artificial shelter or protection 

from enemy fire. Always try to use covered routes and seek cover for each halt or 

overwatch position, no matter how brief the halt is. 

Concealment is protection from observation or surveillance, including concealment 

from enemy air observation. When you are moving, concealment is generally secondary 

to cover; therefore, select routes and positions that protect your teams and minimize the 

potential for enemy forces detecting or shooting at you. Also, ensure that your squad can 

place covering fire on potential threats. 

Obstacles. Obstacles are any obstructions that stop, delay, or divert movement. 

Obstacles can be natural (rivers, swamps, cliffs, or mountains) or they may be artificial 

(barbed wire entanglements, pits, or buildings). Always consider any possible obstacles 

along your movement route and, if possible, try to keep obstacles between the enemy and 

yourself (see Cover & Concealment above). 

Key Terrain. Key terrain is any locality or area that when seized or retained offers 

a marked advantage. Within an urban area, higher buildings may dominate an area, 

offering observation and fields of fire. As mentioned before, narrow areas between 

buildings may offer cover or may be killing zones covered by enemy weapons. Moving

through large open areas must also be avoided. You should always attempt to identify 

and use key terrain to your advantage. 

Avenues of Approach. These are access routes. They may be the routes you can 

use to get to the enemy or the routes they can use to get to you. These include how you 

move to your goal. Basically, an identifiable route that approaches a position or location 

is an avenue of approach to that location. In urban terrain, these avenues may be streets 

or open areas that provide relatively easy movement combined with many opportunities 

for cover and concealment.

In the experiment exercises, the BO is marked in a two-step process. First, you 

make a YELLOW circle around the desired BO position. Next, you place a YELLOW 

“X” in the center of the circle. Figure 2 provides an example. You do not need to 

indicate which team uses which position, as depicted in Figure 1. You do need to 

consider all potential threats to the moving squad and position the BO position 

appropriately. In our exercises you will not be allowed to use building interiors (or 

roofs), so all positions will be outside.

B - 4 
Figure 2. Bounding Overwatch Mark 

Landing Zones. Helicopters need a large amount of space, free of debris and 

obstacles, to land safely. Your goal is to locate and mark an appropriate Landing Zone

(LZ) for a single helicopter. Based on your analysis of the map, you will select a space 

for the LZ, and then plan out a path for the platoon to follow when moving to and 

establishing the LZ. The general rule for clearance for an LZ is an open space that is 100 

by 50 meters, or approximately the size of a football field. The landing area and marks 

(see below) should be centered in this open space. 

The LZ should not be on an uneven slope - one that exceeds seven degrees, so the 

topography should be evaluated for that constraint. A rule of thumb is that if the surface 

drops one meter over a distance of 10 meters, it slopes too much. An inspection of the 

map contours or consideration of the “lay of the land” is used to make this decision. 

Marking an LZ in this experiment, as in the Army, follows a specific protocol that 

is shown in Figure 3. Normally the Army marks an LZ using flags, smoke, or lights for 

night operations. In all LZ marking, four circular marks are arranged in a “Y” or “T” 

pattern (see Figure 3) with the long stem of the Y running in the direction of flight a 

helicopter should take when landing and taking off. The two marks at the top of the Y are 

14 meters apart (remember that each full step in the VE is one meter). These marks 

indicate the landing threshold for the aircraft. From the top of the Y to the bottom of the 

stem is 21 meters. The central landing mark, where the helicopter lands, is placed 14 

meters from the top two. The last mark is placed 7 meters further down. The center mark 

should be placed in the center of the clear landing area (the middle of the 100x50m 

field). This means the center mark should be in the middle of the landing area, with 50 

meters of clearance each way in the direction of flight and 25 meters clearance to each 

side. Figure 4 provides an example LZ mark on an enlarged, but correctly colored map. 

B - 5 
Figure 3. Landing Zone Marks 

Figure 4. Example LZ Marks 

Observation/Fire Post (OFP): Once an LZ is located and marked, it needs to be 

protected from enemy fire. Your final task will be to mark Observation/Fire Posts (OFPs) 

around the LZ to indicate where platoon members can set up posts to cover threats or 

approaches to the LZ. You need to set up enough OFPs to provide complete coverage 

around the LZ (see OCOKA, above). Mark an OFP in the same way the other sites are 

marked, using RED to distinguish the OFP from LZ and BO marks. Figures 5 and 6 

provide examples of the marks and positioning for the OFP sites. 

Note that good OFPs can support each other with overlapping fields of fire, that 

means at least two positions can see each other and shoot at the same approaching force. 

They are not positioned so that they are shooting toward each other in order to shoot at 

the approaching forces. 

B - 6 
Figure 5. Observation/Fire Post Mark 

Figure 6. Example Observation/Fire Posts 

Map Reading For Urban Terrain 

Maps provide a scaled representation of environments using symbols and colors. In 

the MOUT context used in this research, the map is a topographical representation that is 

1:5000 scale. This means that one meter on the map corresponds to 5000 meters on the 

actual terrain. On the map this means one centimeter equals 50 meters on the terrain. 

The map's legend contains the scale and symbols used on the map. The legend will also

provide the contour interval (e.g. 10 meters, the difference in height of the terrain 

between each contour line), and information about the markings for roads, buildings, 

powerlines, railroads, etc. 

B - 7 
Topographic maps use special symbols and colors to indicate features of the 

environment. To facilitate the identification of features on a map, topographical and 

cultural information is printed in different colors. The maps you will use in the exercises 

use the following colors and features: 

Black indicates cultural (man-made) features such as roads, surveyed spot

elevations, and all labels. 

Red-Brown are combined to identify cultural features like buildings.

Blue identifies hydrography or water features such as lakes, swamps, rivers, 

and drainage. Solid blue indicates lakes or rivers, dotted or broken lines indicate 

occasional water like flood zones or intermittent streams. 

Green identifies vegetation with military significance, such as woods, 

orchards, and vineyards. Denser green marks indicate denser vegetation. For 

example, dotted green lines might indicate sparse shrubs and trees, whereas solid 

green areas indicate thick forest regions. 

A contour line is an imaginary boundary line that represents elevation, that is, 

vertical distance, above or below sea level. Index contour lines are thicker than others and 

are marked with an elevation. Moving from a smaller to a larger numbered index contour 

line would indicate an increase in elevation. The lines between the index contour lines 

(normally four) are thinner; each represents a change in elevation of 10 meters. Note that 

some variation in elevation does occur between contour lines, i.e., the terrain between 

these lines can vary in height to some degree. 

Contour lines are helpful in showing geographical features of importance to 

dismounted Soldiers. For example, contour lines that are evenly spaced and close 

together indicate a uniform, steep slope of the terrain. Such a high point might be used as 

an observation/fire post. The closer together the contour lines, the steeper the slope; the 

further apart the contour lines, the gentler the slope. Furthermore, the direction of the 

slope is indicated by the index contour lines. The contour lines in Figure 7 indicate a 

slope upward, from 100 meters to 150 meters. 

B - 8 
Figure 7. Example Contour Lines 

The scale in Figure 7 is 1:5000, meaning the 3cm on the map between the 100 and 

150 index contour lines indicates a rise of 50 meters elevation over 150 meters of 

distance. The slope between the 100 and 150 numbers is quite steep, especially where 

the contour lines are close together. Slopes can be interpreted using boundary estimates. 

For example, a rise of 100 meters over a distance of 100 meters would be a 45 degree 

slope (a 1:1 ratio), 50m rise over 100m (a 1:2 ratio) is a 27 degree slope, and a 10m over 

100m (1:10 ratio) is a 6 degree slope. Other slopes can be evaluated by establishing the 

ratio and interpolating using these boundary values. 

Since the LZ has to be in a flat area, an area with a 1:10 ratio or less is required. 

Note that different slopes can appear between contour lines, as they are only discrete 

measures of the terrain, and don’t provide information about any change in elevation until 

the change exceeds the contour interval. On figure 7, an LZ could only be placed in the 

flat area below the 100 contour line, near the road and building at the bottom right of the 

figure. 

B - 9 
Appendix C 

Training Test & Map 
7.1 Test for IF Training Participant #: _________________

Date: _______________________ 

1. What color are observation fire posts?

A. Red B. Green C. Yellow D. White 

2. What color are Bounding Overwatches?

A. Red B. Green C. Yellow D. White 

3. What color are the Landing Zone markers?

A. Red B. Green C. Yellow D. White 

4. What color of paint is used to erase marks? 

A. Red B. Green C. Yellow D. White 

5. When making marks in the environment, you should be aware of potential enemy threats 

within ____ meters of your position. 

A. 10 B. 100 C. 200 D. 1000 

6. Marks used to indicate where one squad should stop to cover another squad are called: 

A. Observation Fire Posts B. Bounding Overwatches 

C. Cover Points D. Landing Zone Posts 

7. On topographical maps, the colors red or brown are used to indicate: 

A. Cultural features like buildings B. Lakes and streams 

C. Roads and paths D. Vegetation (e.g., trees, bushes, etc.) 

8. On topographical maps, the color black is used to indicate: 

A. Cultural features like buildings B. Lakes and streams 

C. Roads and paths D. Vegetation (e.g., trees, bushes, etc.) 

9. On topographical maps, the color green is used to indicate: 

A. Cultural features like buildings B. Lakes and streams 

C. Roads and paths D. Vegetation (e.g., trees, bushes, etc.) 

C - 1 
10. On the figure below, please mark the appropriate distances, in centimeters, for the 

Landing Zone. 

11. What is the correct order, from first to last, of marks you should make during a exercise?

A. Observation Fire Posts, Bounding Overwatches, Landing Zone 

B. Bounding Overwatches, Landing Zone, Observation Fire Posts 

C. Bounding Overwatches, Observation Fire Posts, Landing Zone 

D. Landing Zone, Observation Fire Posts, Bounding Overwatches 

12. What is your primary goal during the exercises?

A. Protect yourself from enemy fire B. Detect enemy locations 

C. Locate and mark a helicopter landing zone D. Shoot all enemy personnel and tanks 

13. The terrain slope ratio for a Landing Zone should not exceed what ratio between a rise in 

elevation and distance? 

A. 1:1 (100 meters riser over 100 meters distance) 

B. 1:2 (50 meters rise over 100 meters distance) 

C. 1:10 (10 meters rise over 100 meters distance) 

D. 1:100 (1 meter rise over 100 meters distance) 

C - 2 
14. In the figure below, which of the following statements is true?

A. Point A is higher than point B 

B. Point B is higher than point A 

C. Point A and B are at the same elevation

D. Cannot tell if point A is higher or lower than point B 

15. The purpose of an Observation/Fire Posts is to 

A. Provide fire coverage to all areas around a Landing Zone 

B. Provide markers for helicopter pilots 

C. Light up the area surrounding a Landing Zone 

D. Indicate where teams should stop while covering a moving team

Participants missing three questions were required to review the Training manual again. 

Any subsequent mistakes were grounds for disqualification from the research. 

C - 3 
Map Planning 

Find an appropriate space for the LZ and plan a path to that area using OCOKA and the 

movement guidelines presented above to select and mark the BO positions. Lay out the marks 

for the LZ, and the OFPs that will control access and protect the area. Mark all positions, 

appropriately. A small ruler will be furnished, along with colored markers. 

LEGEND 1:12500 

C - 4 
Grading Criteria: 

Many paths to the only available LZ (see below) are available. Bounding Overwatch rules 

provided in the training manual should be applied to evaluate the overlapping coverage and 

OCOKA rules applications. 

Given the scale and configuration of the buildings on the map together with the contour, 

there is only one location for a Landing Zone, the top left open area. Given the scale, the 

placement should be sufficient to clear the buildings in that area and large enough to fit the LZ 

parameters. 

Placement of the Observation/Fire Posts should provide direct observation of all 

approaches and buildings, within 100 meters and without overlapping an adjacent post. 

Violations would be explained to the participant. More than three major errors would

require review of the appropriate section of the training manual, and a repetition of the map 

analysis. Subsequent excessive errors resulted in disqualification of the participant. 

C - 5 
Appendix D 

Virtual Environment Maps 
Ft. Polk, Shugart-Gordon MOUT, Scenario 1. 

D - 1 
Ft. Polk, Shugart-Gordon MOUT, Scenario 2. 

D - 2 
Ft. Benning, McKenna MOUT, Scenario 3. 

D - 3 
Ft. Benning, McKenna MOUT, Scenario 4. 

D - 4