Study 

Report 

2010-02 

 

Game-Based Training Effectiveness 

Evaluation in an Operational Setting 

 Krista Langkamer Ratwani 

Aptima, Inc. 

Kara L. Orvis 

Aptima, Inc.

 

Bruce W. Knerr 

 U.S. Army Research Institute 

 

 

 

 United States Army Research Institute 

 forthe Behavioral and Social Sciences 

 

 

 

 September 2010 

 Approved for public release; distribution is unlimited. 
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the Department of the Army 

Deputy Chief of Staff, G1 

Authorized and approved for distribution: 

 BARBARA A. BLACK, Ph.D. MICHELLE SAMS, PhD. 

Research Program Manager Director 

Training and Leader Development 

 

Research accomplished under contract 

for the Department of the Army 

Aptima, Inc. 

Technical review by 

Robert Albright, TRADOC Capability Manager, Gaming 

Jeffrey E. Fite, U.S. Army Research Institute 

NOTICES 

DISTRIBUTION: Primary distribution of this Study Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926. 

FINAL DISPOSITION: This Study Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Study Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents.
i

REPORT DOCUMENTATION PAGE 

1. REPORT DATE (DD-MM-YYYY) 

September 2010 

2. REPORT TYPE 

Final 

3. DATES COVERED (From – To) 

January 2009 – February 2010 

655803 5a. CONTRACT NUMBER 

W74V8H-04-D-0047 DO 0015 

5b. GRANT NUMBER 

5c. PROGRAM ELEMENT NUMBER 

655803

4. AUTHOR(S) 

Krista Langkamer Ratwani, Kara L. Orvis (Aptima, Inc.) 

Bruce W. Knerr (U.S. Army Research Institute)

5d. PROJECT NUMBER 

D730

5e. TASK NUMBER 

381

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Aptima, Inc. 

12 Gill Street, Suite 1400 

Woburn, MA 01801

8. PERFORMING ORGANIZATION REPORT NUMBER 

1500

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social Sciences 

ATTN: DAPE-ARI-IF 

2511 Jefferson Davis Highway 

Arlington, VA 22202-3926 

10. SPONSOR/MONITOR’S ACRONYM(S) 

ARI

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

Study Report 2010-02 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited 

13. SUPPLEMENTARY NOTES 

Contracting Officer’s Representative and Subject Matter POC: Dr. Bruce W. Knerr 

14. ABSTRACT (Maximum 200 words) 

With high operational tempo and increasingly complex operational environments, , the U. S. Army is increasingly using 

game-based training as a lower cost and more time-effective training method for both individual and collective training of 

tactical skills. However, there has been little empirical evidence to demonstrate the effectiveness of game-based training 

and to help leaders make decisions about their use. In response to the need for more evidence regarding the 

effectiveness of game-based training (GBT), an evaluation of training games supported by Virtual Battle Space 2 (VBS2): 

U.S. Army was conducted in operational settings. This report describes the methods, measures, and results of an 

evaluation with 165 Soldiers participating in GBT. Pre- and post-measures were administered that focused on measuring 

training effectiveness through individual level (e.g., task performance) and unit level (e.g., unit effectiveness) outcomes. 

Results demonstrate that, in general, the training influenced both individual (e.g., task performance) and unit level (e.g., 

unit cohesion) outcomes. In addition, situational characteristics (the amount the unit prepared for the training and the level 

of leader involvement during the training) influenced both types of outcomes. 

15. SUBJECT TERMS

Games, Training Evaluation 

 SECURITY CLASSIFICATION OF: 19. LIMITATION OF 

ABSTRACT 

Unlimited 

20. NUMBER OF 

PAGES 

40 

21. NAME OF RESPONSIBLE PERSON 

Ellen Kinzer 

Technical Publication Specialist 

 703-602-8047 

16. REPORT 

Unclassified 

17. ABSTRACT 

Unclassified

18. THIS PAGE 

Unclassified
ii

 
iii

Study Report 2010-02 

Game-based Training Effectiveness Evaluation in an 

Operational Setting 

Krista Langkamer Ratwani 

Aptima, Inc. 

Kara L. Orvis 

Aptima, Inc. 

Bruce W. Knerr 

U.S. Army Research Institute 

Orlando Research Unit 

Stephen L. Goldberg, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

September 2010 

Army Project Number Personnel and Training 

665803D730 Analysis Activities 

Approved for public release; distribution is unlimited.
iv

Acknowledgements 

Many thanks go to the Soldiers at Ft. Hood, TX and Ft. Lewis, WA who participated in our 

study. Also, thank you to the training facilitators at each of these sites for allowing us to collect 

data and to TCM Gaming for their support and interest in the project. Finally, thank you to Kerri 

Conning Chik and Brandon Beltz fo

r helping with data collection and data entry 
v

GAME-BASED TRAINING EFFECTIVENESS EVALUATION IN AN OPERATIONAL 

SETTING 

EXECUTIVE SUMMARY 

Research Requirement: 

 Army trainers are increasingly turning to emerging technologies and innovative training 

methods to exploit low-cost, technology-based solutions to rapidly fill critical training gaps and 

increase the impact and effectiveness of training for our Soldiers in a time of war. Unfortunately, 

the empirical research linking game-based training to training effectiveness is limited and does 

not extend much beyond extolling the motivational benefits of games in training. The Army 

lacks empirical evidence on which to base future decisions regarding the acquisition and use of 

game-based training. Therefore, the U.S. Army Research Institute, under the sponsorship of the 

U.S. Army Training and Doctrine Command (TRADOC) Capability Manager – Gaming (TCM 

Gaming), conducted an analysis to empirically investigate the effectiveness of game-based 

training in Army operational settings. In addition, this analysis also sought to understand why

games may be useful training tools and if there are situational contexts that dictate the extent of 

their effectiveness. The gaming platform Virtual BattleSpace2 (VBS2): U.S. Army was used as 

a method for evaluating the merits of game-based training. 

Procedure: 

 In order to develop an appropriate evaluation protocol, we observed training at Ft. Hood, 

TX and Ft. Lewis, WA. These observations provided information about the flow of training 

using VBS2: U.S. Army and also provided information about the training objectives normally 

targeted during training. The observations at the training sites also highlighted several constraints 

that had to be taken into consideration when developing the research protocol. First, we could 

not interfere with the training. Second, the amount of time that was made available for data 

collection was limited to approximately one-half hour. Third, we could not ask training 

facilitators to evaluate the performance of the unit. Therefore, based on these constraints and the 

other information gathered during the observations, we developed hypotheses and measures. 

Data were collected from 165 Soldiers engaging in training using VBS2: U.S. Army at 

Ft. Hood and Ft. Lewis. Trainees completed pre- and post-training measures during the course of 

their normal training experience. Measures of learning at both the individual level (e.g., 

individual performance on the tasks) and the unit level (e.g., working together) were included as 

measures of effectiveness. In addition, due to the variance in training both within and across 

training locations, measures to understand the context under which training may be effective 

were also administered. Contextual variables considered included the difficulty of the training 

mission, the level of unit preparation prior to training, the level of leader involvement during the 

training, and time spent on the training. 
vi

Findings: 

 Results of the evaluation demonstrate that the training had a positive impact on individual 

outcomes, as well as on unit learning. Specifically, Soldiers reported being more prepared to plan 

tactical convoy operations following the training. In addition, higher levels of Unit Process and 

Unit Cohesion were reported following the training. These unit-level findings illustrate that the 

training impacted how well the Soldiers felt they could work together as a team, as well as their 

attraction to the unit, including their attraction to their tasks and other group members. In 

addition, contextual variables were shown to influence both individual and unit level outcomes. 

Both the level of unit preparation for the training and the level of unit leadership involvement 

during exercise management positively affected training outcomes. Together these results 

illustrate that game-based training (GBT) can be an effective training tool to the degree that the 

training has specific objectives which are focused on during the training. This focus can come 

from preparing for the training ahead of time and from leaders being involved during the training 

to provide focus and feedback. 

Utilization and Dissemination of Findings: 

 A draft report of the results was provided to TMC Gaming 19 March 2010. Results were 

also presented at the Defense GameTech Conference 31 March 2010. They will be used to 

support future decisions regarding 

the acquisition and use of GBT in the U.S. Army.
vii

GAME-BASED TRAINING EFFECTIVENESS EVALUATION IN AN OPERATIONAL 

SETTING 

CONTENTS 

Page 

INTRODUCTION ...........................................................................................................................1 

METHOD ........................................................................................................................................4 

 Site Observations .......................................................................................................................4 

 Data Collection Protocol and Hypothesis Development ...........................................................5 

 Data Collection ..........................................................................................................................9 

RESULTS ........................................................................................................................................9 

 Individual Level Analyses .......................................................................................................10 

 Unit Level Analyses .................................................................................................................14 

DISCUSSION AND CONCLUSIONS .........................................................................................18 

 Analysis Limitations ................................................................................................................19 

REFERENCES ..............................................................................................................................21 

APPENDIX A EVALUATION MEASURES..............................................................................23 

APPENDIX B SKILL PREPAREDNESS ITEMS – MEANS AND STANDARD 

 DEVIATIONS .....................................................................................................34 

LIST OF TABLES 

TABLE 1 INFORMATION ABOUT INDIVIDUAL AND UNIT LEVEL MEASURES ............7 

TABLE 2 T-TEST RESULTS COMPARING PRE – TO – POST-TRAINING OUTCOME 

 VARIABLES ...............................................................................................................15 

LIST OF FIGURES 

FIGURE 1. A VBS2: U.S. ARMY SCREENSHOT ......................................................................2 

FIGURE 2. PREPAREDNESS LEVELS FOR PLANNING A CONVOY OPERATION .........10 

FIGURE 3. IMPACT OF UNIT PREPARATION ON SKILL PREPAREDNESS ....................11 

FIGURE 4. IMPACT OF UNIT PREPARATION ON TASK PERFORMANCE ......................12 

FIGURE 5. IMPACT OF UNIT PREPARATION ON TRAINING PERFORMANCE .............13 
viii

CONTENTS (continued) 

FIGURE 6. IMPACT OF LEADER INVOLVEMENT ON TASK PERFORMANCE ..............13 

FIGURE 7. IMPACT OF LEADER INVOLVEMENT ON TRAINING MOTIVATION .........14 

FIGURE 8. IMPACT OF UNIT PREPARATION ON UNIT PROCESS ..................................16 

FIGURE 9. IMPACT OF UNIT PREPARATION ON UNIT COHESION ...............................16 

FIGURE 10. .IMPACT OF UNIT PREPARATION ON UNIT EFFICACY ...............................17 

FIGURE 11. IMPACT OF UNIT PREPARATION ON UNIT EFFECTIVENESS....................17 

FIGURE 12. IMPACT OF LEADER INVOLVEMENT ON UNIT PROCESS .........................18 

 
1 

GAME-BASED TRAINING EFFECTIVENESS EVALUATION IN AN OPERATIONAL 

SETTING 

Introduction 

 The U.S. Army faces significant training challenges, now and in the years ahead. Some of 

these challenges involve persistent conflict and deployment pressures on training requirements 

and time. An increased diversity of operational environments, the complexity of possible 

domestic and global missions, and competing requirements for training resources are all key 

factors that have a potential impact on future training budgets (McHugh & Casey, Jr., 2010). The 

Army needs methods for providing Soldiers and leaders with effective training and opportunities 

to practice tasks effectively and efficiently. Consequently, Army trainers are increasingly turning 

to emerging technologies and innovative training methods to exploit low-cost, technology-based 

solutions to rapidly fill critical training gaps and increase the impact and effectiveness of training 

for our Soldiers in a time of war. 

 One technological approach to training that has received increased interest over the last 

decade is the use of game-based training (GBT). Training researchers hypothesize that games are 

instructive because they engage the player/learner, leading to skill acquisition and retention 

(Colquitt, LePine, & Noe, 2000; Mathieu, Tannenbaum, & Salas, 1992; Prensky, 2001; 

Tannenbaum & Yukl, 1992). In fact, empirical research has provided evidence that attributes of 

games (e.g., challenge, realism, and interactivity) influence trainee motivation and the length of 

time that trainees are willing to invest in mastering the skills taught during game play (e.g., 

Corbeil, 1999; Engel, et al., 2009; Garris, Ahlers, & Driskell, 2002). 

 Many Army agencies and organizations, including the U.S. Army Training and Doctrine 

Command (TRADOC), have recognized that games have the potential to augment and improve 

military training for both individuals and collectives. Examples of active GBT applications 

include Army DARWARS Ambush! and Tactical Iraqi. These games are being used to train 

Soldiers on a variety of skills including convoy operations, language, cultural awareness, and 

other tactical drills and tasks. 

 The most recent GBT effort is the acquisition and fielding of Virtual BattleSpace2 

(VBS2): U.S. Army. VBS2: U.S.Army is based on the commercial gaming engine Real 

Virtuality 2 created by Bohemia Interactive. It is an interactive, three-dimensional training 

system to provide a synthetic environment on networked PCs suitable for a wide range of 

military training and experimentation purposes. Trainees typically interact with the simulation 

using a keyboard and mouse, although simulated vehicle control interfaces (steering wheel and 

pedals) can be used. It offers large, realistic terrain areas (up to 100km x 100km in size). VBS2: 

U.S. Army includes after-action review capability, High Level Architecture/Distributed 

Interactive Simulation (HLA/DIS) compliance, and a mission editor that allows scenarios to be 

created and also modified in real time. VBS2: U.S. Army places military trainees in realistic 

urban or rural settings, and provides them with the opportunity to practice protocols relating to 

convoy and ambush operations. Trainees can drive Army vehicles, pilot aircraft, and fire 

weapons across a number of game scenarios. The use of VBS2: U.S. Army is designed to be a 

stepping stone in Soldier training. Ideally, units will use the game to practice tactics, techniques 
2 

and procedures prior to going into the field. By engaging in GBT, Soldiers can improve their 

skills prior to participating in a live training exercise. 

A VBS2: U.S. Army screenshot is shown in Figure 1. Further information about VBS2: U.S. 

Army can be found at the Army MILGaming Portal (https://milgaming.army.mil). 

Figure 1. A VBS2: U.S. Army screenshot. 

While the Army has established a game technology program of record, there are still 

many unanswered questions regarding game technology training effectiveness. Much of the 

common wisdom concerning the effects of GBT is anecdotal, lacking an empirical foundation 

(Hays, 2005). The majority of articles and reports written and published on instructional games 

are based on the writers’ opinions of the promise of instructional games (Rieber, Smith, & Noah, 

1998). Few articles have empirically studied the effectiveness of instructional games for either 

individual or team training (Hays, 2005). The research that does exist has been focused on 

enhancing our understanding of how to design training games to keep trainees interested, with 

less emphasis placed on how games influence the acquisition of specific knowledge and skills. 

Just because a game is motivating to play does not mean that it will be instructional. In fact, what 

is motivating about a game may hinder learning. Research has shown that if the content is not 

correctly embedded into the game, learners are more likely to walk away with increased 

knowledge of how to play the game rather than the transferable, operational knowledge or skill 

(Belanich, Sibley, & Orvis, 2004). 

The existing body of research does not provide insight as to how well GBT works 

relative to other methods, nor why or when (under what conditions) to use it. Decisions to use 

games to train have been based on a “leap of faith,” and there have been few efforts to test that 

faith through rigorous evaluation of the hypothesized benefits of games over other training 
3 

methods (Hays, 2005). Training designers need to understand how to incorporate game-based 

approaches to provide training that is pedagogically sound, as well as motivating and engaging. 

The credibility and impact of GBT programs could rise dramatically if solid empirical research 

was conducted on the effectiveness of such training approaches. 

During FY 2009, the Army distributed copies of VBS2: U.S. Army to many training 

locations all over the world. With such an aggressive application of GBT, there is a need to 

evaluate the effectiveness of the platform in both operational and institutional settings. In 

response to this need, the U.S Army Research Institute for Behavioral and Social Sciences 

(ARI), under the sponsorship of TRADOC Capability Manager Gaming (TCM Gaming), 

conducted a two-part effort to assess the effectiveness of the use of computer games for Army 

training. Both efforts use VBS2: U.S. Army as the evaluation platform. The work described in 

this report focuses on evaluating the use of VBS2: U.S. Army in the operational (unit) setting, 

while the other effort focuses on the use of GBT in the institutional setting. 

The research literature indicates that training effectiveness is influenced by three primary 

determinants: a) the training program itself (including the game), b) the trainee (in terms of 

personal characteristics and attitudes), and c) the situational context in which the training takes 

place (Campbell & Kuncel, 2001; Colquitt, et al., 2000; Mathieu, et al., 1992; Salas & CannonBowers, 2001). In addition to studying the overall effectiveness of VBS2: U.S. Army, the 

purpose of this analysis was to investigate the impact of situational variables on training 

outcomes. Such a multi-pronged approach not only evaluates the overall effectiveness of the 

GBT but also explains “why” the game-based approach may or may not be effective, and 

therefore, provides suggestions for improving future effectiveness of GBT. 

This report first describes the method associated with developing the evaluation protocol. 

This method consisted of making site visits to Army posts using VBS2: U.S. Army on a regular 

basis. Next, the evaluation protocol is described, along with the associated measures and 

hypotheses. Finally, this report presents the evaluation results and also makes recommendations 

for the use of GBT in an operational setting. 

 
4 

Method 

Site Observations 

 Prior to developing the method by which evaluation data would be collected, we 

conducted site visits to locations using VBS2:U.S. Army. The purpose of those visits was to 

learn how training was conducted at each site in order to develop an appropriate evaluation plan 

that matched the training method used. 

On 11-13 August 2009 and 18-19 August 2009, site visits were conducted at Ft. Hood, 

TX and Ft. Lewis, WA, respectively. These sites were chosen because both were using VBS2: 

U.S. Army for training on a regular basis. Throughout the five days of observation, a number of 

units came into the training centers to train using VBS2: U.S. Army. Although the primary 

training objective across the two sites was convoy training, VBS2: U.S. Army also supported 

additional training objectives including crew coordination and teamwork; communications; and 

reinforcing rules of engagement (ROEs) or standard operating procedures (SOPs). Although 

there were some differences across the two sites in terms of the training logistics, the overall 

flow of the training was the same. The next section of this report describes that training. 

Training using VBS2: U.S. Army. At both Ft. Lewis and Ft. Hood, each training facility 

contained computers for each individual Soldier to use during the training session. After the 

training unit entered the training facility, each Soldier was assigned a different role or position 

for the convoy scenario – driver, gunner, truck commander, or observer. One Soldier was also 

assigned the position of convoy commander. In addition, the role of higher headquarters is filled 

by either a training facilitator or someone in a leadership position (e.g., company commander) 

within the unit. By having all of these roles filled during the training, the Soldiers are able to get 

a sense of what each role does, learn how to utilize each position, and also learn to convey the 

appropriate information to the appropriate person at the appropriate time. 

At each training site, there was the potential for wide variability in the computer skills 

and experience with games across all of the Soldiers. Therefore, before beginning actual 

missions, Soldiers were all given initial training on how to use the game in order to get all 

trainees comfortable with the controls. Following this initial training period, the training unit was 

given its mission, and planning for the mission commenced. All Soldiers were then briefed on 

the mission (typically by the convoy commander) and given time to ask questions. Following the 

mission brief, Soldiers then went through the mission. Throughout the mission scenario, training 

facilitators played the role of the opposing forces (OPFOR) and embedded attacks in the form of 

insurgents, snipers, and improvised explosive devices (IEDs) into the scenario. The unit had to 

achieve their mission objectives while contending with attacks from the OPFOR. During the 

observed training, it was not uncommon for a Soldier's avatar to experience a casualty during the 

scenario. All of these elements added a sense of stress to the training mission, encouraging the 

Soldiers to take the training seriously. 

Finally, after the mission was complete, a thorough after-action review (AAR) was 

conducted. After completion of the AAR, units could conduct a second mission, following the 

same procedure as described above. Depending upon the unit commander’s plan, the second 
5 

mission could be identical to the first one conducted or could be different and possibly more 

difficult. 

Across the observed missions, there was a lot of variance in who led and conducted the 

AAR. For some units, the AAR was led by a higher-level leader who had not actually 

participated in the training. For other units, it was led by the individual given the position of 

convoy commander during the training, and hence, was deemed “the leader” of the exercise. In 

other instances, the AAR was conducted by a Soldier who had been selected to conduct the AAR 

at the completion of the exercise. Finally, for some units, the AAR was led by one of the training 

facilitators and not by a Soldier or higher-level leader in the training unit. This variance 

contributed to some AARs being conducted more thoroughly than others. 

One of the unique features of VBS2: U.S. Army is that it has the capability to play back 

the entire mission, including from a birds-eye view, for use during the AAR. This capability 

allows Soldiers to see things like where the insurgents were shooting from, what their truck 

formation looked like, and how each individual Soldier was reacting and behaving. Getting this 

overall view of the mission can provide Soldiers with a better and more objective understanding 

of their behaviors and the results of those behaviors. Although the playback capability is 

available, there were differences in how well the recording was integrated into the AAR. At 

times, the video playback capability was not utilized, or only partially used. 

These observations provided insight into the types of measures that should be used in the 

evaluation. These visits also highlighted the challenges associated with conducting the 

evaluation. One of the things that stood out was the fact that there were differences not only 

between Ft. Hood and Ft. Lewis, but also within each site in terms of how the training was 

conducted. For example, some units spent a much longer time planning their missions than 

others. Some units had the involvement of very senior officers, such as lieutenant colonels, 

whereas other units had no leadership present other than those participating in the training 

scenarios, and others had some level of leadership present only observing, not participating. 

Another source of variability observed was the training scenarios used. Because each unit came 

to the training with its own unique training needs and objectives, it was nearly impossible to 

develop objective measures of performance prior to conducting the evaluation. It became clear 

that the only way to control for all of those factors was to measure them to the greatest extent 

possible while keeping in mind that the actual data collection time was limited. In short, the 

evaluation plan had to be designed to try and overcome some of these challenges associated with 

the training. 

Data Collection Protocol and Hypothesis Development 

 Following training observations at Ft. Lewis and Ft. Hood, the data collection protocol 

was developed. Although time to administer measures was limited, we were given permission 

from each training facility to administer a short set of measures both prior to and after training 

was complete. Therefore, measures to administer both pre- and post-training were developed. 

The hypotheses and measures generated as part of the protocol focused on measuring the 

skills that VBS2: U.S. Army targeted during training. Training seemed to focus on the 

development of two areas: skills associated with individual level learning and skills related to 
6 

learning at the unit level. Therefore, the measures generated for this evaluation also focused on 

measuring these two different types of outcomes. In addition, the data collection protocol 

included measures of the varying contextual differences observed across the training sessions. 

The majority of measures were administered both pre- and post-training. The difference between 

the two administrations was in the question stems. The post-training items asked the trainees to 

respond to the items now that they have engaged in training using VBS2: U.S. Army. 

Individual level measures. Given the variation in the technical training objectives across 

training scenarios, it was difficult to develop measures to assess individual level learning. 

However, based on our observations and conversations with the training facilitators, we 

developed measures associated with specific behaviors and tasks that are typically performed 

while engaging in training using VBS2: U.S. Army. For example, specific behaviors and tasks 

associated with conducting convoy operations (e.g., scanning sectors of fire, communicating 

with unit members, locating sources of enemy fire, and identifying civilians) were identified and 

used in measures associated with performing these skills during training. 

The individual level measures administered consisted of a demographics measure (8 

items; Appendix A, pg. 23), skill preparedness measure (13 items; Appendix A, pg. 24), training 

motivation (10 items; Appendix A, pg. 25) and task performance (15 items; Appendix A, pg. 26). 

For the majority of these measures, trainees responded on a 5-point scale (1 = Strongly Disagree; 

5 = Strongly Agree). When necessary, items were re-coded so that higher numbers represented a 

more positive score. Table 1 lists when each measure was administered (e.g., pre- or posttraining), a description of the measure, and an example item for each measure. 

Unit level measures. Several measures were used to assess unit level outcomes. 

Examining these measures over time provides evidence of the training impact on unit outcomes. 

These unit level variables included the following: unit process measure (13 items; Appendix A, 

pg. 27), unit cohesion measure (task and interpersonal cohesion for a total of 11 items; Appendix 

A, pg. 28), unit efficacy measure (7 items; Appendix A, pg. 29), and overall unit effectiveness 

measure (3 items; Appendix A, pg. 29). All of these measures were administered both pre- and 

post-training. More information about each of these measures is in Table 1. 

Contextual variables. In order to understand the context under which GBT may be more 

or less effective, we also generated several measures to assess variables that may impact the 

effectiveness of the training. These measures were completed by outside observers to the 

training. All measures assessing contextual variables are in Appendix A. 

 
7 

Table 1. Information about Individual and Unit Level Measures 

Measure Measure Description When 

Administered 

Example Item(s) 

Skill Preparedness Assess Soldier belief in level of 

preparation to perform certain 

general tactical tasks often a part 

of VBS2 training missions 

Pre- and Post 

Training 

How prepared do you feel 

to... 

“Assess the tactical 

situation?” 

“Plan a tactical convoy 

operation?” 

Training 

Motivation 

Assess Soldier motivation level to 

participate in VBS2 training 

Post-Training “I tried very hard during 

this exercise.” 

Task Performance Assess Soldier belief about 

performance on specific tasks 

performed during training 

Post-Training Please rate your ability to 

perform each of the 

following tasks in the 

simulator... 

“Scan for threats.” 

“Identify insurgents.” 

Unit Process Assess Soldier belief about how 

well the unit worked together 

during the training in terms of 

different teamwork skills (e.g., 

communication, monitoring 

progress towards goals). Research 

has hypothesized that engagement 

in effective unit processes are 

directly linked to increased unit 

performance (e.g., Marks, 

Mathieu, & Zaccaro, 2001). 

Pre- and Post 

Training 

“My training unit members 

and I understand how one 

another prefer to 

communicate information 

to other members.” 

“My training unit members 

and I understand how 

members are going to 

work together to achieve 

our goals.” 

Unit Cohesion Assess Soldier attraction to the 

unit, including their tasks (task 

cohesion) and other group 

members (interpersonal cohesion). 

Research has shown that unit 

cohesion influences unit 

performance (e.g. Gully, Devine, 

& Whitney, 1995). 

Pre- and Post 

Training 

Task Cohesion: “When 

one person is struggling 

with a task, another 

member of the training 

unit will step in to help.” 

Interpersonal Cohesion: 

“The members of my 

training unit get along with 

one another.” 

Unit Efficacy Assess Soldier confidence level in 

ability of the unit to work together 

and perform well during the 

training. Research has shown a 

positive relationship between unit 

efficacy and performance (e.g. 

Stajkovic, Lee, & Nyberg, 2009). 

Pre- and Post 

Training 

“I am confident that the 

members of my training 

unit and I can effectively 

set contingency plans.” 

Unit Effectiveness Assess Soldier belief about how 

effective the unit was during the 

training mission 

Pre- and Post 

Training 

“This training unit is 

effective.” 
8 

Training facilitator measures. The training facilitators completed measures regarding the 

level of unit preparation of the unit prior to training (Appendix A, pg. 30). Facilitators completed 

these ratings at the end of each mission completed by one training unit. This measure was 

designed to account for differences in how much effort the unit put into preparing for the VBS2: 

U.S. Army training prior to coming to the training facility. Differences may exist in terms of how 

much the unit communicates with the training facilitators ahead of time in regard to mission 

objectives, training scenarios, etc. For this measure, facilitators rated each unit on a 5-point scale 

(1 = Completely Unprepared; 5 = Completely Prepared). These ratings were averaged across 

facilitators if more than one facilitator rated the level of preparation. In order to classify the units 

into more meaningful groups, a dichotomous variable that classified those units who received an 

average rating of 3 or less as “Less Prepared” and those units who received an average rating of 

over 3 as “More Prepared.” This dichotomy was used throughout all analyses involving this 

variable. Given that this measure was meant to tap into the unit’s initial level of preparation, only 

the rating given by facilitators regarding the first mission completed at the training center was 

used. 

Observer measures. Training observers from ARI, Aptima, and TCM Gaming completed 

several measures as they observed each mission. Namely, observers completed Time Spent on 

Training and Leader Involvement measures (Appendix A, pgs. 31). For the Time Spent on 

Training measure, observers simply recorded the number of minutes each training unit spent on 

1) initial VBS2: U.S. Army training (e.g., buttonology), 2) the mission brief, and 3) the actual 

training mission. The time spent on the initial training was only taken into account once, whereas 

the time spent on the mission brief and the actual training mission was taken into account each 

time the unit began a new mission. 

The Leader Involvement measure consisted of six questions targeted at recording the 

level of involvement from unit leadership (not actually involved as a participant in the training) 

during each training mission. The qualitative information from each of these measures was 

transformed to a quantitative three-point scale to measure the level of involvement. A rating of 1 

meant that unit leadership was not present at the training exercise; a rating of 2 meant that unit 

leadership participated in either the mission brief or the AAR portion of training; and a rating of 

3 meant that unit leadership was present and participating in both of those important aspects of 

the training. 

Hypotheses. Based on all these measures, we generated several hypotheses about how 

training would impact individual and unit level learning, as well as how the contextual variables 

would influence training effectiveness. 

1. VBS2: U.S. Army training positively impacts the training outcome of Skill Preparedness. 

2. The level of Unit Preparation for the training positively impacts the training outcomes of 

a) Skill Preparedness, b) Task Performance, and c) Training Motivation. 

3. Leader Involvement in training positively impacts the training outcomes of a) Skill 

Preparedness, b) Task Performance, and c) Training Motivation. 
9 

4. VBS2: U.S. Army training positively impacts the training outcomes of a) Unit Process, b) 

Unit Cohesion, c) Unit Efficacy, and d) Unit Effectiveness. 

5. The level of Unit Preparation for the training positively impacts the training outcomes of 

a) Unit Process, b) Unit Cohesion, d) Unit Efficacy, and d) Unit Effectiveness. 

6. Leader Involvement in training positively impacts the training outcomes of a) Unit 

Process, b) Unit Cohesion, d) Unit Efficacy, and d) Unit Effectiveness. 

Data Collection 

 Participants. One-hundred and sixty five Soldiers participated in the evaluation study 76 

at Ft. Hood and 89 at Ft. Lewis. These 165 Soldiers represented 9 different platoons. 95% of 

these Soldiers were enlisted; 4% were officers; and less than 1% were warrant officers. The 

mean age of the trainees was 25.20 (SD = 5.68). In addition, 33% of the trainees reported having 

engaged in training using VBS2: U.S. Army in the past. 

Procedure. Soldiers were given a brief overview of the project upon entering the training 

facility. They were informed that their participation was voluntary and that all of their responses 

were anonymous and would not be linked back to any single individual. Soldiers read and signed 

the informed consent form, and then were given the pre-training questionnaire packet to 

complete. This set of measures took approximately 15 to 20 minutes for each Soldier in the unit 

to complete. Following completion of these measures, the unit engaged in training as usual using 

VBS2: U.S. Army. As described above, this training consisted of a practice mission, to be 

followed by the actual training mission, and an AAR. Soldiers completed a second set of 

measures after the final AAR of the day. 

 In addition to the Soldiers completing measures, after each mission, training facilitators 

provided an overall rating of how prepared the unit was for the training. Aptima and ARI 

personnel present during the data collection also recorded the time each unit spent on training 

and conducting the mission. All of these measures served to control for some of the variability 

present across the training events. 

Results 

 Before conducting any analyses, the measures themselves were examined to ensure that 

they were reliable. Statistically, all of the items in each of the measures fit well together, as 

demonstrated by a measure of internal consistency (alpha levels ranging from .76 to .96). This 

statistic indicates that the items within one measure all tap into the same construct. 

Trainees spent an average of 86 minutes (SD = 39.65) on the initial VBS2: U.S. Army 

training; this time was spent teaching trainees how to use the simulation prior to conducting their 

training mission. The average time spent conducting the mission brief and planning for the 

mission was 17.88 minutes (SD = 8.96), and the average training mission lasted 83.64 minutes 

(SD = 61.18). 
10 

Overall, the results demonstrated that training with VBS2: U.S. Army had some impact 

on both individual-level and unit-level outcomes. The context in which this training occurred 

also impacted both sets of variables. This results section first discusses the analyses associated 

with the individual level variables, followed by a discussion of the results with the unit level 

variables. 

Individual Level Analyses 

 Hypotheses 1 through 3 focused on the influence of the training and the training context 

on the individual level variables (Skill Preparedness, Task Performance, and Training 

Motivation). First, in order to generally understand how trainee scores changed from pre- to posttraining measures (and test hypothesis 1), a paired sample t-test was conducted on the Skill 

Preparedness measure (since pre-measures were not administered for Task Performance or 

Training Motivation, no analyses could be conducted on these measures). Although the trainees 

reported feeling slightly more prepared to conduct the skills after training (M = 3.97, SD = .76) 

compared to pre-training (M = 3.93, SD = .67), this difference was not statistically significant, 

t(141) = -.61, n. s. However given that the skills each unit performed during the training mission 

were aligned with the diverse training objectives of each unit, we looked more deeply at the 

individual items in this measure. Upon examination, we found that trainees reported feeling more 

prepared to “plan a tactical convoy operation” after the training (M = 3.78, SD = 1.06) than prior 

to the training (M = 3.50, SD = 1.15), t (141) = -2.61, p = .01. Figure 2 depicts this relationship. 

These results demonstrated that although trainees did not report feeling more prepared to 

perform a variety of skills following the training, they did report that the training made them feel 

more prepared to engage in one of the more general objectives of VBS2: U.S. Army – planning 

for a convoy mission. The results of the analyses for each individual Skill Preparedness item are 

in Appendix B. 

* p = .01 

Figure 2. Preparedness levels for planning a convoy operation. 

Hypotheses 2 and 3 focused on the impact of situational factors on training effectiveness. 

To examine the influence of Unit Preparation and Leader Involvement on the individual level 

3.50*

3.78*

2.5

3

3.5

4

Pre-Training Post-Training

Preparedness to Plan a 

Convoy Operation
11 

outcomes, Analysis of Covariance (ANCOVA) was conducted. This statistical analysis technique 

was utilized multiple times while analyzing the data. ANCOVA statistically controls for 

variables that may mask the direct relationship between two other variables. In all instances 

where ANCOVA was used, the impact of the platoon that the trainees were in, as well as pretraining levels of the variables where relevant, were controlled for. For example, hypothesis 2a 

looks at the impact of Unit Preparation on Skill Preparedness. In order to understand the more 

direct relationship between these two variables, we accounted for random variance associated 

with the training unit to which the trainees belonged and their initial preparedness levels. This 

type of analysis allows for a cleaner examination of the situational variables on the training 

outcome. Therefore, the results reported here for Hypotheses 2 and 3, as well as for other 

applicable hypotheses, are based on accounting for those extraneous variables. 

For Hypotheses 2, ANCOVA results showed that Unit Preparation significantly impacted 

a) Skill Preparedness, b) Task Performance, and c) Training Motivation. Figures 3, 4, and 5 

depict the nature of these relationships. As shown in these figures, trainees in units which had 

prepared more for the training reported being more prepared to conduct certain skills and 

perform various tasks throughout the training; they also reported higher motivation for the 

training. 

* Adjusted means significantly different from one another, F (1, 126) = 5.48, η2 = .04, p < .05 

Figure 3. Impact of unit preparation on skill preparedness. 

3.88*

4.38*

3

3.5

4

4.5

Less Prepared More Prepared

Skill Preparedness

Unit Preparation
12 

* Adjusted means significantly different from one another, F (1, 124) = 7.88, η2

 = .06, p < .01 

Figure 4. Impact of unit preparation on task performance. 

 

* Adjusted means significantly different from one another, F (1, 126) = 5.73, η2 = .04, p < .05 

Figure 5. Impact of unit preparation on training motivation. 

Hypothesis 3 was similar to Hypothesis 2 and predicted that the level of Leader 

Involvement in the training positively influences a) Skill Preparedness, b) Task Performance, and 

c) Training Motivation. Parts b and c of this hypothesis were supported; an ANCOVA indicated 

that there was no statistically significant relationship between the level of Leader Involvement in 

the training and Skill Preparedness. Figure 6 illustrates the significant relationship between the 

level of Leader Involvement during training and Task Performance. According to LSD post-hoc 

tests, Soldiers in units with substantial levels of Leader Involvement (Madj = 4.32, SE = .11) were 

3.97*

4.36*

3

3.5

4

4.5

Less Prepared More Prepared

Task Performance

Unit Preparation

3.83*

4.26*

3

3.5

4

4.5

Less Prepared More Prepared

Training Motivation

Unit Preparation
13 

more likely to report higher task performance compared to Soldiers in those units with only some 

level of Leader Involvement (Madj = 3.95, SE = .06, p = .01) and units with no Leader 

Involvement (Madj = 3.83, SE = .13, p = .01). The difference in reported levels of Task 

Performance is not statistically significant for those Soldiers in units with some level of Leader 

Involvement compared to those Soldiers in units with no Leader Involvement during training. 

* Adjusted means significantly different from one another, F (2, 136) = 3.67, η2 = .05, p < .05 

Figure 6. Impact of leader involvement on task performance. 

 Figure 6 depicts the relationship between the level of Leader Involvement during training 

and Training Motivation. As can be seen in this figure, the highest Training Motivation levels 

were reported by those trainees with substantial levels of Leader Involvement (Madj = 4.24, SE = 

.14). LSD post-hoc tests demonstrated that both trainees with some (Madj = 3.82, SE = .08, p < 

.05) and no (Madj = 3.41, SE = .17, p < .01) Leader Involvement reported significantly lower 

levels of motivation. The difference in motivation levels for Soldiers in units with no Leader 

Involvement and some Leader Involvement is also statistically significant (p < .05). 

In general, these results demonstrate that GBT can have the power to impact individuallevel learning, especially in an appropriate context. Being prepared for the training seems to be 

one key element for individuals to learn from GBT. Also, having unit leadership involved during 

the training seems to provide support for individuals to be more motivated and perform better 

during the training. The next section examines similar relationships with unit-level variables. 

3.83*

3.95*

4.32*

3

3.5

4

4.5

No Leader 

Involvement

Some Leader 

Involvement

Substantial Leader 

Involvement

Task Performance

Level of Leader Involvement during Training
14 

* Adjusted means significantly different from one another, F (2, 138) = 5.41, η2 = .07, p < .01 

Figure 7. Impact of leader involvement on training motivation. 

Unit Level Analyses 

Hypotheses 4 through 6 focused on the influence of the training and the training context 

on unit level variables (Unit Process, Unit Cohesion, Unit Efficacy, and Unit Effectiveness). The 

first analysis with the unit level variables was to examine the general impact of the training. 

Paired sample t-tests were conducted to examine whether trainees perceived higher levels of 

these unit level variables following the training as opposed to prior to the training. Although all 

of the post-test scores were higher than the pre-test scores, the only significant differences 

existed with Unit Process and Unit Cohesion. Table 2 showcases the results of this analysis for 

all four unit level outcomes. These results indicate that Soldiers reported feeling better about 

how their unit engages in various team processes (e.g., communication, backing each other up) 

and the level of cohesion present among unit members after having participated in training with 

VBS2: U.S. Army. 

3.41*

3.82*

4.24*

3

3.5

4

4.5

No Leader 

Involvement

Some Leader 

Involvement

Substantial Leader 

Involvement

Training Motivation

Level of Leader Involvement During Training
15 

Table 2 

T-Test Results Comparing Pre- to Post-Training Outcome Variables 

Comparison M SD t 

Unit Process 

Pre Training 3.30 .70 -2.71* Post Training 3.46 .78 

Unit Cohesion 

Pre Training 3.41 .68 -1.93* Post Training 3.54 .89 

Unit Efficacy 

Pre Training 3.46 .78 -1.33 Post Training 3.55 .82 

Unit Effectiveness 

Pre Training 3.07 .78 -1.40 Post Training 3.16 .83 

* p < .05 

Hypotheses 5 and 6 focused on the impact of situational factors on training effectiveness 

at the unit level. As with the individual level variables, the effect of Unit Preparation and Leader 

Involvement on the unit level outcomes was examined through ANCOVA, controlling for the 

impact of the platoon that the trainees were in as well as pre-training levels of the variables. 

Results showed that the level of Unit Preparation significantly impacted a) Unit Process, b) Unit 

Cohesion, d) Unit Efficacy, and d) Unit Effectiveness. Figures 7 through 10 depict the nature of 

these relationships. As seen in the four graphs, higher levels of unit level variables were reported 

by Soldiers in units that had prepared more for the training exercises. These analyses indicate 

that the VBS2: U.S. Army training was particularly effective at increasing how unit members 

work together and feel about working together when units have adequately prepared for that 

training. 

 
16 

* Adjusted means significantly different from one another, F (1, 126) = 10.23, η2 = .08, p < .01 

Figure 8. Impact of unit preparation on unit process. 

* Adjusted means significantly different from one another, F (1, 126) = 6.72, η2 = .05, p < .05 

Figure 9. Impact of unit preparation on unit cohesion. 

3.34*

4.00*

3

3.5

4

4.5

Less Prepared More Prepared

Unit Process

Unit Preparation

3.43*

4.05*

3

3.5

4

4.5

Less Prepared More Prepared

Unit Cohesion

Unit Preparation
17 

* Adjusted means significantly different from one another, F (1, 125) = 3.95, η2 = .03, p < .05 

Figure 10. Impact of unit preparation on unit efficacy. 

* Adjusted means significantly different from one another, F (1, 125) = 5.00, η2 = .04, p < .05 

Figure 11. Impact of unit preparation on unit effectiveness. 

 Finally, Hypothesis 6 predicted that Leader Involvement would positively impact the four 

unit level variables. Support was found for only one part of that hypothesis; specifically, the 

level of Leader Involvement was significantly related to Unit Process. This relationship is 

illustrated in Figure 11. As can be seen in this graph, Soldiers in units with substantial levels of 

leader involvment perceived the highest levels of Unit Process (Madj = 3.90, SE = .17). Post-hoc 

LSD tests demonstrate that this reported level of Unit Process is statistically different compared 

to Soldiers in units with some Leader Involvement (Madj = 3.30, SE = .09, p < .05) and no Leader 

Involvement (Madj = 3.19, SE = .20, p < 01). The difference in Unit Process levels perceived by 

3.46*

3.92*

2.5

3

3.5

4

4.5

Less Prepared More Prepared

Unit Efficacy

Unit Preparation

3.07*

3.60*

2.5

3

3.5

4

4.5

Less Prepared More Prepared

Unit Effectiveness

Unit Preparation
18 

Soldiers in units with no Leader Involvement and some Leader Involvement was not statistically 

significant. 

* Adjusted means significantly different from one another, F (2, 137) = 4.02, η2 = .06, p < .05 

Figure 12. Impact of leader involvement on unit process. 

Discussion and Conclusions 

 This report provides some evidence that training using VBS2: U.S. Army is effective in 

improving both individual-level and unit-learning learning. In terms of individual-level learning, 

the training did not directly impact how prepared Soldiers felt to perform certain skills; however, 

this non-significant finding seems to be more of a function of measurement than the actual 

training. The lack of consistency across the training scenarios, and consequentially the training 

objectives, makes it difficult to measure the appropriate skills. That is, our list of skills, prepared 

in advance of the training and without knowledge of the unit’s training objectives, may simply 

not have matched well with the skill(s) the unit wished to train. However, after engaging in the 

training, Soldiers reported feeling more prepared to plan a convoy operation than prior to 

training. Even though specific training objectives differed from unit to unit, one commonality 

across all of the training sessions was that each mission focused on planning for and conducting 

convoy operations. Therefore, it makes sense that Soldiers would report feeling more prepared to 

do this type of planning after actually having done it through training using VBS2: U.S. Army. 

This finding has implications for evaluating GBT, as well as presents evidence for the validity of 

GBT. First, it points to the importance of targeting and measuring clear learning objectives when 

using GBT. Second, this finding illustrates that when clear learning objectives are present in 

games, games can effectively prepare trainees to some extent and make them feel more confident 

in their ability to perform skills targeted during the training. 

 Results associated with understanding the impact of GBT on individual level learning 

also point to the importance of the context surrounding the training. Particularly, the extent to 

which units prepared for the training and how involved unit leadership was during the training 

influence training effectiveness. Although both of these contextual variables positively impacted 

3.19*

3.30*

3.90*

2.5

3

3.5

4

No Leader 

Involvement

Some Leader 

Involvement

Substantial Leader 

Involvement

Unit Process

Level of Leader Involvement during Training
19 

the individual level variables, unanswered questions still remain about their impact. Future 

research needs to further examine the nature of Leader Involvement in GBT to understand 

exactly how it contributes to effectiveness. For example, is it enough for leaders to just be 

present during training, or do they need to provide a certain type of feedback to be influential? 

Additional research should focus on uncovering this relationship in more depth. 

 Results of this analysis suggest that the GBT investigated here is particularly useful for 

increasing unit-level outcomes. In general, the training influenced how the unit felt they worked 

together (Unit Process), unit motivational states (Unit Cohesion and Unit Efficacy), and the 

perceived effectiveness and performance of the unit. Trainees reported more effective unit 

processes and increased levels of unit cohesion from pre- to post-training. In addition, Soldiers in 

units that were more prepared for the training reported higher levels of Unit Process, Unit 

Cohesion, Unit Efficacy, and Unit Effectiveness compared to Soldiers in units with lower levels 

of preparation. It is likely that units who spent time preparing for the training exercises 

emphasized and focused on getting the unit to work together. Anecdotal evidence from our 

training observations and data collections indicates that unit leaders were using the training as a 

chance to focus on unit level behaviors. They continually emphasized the importance of 

communicating with each other during the training and understanding the role and position of 

each unit member. The results with the unit level variables mirror this emphasis by the units 

prior to and during the training. 

 In contrast to the influence of Unit Preparation on all four unit-level variables, the level 

of Leader Involvement in the training only influenced the level of Unit Process reported by the 

trainees. As pointed out with the individual-level variables, understanding the nature of this 

involvement may help to explain why process was the only variable influenced. When unit 

leadership was involved during the training, their involvement likely came in the form of 

feedback about how the unit was working together (i.e., their team processes) and was not 

centered on motivational states or overall performance feedback. 

Analysis Limitations 

There are several limitations associated with this analysis, many of them associated with 

the challenges in collecting data. As indicated earlier in this report, there was a very limited 

amount of time available to collect data. Therefore, the measures administered had to be limited 

and mainly self-report in nature. Second, there was no opportunity for a control group. Although 

we were able to gather both pre- and post-training data, the results of this analysis would have 

been stronger if we had been able to compare the GBT to a group that did not receive GBT. In 

addition, collecting data from the same training units following a field training exercise would 

provide a useful analysis of transfer of training. The performance of units which engaged in GBT 

could be compared to the performance of units which did not engage in GBT to more deeply 

understand how effective the training was. Finally, it was difficult to measure objective training 

performance given the differences in training objectives across the units. Each of these 

limitations made it more difficult for us to identify significant changes or improvements as a 

result of training with VBS2: U.S. Army. Despite these limitations, several interesting findings 

emerged to lend credence to the use of games for training purposes. Under the correct conditions, 

GBT can be an engaging, useful training mechanism. 
20 

 
21 

References 

Belanich, J., Sibley, D. E. & Orvis, K. L. (2004). Instructional characteristics and motivational 

features of a PC-based game (Research Report 1822). Alexandria, VA: U.S. Army 

Research Institute for the Behavioral and Social Sciences. 

Campbell, J. P., & Kuncel, N. R. (2001). Individual and team training. In N. Anderson, D. S. 

Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of work and organizational 

psychology. London: Blackwell. 

Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training 

motivation: a meta-analytic path analysis of 20 years of research. Journal of Applied 

Psychology, 85, 678-707. 

Corbeil, P. (1999). Learning from the children: practical and theoretical reflections on playing 

and learning. Simulation & Gaming, 30, 163-180. 

Engel, K., Langkamer, K. L., Estock, J. L., Orvis, K. L., Salas, E., Bedwell, W., & Conkey, C. 

(2009). Investigating the Effectiveness of Game-Based Approaches to Training. 

Proceedings of the 31st Annual Interservice/Industry Training, Simulation, and 

Education Conference. Orlando, FL: I/ITSEC. 

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation and learning: A research and 

practice model. Simulation & Gaming, 33, 441-467. 

Gully, S. M., Devine, D. J., & Whitney, D. J. (1995). A meta-analysis of cohesion and 

performance: Effects of level of analysis and task interdependence. Small Group 

Research, 26, 497-529. 

Hays, R. T. (2005). The effectiveness of instructional games: a literature review and discussion

(Technical Report 2005-004). Naval Air Warfare Center Training Systems Division: 

Orlando, FL 

McHugh, J.M., & Casey, Jr., G.W. (2010). 2010 U.S. Army Posture Statement. Downloaded 

from https://secureweb2.hqda.pentagon.mil/VDAS_ArmyPostureStatement/2010/ 

Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and 

taxonomy of team processes. Academy of Management Review, 26, 356-376. 

Mathieu, J. E., Tannenbaum, S. I., & Salas, E. (1992). Influences of individual and situational 

characteristics on measures of training effectiveness. Academy of Management Journal, 

35, 828-847. 

Prensky, M. (2001). Digital game-based learning. New York: McGraw-Hill. 
22 

Rieber, L. P., Smith, L., & Noah, D. (1998). The value of serious play. Educational Technology,

38, 29-37. 

Salas E., & Cannon-Bowers J. A. (2001). The science of team training: A decade of progress. 

Annual Review of Psychology, 52, 471– 499. 

Stajkovic, A. D., Lee, D., & Nyberg, A. J. (2009). Collective efficacy, group potency, and group 

performance: Meta-analyses of their relationships, and test of a mediation model. 

Journal of Applied Psychology, 94(3), 814-828. 

Tannenbaum, S. I., & Yukl, G. (1992). Training and development in work organizations. In M. 

R. Rosenweig, & L. W. Porter (Eds.), Annual Review of Psychology (Vol. 43, pp. 399-

441). Palo Alto, CA: Annual Reviews. 

 
23 

Appendix A-1 

Evaluation Measures 

PARTICIPANT INFORMED CONSENT FORM 

Date: 1 October 09

Title: Game-Based Training Effectiveness (Unit) 

Purpose: This research is being conducted by the Army Research Institute (ARI). The purpose is to 

obtain information about the effectiveness of computer simulations for Army training. This information 

will be used to help determine the future use of similar simulations in the Army. 

What you will be asked to do in this study: In addition to normal training activities, you will be asked 

to complete a short background questionnaire, several paper and pencil tests, and several questionnaires 

about your training. 

Time required: It is estimated that all questionnaires will take no longer than 30 minutes to complete in 

total. 

Cost/Risk: There are no associated risks with participating in this research. Providing information is 

voluntary, and you have the right to withdraw from the research at any time without consequences. 

Failure to respond to any particular question will not result in any penalty. All of your responses will be 

kept confidential and will not be shared with other individuals participating in this research. Any 

identifying information collected will be used only by persons engaged in, and for the purposes of, the 

research. 

Benefits: The benefits that you may expect from your participation in this research are minimal. Your 

participation will aid efforts to improve the effectiveness of Army training. 

Voluntary consent: Your signature below means that you have read this consent form and agree to 

voluntarily participate in this study. Your signed consent form will be kept separate from the rest of the 

research materials. Thus, your name will not be associated with your responses, and the data that you 

provide will be anonymous. You must be 18-years of age or older to participate.

• I have read the procedure described above. 

• I have been informed of the reasons for this research. 

• I have had the research explained to me. 

• I have had all of my questions answered. 

• I give consent voluntarily to participate in this research. 

• By signing this consent form, I have not given up any of my legal rights. 

 

Signature of Participant Date 

_________________________ 

Printed Name 

 

Signature of Person Conducting Informed Consent Discussion Date 
24 

Privacy Act Statement 

Title: Game-Based Training Effectiveness (Unit) 

Date: 1 October 09 

AUTHORITY: The Department of the Army may collect the information requested in this session under 

the authority of 10 United States Code, Section 2358, “Research and Development Projects.” In 

accordance with the Privacy Act of 1974 (Public Law 93-579), this notice informs you of the purpose, 

use, and confidentiality of this session. 

PURPOSE: The purpose is to collect obtain information about the effectiveness of computer simulations 

for Army training. 

ROUTINE USES: This information will be used to help determine the future use of similar simulations in 

the Army. 

DISCLOSURE: Participating in this session is voluntary and you may choose at any time not to 

participate. There is no penalty for choosing not to participate.

CONFIDENTIALITY: We will not identify you, or attribute responses to any particular participant 

made during this session, and we will NOT include your name or other personally identifiable 

information in our interview notes or report. Likewise, we ask that you respect the confidential 

nature of this session, by not identifying individual participants with comments made or heard 

during this session. 

We cannot provide "confidentiality" or "non-attribution," to a participant regarding comments involving 

criminal activity/behavior, or statements that pose a threat to yourself or others. Do NOT discuss or 

comment on classified or operationally sensitive information during this session. 

CONTACT: For further information about this project or your rights as a participant, send e-mail to: 

ARI_RES@conus.army.mil (type “GBT Unit” in the subject line).

RETAIN FOR YOUR RECORDS 

 
25 

Demographics 

1. Age 2. Civilian Education (check highest level attained): 

 GED  High 

School 

 Some 

College

 B.S. /B.A.  Postcollege 

3. Time in 

Service: Yrs Mos. 

4. Grade/ 

Rank: 

5. MOS/ 

Branch 

6. Deployment Experience (Provide information for all that apply) 

Position(s) Time (months) 

OIF 

OEF 

Bosnia 

7. Have you participated in VBS2 training before today? Yes No 

If yes, how many times prior to today have you participated in VBS2 training? ___________________ 

8. Using the scale below, indicate how familiar are you with the other Soldiers that you are training with 

today; please think about familiarity in terms of training together, deployed together, and even just 

knowing each other in a more social aspect: _____________ 

1 – Completely unfamiliar (e.g., I have not met most of them prior to today). 2 – Slightly unfamiliar. 3 – 

Neutral. 4 – Slightly familiar. 5 – Very familiar (e.g., I know most of them very well)  
26 

Skill Preparedness 

Prior to training with VBS2/Now that you have trained with VBS2, please respond to the following items about 

how prepared you feel to conduct the following tasks; please use the following scale in your assessment of 

each task: 

1 – Unprepared, 2 – Slightly Unprepared, 3 – Neither Unprepared nor Prepared, 4 – Slightly Prepared, 5 – 

Prepared

_____ 1. React to Attack (Near or Far). 

_____ 2. Assess the tactical situation. 

_____ 3. Consolidate and Reorganize. 

_____ 4. Locate known or suspected enemy positions. 

_____ 5. Scan my sector appropriately. 

_____ 6. Comply with rules of engagement. 

_____ 7. Control convoy movement while NOT in contact with the enemy. 

_____ 8. Plan a tactical convoy operation. 

_____ 9. Break contact. 

_____ 10. Coordinate activities with your chain of command. 

_____ 11. Communicate with members of your unit. 

_____ 12. Conduct CASEVAC/Recovery Operations. 

_____ 13. React to IED 
27 

Training Motivation 

Use the following scale for the following items: 

1 - Strongly disagree, 2- Disagree, 3 – Neither Agree nor Disagree, 4 - Agree, and 5 - Strongly 

Agree 

1. I enjoyed this training exercise very much. 

2. I put a lot of effort into this training exercise. 

3. It was important to me to do well in this training exercise. 

4. I tried very hard during this exercise. 

5. This training exercise was fun. 

6. I would describe this training exercise as very interesting. 

7. I am satisfied with my performance in this training exercise. 

8. I did not try very hard during this exercise. 

9. During this training exercise, I was thinking about how much I enjoyed it. 

10. This training exercise did not hold my attention. 
28 

Task Performance

Please rate your ability to perform each of the following tasks in the simulator at the end of today’s 

training. Please respond using the following scale: 

1 –Did Not Attempt, 2 – Very Poor, 3 – Poor, 4 – Good, 5 – Very Good 

Did Not 

Attempt

Very 

Poor Poor Good 

Very 

Good 

1. Move tactically while dismounted. c d e f g

2. Engage a target with my individual weapon. c d e f g

3. Enter a building. c d e f g

4. Drive a vehicle. c d e f g

5. Change my posture (prone-kneeling-standing) c d e f g

6. Detect an IED. c d e f g

7. Scan for threats. c d e f g

8. Identify civilians. c d e f g

9. Communicate with others. c d e f g

10. Locate the source of enemy fire. c d e f g

11. Mount a vehicle. c d e f g

12. Distinguish between friendly and enemy fire. c d e f g

13. Engage a target with a vehicle-mounted weapon. c d e f g

14. Identify insurgents. c d e f g

15. Dismount a vehicle. c d e f g

   
29 

Unit Process 

Use the following scale for the following items: 

1 - Strongly disagree, 2- Disagree, 3 – Neither Agree nor Disagree, 4 - Agree, and 5 - Strongly 

Agree 

1. My training unit members and I understand how one another prefer to communicate 

information to other members. 

2. My training unit members and I understand when we need to consult each other. 

3. My training unit members and I have a good understanding of the roles played by each 

member. 

4. My training unit members and I sometimes feel confused about how members should 

work together. 

5. My training unit members and I understand how members of the unit should help each 

other out. 

6. My training unit members and I understand how members are going to work together to 

achieve our goals. 

7. My training unit members and I are clear about what tasks each member is responsible 

for. 

8. My training unit members and I are able to prevent conflict from occurring between 

members. 

9. My training unit members and I are able to manage the emotions of the team (e.g. 

frustration, excitement). 

10. My training unit members and I are able to keep confidence high within the unit. 

11. My training unit members and I are able to keep motivation high within the unit. 

12. When something about the situation is uncertain, more knowledgeable members of the 

training unit will help the other members make sense of the situation. 

13. When one person is struggling with a task, another member of the training unit will step 

in to help. 

14. My training unit is engaged in or “into” our mission. 
30 

Unit Cohesion 

Use the following scale for the following items: 

1 - Strongly disagree, 2- Disagree, 3 – Neither Agree nor Disagree, 4 - Agree, and 5 - Strongly 

Agree 

1. My training unit is engaged in or “into” our mission. 

2. My training unit members enjoy working on the mission/tasks we must collaborate on. 

3. It is important for my training unit to produce high quality goals. 

4. It is important for my training unit members to perform well when given a mission and/or 

task. 

5. The members of my training unit consider the mission/tasks we must work on together as 

meaningful and important. 

6. My training unit expects to benefit from successful mission/task performance. 

7. The members of my training unit like each other. 

8. The members of my training unit get along with one another. 

9. The members of my training unit feel similar to one another. 

10. It is important that the members of my training unit socialize while working together. 

11. The members of my training unit like being a part of this unit. 
31 

Unit Efficacy 

Use the following scale for the following items: 

1 - Strongly disagree, 2- Disagree, 3 – Neither Agree nor Disagree, 4 - Agree, and 5 - Strongly 

Agree 

1. I am confident that the members of my training unit and I can effectively set contingency 

plans. 

2. I am confident that the members of my training unit and I can develop good strategies. 

3. I am confident that the members of my training unit and I can understand the tasks at 

hand. 

4. I am confident that the members of my training unit and I can coordinate effectively with 

all members. 

5. I am confident that the members of my training unit and I can determine our progress 

towards goals. 

6. I am confident that the members of my training unit and I can help each other out if 

necessary. 

7. I am confident that the members of my training unit and I can manage conflict and 

frustration.

Unit Effectiveness 

1. My training unit performs well at most missions/tasks. 

2. This training unit is effective. 

3. Relative to other training units performing similar functions, this training unit performs: 

a) Better 

b) The Same 

c) Worse 
32 

Unit Preparation for Training 

Please rate the preparedness of the training unit using a 5-point scale (1 = Very easy; 2 = 

Easy; 3 = Average; 4 = Difficulty; 5 = Very difficult). Please circle your answer. 

You will see anchors below to help you make your assessment. Each rating has several factors 

for you to consider in your assessment. Please note that not every factor has to be true in order 

for you to make your rating. In addition, it is likely that some additional factors may influence 

your assessment of Unit Preparedness. Please do not consider the factors below as the definitive 

criteria, but instead, use them as a guide to help you make your rating. 

Unit Preparedness Rating Factors to Consider in Rating 

1 (Completely Unprepared) • Arrived at the training center with no prior communication with 

the training facilitators about training objectives 

• No definitive training objectives 

• Spent no time planning for the mission once in the training center 

2 (Slightly Unprepared) • Arrived at the training center with no prior communication with 

the training facilitators 

• Had some slightly defined training objectives 

• Spent no more than 10 minutes planning for the mission once in 

the training center 

3 (Neither Prepared nor 

Unprepared) 

• Talked with training facilitators once to convey training objectives 

• Spent no more than 1 hour in the training center on mission 

planning 

4 (Slightly Prepared) • Talked with training facilitators more than once to convey training 

objectives 

• Discussed ideas with the training facilitators about how training 

missions may be modified to meet objectives 

• Spent at least a half day planning for the mission in the training 

center prior to conducting mission 

5 (Completely Prepared) • Met with training facilitators multiple times prior to the training 

exercise 

• Provided real-world missions for the training facilitators to 

incorporate into the training 

• Worked with, and provided suggestions to, the training facilitators 

on how best to accomplish training objectives 

• Spent one whole day in the training center conducting mission 

planning prior to conducting the actual mission 
33 

Time Spent on Training 

1. Time spent on VBS2 training: ____________ minutes 

2. Time spent on mission briefing: ___________ minutes 

3. Time spent on actual mission: _____________ minutes

Exercise Management – Leader Involvement 

Please answer each of the below questions according to your observations during the unit training. 

1. What was the highest level of unit leadership present during the training exercise? 

______________________________ 

2. Who conducted the mission briefing (e.g., the platoon sergeant, the company commander, a 

training facilitator)? _________________

Aside from the person named in the line above, who else helped with the briefing? 

______________________________

3. If feedback was provided to the trainees during the exercise, who provided it? 

_____________________

4. If someone from the unit participated in the training exercise by serving as headquarters, who was 

it? ___________________________

5. If someone from the unit communicated with the training facilitators during the exercise to 

contribute to the events that occurred during the mission, who was it? ____________

6. Who conducted the AAR? _______________________

Aside from the person named in the line above, who else helped with the AAR? 

______________________________

 
34 

Appendix B-1 

Skill Preparedness Items – Means and Standard Deviations 

Comparison M SD t 

React to an attack 

Pre Training 3.96 .91 1.09 Post Training 3.85 1.04 

Assess the tactical situation 

Pre Training 3.89 .90 -.47 Post Training 3.94 .94 

Consolidate and reorganize 

Pre Training 3.84 .94 .14 Post Training 3.83 .99 

Locate known or suspected enemy positions 

Pre Training 3.89 .90 .83 Post Training 3.82 .99 

Scan my sector appropriately 

Pre Training 4.39 .83 .51 Post Training 4.35 .92 

Comply with rules of engagement 

Pre Training 4.12 1.06 -1.69 Post Training 4.29 .91 

Control convoy movement while NOT in 

contact with the enemy 

Pre Training 4.08 .93 -.64 Post Training 4.14 .96 

Plan a tactical convoy operation 

Pre Training 3.50 1.15 -2.61* Post Training 3.78 1.06 

Break contact 

Pre Training 3.77 1.05 -.54 Post Training 3.82 .99 

Coordinate activities with your chain of 

command 

Pre Training 3.84 .96 -.64 Post Training 3.90 .98 

Communicate with members of your unit 

Pre Training 4.20 .82 .68 Post Training 4.13 .92 

Conduct CASEVAC/Recovery operations 

Pre Training 3.68 1.05 -.66 Post Training 3.75 1.02 

React to IED 

Pre Training 3.94 1.02 -.98 Post Training 4.04 .99 

* p < .01