Technical Report 1135

UTILITY OF GAME INSTRUCTIONS

Jessie Y.C. Chen

Consortium of Universities of the Washington Metropolitan Area

Consortium Post-Doctoral Fellows

April 2003

United States Army Research Institute

for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

ZITA M. SIMUTIS

Director

Technical Review by

John S. Barnett, ARI

Brooke Schaab, ARI

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001

Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Technical Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
i

REPORT DOCUMENTATION PAGE

1. REPORT DATE:

April 2003

2. REPORT TYPE:

Final

3. DATES COVERED:

September 2001  September 2002

4. TITLE AND SUBTITLE: 5a. CONTRACTOR OR GRANT NUMBER:

Utility of Game Instructions

5b. PROGRAM ELEMENT NUMBER:

0602785A

5c. PROJECT NUMBER:

2O262785A790

5d. TASK NUMBER:

233

6. AUTHOR(S):

Jessie Y. C. Chen (Consortium of Universities of the

Washington Metropolitan Area)

5e. WORK UNIT NUMBER:

H01

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social Sciences

ATTN: TAPC-ARI-IF

5001 Eisenhower Avenue

Alexandria, VA 22333-5600

8. PERFORMING ORGANIZATION REPORT NUMBER:

10. MONITOR ACRONYM:

ARI

9. SPONSORING / MONITORING ABENCY NAME(S) AND ADDRESS(ES):

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue

Alexandria, VA 22333-5600

11. MONITOR REPORT NUMBER:

Technical Report 1135

12. DISTRIBUTION / AVAILABILITY STATEMENT:

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES:

14. ABSTRACT (Maximum 200 words):

Due to the increasing usage of computer games for military training purposes, it is more important than ever

to understand how computer games can be utilized in an effective and efficient manner. One important issue

facing the military training community is that training time is at a premium and trainees need to be able to

play the game within as short a time as possible so they can start using the game to train the skills of interest.

This report describes an experiment that examined various game-related performance measures and

concluded that the two different instructional techniques (i.e., computer-based tutorials and game tips)

appeared to be effective in different ways, and players with access to both learned the game most effectively.

To be more specific, computer-based tutorials appeared to be more beneficial for motor elements such as

maneuvering and actions. On the other hand, those who had access to game tips performed better in cognitive

segments of the game such as setting up game plans and familiarity with the game interface. These results

can be incorporated in military training programs where computer games are part of the curriculum. Future

military game development can also utilize these results to determine which type of instructional material to

be included in the games.

15. SUBJECT TERMS:

Computer games, computer-based tutorial, game tips, instructions

SECURITY CLASSIFICATION OF

16. REPORT

Unclassified

17. ABSTRACT

Unclassified

18. THIS PAGE

Unclassified

19. LIMITATION OF

ABSTRACT

Unclassified

20. NUMBER OF

PAGES

38

21. RESPONSIBLE PERSON

(Name and Telephone Number)

Dr. Stephen L. Goldberg

407/384-3980
ii
Technical Report 1135

Utility of Game Instructions

Jessie Y. C. Chen

Consortium of Universities of the Washington Metropolitan Area

Consortium Post-Doctoral Fellows

Simulator Systems Research Unit, Orlando FL

Stephen L. Goldberg, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

April 2003

Army Project Number Personnel Systems

2O262785A790 and Performance Technology

Approved for public release; distribution is unlimited.

iii
iv
FOREWORD

The U.S. Army has shown increasing interest in utilizing computer games for training

purposes. Some games have been modified from existing COTS (commercial off-the-shelf)

wargames and some are being developed exclusively for various Army training activities. An

important issue facing the military training community is that training time is at a premium and

trainees need to be able to play the game within as short a time as possible so they can start using

the game to train the skills of interest.

The goal of this research is to examine the usefulness of various tutorial techniques used

in current COTS computer games in helping novice players reach an acceptable level of skill

where they could play the game effectively. The focus of this research effort is to investigate

how two often-used instructional media (i.e., game tips and computer-based tutorials) affect the

acquisition of basic computer game-playing skills. Computer-based tutorials appeared to be

more beneficial for motor elements such as maneuvering and actions. Those who had access to

game tips performed better in cognitive segments of the game such as setting up game plans and

familiarity with the game interface. These results can guide the introduction of computer games

in military training programs. Future military game development can also utilize these results to

determine which type of instructional material to be included in the games.

The work described here is a product of the newly established post-doctoral program of

the Consortium of Universities of the Washington Metropolitan Area. The U.S. Army Research

Institute for the Behavioral and Social Sciences, Simulator Systems Research Unit, supervised

this research effort. The findings were briefed to the cooperating agency, the ADL (Advanced

Distributed Learning) lab of the Simulation Technology Center of RDECOM (Research,

Development, and Engineering Command).

KATHLEEN A. QUINKERT

Acting Technical Director

v
ACKNOWLEDGMENT

This research could not have been accomplished without the guidance and mentoring by Drs.

Michael J. Singer and Stephen L. Goldberg. The author also wishes to express appreciation to

RDECOM (Research, Development, and Engineering Command) -Simulation Technology

Center and Mr. Bill Pike, in particular, for providing equipment and facility for this research

effort.

vi
UTILITY OF GAME INSTRUCTIONS

EXECUTIVE SUMMARY

Research Requirement:

The U.S. Army has made substantial efforts on computer games development for training

purposes over recent years. Some games are modified from existing commercial off-the-shelf

(COTS) computer games and some are developed exclusively for various Army training

activities. As computer games become more and more complex, it is necessary to examine

issues such as how these games, which increasingly employ complex plots and sophisticated

response mechanisms, can be included in Army training in an effective and efficient fashion.

Since training time is at a premium, trainees need to be able to progress from learning the game

to using the game to achieve training objectives within a short time. The goal of this research is

to examine the usefulness of various techniques used in current COTS in helping novice players

reach an acceptable level of skill where they could play the game effectively. The focus of this

research effort is to investigate the effectiveness of two often-used instructional media, game

playing tips and computer-based tutorials (CBT), for the acquisition of basic computer gameplaying skills.

Procedure:

In this experiment, novice players went through one of the following training regimes for

learning a first-person-shooter game: both types of instructions (i.e., game tips and CBT), only

one type of instructions (tips or CBT), or no instructions. Training effectiveness was evaluated

by testing the participants, after three hours of training, on the game interface as well as game

playing. More specifically, the training outcomes were defined as how sophisticated the

participants game plan was, how familiar the participant was with the game interface, the game

scores, and whether critical commands were incorporated in the game plan.

Findings:

Overall, the results support three conclusions: (1) a combination of both tips and CBT are

the most effective in enhancing overall game performance compared to the tips-only, CBT-only,

and the free play (No Training) conditions; (2) game playing tips are effective in training novice

player to become familiar with various aspects of game playing, including game interface and

how to set up more elaborate game plans; (3) CBT can be effective in enhancing novice players

game scores.

vii
Utilization of Findings:

Army training that incorporates PC games will proceed most rapidly by introducing the

game through highly structured computer-based instruction combined with goal and strategy

oriented tips. The experimental results reported in this study should be used by Training

specialists to select the more appropriate instructional media. Future military game development

should utilize these results to guide the development of instructional material to be included in

the games used for Army training.

viii
UTILITY OF GAME INSTRUCTIONS

CONTENTS

Page

INTRODUCTION................................................................................................................. 1

Computer-based Instruction............................................................................................. 2

Part-task vs. Whole-task Training.................................................................................... 2

Video Game Performance Training................................................................................. 3

Military Games................................................................................................................ 4

Current Study................................................................................................................... 6

Hypotheses....................................................................................................................... 7

METHOD.............................................................................................................................. 7

Participants...................................................................................................................... 7

Materials.......................................................................................................................... 8

Procedure......................................................................................................................... 8

Measures.......................................................................................................................... 9

RESULTS.............................................................................................................................. 10

Qualitative Data............................................................................................................... 14

DISCUSSION........................................................................................................................ 15

Implications for Military Training................................................................................... 17

REFERENCES.......................................................................................................................19

APPENDIX A. Group matching and power analysis........................................................ A-1

B. Questionnaire on Computer/Gaming Experience..................................... B-1

C. Quiz on Game Interface............................................................................ C-1

D. Usability Questionnaire............................................................................ D-1

E. Participants Comments............................................................................ E-1

ix
CONTENTS (continued)

Page

LIST OF FIGURES

Figure 1. PC game playing instructional approaches ............................................................ 5

2. Two dimension Training Scale presentation of experimental groups .................. 6

3. Screen shots from Rogue Spear® .......................................................................... 8

4. Average number of Special Functions employed in the plan for the four groups.. 11

5. Mean scores of Quiz on the game interface for the four groups............................ 12

6. Average number of Kills (game scores) for the four groups.................................. 12

7. Average Special Functions, Quiz scores, and number of Kills (game scores)

for the four groups.................................................................................................. 13

8. Relationships between Special Functions and Tips for frequent FPS game

players and infrequent players................................................................................ 13

9. Relationships between Quiz scores and Tips for frequent FPS game players

and infrequent players.............................................................................................14

10. Percentages of participants who included complete Escort command vs.

those who included incomplete Escort command for the Tips/non-Tips groups...14

x
Introduction

The utility of commercial off-the-shelf (COTS) personal-computer (PC) games for

military training has been explored over the years. One of the first efforts was the Marine Corps

report on assessment of COTS wargames and the subsequent development of Marine Corps

Doom® (Loughran, 1999). Although Marine Corps Doom® was never widely implemented in

formal training programs, the Marine Corps has continued to investigate the value of using PC

games for training (e.g., Marine Expeditionary Unit 2000®). The Army is also evaluating the

merits of PC games for training purposes, which range from adapting PC games for inclusion in

courses (e.g., SPEARHEAD® application within the Armor Captains Career Course at Ft. Knox)

to investigating first-person shooter games for cognitive skill practice and doctrinal correctness

(Tarr, Morris, & Singer, in preparation). Some games are modified from existing COTS

wargames and some are developed exclusively for various Army training activities. One of the

examples of modified COTS game is FBCB2/SPEARHEADII®, which was extended from

SPEARHEAD II® (co-developed by MAK Technologies and Zombie), an M1 Abrams tank

simulation wargame. MAK Technologies is also developing a tactical decision-making game,

BC2010®, for training future Army Brigade and Battalion commanders. One of the Armys

most recent wargame development efforts is the joint-venture among the Program Executive

Office- Simulation, Training, and Instrumentation (PEO STRI), Institute for Creative

Technologies (ICT) of University of Southern California and a team of game-making companies

(Shachtman, 2001). Two games, Full Spectrum Command® and Full Spectrum Warrior®, will

be available in the near future, and both employ urban warfare settings (for more information,

see the ICT site at http://www.ict.usc.edu/disp.php?bd=proj_games). These games, potentially,

can be used to train complex military leadership tasks ranging from rescuing hostages to dealing

with social/political issues.

Commercially available PC games are generally designed to attract customers; beating

the game is the main challenge and reason for playing. It is not unusual for players to spend

long hours playing the game before an adequate level of proficiency can be achieved. For

military training purposes, however, excessive difficulty in initially learning a games interface

and rules may degrade or prohibit its usefulness as a tool for acquisition and/or skill maintenance

since training time is at a premium in the military. If the user can learn to play the game quickly

and efficiently, they will be able to progress from learning the game to using the game to achieve

training objectives within a shorter time. For the military, the purpose of using COTS PC games

is to both challenge users and provide a tool for acquiring or practicing needed skills. One basic

issue in the application of every PC game is ease of acquisition of game skills vs. military skills.

In other words, how much time and effort must be spent learning the game interface before being

able to exercise the PC game in meeting Army training requirements. Therefore, a careful

analysis of the game operation training available in PC games is needed if they are to be

effective instruments for military training.

Literature from video game-related performance suggests that game knowledge and

game-related motor skills play an important role in determining game performance (Baba, 1993).

Baba found that, especially early in learning, game specific knowledge was more important than

motor-skill training as a determinant of game performance. Novice players who were trained

with game-playing tips performed significantly better than did those without the knowledge.

1
Although game-related motor skill training was not found to be as effective as knowledge

training in Babas studies, where a maze-running game similar to the popular Pac Man® game

was employed, it is questionable if the same results would be found with different types of

games. As computer games become more and more complex, it is necessary to examine whether

findings from studies employing simplistic games are applicable to those current games with

much more complex plots and sophisticated response mechanisms.

The goal of this research is to examine the usefulness of various tutorial techniques used

in current COTS in helping novice players reach an acceptable level of skill where they could

play the game effectively. The focus of this research effort is to investigate the effectiveness of

two often-used instructional media (i.e., game playing tips and computer-based tutorials) for the

acquisition of basic PC game-playing skills. This effort will examine different combinations of

instruction with a goal of reducing time required for the user to achieve a sufficient level of

proficiency to be able to play the game. Literature in instructional media and training strategies

has been reviewed and will be briefly discussed next.

Computer-based Instruction

Literature in instructional media has not provided a consistent conclusion regarding the

superiority of instructional effectiveness of one certain medium over other media, and few valid

guidelines are available for instructional designers (Swezey & Llaneras, 1997). However, a

substantial number of studies have demonstrated that innovative instructional technologies such

as computer-based instruction (CBI) can be effective in enhancing student performance and

reducing time required for training. For example, Kulik and Kulik (1991) reviewed 254

controlled studies in a meta-analysis on CBI effectiveness and reported a moderate but

significant positive effect (i.e., improvement of .30 standard deviations) of CBI on student

performance. All studies reviewed involved teaching in real classrooms, and the outcome

measure employed most often was achievement examination administered at the conclusion of

the CBI training program. In 29 out of 32 studies, in which instructional time was compared,

results showed that CBI generally saved about one-third of instructional time compared with

conventional instructional techniques (Kulik & Kulik, 1991). In another meta-analysis of 28

military-training related studies by Fletcher (1990), the result shows that interactive video-based

instruction improved student achievement by about 0.50 standard deviations over more

conventional instructional means (e.g., text, lecture, on-the-job training, videotape). Another

meta-analysis of 63 studies was conducted by McNeil and Nelson (1991) on the effects of

interactive video on learning, and an average of 0.53 standard deviations improvement in

performance was reported. In the context of PC game playing, the above research findings

suggests that computer-based tutorials may be more effective in training players to play a PC

game compared with other passive media such as a user manual or paper-based tutorial, which is

employed in many PC games developed for military training.

Part-task vs. Whole-task Training

Literature in training procedural and psychomotor tasks was also reviewed, since most

PC/video game playing involves both knowledge and perceptual-motor execution of game

strategies (Baba, 1993). A general finding from this literature is that complex tasks involving

2
procedural items or perceptual-motor manipulations greatly benefit from part-task training

(Kirlik, Fisk, Walker, & Rothrock, 1998). Kirlik and his colleagues suggest that beginning a

training program with part-task training and then proceeding to dual-task or whole-task training

may be the most efficient training method (p. 95). In the context of PC game playing, part-task

or segmentation training is utilized in the computer-based tutorials in many commercial games.

Those tutorials typically select a number of difficulty elements in the game and suggest that

players practice those elements before they play the game. From the standpoint of skill

acquisition, opportunities to practice critical/difficult elements of the games should be more

effective in training the players compared with simply consulting the user manual or playing the

game without the tutorial. Additionally, games that do not have computer-based tutorials cannot

provide their users opportunities for part-task training and have to be played in their entirety,

which is difficult for some novice players even with the game difficulty adjusted to an easier

level.

A series of studies, also known as the Learning Strategies Project, were conducted using

Space Fortress®, a video-game-like research tool (Donchin, 1995). Some of these studies

investigated the differential utility of part-task vs. whole-task training (Fabiani, Buckley,

Gratton, Coles, Donchin, & Logie, 1989; Frederiksen & White, 1989; Gopher, Weil, & Siegel,

1989; Mané, Adams, & Donchin, 1989). A predominant finding from these studies is that parttask training is superior to whole-task training in terms of enhanced game performance and

reduced training time. For example, Fabiani et al. demonstrated that different game playing

strategies were developed depending on the type of training received. The authors compared the

strategies used by the control subjects, who only practiced the game in a whole-task fashion,

with those developed by the experimental subjects, who received part-task game-element

training. They found that the control subjects strategies were qualitatively different from those

developed by their experimental counterparts, and the control subjects less efficient strategies

resulted in inferior game performance. In addition, three of the Learning Strategies Project

studies (Fabiani et al., 1989; Frederiksen & White, 1989; Gopher et al., 1989) report that lowability subjects benefited more from the part-task regimes than high-ability subjects in improving

their game performance. In fact, low-ability experimental subjects in Frederiksen and Whites

study performed almost as well as the high-ability control subjects playing the criterion game

following the completion of training. The high ability subjects did not receive any part-task

training other than practicing the game. Therefore, Frederiksen and White suggest that welldesigned part-task training may substantially reduce if not eliminate differences in training

performance associated with starting ability (pp.110-111).

Video Game Performance Training

One study from the Learning Strategies project (Foss, Fabiani, Mané, & Donchin, 1989)

investigated the performance of the control group participants, who were only given standard

game instructions without any explicit training regimes. They found that although participants

were able to continuously improve on game performance, there were systematic individual

differences among participants in the strategies they adopted. These different strategies had a

significant impact on their game performance and the shapes of their learning curves. In other

words, those participants who figured out an effective way of playing the game (Space

Fortress®) were able to achieve a better learning outcome than those whose strategies were less

3
useful. Another study from the same project (Shapiro & Raymond, 1989) examined whether

efficient oculomotor movement, which expert game players tend to use, could be trained to

enhance game performance. The results showed that the group receiving efficient oculomotor

training achieved a significant higher game score than did the group receiving inefficient

oculomotor training or the control group, who simply practiced the game. Since novice players

tend to use inefficient oculomotor movement if not trained otherwise, it was concluded that

efficient eye movement training could benefit novice players in improving their game

performance.

Baba (1993) compared the effectiveness of different instructional techniques for

enhancing PC game performance. She found that novice game players coached with game

specific knowledge/strategies (i.e., game playing tips provided by experts) performed much

better and had much steeper learning curves compared with those who were not coached, in

playing a maze-running video game, Lady Bug®, similar to the popular Pac Man® game.

Game-related motor skill training did not contribute as much to game performance. Essentially,

those who received both game knowledge and motor skill training did not perform better than

those who only received game knowledge training only. However, both groups performed

significantly better than those who received motor skill training only, and better than the control

group, who did not receive any training. The latter groups never discovered some crucial gamewinning strategies on their own, even over the course of 50 game sessions. A related study from

the Learning Strategies Project (Newell, Carlton, Fisher, & Rutter, 1989) also examined different

training approaches to enhancing video game (Space Fortress®) performance. The results are

consistent with Babas findings and show that information regarding procedural game strategies

is more beneficial than practice of the motor components.

The studies cited above demonstrate the importance of game playing knowledge and

strategies in novice players game skill acquisition. The authors also provide some suggestions

regarding instructional approaches based on their findings. For example, Newell et al. (1989)

suggest that optimal approaches to instruction intervention must combine...both the part training

of the response dynamics with the development of the appropriate task strategy for completing

the task at hand (p. 214). However, for games different from those employed in these two

studies, it still remains unclear whether the combined instructional approach (i.e., game

knowledge/strategies plus part-task) is superior to the situation where only one or neither of the

training is available.

Military Games

PC games that are being used or developed for use at the Advanced Distributed Learning

(ADL) lab of the Simulation Technology Center of Research, Development, and Engineering

Command (RDECOM) were evaluated based on the characteristics of their instructions for game

playing. The evaluation showed that PC games developed for military use (e.g., BC2010®,

FBCB2/Spearhead II®) generally lacked computer-based tutorials, which were available in many

commercial PC games. Some military-developed games (e.g., TacOps®) used paper-based

tutorials or very rudimentary computer-based tutorials, which did not provide opportunities for

practice. As for user manuals, those provided in commercial PC games tended to be more

detailed while those of military games varied in their degrees of detail. Finally, none of the

4
military-developed games provided game-playing tips, although some games had brief paperbased walkthroughs in their user manuals. For the two military-developed games available at the

ADL lab (i.e., BC2010® and TacOps®), no game-playing tips were found on any military or

game-related web sites. For popular commercial games, on the other hand, game-playing tips

could be found on many game-related web sites and in video game magazines.

Based on the results of the above evaluation, a multi-dimensional scale (Figure 1) was

developed to describe PC game playing instructional approaches and is depicted below (i.e.,

availability of computer-based tutorial, degree of detail of user manual and availability of game

playing tips). Four games available at the ADL lab are included here based on the initial

examination of their instructional approaches.

Computer

-based

Tutorial

Comprehensive

Limited

Moderately

detailed

None

Rainbow 6-

Rogue Spear

TacOps

BC2 Spearhead 010

Manual +

Tips

Simple

manual

Detailed

manual

User Manual

None

Figure 1. PC game playing instructional approaches.

Based on the literature reviewed previously, computer-based tutorial with opportunities

for the player to practice critical/difficult elements of the game (i.e., part-task training) is

expected to be superior to paper-based tutorial or user manual in its training effectiveness.

Game-playing tips are also expected to be beneficial for training novice players. An antiterrorism PC game, Rainbow Six-Rogue Spear®, was selected as the experimental tool for its

availability of instruction components on both dimensions. The Secret Service and the Marine

Corps have been using Rogue Spear® for interactive training, and the Army is working on

linking One Semi Automated Force (OneSAF) with Rogue Spear® (Bill Pike, personal

communication). The modified Rogue Spear® mainly focuses on training decision-making

skills at the small-unit level (Kennedy, 2002). The Army is also evaluating the utility and

doctrinal correctness of Rogue Spear® for training purposes (Tarr, Morris, & Singer, in

preparation).

5
Current Study

An experiment was carried out to examine the effects of PC game instruction techniques

(i.e., computer-based tutorial and paper-based game playing tips) on training outcomes, which

was defined as players game performance after being exposed to various instructional methods.

The focus was on which combination of instruction techniques resulted in a better performance

in initial game playing.

The independent variable in this experiment was the type and extent of instruction for PC

game playing. Participants experience in first-person-shooter (FPS) game playing was used as a

covariate, since most FPS games share many similarities in game playing and, therefore, the

players experiences were expected to have an effect on their learning. There were three

experimental groups and a control group. The first experimental group, CBT&T (CBT and Tips)

received the computer-based tutorial to practice essential game elements and, in addition, had

access to the full-length user manual plus game playing tips. The second experimental group,

CBT, received the computer-based tutorial and had access to the full-length user manual but not

game-playing tips. The third experimental group, Tips, only had access to the full-length user

manual plus game playing tips and played the game instead of receiving any computer-based

tutorial. The fourth group, NoTraining, only had access to the full-length user manual and

played the game without any computer-based tutorial or game playing tips. The four groups are

depicted graphically below (Figure 2).

NoTraining

(Manual)

User M

+ Tips

User M anual

only

anual

User Manual

None

Full

None

CBT&T (CBT +

Manual + Tips)

CBT (CBT +

Manual)

Computer

-based

Tutorial

Tips (Manual +

Tips)

Figure 2. Two dimension training scale presentation of experimental groups.

The dependent variable was participants performance in the final test session after three

hours of training. It was determined that the performance measures would include participants

game scores, quiz scores on game interface, the number of special functions utilized in the game

test session, and whether participants game plans include a critical command Escort. (The

rationale for selecting these measures is presented in the Method section). Game scores might

not be entirely reliable for they tend to be contaminated by the artificial intelligence (AI)

employed in the game. In other words, some participants may have had higher game scores

simply because their plan called for more AI in their games. However, it was decided that game

6
scores should be one of the measures based on the rationale that they indicated how good the

participants plan was in addition to reflecting the action (i.e., FPS) component of the game.

Hypotheses

Based on the literature review and reasoning presented above, the following hypotheses

were generated,

The CBT&T group (CBT + manual + tips) will perform better than the Tips (manual + tips)

group, which will perform better than the CBT group (CBT + manual). The CBT group will

perform better than the NoTraining group (manual only).

These hypotheses are based on the research findings that part-task training is beneficial for skill

acquisition in complex tasks and CBTs are effective in enhancing student performance and

reducing training time. In addition, players who have access to game-playing tips are expected

to perform better than those who do not (Baba, 1993; Newell et al., 1989). Baba and Newell et

al. demonstrated that those subjects who had access to game-tips outperformed those who only

received game-related motor skill training. Therefore, the Tips (manual + tips) group was

expected to outperform the CBT group (CBT + manual).

Method

Participants

A total of 44 males recruited from the University of Central Florida participated in the

study. The ages of the participants ranged from 17 to 40 (M = 22.6, SD = 5.24). Participants

were compensated at a rate of $7.50 per hour or class credit for their participation in the

experiment.

Users experience in PC gaming, specifically FPS gaming, was expected to covary with

their game performance and, therefore, was included in the analysis. The procedure for

matching participants for each group based on their FPS game experiences and the power

analysis performed to obtain the appropriate size of each group are reported in Appendix A.

Based on the literature reviewed on gender differences in video game performance

(Sanchez-Ku & Arthur, 2000; Brown, Hall, Holtzer, Brown, & Brown, 1997), it was decided that

only male participants would be included in the experiment to prevent additional variance from

gender differences on dependent measures. Results from the studies by Brown et al. indicated

that male participants performed better in playing a video game than did the female participants

with a comparable video game experience level. Sanchez-Ku and Arthur evaluated female

participants performance in playing Space Fortress®, which has only been played previously by

male participants in the series of studies conducted in the Learning Strategies project (Donchin,

1995). Their results showed that their female participants performance was lower than that

obtained for males in previous studies, although both received the same training protocol and

their demographic characteristics and general attitudes toward video games were similar.

7
Materials

An anti-terrorism FPS PC game, Rainbow Six-Rogue Spear®, and its users manual and

training component were used. The CBTs consisted of skill training and course familiarization

training. Skill training covered drills of fire and movement, room clearing, hostage rescue, open

training, and terrorist hunt. Users were advised to select a skill set and practice it using various

course setting (shooting ranges, obstacle course, grenade practice, door breaching, etc.). Limited

on-line performance feedback was available in the tutorial. Game-playing tips were gathered

from several Rogue Spear®-related web sites. The tips included detailed game-playing

techniques (e.g., Dos and Donts) and mission walk-throughs with illustrations of screen shots

from the game.

The game consisted of two phases: Planning and Actions. During the Planning phase, the

player created a plan, in which elements such as Roster selection, Kit selection, Team set up, and

Waypoints planning were incorporated. The Waypoints planning was the critical part of the

Planning phase and determined the paths for the teams not controlled by the player (i.e., the

teams controlled by AI) and the orders for those teams. The game could be played in either

single-player or multiple-player mode. In this study, only the single-player mode was used. A

screen shot form the planning phase and a screen shot from the action (game) phase are

presented in Figure 3.

.

Figure 3. Screen shots from Rogue Spear® (left: planning phase; right: action phase).

Procedure

The experiment protocol was set up as follows:

Consent and Questionnaire on Computer/Gaming Experience. All participants listened to a

recorded message about the experiment procedure and then filled out the consent form. After

consenting to participate in the study, the participants filled out a questionnaire on their computer

and gaming background (see Appendix B).

8
Initial training according to group assignment (2 hours). Participants in the CBT&T and the

Tips groups were asked to start by studying the game playing tips (including paper-based walkthrough). The CBT&T group then completed the tutorial and the Tips group started playing the

first mission using the tips and user manual. The CBT&T participants were allowed to start

playing the same game mission with access to the tips and manual after they finished the tutorial.

The CBT group started by going through the tutorial and were allowed to play the game once the

tutorial was completed. The NoTraining group practiced playing the game during the entire

initial training session. The game included two components: Planning phase and Action phase.

In the Planning phase, the participant had to devise a plan according to the scenario briefing, and

the plan included team member selection, weapon selection, and waypoint planning, which

included Special Functions the participants thought to be beneficial for the outcome. In the

Action phase, the participant essentially executed the plan he devised and played the game in a

FPS fashion. All four groups had access to the user manual at all time during this phase, and

those who achieved satisfactory results on the first mission were allowed to go on to the

remaining missions. All games played in this segment were at the Recruit (easy) level.

Practice of game playing (1 hour). After the initial training, all participants continued practicing

the game at the Recruit level. All participants were allowed to consult their user manual while

playing, and the CBT&T and the Tips groups also had access to the game tips. The game

statistics were saved for analysis.

Quiz on game interface (15 minutes). All participants were administered a quiz with 10 short

questions on the game interface (see Appendix C).

Test on game playing (about 40 minutes). Participants were asked to complete two game tests,

which were the first mission played twice at the Veteran (intermediate) level. They had access to

the user manual, but the CBT&T and the Tips groups no longer had access to the game tips. All

game scores (including successful completions of games) and time spent on game playing were

recorded.

Survey and Debriefing (about 10 minutes). After the tests, all participants filled out a

questionnaire on the usefulness of the instructions they received, ease of use of the game

interface, and other lab-related issues (see Appendix D for a sample). During the debriefing

session, the participants were informed of the nature of the study and were also asked to recall

two of the most difficult areas in their game learning.

Measures

The following measures were employed in the study.

Special Functions. The number of special functions used in the Waypoints planning was used as

a dependent measure. A larger amount of special functions indicates a higher level of

sophistication of the plan. The special functions include: Go-codes, Rules of Engagement

(ROEs), Speeds, and actions such as Snipe, Cover, Defend, Flashbang usage, Frag grenade

usage, and Breaching charges.

9
Quiz scores. Participants quiz scores were used for measuring their familiarity with the game

interface. The quiz consisted of ten short-answer questions, covering both the Planning and

Action phases (see Appendix C). A full score would be 10 points.

Game Scores. The number of kills was the game score and it determined the outcome of the

game unless the participant successfully escorted the hostages to a safe zone (i.e., Extraction

Zone), in which case a full score of 15 points would be assigned. Although the game scores

partially depended on the AI, which in turn depended on the soundness of the plan, it was

decided that it could still provide a comprehensive indicator of both planning quality and game

skill.

Escort. As stated above, the game could also be won by escorting the hostages to the Extraction

zone instead of killing all the terrorists. Therefore, it was critical to include the Escort command

in the plan and plot the waypoints leading to the Extraction Zone so that the AI teams could

execute the plan accordingly. A plan without this order was considered incomplete. More

specifically, a plan was considered complete only if the Escort command was included and the

final waypoint was placed in the Extraction Zone.

Results

A two-factor (CBT x Tips) multivariate analysis of covariance (MANCOVA) was

performed to analyze the data with Special Functions, Quiz scores, and Game Scores as the

dependent measures and FPS Game experience as the covariate. An additional one-way analysis

of variance (ANOVA) and post-hoc tests were performed to examine the differences in the

dependent measures among the groups. Chi-square tests were used to determine if there was an

association between the instruction type (e.g., Tips or No Tips) and the number of participants

whose plans included the Escort command. Statistical significance is reported for all tests at a

probability equal to or less than .05. Descriptive summary (means and standard deviations) for

the first three measures are provided in Table 1.

TABLE 1. Descriptive Summary (Means and Standard Deviations) for Number of Special

Functions, Quiz Scores, and Game Scores

Special Functions Quiz Game Scores

NoTraining 2.36 (2.20) 4.73 (2.23) 10.45 (2.50)

CBT 2.36 (3.04) 5.32 (1.94) 11.64 (1.80)

Tips 4.91 (1.81) 6.82 (1.90) 10.73 (2.90)

CBT&T 5.45 (1.44) 7.32 (2.03) 12.82 (3.43)

Note. Standard deviations appear in parentheses.

10
The percentages of participants who included Escort command in their plans are

presented in Table 2.

TABLE 2. Percentages of Participants Who Included Escort Command in the Plan

NoTraining CBT Tips CBT&T

Percentage for

employing

Escort command

18% 9.1% 100% 82%

Levenes tests of equality of error variances were performed on the variances of the three

dependent measures to examine their homogeneity. The results did not show the variances of

any of the measures to be heterogeneous (all ps > .05). In addition, Boxs test of equality of

covariance matrices indicated that the observed covariance matrices of the dependent variables

were not significantly different across groups. Normality tests were performed on all three

dependent measures for all four groups. It was discovered that for Special Functions, two of the

distributions (i.e., NoTraining and CBT) were not normally distributed; for Game Scores, two of

the distributions (i.e., NoTraining and CBT&T) were not normally distributed. Since these were

only moderate departures from normality, it was decided that ANOVAs should be robust enough

for testing the differences (Howell, 2002). The kurtosis and skewness scores of all 12

distributions were within 2 and 2 except for the kurtosis score for Special FunctionsNoTraining, which was slightly lower than 2.

The main effect of Tips was significant, F(3, 37) = 7.23, p < .005. None of the other

main effect, the covariate, and interactions among the factors was statistically significant.

Further analyses reveal that groups with Tips employed significantly more Special Functions in

their plans than did the groups without Tips, F(1, 39) = 19.05, p < 001. Figure 4 displays the

average number of Special Functions employed in the plan for the four groups. Post hoc (LSD)

tests revealed that the differences between the NoTraining (mean = 2.36) and the Tips (mean =

4.91) groups as well as between NoTraining and the CBT&T (mean = 5.45) groups were both

statistically significant, p < .05. In addition, the differences between the CBT (mean = 2.36) and

the Tips groups was also statistically significant, p < .05.

CBT

Tips or NoTips

NoTips Tips

6.0

5.5

5.0

4.5

4.0

3.5

3.0

2.5

2.0

CBT or NoCBT

NoCBT

CBT

CBT

No Tips Tips

No CBT

Figure 4. Average number of Special Functions employed in the plan for the four groups.

11
The groups with Tips also scored significantly higher on their Quiz of game interface

than did the other two groups without Tips, F(1, 39) = 12.20, p < .005. Figure 5 shows the mean

scores (out of 10 points) of Quiz on the game interface for the four groups. Post hoc (LSD) tests

revealed that the differences between the NoTraining (mean = 4.73) and the Tips (mean = 6.82)

groups as well as between NoTraining and the CBT&T (mean = 7.32) groups were both

statistically significant, p < .05. In addition, the difference between the CBT (mean = 5.32) and

the CBT&T groups was also statistically significant, p < .05.

Tips or NoTips

N Tips

7.5

7.0

6.5

6.0

5.5

5.0

4.5

oTips

CBT or NoCBT

NoCBT

CBT

No Tips Tips

No CBT

CBT

Figure 5. Mean scores of Quiz on the game interface for the four groups.

The groups with CBT were found to have higher Game Scores than did the other two

groups without CBT training, although the differences were only moderate, F(1, 38) = 4.105, p =

.055. Figure 6 shows the average Game Scores (highest possible score being 15) for the four

groups. Post hoc (LSD) tests revealed that only the difference between the NoTraining (mean =

10.45) and the CBT&T groups (mean = 12.82) was statistically significant, p < .05. Figure 7

shows the average number of Special Functions, Quiz scores, and Game Scores for each group.

Figure 6. Average Game Scores for the four groups.

Tips or NoTips

NoTips Tips

13.0

12.5

12.0

11.5

11.0

10.5

10.0

CBT or NoCBT

NoCBT

CBT

No Tips Tips

No CBT

CBT

12
Group

NoTrain CBT Tips CBT+T

14

12

10

8

6

4

2

0

SpecialFunctions

Quiz

Kills

Game

Quiz

Special

Functions

Figure 7. Average Special Functions, Quiz scores, and Game Scores for the four groups.

The covariate, FPS Game experience, was found to be a significant factor for different

performances on Quiz, F(1, 39) = 5.30, p < .05. Figure 8 shows the relationships between

Special Functions and Tips for frequent FPS game players (those who play FPS games at least

once a month) and infrequent players (those who never or rarely play FPS games and those who

play FPS games once every few months). Infrequent FPS players who had access to game tips

employed more Special Functions (mean = 4.6) than did frequent FPS players who did not have

access (mean = 2.9), although the difference was not statistically significant, p > .05.

No Tips Tips No Tips Tips

Infrequent Frequent

Figure 8. Relationships between Special Functions and Tips for frequent FPS game players and

infrequent players.

Figure 9 shows the relationships between Quiz scores and Tips for frequent FPS game

players and infrequent players. Infrequent FPS players who had access to game tips had slightly

higher Quiz scores (mean = 6.2) than did frequent FPS players who did not have access (mean =

5.9), although the difference was not statistically significant, p > .05.

13
Figure 9. Relationships between Quiz scores and Tips for frequent FPS game players and

infrequent players.

Infrequent Frequent

No Tips Tips No Tips Tips

Escort Command

Chi-square tests revealed that the groups with Tips had a significantly higher tendency to

include Escort commands in their plans, x

2

= 16.6, p < .005. 91% of the participants who had

access to the game tips included critical commands such as Escort in their planning, while only

13.5% of those who did not have access to game tips employed the Escort command. Figure 10

shows that percentages of participants who included complete Escort command in their plans vs.

those who included incomplete Escort command for the Tips groups (i.e., Tips and CBT&T) and

non-Tips groups (NoTraining and CBT).

Tips or NoTips

NoTips Tips

100

80

60

40

20

0

Extraction

No

Co

Complete

Not

Comp Complete lete

Not Complete

No Tips Tips

t complete

mplete

Not Complete

Complete

Figure 10. Percentages of participants who included complete Escort command vs. those who

included incomplete Escort command for the Tips groups and non-Tips groups.

Qualitative Data

Participants comments as to the usefulness of the training session (i.e., practice, game

tips, and CBT), the amount of information during training, the ease of use of the program, and

14
their own performance during the game were obtained from the usability survey. A common

comment from all four groups was that the planning phase was complicated and assistance in

addition to the users manual would be very helpful. One of the NoTraining group participant

responded: I spent about an hour trying to get to the actual game playing because I didnt

realize that I was setting up a path on the floor plan for a wrong team. I never played this game

and could have used a little assistance. A CBT group participant responded: The CBT was a

great way to learn how to play. Another CBT group participant stated: In training section, the

game must show some useful windows and tips and how to command the teams. There is not

enough information available in the manual.... There must be a section where we can find the

basic steps that must be followed to achieve certain goals. Many Tips group participants

responded that the game tips were very helpful, and one stated: Walkthrough was short and

quick and got me playing quick! However, one Tips group participant responded: There

should be tutorial missions to orient the player to the game and controls. One CBT+T group

participant responded: The game tips were good and easy to understand. The time duration to

get acquainted with the CBT was short; perhaps a bit extra time should be given. The program is

very user friendly. These comments suggest that a combination of CBT and Tips might provide

the most comprehensive instructions for a novice player to get familiar with the game. The

comments are partially presented in Appendix E.

Discussion

The goals of the present research were to examine the effectiveness of PC game

instruction techniques (i.e., computer-based tutorial and game tips) for training novice players to

learn a FPS game. The training outcomes were defined as how many Special Functions were

used in the game plan, how familiar the participants were with the game interface (Quiz scores),

the Game Scores, and whether critical commands such as Escort were incorporated in the game

plan. Overall, the results support three conclusions:

a. a combination of both tips and CBT is the most effective in enhancing overall game

performance compared to the tips-only, CBT-only, and the free play (No Training)

conditions;

b. game playing tips (walkthroughs) are effective in training novice player to become

familiar with various aspects of game playing, including game interface and how to set

up more elaborate game plans by employing more Special Functions and game critical

commands (e.g., Escort);

c. computer-based tutorial (CBT) can be effective in enhancing novice players game

scores.

The findings of this research suggest that game tips and CBT are effective in quite

different ways. Game tips seem to be more effective than CBT in enhancing cognitive aspects of

game performances (e.g., planning and familiarity with game interface); while CBT, on the other

hand, is more effective in enhancing game scores. Participants who had access to the game tips

employed more Special Functions in their plans, indicating that their plans were more elaborate

than those of the groups that did not have access to game tips. In fact, infrequent FPS players

who had access to game tips employed more Special Functions than did frequent FPS players

who did not have access, although the difference failed to achieve statistical significance. From

15
military training perspective, however, these results indicate that game tips might be more

effective in teaching the trainees how to develop an elaborate plan for the game.

Over ninety percent of the participants who had access to the game tips included critical

commands such as Escort in their planning, while only 13.5% of those who did not have access

to game tips employed the Escort command. Further examination of the time spent reading

users manual revealed that those who did not have access to the game tips but employed the

Escort command spent an average of 38 minutes reading the users manual, while those who did

not have access to the game tips and did not employ the Escort command spent an average of 10

minutes reading the manual. These results suggest that the users were less likely to be aware of

the command if they did not spend more time reading the users manual.

Game tips were also found to be effective in enhancing Quiz scores. FPS experience was

also significantly correlated with Quiz scores. The latter result should not be surprising,

considering that most FPS games share similar command structures. Experienced FPS players

have the advantage of well-established schemas for game playing, while inexperienced FPS

players tend to learn the task/interface by using trial-and-error methods (Pillay, 2003). What is

worth noticing is that inexperienced FPS players who had access to game tips were able to

achieve Quiz scores that were at the same level as the experienced FPS players who did not have

access to the game tips. These results suggest that game tips might be useful in helping

inexperienced FPS players learn the game interface in a more efficient manner and, therefore, in

reducing training time.

CBT was effective in enhancing game scores, although the differences between CBT

groups (i.e., CBT and CBT&T) and non-CBT groups (i.e., NoTraining and Tips) were only

marginally significant. However, CBT&T group did have significantly higher game scores than

did the NoTraining group, indicating that a combination of CBT and game tips resulted in more

effective learning of game playing and higher game scores. Although the measure of game

scores was contaminated by the artificial intelligence component of the game engine, it still

provides a comprehensive indicator of game performance, including both the planning and the

action components. Presumably, CBT&T group had the highest game scores because they

benefited from both the game tips for the planning component and the part-task tutorials for the

action component.

Overall, the results of this research are consistent with some previous studies on video

game performance training and demonstrate that game tips and CBT can be effective in

enhancing game performance (e.g., Baba, 1993; Newell et al., 1989). In addition, our data show

that the effectiveness of game tips and CBT appears to be task-specific. Tasks involving mainly

cognitive components (e.g., planning and game interface) benefit from game tips while a

combination of game tips and CBT appears to be most effective in enhancing the overall game

performance (i.e., game scores). As stated in Newell et al., mere game scores may not provide a

sufficient basis from which to assess all aspects of game performance. Previous gaming studies

that only show advantages of game tips or part-task training may not have examined all the

aspects of the games or the games employed in the studies may be more cognitive-oriented or

motor-oriented. When both cognitive and motor components are critical for the game

16
performance, a combination of game tips and part-task training would probably be the optimal

approach. This conclusion is consistent with the suggestion by Newell et al.

Implications for Military Training

Due to the increasing usage of computer games for military training purposes, it is more

important than ever to understand how computer games can be utilized in an effective and

efficient manner. As stated previously, one important issue facing the military training

community is that training time is at a premium and trainees need to be able to play the game

within as short a time as possible so they can start using the game to train the skills of interest.

The results reported in this study suggest that the two different instructional techniques (i.e.,

CBT and game tips) seem to be effective in different ways, and players with access to both seem

to learn the game most effectively. To be more specific, CBT may be more beneficial for games

involving more maneuvering and actions such as those in FPS games; on the other hand, it may

be more advantageous to employ game tips for games with predominantly cognitive components

such as setting up game plans. These results can be incorporated in military training programs

where computer games are part of the curriculum. Future military game development can also

utilize these results to determine which type of instructional material to be included in the games.

17
18
References

Baba, D. M. (1993). Determinants of video game performance. In J. L. Starkes & F.

Allard (Eds.), Cognitive Issues in Motor Expertise (pp. 57-74). Amsterdam,

Holland: Elsevier Science Publishers.

Brock, J. F. (1997) Computer-based instruction. In G. Salvendy (Ed.), Handbook of

Human Factors and Ergonomics (pp. 578-593). NY, NY: Wiley and Sons.

Brown, R. M., Hall, L. R., Holtzer, R., Brown, S. L., & Brown, N. L. (1997). Gender and

video game performance. Sex Roles, 36(11/12), 793-812.

Donchin, E. (1995). Video games as research tools: The Space Fortress game. Behavior

Research Methods, Instruments, & Computers, 27(2), 217-223.

Eberts, R. E. (1994). User Interface Design. Englewood Cliffs, NJ: Prentice-Hall.

Fabiani, M., Buckley, J., Gratton, B., Coles, M. G. H., Donchin, E., & Logie, R. (1989).

The training of complex task performance. Acta Psychologica, 71, 259-299.

Fletcher, J. D. (1990). Effectiveness and Cost of Interactive Videodisc Instruction in

Defense Training and Education (IDA paper P-2372). Alexandria, VA: Institute

for Defense Analyses.

Fober, G. W., Bredthauer, J. L., & Dyer, J. L. (2000). The Computer Background of

Soldiers in Infantry Courses: FY99-00 (Research Report 1762). Alexandria, VA:

US Army Research Institute for the Behavioral and Social Sciences.

Foss, M. A., Fabiani, M., Mané, A. M., Donchin, E. (1989). Unsupervised practice: The

performance of the control group. Acta Psychologica, 71, 23-51.

Frederiksen, J. R. & White, B. Y. (1989). An approach to training based upon principled

task decomposition. Acta Psychologica, 71, 89-146.

Gopher, D., Weil, M., & Siegel, D. (1989). Practice under changing priorities: An

approach to the training of complex skills. Acta Psychologica, 71, 147-177.

Howell, D. C. (2002). Statistical Methods for Psychology (5th ed.). Pacific Grove, CA:

Wadsworth.

Kennedy, H. (November, 2002). Computer games liven up military recruiting, training.

National Defense, p. 58.

Kirlik, A., Fisk, A. D., Walker, N., & Rothrock, L. (1998). Feedback augmentation and

part-task practice in training dynamic decision-making skills. In J. A. Cannon19
Bowers & E. Salas (Eds.), Making Decisions Under Stress (pp. 93-113). Washington,

DC: American Psychological Association. Washington, DC: American Psychological

Association.

Kulik, C. C. & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An

updated analysis. Computers in Human Behavior, 7, 75-94.

Loughran, J. (1999). Simulation Across the Spectrum: The Island of Entertainment

Simulations. Retrieved Oct. 29, 2001 from Simulation Technology Magazine:

http:www.sisostds.org/webletter/siso/Iss_78/index.htm, June, 1999 (published by the

Simulation Interoperability Standards Organization).

Mané, A. M., Adams, J. A., & Donchin, E. (1989). Adaptive and part-whole training in

the acquisition of a complex perceptual-motor skill. Acta Psychologica, 71, 179-

196.

McNeil, B. J. & Nelson, K. R. (1991). Meta-analysis of interactive video instruction: A

10 year review of achievement effects. Journal of Computer Based Instruction,

18, 1-6.

Newell, K. M., Carlton, M. J., Fisher, A. T., & Rutter, B. G. (1989). Whole-part training

strategies for learning the response dynamics of microprocessor driven simulators.

Acta Psychologica, 71,197-216.

Pillay, H. (2003). An investigation of cognitive processes engaged in by recreational

computer game players: Implications for skilled of the future. Journal of Research

on Technology in Education, 24(3), 336-350.

Sanchez-Ku, M. L. & Arthur, W., Jr. (2000). A dyadic protocol for training complex

skills: A replication using female participants. Human Factors, 42 (3), 512-520.

Shachtman, N. (2001). New Army Soldiers: Game Gamers. Retrieved Oct. 29,

2001 from Wired News:

http://www.wired.com/news/conflict/0%2C2100%2C47931%2C00.html.

Shapiro, K. L. & Raymond, J. E. (1989). Training of efficient oculomotor strategies

enhances skill acquisition. Acta Psychologica, 71, 217-242.

Shavelson, R. J. (1996). Statistical Reasoning for the Behavioral Sciences (3rd ed.).

Needham Height, MA: Allyn and Bacon, Inc.

Swezey, R. W. & Llaneras, R. E. (1997). Models in training and instruction. In G.

Salvendy (Ed.), Handbook of Human Factors and Ergonomics (pp. 516-577).

NY, NY: Wiley and Sons.

20
Tarr, R. W., Morris, C. S., & Singer, M. J. (in preparation). Low-cost PC gaming and simulation:

Doctrinal survey. Alexandria, VA: U. S. Army Research Institute for the Behavioral and

Social Sciences.

21
22
Appendix A

Group matching and power analysis

Users experience in PC gaming, specifically first-person-shooter gaming, was expected

to covary with their game performance and, therefore, was included in the analysis. Based on

the information obtained from the U.S. Army (e.g., The Computer Background of Soldiers in

Infantry Courses: FY 99-00 by Fober, Bredthauer, & Dyer, ARI Research Report 1762), a

sample that approximated the range and level of experience in the U.S. Army was selected and

tested. The military data on soldiers' game experiences is as follows:

Daily 18%

Weekly 24%

Monthly 14%

<Monthly 22%

Never 22%

The Daily, Weekly, and Monthly categories was collapsed to form the Frequent players category

(56%); the <Monthly and the Never categories formed the Infrequent players category (44%).

Potential participants filled out questionnaires so their PC/video game and first-person-shooter

game playing experiences could be classified into these categories, and they could be assigned to

conditions to replicate the approximate distribution found in the Army today. However, due to

the extreme difficulty of recruiting college male students with limited experience in PC/video

games, it was decided that participants first-person-shooter game experience, instead of

PC/video game experience, would be used to match participants for the four groups.

A power analysis was performed to determine the minimum number of participants

needed to detect differences among the three experimental groups (Shavelson, 1996). Data from

Baba (1993) was used to estimate the necessary sample size based on the rationale that the

experimental design (i.e., independent and dependent variables) of her study was similar to that

proposed in this study. The four groups in her study (Control, Movement training, Game

Strategy training, and Movement plus Game Strategy training) are similar to the groups of this

current study (Control, CBT, Tips, and CBT + Tips, respectively). She used game scores as the

main performance measurement, which is slightly different from what will be used in this current

study (i.e., successful completions of game missions as well as game scores) but is not expected

to be greatly dissimilar. Therefore, it was decided that using Babas data should be adequate for

estimating the required sample size. In Babas study, there were 4 subjects in each group

(Control, Movement, Strategy, and Movement plus Strategy) and an estimated effect size of .94

standard deviation unit was obtained when the rate of improvement was compared among the

groups. The improvement scores were calculated by subtracting the pre-training game scores

from the post-training game scores. The effect size could only be estimated because all groups

were not given equal training times. Therefore, the rates of improvement and their standard

deviations could only be inferred from the data reported.

The power analysis revealed that 14 participants per group would be sufficient to obtain a

power of .80 (i.e., 80% chance of correctly rejecting the null hypothesis). However, due to the

difficulty of obtaining participants with limited gaming experience (i.e., Infrequent players), it

was decided that each group would include 11 participants. A total number of 44 participants

A-1 23
was selected based on the percentages in the five categories indicated previously but was

classified as one of the two categories (i.e., Frequent, Infrequent). These groupings were

employed to avoid low numbers of participants in the five categories and to increase the power

of statistical analysis. The 44 participants, then, were assigned to the three experimental groups

and the control group with a resulting assignment of 6 Frequent and 5 Infrequent participants in

each group. Participants were screened to ensure that they did not have prior experience in

playing Rogue Spear® or Rainbow Six®, to which Rogue Spear® is a sequel.

A-2 24
Appendix B

Questionnaire on Computer/Gaming Experience

Participant # _________ Age __________ Major _____________________

1. What is the highest level of education you have had?

Less than 4 yrs of college ____ Completed 4 yrs of college ____ Other ____

2. When did you use computers in your education? (Circle all that apply)

Grade School Jr. High High School

Technical School College Did Not Use

3. Where do you currently use a computer? (Circle all that apply)

Home Work Library Other________ Do Not Use

4. For each of the following questions, circle the response that best describes you.

a. Do you own a personal computer? Yes No

b. How often do you:

Use a mouse? Daily, Weekly, Monthly, Once every few months, Rarely, Never

Use icon-based programs/software?

Daily, Weekly, Monthly, Once every few months, Rarely, Never

Use programs/software with pull-down menus?

Daily, Weekly, Monthly, Once every few months, Rarely, Never

Use graphics/drawing features in software packages?

Daily, Weekly, Monthly, Once every few months, Rarely, Never

Use E-mail? Daily, Weekly, Monthly, Once every few months, Rarely, Never

Use the Internet? Daily, Weekly, Monthly, Once every few months, Rarely, Never

Play computer games? Daily, Weekly, Monthly, Once every few months, Rarely, Never

Play video games? Daily, Weekly, Monthly, Once every few months, Rarely, Never

Play first-person-shooter video/computer games?

Daily, Weekly, Monthly, Once every few months, Rarely, Never

5. Which of the following best describes your typing ability? (check one)

_____ Hunt and peck slowly

_____ Hunt and peck quickly

_____ Type slowly while not looking at the keyboard

_____ Type quickly while not looking at the keyboard

6. Which of the following best describes your expertise with computer? (check one)

_____ Novice

_____ Good with one type of software package (such as word processing or slides)

_____ Good with several software packages

_____ Can program in one language and use several software packages

_____ Can program in several languages and use several software packages

25 B-1
Appendix C

Quiz on Game Interface

1. 1. What are the steps for making Red teams member Bogart the leader of Blue team?

2. What are the steps for changing the primary weapon and assigning it to all team members?

3. How do you change the directions for orders such as Snipe and Cover?

C-126
For questions 4-10, use the screen shot below:

4. How do you select another weapon?

5. How do you zoom in/out the map?

6. What do the numbers 3 and 2 mean?

7. How do you peek?

8. How do you change magazine?

9. How do you change the Rule of Engagement (ROE) during mission?

10. How do you issue the Alpha Go-Code for Green Team?

C-227
Appendix D

Usability Questionnaire

Usability Questions

For the following, circle the number that best describes the degree to which you agree with each statement.

Training Session:

1. The training adequately prepared me for the test (playing the game in the test session).

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

2. The users manual was useful for helping me learn the game.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

3. The game tips (walkthrough) were useful for helping me learn the game.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

4. The computer-based tutorials were useful for helping me learn the game.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

5. There was too much information in the training session to learn all at once.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

D-1 28
Game Performance:

6. The program was easy to use overall.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

7. I was able to easily recognize objects in the game environment.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

8. I was able to identify the characters (teammates, terrorists, and hostages) without difficulty.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

9. I was able to move through the game environment without difficulty.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

10. The planning phase (Roster selection, Kit selection, and Waypoint plotting) is too

complicated.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

Lab Environment:

11. I was able to concentrate on my task without much distraction or disturbance.

1 2 3 4 5 6 7

Strongly

Disagree

Disagree Somewhat

Disagree

Neutral Somewhat

Agree

Agree Strongly Agree

Comment:

D-2 29
Appendix E

Participants Comments

NoTraining Group:

1. It took a long time to get used to the game, and I still do not know what a lot of it means.

Without the manual, I would have never been able to figure out the game.

2. A lot of things to remember. During gameplay, multiple things are going on at once, creating

a lot of confusing gameplay. The planning phase takes up so much time and it is extremely

confusing.

3. The users manual had all the information; there was just too much to take in at once.

Knowledge of the controls and having practice of using the controls is very different. There

are a lot of commands that are forgotten while playing. The movement commands are

uncomfortable to use (my hand was starting to cramp) and the movement of the sights was

too fast at times. The planning phase would become easy after using if awhile, but it is very

hard to pick up right away. It was also a little hard to tell the terrorists apart form certain

objects (lampposts, paintings, etc.).

4. I spent about an hour trying to get to the actual game playing because I didnt realize that I

was setting up a path on the floor plan for a wrong team. I never played this game and could

have used a little assistance. The manual helped me understand the overall picture of the

game but I mainly figured out how to play the game on my own. There could have been too

much information in the training session to learn all at once for a novice such as myself. The

key functions must be known in order to navigate during the game. The planning phase was

a little tricky and difficult at first but eventually I got accustomed to it.

5. The planning phase is near impossible! The section on planning in the users manual was

very unclear.

CBT group:

1. In training section, the game must show some useful windows and tips and how to command

the teams. There is not enough information available in the manual. Some important terms

must be explained, such as ROE. There must be a section where we can find the basic steps

that must be followed to achieve certain goals.

2. The training did not adequately prepare me for setting up missions.

3. An outline or table of contents would help. The CBT was a great example of game. I was

unaware of own team at first. I would like to the map to show the entire location during

game. During the planning phase, the waypoint plotting is too complicated; I hardly

followed through with it.

4. The CBT was a great way to learn how to play. The planning phase needs to be faster.

5. More training in the planning segment would have helped, I think. The planning phase needs

more explanation, but it isnt too complex.

6. There was not enough information in the training session. The planning phase should be

already set on certain levels.

Tips group:

1. The planning phase could use some work to make it easier to use.

2. The training material could have been more legibly written and the training time more

extensive. The control codes are not intuitive so it takes long to become intuitive with their

30E-1
use. Planning vs. executionnot sure how correlated (i.e., is there penalty for diverging

from plan?). Not being intimately familiar with firearms, the weapons are so much of a blur

that picking weaponry is pretty much guesswork (i.e., it helps but it takes a while to get

working knowledge to select confidently and appropriately and one doesnt know how if it

affects underlying algorithm, if at all). The game tips (walkthrough) were helpful. I was

frequently referring to movement codes while trying to watch for and fend off the bad guys!

Im not sure how the planning phase could be simplified but its definitely a good exercise to

have a plan to start from. The up and down stairs were hard to figure out.

3. There should be tutorial missions to orient the player to the game and controls.

4. The game tips (walkthrough) were very helpful. There was too much information in the

training session to learn all at once; it was confusing, and I did not master all the things.

During the game, it was easy to lose orientation.

5. During the training session, I learned much more than I knew, and was better prepared but

not fully. The walkthrough taught me everything I needed for a general understanding. I

would rather space out the time than do it all in one session. During the planning phase,

roster, kit and waypoint are okay but the alpha plans were confusing.

6. Itd be nice to have shortcut key that showed what victory conditions/successes were. The

users manual was only useful for the weapon specifications. Nobody likes reading

instructions; walkthrough was short and quick and got me playing quick! There was not too

much information in the training session; I built from session to session. Go-codes were

complicated at first.

7. During the training session, more actions and commands need to be shown. During the

planning phase, planning for actions and go-codes werent clear.

8. Best preparation is familiarity with a particular mission. I didnt look at the users manual;

quicksheets covered the important stuff. The walkthrough was good, but for a very straightforward and relatively simple mission.

9. During the training session, there was a lot of info but not too much. Set up the game was

difficult. The planning phase have a lot of objects and the go-codes were hard to understand

and use.

10. There are a lot of controls to remember. The game tips were very helpful. When using the

program, there was a lot to do all at once. The planning phase helps you to plan strategies.

CBT+T group:

1. I was not familiar enough with key-controls. The users manual helps, but a booklet with

explanations would be better. The game tips were useful, but only for the scenario played.

The CBT help if Id known how to use them properly. The planning phase is not too

complicatedproper planning leads to higher mission success.

2. The users manual was very helpful. The game tips were good and easy to understand. The

time duration to get acquainted with the CBT was short; perhaps a bit extra time should be

given. The program is very user friendly.

3. The users manual was okay except it was unnecessarily full of not really necessary info.

The game tips were good intro for the game and tutorial. Overall, the icon-based things

made the program easy but the planning part was difficult.

4. The summary sheet with the controls was very useful.

5. I did not know what to expect before I used the CBT. There was too much to remember

without playing the game first.

31 E-2