U.S. Army Research Institute

for the Behavioral and Social Sciences

Research Report 1865

Performance in Non-Face-to-Face Collaborative

Information Environments

Brooke B. Schaab, J. Douglas Dressel,

and Mark A. Sabol

U.S. Army Research Institute

Andrea L. Rittman

George Mason University

Consortium Research Fellows Program

January 2007

Approved for public release; distribution is unlimited.

20070402121
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the Department of the Army

Deputy Chief of Staff, G1

Authorized and approved for distribution:

SCOTT E. GRAHAM MICHELLE SAMS

Acting Technical Director Acting Director

Technical review by

Dan Horn, U.S. Army Research Institute

Gregory A. Ruark, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS,

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Research Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

January 2007 Final June 2006-December 2006

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Performance in Non-Face-to-Face Collaborative Information

Environments 5b. PROGRAM ELEMENT NUMBER

20262785A

6. AUTHOR(S) 5c. PROJECT NUMBER

A790

Brooke B. Schaab, J. Douglas Dressel, and Mark A. Sabol,

US Army Research Institute for the Behavioral and Social 5d. TASK NUMBER

Sciences; Andrea L. Rittman, George Mason University HO1

Consortium Research Fellows Program.

5e. WORK UNIT NUMBER

217

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

US Army Research Institute for the Behavioral & Social

Sciences, ATTN: DAPE-ARI-RK

2511 Jefferson Davis Highway

Arlington, VA 22202-3926

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

US Army Research Institute for the Behavioral & Social ARI

Sciences

2511 Jefferson Davis Highway 11. MONITOR REPORT NUMBER

Arlington, VA 22202-3926 Research Report 1865

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited

13. SUPPLEMENTARY NOTES

Subject Matter POC: Brooke B. Schaab

14. ABSTRACT (Maximum 200 words):

Using technology to obtain and process information requires training not only in human-computer interaction but also

in human-human-computer (collaborative) interaction. Warfighters must not only develop their own situational

awareness (SA), they must understand each others' SA (Pew, 1995). This common ground is what each

collaboration participant assumes about the others to ensure effective interactions (Ross, 2003; Wellons, 1993).

Communication is key. Collaborators must coordinate and share information; Collaboration influences military

operations at all levels. Technical interoperability is not enough to produce the synchronization required.

15. SUBJECT TERMS

Collaborative, situational awareness, leader development, information environments

SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES

16. REPORT 17. ABSTRACT 18. THIS PAGE Ellen Kinzer

Technical Publication Specialist

Unclassified Unclassified Unclassified Unlimited 26 (703)602-8047

Standard Form 298
ii
Research Report 1865

Performance in Non-Face-to-Face Collaborative

Information Environments

Brooke B. Schaab, J. Douglas Dressel,

and Mark A. Sabol

U. S. Army Research Institute

Andrea L. Rittman

George Mason University

Consortium Research Fellows Program

Leader Development Research Unit

Stanley M. Halpin, Chief

U. S. Army Research Institute for the Behavioral and Social Sciences

2511 Jefferson Davis Highway, Arlington, VA 22202-3926

January 2007

Army Project Number Personnel Performance

622785A790 and Training Technology

Approved for public release; distribution is unlimited.

iii
iv
PERFORMANCE IN NON-FACE-TO-FACE COLLABORATIVE INFORMATION

ENVIRONMENTS

EXECUTIVE SUMMARY

Research Requirement:

The U.S. Army is increasingly operating in a network centric environment where

Soldiers work together towards a common goal. Although the Soldiers are working

together, they may be miles or thousands of miles apart using digital systems to share

task information and communications. This collaboration of distal Soldiers each having

their own duties and tasks, raises many questions including: how knowledge of other

collaborative team members' tasks influences overall team performance; how mode of

communication influences performance, and how different types of communication are

associated with successful performance. The experimental research reported here

attempts to provide answers to these important questions.

Procedure:

Twenty-eight pairs of college students worked together in pairs over a computer

network to locate scud missile launchers in the computer-based game SCUDHunt. The

students operated from different rooms in a research suite; they were physically isolated

from each other. Each partner had a different array of intelligence gathering assets,

with different capabilities and reliabilities, to employ to locate the launchers. After

completing a demographics questionnaire, each isolated team member was taught the

rules of the game and the characteristics of their assets. After the first training module,

the team member completed a paper and pencil test on the training material and was

given immediate test feedback. Each team member then received a second training

module. For half the teams, this was a repeat of the first module in which each team

member learned about the assets he/she would manage in the game; the other half of

the teams learned about their partner's assets during this second training module.

These conditions were called Own and All respectively. Again, each team member was

given a test on the training material just presented and given immediate feedback.

Teams then practiced the game for one turn. Experimenters answered any questions

regarding operation of the game or the assets. Teams then played two 5-turn games of

SCUDHunt selecting the three most likely launcher locations at the end of each turn.

For one game each team communicated via voice over headphones (Oral) and used

text chat (Chat) via the computer for the other game. After each game, partners

completed a task load index. The number of launchers located on each turn was

recorded (quality score) as well as the number of map grid locations selected in

common by each partner (shared situational awareness).
Findings:

Method of communication had no significant (p>.05) effect upon either the quality

or the shared situational awareness scores. The training condition did influence

performance with the All teams locating significantly more launchers in game 2 than

Own teams which had a significantly higher level of shared situational awareness. The

Own teams also reported a higher workload after game 1. Analysis of the text

messages indicated greater success in locating launchers in those teams with the

highest frequency of messages in the communication categories regarding: game

situation, player status and non-task related/social interactions.

These results indicate that performance on collaborative tasks benefits when

collaborating participants are cross-trained to have a broader, system-wide view of the

entire situation. Additionally, this cross-training reduces the initial workload for the

cross-trained teams. This research also indicated identifiable communication patterns

related to higher performance. Understanding the types of communication that develop

more accurate situational awareness provides insights into how to train Soldiers to

communicate effectively in non-face-to-face environments.

Utilization and Dissemination of Findings:

These findings can be used by Army trainers and training developers when

designing or conducting training for collaborative network centric distal operations.

Emphasis must be placed on the value of cross-training and effective communication

among the collaborative team.

The findings of this research have been briefed and discussed at numerous

TRADOC, HEL, and JFCOM meetings and conferences.

vi
PERFORMANCE IN NON-FACE-TO-FACE COLLABORATIVE INFORMATION

ENVIRONMENTS

CONTENTS

Page

In tro d u ctio n ...................................................................................................................... 1

Research Concepts .................................................................................................... 2

Research Venue ............................................................................................. 2

Cognitive Task Analysis .................................................................................. 3

Com mon Knowledge ...................................................................................... 3

Com m unication Patterns and Mode ................................................................ 4

M e th o d ............................................................................................................................. 4

Participants ................................................................................................... 4

Instruments ..................................................................................................... 4

Questionnaire ...................................................................................... 4

W orkload ............................................................................................... 4

Gam e Description ...................................................................................... 5

Design ................................................................................................. 6

Procedure ............................................................................................. 6

Results and Discussion ............................................................................................. 7

Performance by Mode of Com m unication ....................................................... 7

Performance by Training Condition .................................................................. 7

W orkload ........................................................................................................ 8

Age, Com puter Experience, and Gender Differences ...................................... 9

Com m unication Patterns ................................................................................ 11

Conclusions .................................................................................................................. 13

References ................................................................................................................... 15

A p p e n d ix A ................................................................................................................... 1 7

A p p e n d ix B ................................................................................................................. 1 9

vii
CONTENTS (continued)

List of Tables

Table 1. Experience with Web-based Gaming ......................................................... 10

Table 2. Experience with Computer/Video Games .................................................... 11

List of Figures

Figure 1. Symbols for air and ground assets controlled by participants ...................... 5

Figure 2. Quality score for participants trained in the All versus the Own condition ........ 7

Figure 3. Shared-situational awareness for participants trained in the Own versus the

A ll co nd itio n ................................................................................................ . . 8

Figure 4. Comparison of workload in the Own and All conditions .............................. 9

Figure 5. Survey results describing participants' computer experience ..................... 10

Figure 6. Number of communications exchanged compared to number of

S C U D s lo cated ......................................................................................... . . 12

viii
PERFORMANCE IN NON-FACE-TO-FACE COLLABORATIVE

INFORMATION ENVIRONMENTS

Technological innovations play a paradoxical role in military transformation. While

they help to resolve existing battlefield challenges on the one hand, they invariably

introduce new challenges on the other hand. Network-centric operations (NCO)

are just such an innovation.

(LTG W.S. Wallace, June 2005, p.2)

INTRODUCTION

Today's military operates in a network-enabled environment where technological

advances exploit gathering and sharing information to building shared-situational

awareness. These technologies have transformed the nature of traditional collaborative

work (Easley, Devaraj, and Crant, 2003). The research in this report helps us

understand what is different in these network-enabled environments.

Sonnenwald & Pierce (1998) observed that using technology to obtain and

process information requires training not only in human-computer interaction but also in

human-human-computer (collaborative) interaction. Warfighters must not only develop

their own situational awareness (SA), they must understand each others' SA (Pew,

1995). This common ground is what each collaboration participant assumes about the

others to ensure effective interactions (Ross, 2003; Wellons, 1993). Communication is

key. Collaborators must coordinate and share information (Hevel, 2002; Salas, Prince,

Baker, and Shrestha, 1995; Sonnenwald & Pierce, 1998). Griffin and Reid caution,

Warfare is not 'network centric'. It is either 'people centric', or it has not centre at all.

(Griffin & Reid, unpublished manuscript).

Collaboration influences military operations at all levels. Technical interoperability

is not enough to produce the synchronization required. Major General Gordon C. Nash,

Commander, Joint Warfighting Center, Director, Joint Training J7 states:

Collaboration capabilities can affect all aspects of joint operations.

... These capabilities are important to Department of Defense efforts to transform

the way we plan and execute joint operations. To accomplish this transformational

task, we must improve collaboration among combatant commands, Services,

agencies, and multinational partners. An environment of collaboration can

enable and integrate such a cooperative effort among these organizations and

help the joint force achieve decision superiority.

1
Information sharing while working towards a common goal is a key aspect of

collaboration (Alberts 2003). To gain insights into issues involving collaboration and

training, the U.S. Army Research Institute for the Behavioral and Social Sciences

(USARI) conducted observations and interviews of U.S. Army personnel who operated

digital systems incorporated into their units (Schaab & Dressel, 2003; Schaab, Dressel,

& Hayes, 2005). It soon became clear that classroom training on how to use digital

systems was not enough. Soldiers understood that job success requires an

understanding of how their system interacts with other systems. Developing both a clear

sense of how to collaborate with people operating those other systems and an

appreciation of how important collaboration is in achieving and maintaining situational

understanding requires experience in multiple training exercises that incorporate a

variety of scenarios. In one command center, Soldiers actually place two different

systems side-by-side and cross-train each other to promote collaboration. They

comprehended the need to understand the interrelationship between their roles. Such

opportunities to foster mutual understanding become more difficult, of course, when

members are dispersed. Alberts (2003) cautions that, without appropriate training and

practice, the network-centric environment might actually increase the fog of war rather

than provide superior situational understanding. To insure the latter result and avoid the

former, the training side of the military, in particular, needs to understand the dynamics

of this new environment, where warfighters increasingly interact with their peers and

leaders digitally.

To improve the warfighter's ability to perform effectively in this information-rich and

changing environment, the military needs to train personnel at all levels to do essential

coordination with others across horizontal and vertical networks. The current research

examines characteristics of network-collaborative environments (e.g., personnel

geographically dispersed, different knowledge sets, unacquainted) and how this

environment impacts performance. A second phase of this research will examine

different techniques to train participants to collaborate.

RESEARCH CONCEPTS

Researchers first selected a game that both allowed performance, using different

communication modes and knowledge sets, to be examined and the collection of

communication exchanges. Descriptions of the practical features of the game and

several relevant underlying research concepts follow.

Research Venue

SCUDHunt, a game developed by Thought link, INC, was selected for this research

on collaboration because it provided a simplified model of the interplay of shared

awareness and communication, while permitting independent manipulation of variables

thought to affect them. SCUDHunt requires participants to (1) collaborate from

distributed locations and (2) share unique information from their intelligence assets for

optimal game performance. The goal of the game is simple, to locate three SCUD

2
missile launchers on a map. The game requires geographically-dispersed players to

collaborate while executing digital tasks in order to achieve a shared goal.

Cognitive Task Analysis

The research included a cognitive task analysis of this game to identify critical points

where collaboration would be beneficial (Ross, 2003). In general, players need to

communicate planning strategies and to share gathered information in order to perform

effectively. The collaboration areas identified were:

Coordinating deployment: Players discuss where best to place their assets on the

map grid, with the goals of (1) maximizing coverage of the area remaining to be

searched, and (2) using certain assets to verify the results of earlier searches;

Interpreting results: Players discuss the reliability of reports from different

intelligence-gathering assets, leading to a determination of the likelihood that a SCUD

launcher is at any particular location. This involves interpretation of results from the

current turn, as well as integration of findings from previous searches.

Common Knowledge

This situation of networked individuals who have shared goals but unique roles and

responsibilities raises a new question for research on collaboration at a distance: To

what extent does knowledge about a partner's role influence an individual's

performance effectiveness? Knowledge about others' roles, responsibilities, and job

requirements has been termed "interpositional knowledge." One effective training

strategy for increasing interpositional knowledge among team members is cross-training

(Blickensderfer, Cannon-Bowers, & Salas, 1998).

Volpe, Cannon-Bowers, Salas, and Spector (1996) defined cross-training as a

strategy where "each team member is trained on the tasks, duties, and responsibilities

of his or her fellow team members" (p. 87). This involves having team members

understand and sometimes practice each other's skills. The Volpe et al. (1996) initial

research on cross-training, as well as an extension (Cannon-Bowers, Salas, &

Blickensderfer, 1998), showed that those team members who received cross-training

were better able to anticipate each other's needs, shared more information, and were

more successful in task performance. Additional research has found that cross-training

and a common understanding of roles contributes to shared mental model development,

effective communication, and improved coordination (McCann, Baranski, Thompson, &

Pigeau, 2000; Marks, Sabella, Burke, & Zaccaro, 2002).

Similarly, U.S. Army trainers found that Soldiers needed an understanding of how

their digital system interacted and complemented other Army systems (Schaab &

Dressel, 2003; Schaab, Dressel & Hayes, 2005). Computer systems training has been

modified to include an introductory module on the unique contributions of each system

and how they work together to achieve optimal situational understanding.

3
To investigate such issues, cross-training (called the All condition) versus intensive

training in one role (called the Own condition) was included as a variable in this

research. Players received either a double dose of training on the tasks they would

perform (deployment and interpretation of their own assets) or a single dose of training

on both their tasks and on the tasks of their partner; this is referred to as the All training

condition.

Communication Patterns and Mode

Recent research has shown that communication content (Urban, Weaver, Bowers, &

Rhodenizer, 1996) can influence team coordination and performance. To investigate

these relationships, researchers explored how the type of communication during a

collaboration effort was associated with success. A two tier communication

categorization was used to analyze text chat communication between players (see

Appendix A). The first tier consisted of 7 types of communication actions while the

second tier considered the game related object of the first tier action. For example, a

message could be categorized as a request for information (action) regarding asset

capability (object of action). More specifically, consider the communication, "How

reliable is your satellite in detecting a SCUD?" First tier (action) is a request for

information and the second tier (object of the action) is asset capability.

In addition, communication mode was manipulated. All pairs wore headsets that

allowed oral communication during one of the two games they played; during the other

game, participants communicated by sending typed messages via an on-screen "chat"

box. For a random half of the pairs, the "chat" game came first. The data analyzed

included measures of the types and frequency of communication between participants

in the "chat" game and game performance.

METHOD

Participants

Fifty-six undergraduate students, 31 females and 25 males, received course credit

for three hours of participation. Participants ranged in age from 18 to 35 years, with the

overwhelming majority between 18 to 25 years. Five participants had military

experience. Participants were paired for the research. Forty-two individuals forming 21

pairs had never met. The remaining participants were acquaintances or friends.

Instruments

Questionnaire: At the beginning of the session, participants completed a questionnaire

that requested demographic information (e.g., gender, age, and military experience) and

computer experience.

Workload: The NASA Task Load Index (TLX) (Hart & Staveland, 1988) is a multidimensional rating tool that provides workload scores based on six subscales: Mental

Demands, Physical Demands, Temporal Demands, Own Performance, Effort, and

4
Frustration. A definition of each subscale is provided in Appendix B. The NASA TLX

was administered to each participant after completing the first and second game of

SCUDHunt.

Game Description: The SCUDHunt game presents players with the mission of

determining where - on a five-by-five grid board representing the map of a hostile

country - the launchers for SCUD missiles are located. Participants are told that there

are three SCUD launchers, each in a different fixed location among the 25 squares on

the board. On each of five turns, participants deploy intelligence-gathering assets (for

example, a reconnaissance satellite or a team of Navy Seals), receive reports from

those assets, and create a "strike plan" (to be sent to their fictional commander)

indicating their independent best guesses based on all the information received both

from that turn and previous turns as to the SCUD launcher locations. They are told that

the final strike plans - after the fifth turn - will be used by their commander to direct an

attack on the SCUD launchers, and they are given the results of this final strike plan in

terms of which bombed location held a now-destroyed launcher. Participants control

either air or ground assets (see Figure 1) with each asset having unique capabilities and

returning intelligence reports of different reliabilities. For example, eyes on target reports

from Human Intelligence assets would be more reliable than reports generated from

satellites where sensors must interpret images from great distances. This game then is

a representation of a situation in which Soldiers would use digital systems to execute

tasks requiring collaboration for successful performance.

Air Control Ground Control

Satellite UAV HUMINT SEAL

COMINT

Manned Aircraft Special Operations

Figure1. Symbols for air and ground assets controlled by participants.

5
Measures generated as the game is played are 1) the number of SCUD launcher

locations identified, called quality score, 2) the degree to which the two participants on

a team chose the same grid squares or location in their independent strike plans, called

shared-situational awareness, 3) the number of text chat communications taking

place, 4) the categorized content of the text chat, 5) measures of subjective workload

reported on the NASA TLX, and 6) responses to questionnaire items on demographics

and computer experiences.

Design. Three primary independent variables for this experiment were training

condition, mode of communication and communication patterns. All versus Own

training conditions involved training on the characteristic of the information-gathering

assets used in the SCUDHunt game. Every participant received, as their first training

module, an explanation of the characteristics of the assets they would be controlling.

Half of the pairs (the Own condition) received a second exposure to the same asset

training; the other half (the All condition) received training in which each participant

learned the characteristics of the assets to be controlled by that participant's partner.

Mode of communication (Oral or Chat) was analyzed to determine if communication

mode influenced game performance.

Communication patterns, as described in Appendix A, were also evaluated to

determine if the frequency of certain types of communication were associated with

higher quality scores and increased shared-situational understanding.

Procedure. Upon arrival at the laboratory, individual participants were ushered into

separate rooms where they read and signed a standard consent form describing the

experiment and their rights as participants. This was followed by each participant

completing a questionnaire on demographics and their experience with computers and

computer games. Researchers then explained that the experiment involved participants

playing a computer game with a partner who was located in another room.

Several computer-based training modules were then presented on 1) the overall

aspects of playing the SCUDHunt game and 2) the characteristics of the informationgathering assets used in playing the game. Following each training module, participants

took paper and pencil quizzes on the material just presented and were given immediate

corrective feedback, if necessary, to ensure that they understood how to play the game

and the capabilities of their assets. After this training, the pair played a one-turn practice

game, to ensure that the mechanics of playing the game were understood. After the

experimenters answered any question the participants might have, the pair played two

complete five-turn games of SCUDHunt. During these games, data were automatically

collected on 1) the messages participants sent to each other, 2) the degree to which

grid squares chosen as targets in the "strike plans" (submitted at the end of each turn)

were identical for the two members of the pair, and 3) the number of those chosen

target squares that actually contained missile launchers.

6
RESULTS and DISCUSSION

Performance by Mode of Communication

All participants played the game in each of the two communication modes (Oral

or Chat) with the order of presentation balanced across pairs. Analyses of variance

revealed no significant effect (p>.05) of mode of communication upon either the number

of Scud launcher identified (the quality score) or the concordance of best guesses of the

partners (shared situational awareness).

Performance by Training Condition

Participants were trained either on both their tasks and their partner's tasks (All

condition) or they were trained twice on their own tasks (Own condition).

During the first of the two games played, no differences in performance were

seen between those trained in the All or the Own conditions in either quality score or

shared-situational awareness. In the second game, participants in the All condition

located significantly more SCUD launchers than did those in the Own condition (see

Figure 2), but those in the Own condition had significantly higher levels of sharedsituational awareness (see Figure 3). This means that participants who were crosstrained in both their role and in their partner's role were more successful in locating

SCUD launchers. Participants trained solely in their role achieved higher levels of

agreement with their partner on where they thought the SCUD launchers were located,

but were wrong more frequently than those who were cross-trained.

100 -

w Own

80- o All

, 60-

0

_j

4040

IL 20-

Game 1 Game 2

Figure 2. Quality score for participants trained in the All versus the Own condition.

Participants in the All condition located significantly more SCUD launchers in Game 2

than those in the Own condition (F (1, 54) = 4.435, p<.05).

7
0 2

Ur) 0 m Own

SI-1 All

=' 1.5

1

0

CU

2 0.5-

"ii

*0

(n Game 1 Game 2

Figure 3. Shared-situational awareness for participants trained in the Own versus the

All condition. Participants in the Own reported more SCUD launcher location in

common in Game 2 than did those in the All condition (F (1, 54) = 4.49, p<.05). No

locations in common =0; All 3 locations in common=2

Workload.

Participants completed the NASA TLX after game 1 and game 2. During game 1,

those in the Own condition reported higher levels of time demand (F (1, 53) = 11.162, p

< .05) and frustration (F (1, 53) = 6.275, p <.05) than those in the All condition (see

Figure 4). No differences in workload were reported during Game 2. This suggests that

participants in the Own condition initially experienced higher levels of workload than

those trained in the All condition.
20

18

U Own

16 fl All

•- 14 2n

11 12 0

C14

S10 0

,_j

II

S 8

0)

"" 6

4

2

0

Mental Demand Time Demand* Performance Effort Frustration*

Figure 4. Comparison of workload in the Own and All conditions. After Game 1,

participants in the Own condition reported significant higher levels of Time Demand and

Frustration than those in the All condition.

Age, Computer Experience, and Gender Differences

Fifty-one of the 56 participants (91 percent) were 25 years old or younger with none

of the participants over age 25 reporting "a lot" of experience with either computer/video

games or web-based gaming. Although differences in gaming experience were

reported, no significant differences in performance between "older" and younger players

were found. Comparisons between older and younger participants are tenuous

because these groups are dissimilar in size and the "older" group was between 26-35

years of age.

Relationships between computer experience in gaming and gender further our

understanding of how these variables interact. Most participants used computers for

email, searching the internet and instant messaging (see Figure 5). Thirteen of 56

participants (23 percent) reported "a lot" of experience with computer/video games,

while only 6 of the 56 (11 percent) had "a lot" of experience with web-based gaming. A

significant correlation was found (r-- -.264, p<.05) between experience with web-based

gaming and the number of SCUD launchers located in the first game played.

Participants with more web-based gaming experience actually performed worse during

the first game. This difference was not seen in the second game. It is possible that prior

gaming experience initially resulted in negative transfer to the new game. Additional

research would be needed to understand this relationship more fully. No other

significant correlations were found between any of the types of computer experience

reported and performance.

9
A lot 4-

Some 3-

A little 2

None 1---- r

Email Internet Instant Computer Web-based Distance

Messaging Gaming Gaming Learning

Figure 5. Survey results describing participants' computer experience.

Significant gender differences were found, with males reporting more experience in

the categories of web-based gaming (X,2 1, 3= 6.47) and computer/video games (X,2 .05, 3=

15.37) than did females (see Tables 1 and 2) Despite these significant differences in

gaming experience, no difference in performance, either in level of shared-situational

awareness or number of SCUD launchers located, was found between males and

females. Although males engage in more computer gaming activities than females,

these experiences did not lead to better performance on this game.

Additionally, when looking at males only, no differences in game performances were

found as a function of relative gaming experience. These findings imply that regardless

of prior gaming experience, people can benefit from training that incorporates serious

gaming activities.

Table 1. Experience with Web-based Gaming

Males (Percentage) Females (Percentage)

None 20 45

A little 44 36

Some 16 16

A lot 20 3

I0
Table 2. Experience with ComputerNideo Games

Males (Percentage) Females (Percentage)

None 0 13

A little 16 45

Some 40 36

A lot 44 6

Also, no significant differences were found in quality scores or shared-situational

awareness based on computer experience, military background, or acquaintance with

partner.

Communication Patterns

Players with a higher frequency of communications in the categories of game

situation, player status, and non-task related/social, identified a significantly greater

number of SCUD launchers (Game Situation F (3, 51) = 4.78, p<.05; Player Status (F

(3, 51) = 6.43, p<.05; Non-task Related/Social F(3, 51) = 8.166, p<.05). Examples of

these types of communication patterns follow.

Game Situation: The specific game context the players have created by their play of

the current game.

"Not much going on in E2." or

"Grid square C3 looks like a possible target."

Turn: 1 Phase: Search Plan History of Search Results

Status: Fos the UAV. click on any contiguous 5 cells,

stlating and ending at an edge. Thn 1

Saleel Md-•,•i.• lt UAV IN"£Seal: I b 41 Slet~ek Man-edAicrltj UAV I NalvSea4I Jokrt L

1 2 3 4 5 1 2 3 4 5

A Az

Search Results

B B

C 4 5I C Ci x

E E 00

$t Skip Turn pli BlkGafe Bo,••

11
Non-task Related Social: Social remarks not game related.

"Anyone

know a

good pizza

place?"

Interestingly, the second highest number of communications in each of these

three categories occurred when only one Scud launcher was located. This difference

was significant only for the Game Situation category (see Figure 6)

30 -

25

-o SCI

.2_ 20 Loc

200

E

E rl 0 15

0

,. 83J

E 10-

Z

a, B

0 - - ~ --

Player Status Game Situation Non-task Related

Social

Figure 6. Number of communications exchanged compared to number of SCUDs

located. Significantly more communications took place when all three SCUD launchers

were located.

12
CONCLUSIONS

These results demonstrate that collaborative tasks benefit when collaborating

participants are cross trained to have a broader, system-wide view of the entire

situation. Participants trained on both their role and their partner's role (All condition)

were more successful in locating SCUD launcher locations and, in Game 1, did so with

lower levels of Time Demand and Frustration than found in those trained solely on their

role (Own condition). In addition, while those in the Own condition identified more

suspected SCUD launcher locations in common (shared- situational awareness) than

those in the All condition, these concurrences were more likely to be incorrect locations.

Training Soldiers of the 2 1 st century to effectively communicate non-face-to-face in

order to develop accurate shared-situational awareness is essential in our transforming

military. This research implies that Soldiers achieve higher levels of accurate situational

awareness when they understand each other's role. Without this shared understanding,

Soldiers may agree on what is taking place, but this agreement is more likely to be

incorrect.

Participants with high levels of gaming experience performed similarly to those with

little or no experience. Males and females performed equally well despite males

reporting significantly more experience with gaming than did females. When looking at

males only, those with high levels of gaming experience did not perform any differently

than those with little or no experience.

Gaming, which was used in this research, is being investigated as a training tool

because it may motivate trainees to train more frequently and for longer periods.

Although additional research is needed to understand how to maximize the training

benefits of gaming, the current research suggests that learning benefits of serious

gaming may apply to both gamers and non-gamers.

Identifiable communication patterns are related to performance. Better

performers communicated more often. Communication categories related to higher

performance include player status, game situation, and non-game related social.

Understanding the types of communication that develop more accurate

situational awareness provides insights into how to train Soldiers to communicate

effectively in non-face-to-face environments. Good performers keep their partner

informed of their status and the current situation. Interestingly, better performers

engaged in higher level of non-game related social interaction. Perhaps these social

communications aid in building understanding and trust.

13
14
References

Alberts, D. S. (2003). Information age transformation: Getting to a 2 1st century

military. Washington, DC: DoD Command and Control Research Program.

Blickensderfer, E., Cannon-Bowers, J. A., & Salas, E. (1998). Cross-training and team

performance. In J. A. Cannon-Bowers & E. Salas (Eds.), Decision making under stress

Implications for training and simulation (pp. 299-312). Washington, DC: American

Psychological Association.

Cannon-Bowers, J. A., Salas, E., & Blickensderfer, E. (1998). The impact of crosstraining and workload on team functioning: A replication and extension of initial findings.

Human Factors, 40, 92-101.

Easley, R. F., Devaraj, S., & Crant, M. J. (2003). Relating collaborative technology use

to teamwork quality and performance: An empirical investigation. Journal of

Management Information Systems, 19, 247-268.

Griffin, R. E. (Lieutenant Colonel, CA AF) & Reid, D. J. (Australian MoD). A woven wed

of guesses, canto one. Network centric warfare and the myth of the new economy (p.

21). Unpublished manuscript. Cite in: Wallace, W. S. (May-June 2005). Networkenabled battle command. (pp. 2-5) Military Review.

Hart, S. G., & Staveland, L. E. (1988). Development of a multi-dimensional workload

rating scale: Results of empirical and theoretical research. In P. A. Hancock & N.

Meshkati (Eds.), Human Mental Workload. Amsterdam, The Netherlands: Elsevier.

Hevel, J. R. (2002). The Objective Force Battle Staff? Monograph, School of Advanced

Military Studies, U. S. Army Command and General Staff College, Fort Leavenworth,

Kansas.

Marks, M. A., Sabella, M. J., Burke, C. S., & Zaccaro, S. J. (2002). The impact of crosstraining on team effectiveness. Journal of Applied Psychology, 87, 3-13.

McCann, C., Baranski, J. V., Thompson, M. M., & Pigeau, R. A. (2000). On the utility of

experiential cross-training for team decision-making under time stress. Ergonomics, 43,

1095-1110.

Nash, G. C. (2005). Operational Implications of the Collaborative Information

Environment. JWFC Doctrine Pam 5.

Pew, R. W. (1995). The state of situational awareness measurement: Circa 1995. In

D.J. Garland & M. R. Endsley (Eds.), Experimental analysis and measurement of

situational awareness (pp. 7-15). Daytona Beach, FL: Embry-Riddle Aeronautical

University Press.

15
Ross, K.G. (September, 2003). Perspectives on Studying Collaboration in Distributed

Networks, Contractor Report prepared for the U. S. Army Research Institute for the

Behavioral and Social Sciences by Klein Associates Inc., Fairborn, Ohio, under Contract

PO DASW01-02-P-0526.

Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situational awareness in team

performance: Implications for measurement and training. Human Factors, 37, 123-126.

Schaab, B.B., & Dressel, J.D. (2003). Training the troops: What today's Soldiers tell us

about training for information age digital competency. (Research Report 1805).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Schaab, B.B., & Dressel, J.D. (2005). Digital Skills Training for Collaborative

Environments. Arlington, VA: U.S. Army Research Institute for the Behavioral and

Social Sciences.

Sonnenwald, D. H., & Pierce L. G. (1998). Optimizing collaboration in battalion staff

elements. (Contract Report ARL-CR-435). Chapel Hill, NC: University of North Carolina

at Chapel Hill.

Thoughtlink at www.thoughtlink.com

Urban, J.M., Weaver, J.L., Bowers, C.A., & Rhodenizer, L.A. (1996). Naturalistic team

decision making: Effects of structure and workload on team process and performance.

Human Factors, 38, 300-310.

Volpe, C. E., Cannon-Bowers, J. A., Salas, E., & Spector, P. E. (1996). The impact of

cross-training on team functioning: An empirical investigation. Human Factors, 38, 87-

100.

Wallace, W. S. (Lieutenant General, U.S. Army) (May-June 2005). Network-enabled

battle command. (pp. 2-5) Military Review.

Wellons, A. R. (1993). Group situational awareness and distributed decision making:

From military to civilian applications. In N.J. Castellan (Ed.), Individual and group

decision making: Current issues (pp. 267-291). Mahwah, NJ: Lawrence Erlbaum.

16
Appendix A

Communications Category Definitions

1 st Tier (Action)

Request information: To ask game-related questions.

Response to information request: To answer game-related questions.

Share information - unsolicited: To provide game-related information without being

asked.

Direct actions: To dictate the action of the partner.

WILCO: To comply with the directed action recommended.

Non-task related (social): Social remarks directed either toward the play of the game

(game related) or something other than the game (non-game related).

Equipment and Technical Problems: Refers to the operation of the experimental

apparatus rather than the play of the game.

2 nd Tier (Object)

Asset capability refers to how the asset can be moved (deployed) and the reliability of

its findings (search results).

-Can your assets be placed individually?

-Manned aircraft, they are excellent at detecting?

Player status refers to the condition of the player: is the player ready, has the player

deployed assets, has the search plan been submitted.

-Already submitted.

-There, I submitted my strike plan.

-My SpecOps is very accurate.

Game situation refers to the specific game context the players have created by their

play of the current game.

-D I looks like a target

-That's a place where there's not much going on

Search strategy refers to the plan of how assets can be deployed; i.e., placed on the

grid, to the best advantage for target detection.

-4 covered 2 sides of the square

-Where are you searching?

-For next turn, which cells do you want to take?

17
Strike plan refers to the plan of selecting which grid squares are most likely to contain

targets as a function of previous searches.

-4 will strike C1.

-Where are you thinking about striking?

Game rules/operation refer to the general rules which govern the play of the game

-This next turn is our last.

-It's only the last strike that matters.

Note: A communication may contain several components each of which needs to be

coded; however, each component will have only one code.

18
Appendix B

RATING SCALE DEFINITIONS

Title Endpoints Descriptions

MENTAL DEMAND Low/High How much mental and perceptual activity

was required (e.g., thinking, deciding,

calculating, remembering, looking,

searching, etc.)? Was the task easy or

demanding, simple or complex, exacting or

forgiving?

PHYSICAL Low/High How much physical activity was required

DEMAND (e.g., pushing, pulling, turning, controlling,

activating, etc.)? Was the task easy or

demanding, slow or brisk, slack or

strenuous, restful or laborious?

TEMPORAL Low/High How much time pressure did you feel due to

DEMAND the rate or pace at which the tasks or task

elements occurred? Was the pace slow and

leisurely or rapid and frantic?

EFFORT Low/High How hard did you have to work (mentally

and physically) to accomplish your level of

performance?

PERFORMANCE Good/Poor How successful do you think you were in

accomplishing the goals of the task set by

the experimenter (or yourself)? How

satisfied were you with your performance in

accomplishing these goals?

FRUSTRATION Low/High How insecure, discouraged, irritated,

LEVEL stressed and annoyed versus secure,

gratified, content, relaxed and complacent

did you feel during the task?

19