U.S. Army Research Institute 

for the Behavioral and Social Sciences 

Research Report 1849 

Global Teams: Enhancing the Performance of 

Multinational Staffs Through Collaborative 

Online Training 

Angela O’Dea, Karol G. Ross 

Anna McHugh, Jennifer K. Phillips 

Klein Associates

May H. Throne 

U. S. Army Research Institute 

Michael McCloskey 

361 Interactive LLC 

J. Andrew Mills

Continental Realty 

March 2006 

Approved for public release; distribution is unlimited.
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the Department of the Army 

Deputy Chief of Staff, G1 

Authorized and approved for distribution: 

 

 MICHELLE SAMS ZITA M. SIMUTIS 

 Technical Director Director 

Research accomplished under contract 

for the Department of the Army 

Klein Associates 

Technical review by 

Scott Beal, U.S. Army Research Institute 

 

Teresa Taylor, U.S. Army Research Institute

NOTICES 

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI. 

Please address correspondence concerning distribution of reports to: U.S. Army 

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS, 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer 

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

NOTE: The findings in this Research Report are not to be construed as an official 

Department of the Army position, unless so designated by other authorized documents
i

Continental Realty

150 E Broad Street 

Columbus, OH 44215 

361 Interactive, LLC 

408 Sharts Rd, Ste 7 

Springboro, OH 45066

U.S. Army Research Institute for the Behavioral and Social Services

ATTN: DAPE-ARI-IK 

121 Morande Street 

Fort Knox, KY 41021-4141 

REPORT DOCUMENTATION PAGE 

1. REPORT DATE (dd-mm-yy)

March 2006 

2. REPORT TYPE

Final 

3. DATES COVERED (from. . . to)

July 2004 – January 2006 

5a. CONTRACT OR GRANT NUMBER 

W74V8H-04-C-0036 

4. TITLE AND SUBTITLE

Global Teams: Enhancing the Performance of Multinational 

Staffs Through Collaborative Online Training 5b. PROGRAM ELEMENT NUMBER 

622785 

5c. PROJECT NUMBER 

A790 

5d. TASK NUMBER 

211 

6. AUTHOR(S) 

Angela O’Dea, Karol G. Ross, Anna McHugh, Jennifer K. Phillips 

(Klein Associates), May H. Throne (U.S. Army Research 

Institute), Michael McCloskey (361 Interactive LLC), and 

J. Andrew Mills (Continental Realty) 

5e. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER 

10. MONITOR ACRONYM

ARI 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

U.S. Army Research Institute for the Behavioral and Social Sciences 

ATTN: DAPE-ARI-IK 

2511 Jefferson Davis Highway

Arlington, VA 22202-3926 

11. MONITOR REPORT NUMBER 

Research Report 1849 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited. 

13. SUPPLEMENTARY NOTES 

Contracting Officer’s Representative and Subject Matter POC: May H. Throne 

This research was supported by The Technical Cooperation Program (http://www.dtic.mil/ttcp/). 

14. ABSTRACT (Maximum 200 words): 

This research report describes the research, development, and evaluation of a web-based, scenario-based training tool, 

designed to support the development of expertise in coordination and decision making for multinational forces in

coalition operations. The tool allows discussion and collaborative problem solving between at least two coalition 

partners, with the support of a facilitator. The project team used the operational experience of officers from Englishspeaking nations (U.S., U.K., Canada, and Australia to identify the cognitive challenges inherent in coalition operations

and to drive the development of context-rich scenarios. Evaluation of the training highlighted six critical factors which

impact the effectiveness of the training. 1. Clarify the learning objectives in advance. 2. Emphasize the problem 

solving and coordination aspects of the exercise. 3. Capitalize on the opportunity for interaction by allowing partners to

interact over discussion and problem solving. 4. Set the training at the appropriate level. 5. Use an experienced

facilitator to direct the training. 6. Tie tool functionality to the learning strategy. Overall, the training was shown to

improve participants’ awareness of, and ability to respond to, the key themes of coalition coordination. The tool 

provides an easy-to-implement and cost-efficient means for coalition partners to train in a distributed environment. 

15. SUBJECT TERMS 

Scenario-based training, web-hosted training, distributed training, coalition operations, multinational forces 

 SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

16. REPORT

Unclassified 

17. ABSTRACT 

Unclassified 

18. THIS PAGE

Unclassified 

ABSTRACT 

Unclassified 

OF PAGES

 43 

Ellen Kinzer 

Technical Publications

Specialist 

(703) 602-8047 

Klein Associates

1750 Commerce Center Blvd. N 

Fairborn, OH 45324-6362 
ii 
Research Report 1849 

Global Teams: Enhancing the Performance of Multinational 

Staffs Through Collaborative Online Training 

Angela O’Dea, Karol G. Ross

Anna McHugh, Jennifer K. Phillips 

Klein Associates 

May H. Throne 

U. S. Army Research Institute 

Michael McCloskey

361 Interactive LLC 

J. Andrew Mills

Continental Realty 

Armored Forces Research Unit 

Barbara A. Black, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences 

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926 

March 2006 

Army Project Number Personnel Performance 

622785A790 and Training Technology

Approved for public release; distribution is unlimited.

iii 
ACKNOWLEDGMENTS

The authors would like to thank all those who contributed to the development and 

implementation of the training method and materials used in this effort. We would especially 

like to thank the Joint Services Command and Staff College (JSCSC) in Wiltshire, United 

Kingdom. The former Commandant of the JSCSC, Major General Parker, provided instructors 

and students for our initial interviews, which led to a better understanding for us on how 

coalitions operate. The current Commandant of the JSCSC, Air Vice-Marshal Maddox, 

continued the support by providing us with participants in the final implementation of the 

training method. 

We would also like to thank the Commander of the Joint Readiness Training Center 

(JRTC) Operations Group, Colonel (P) Rounds, for providing us with officers with coalition 

experience to test our training materials and provide us with feedback. Their willingness to 

participate and provide us with comments and suggestions led to more accurate and useful 

training materials. 

Finally, Major Christopher Young, the Canadian Liaison Officer at Fort Knox, Kentucky 

contributed his time to provide us with his coalition experiences. He also was essential in 

finding Canadian Officers to participate in the final implementation of the training method. 

iv 
GLOBAL TEAMS: ENHANCING THE PERFORMANCE OF MULTINATIONAL STAFFS

THROUGH COLLABORATIVE ONLINE TRAINING 

EXECUTIVE SUMMARY 

Research Requirement: 

Developing and delivering high-quality training for the variety of contingencies that may 

arise in the contemporary operating environment is vital to preparing a stronger Future Force. 

Currently there is a need for more research to identify the key challenges in the operational 

environment and for the development of focused and relevant training that can support improved 

decision making and coordination. To prepare for the challenges of coalition operations, costeffective training is needed to build trust and improve decision making and coordination among 

coalition partners. There were two primary objectives for this research effort. The first was to 

develop high quality training to build expertise in coalition coordination and decision making by 

supporting participants’ awareness of the key challenges of coalition coordination and how they 

can be overcome in mission specific circumstances. The second major objective was to explore 

the feasibility of online collaboration platforms as a training medium for coalition operations. 

Procedure: 

We used the considerable operational experience of English-speaking (American, British, 

Canadian, Australian) coalition officers to identify the key cognitive challenges inherent in 

coalition operations. The key challenges were used to drive the development of context-rich 

scenarios and to guide development of a training program. To facilitate the training, we 

developed a collaborative web-based distributed training tool that would allow at least two 

coalition partners, in addition to a facilitator, to practice working together as a coalition staff 

using a scenario-based training program. 

Findings: 

The evaluation objective was to explore the feasibility of online collaboration platforms 

as a training medium for coalition operations. As a result of the evaluation, six critical factors 

were identified, which impact the effectiveness of the training. 1. Clarify the learning objectives 

in advance; 2. Emphasize the problem solving and coordination aspects of the exercise; 3. 

Capitalize on the opportunity for interaction by allowing partners to interact over discussion and 

problem solving; 4. Set the training at the appropriate level; 5. Use an experienced facilitator to 

direct the training; 6. Tie tool functionality to the learning strategy. 

Utilization and Dissemination of Findings: 

Overall, the training was shown to improve participants’ awareness of, and ability to respond to, 

the key themes of coalition coordination. Briefings were provided to the Joint Readiness 

Training Center (JRTC) Operations Group, September 7, 2005, and to the Joint Services 

Command and Staff College (JSCSC), March 1, 2005. The developed method and tool provide 

v 
an easy-to-implement and efficient means for coalition partners to train in a distributed

environment, on short notice, prior to deploying to the operating environment.

vi 

GLOBAL TEAMS: ENHANCING THE PERFORMANCE OF MULTINATIONAL STAFFS

THROUGH COLLABORATIVE ONLINE TRAINING 

CONTENTS 

Page 

Introduction.................................................................................................................................. 1

Identification of Training Need ............................................................................................. 3

Research Objectives............................................................................................................... 3

Developing Training for Multinational Coalition Operations Teams.................................... 3

Requirements Analysis and Training Development .................................................................... 4

Phase 1: Data Collection to Identify Training Requirements............................................... 5

Domain Analysis.............................................................................................................. 5

Critical Decision Method Interviews............................................................................... 5

Phase 2: Training Requirements Analysis and Scenario Development................................ 6

Training Requirements Analysis...................................................................................... 6

Scenario Development..................................................................................................... 10

Phase 3: Developing the Learning Strategy and Training Method....................................... 11

Learning Strategy and Facilitation Guides ...................................................................... 12

Training Method-The Global Teams Collaborative Training Tool ................................. 13

Training Interface............................................................................................................. 14

Phase 4: Training Evaluation.............................................................................................. 16

Level 1: Reactions: Survey and Discussion................................................................... 16

Level 2: Learning: Situation Awareness (SA) Exercise ................................................ 17

Level 3: Behavior: Behavioral Markers......................................................................... 17

Level 4: Organization ..................................................................................................... 20

Results of the Training Evaluation .............................................................................................. 21

Preliminary Evaluation .......................................................................................................... 21

Preliminary Evaluation Reactions Survey Results .......................................................... 21

 Preliminary Evaluation Discussion Results..................................................................... 22

Main Evaluation..................................................................................................................... 22

Level 1: Reactions: Survey and Discussion................................................................... 23

Level 2: Learning: Situation Awareness (SA) Exercise ................................................ 25

Level 3: Behavior: Behavioral Markers......................................................................... 26

Level 4: Organization ..................................................................................................... 27

Lessons Learned........................................................................................................................... 27

Technology ............................................................................................................................ 28

vii 
CONTENTS (Continued) 

Page 

The Learning Strategy and the Training Audience................................................................ 28

Online Facilitation of Collaborative Learning....................................................................... 30

Conclusions.................................................................................................................................. 31

References .................................................................................................................................... 33

Appendix A Global Teams Training Tool: Primary Supported Functionalities..................... A-1 

Appendix B Reactions Survey....................................................................................................B-1 

Appendix C Discussion Questions..............................................................................................C-1 

Appendix D Behavioral Marker System—Score Forms ........................................................... D-1 

List of Tables 

Table 1 High Level Focus Areas and Awareness Points............................................................ 7

Table 2 Framework of Desirable Behaviors............................................................................... 18 

Table 3 Mean Observer Ratings for Five Behavioral Categories............................................... 27 

Figure 

Figure 1 Global Teams main page.............................................................................................. 15

viii 
GLOBAL TEAMS: ENHANCING THE PERFORMANCE OF MULTINATIONAL STAFFS

THROUGH COLLABORATIVE ONLINE TRAINING 

Introduction 

Coalition operations have emerged as a key feature of military operations in the postCold War era. While the U.S. Army has participated in coalition operations throughout its 

existence, it is now the standard mode across the spectrum of military interventions. “Virtually 

all U.S. military operations since 1989 have been conducted as part of a multinational operation” 

(Bensahel, 2003, p. 112). A coalition operation usually entails an ad hoc arrangement between 

two or more nations acting together in order to pursue a common objective American-BritishCanadian-Australian ([ABCA], 2001). They are formed where a commonality of interests exists 

between nations, be it political, economic, or military, allowing those nations the benefit of

mutual aid in promoting their national interest and securing against real or perceived threats. 

They are usually created for specific purposes and for a limited duration. To date, the English 

speaking coalition nations, the focus of this report, have served together in ad hoc coalitions on 

several occasions to pursue common objectives (ABCA, 2001). 

Coalition operations are very different from unilateral operations. They engender a 

number of organizational, operational, political, and cultural challenges which are unique to the 

operational environment. Organizationally, coalitions can be very complex. They often include 

various branches of the U.S. military and foreign militaries, as well as nonmilitary organizations

such as non-governmental organizations (NGOs) and private voluntary organizations (PVOs), 

each bringing their own unique core competencies and political legitimacy. This blending of 

capabilities makes possible certain operations that a single nation could not, or would not 

conduct unilaterally. However, coalitions also engender significant coordination challenges. For 

example, consensus can be difficult to achieve because national interest may compete with 

standard practice as outlined in doctrine (Joint Chiefs of Staff, 2000). 

Operationally, coalitions cross the entire range of military operations from combat to 

humanitarian to stability and support operations (SASO). The precise role of armies in these 

operations varies according to each political and military situation. In some situations military 

operations are subordinate to political and diplomatic efforts. In such circumstances, coalitions 

have to be very creative to find ways to deliver appropriate and effective levels of military 

leverage that support political-diplomatic initiatives within the prescribed limitations (Peters, 

Johnson, Bensahel, Liston & Williams, 2001). In other situations, some coalition partners may 

participate in the humanitarian and peacekeeping phases of an operation, but not in combat, 

should that occur. Moreover, because of their limited purpose and life span, coalitions do not 

afford the same political resolve and commonality of aim as other types of military operations. 

Within this realm, sovereignty issues are often the most difficult issues for coalition forces to 

deal with. Often, the commander of a coalition force is a commander in title only; mission 

accomplishment is achieved through coordination, communication, and consensus rather than by 

traditional command concepts (i.e., giving an order) (Joint Chiefs of Staff, 2000). 

Political and sovereignty issues make planning and coordination of coalition operations 

particularly difficult. Commanders and planners must learn the capabilities of partner nations or 

1
organizations and must design plans in such a way as to allow all units to contribute (ABCA, 

2001). In many cases, the coalition leadership will have to be innovative to find effective roles 

for all participants (Peters et al., 2001). Army doctrine emphasizes the importance of including 

NGO and PVO representatives at the earliest possible phase of mission planning. The sensitivity 

and understanding gained by including these organizations early in the planning process can 

contribute to more effective working relationships once the mission begins (Joint Chiefs of Staff, 

2000). 

Coalition doctrine also recognizes the role of intangible factors in multinational 

collaboration. In particular, developing effective partnerships, confidence, rapport and respect 

are considered to be the cornerstones of successful coalition operations. In fact, after World War 

II, General Dwight D. Eisenhower said that “mutual confidence” is the “one basic thing that will 

make allied commands work” (Joint Chiefs of Staff, 2000, p. I-9). 

Good rapport between coalition members results in successful teamwork and overall 

unity of effort (Joint Chiefs of Staff, 2000). The first concern when establishing rapport is an 

understanding of the characteristics, personalities, capabilities, ambitions, and cultural habits of 

the various coalition partners. Rapport is more easily established when nations combining forces 

share similar cultural backgrounds. Respect for the partners’ culture, religion, customs, and 

values combined with understanding and consideration of their ideas will solidify the 

partnership. Without such genuine respect of others, rapport and mutual confidence cannot exist; 

lack of respect may lead to friction, jeopardizing mission accomplishment. While these factors 

cannot guarantee success for the coalition, ignoring them can usually guarantee failure of the 

coalition in accomplishing its mission (Joint Chiefs of Staff, 2000). 

Commanders must possess the knowledge and experience that is critical for effective 

decision making and teamwork in these types of operations. The key challenge for commanders 

appears to be coordinating with coalition partners, international and civilian organizations, and 

the host nation government and local citizens. The literature describes this as a problem of 

“interoperability.” To date, a clear understanding of how to improve interoperability in these 

operations has not yet been established (ABCA, 2001; Ryan, 2000). For example, what is the 

nature of effective coordination among such diverse players? What are the organizational 

arrangements by which coalition operations solve problems? Who needs to interface with 

whom, by what means do they do it, and what are the critical timeframes for particular kinds of 

communications among particular players? How should commanders approach decision making 

in peace operations with limited resources and restrictions on unity of command? What 

opportunities must commanders create for themselves in the face of seemingly impossible 

conditions? These and other critical questions about the judgments, decisions, creativity, 

organizational arrangements, and control faced by coalition commanders need to be identified 

and answered from the perspective of the commander. 

Multinational exercises are thought to be extremely important for ensuring

interoperability with potential coalition partners and for working out command issues. 

Moreover, researchers believe that NATO should improve its capability for exercises so that it 

can routinely integrate coalition-building and maintenance activities with military action (Peters 

et al., 2001). 

2
Identification of Training Need 

Developing and delivering high-quality training for the variety of contingencies that may 

arise in the contemporary operating environment is vital to preparing a strong Future Force. 

Indeed the U.S. Army planning documents for the Future Combat System of Systems 

specifically call out the requirement to “develop, through training and experience, the thinking, 

confident, versatile, adaptive, and seasoned leaders at the tactical level required for the digitized, 

rapidly deployable objective force” (U.S. Army Training and Doctrine Command, 2001, p. 6-

29). In order to achieve this objective there is a recognized need for more research to identify 

the key challenges in this operational environment, and for the development of focused and 

relevant training that can support improved decision making and coordination. 

Research Objectives 

There were two primary objectives for this research effort. The first was to develop high 

quality training to build expertise in coalition coordination and decision making by supporting 

participants’ awareness of the key challenges of coalition coordination and how they can be 

overcome in mission specific circumstances. The second major objective was to explore the 

feasibility of online collaboration platforms as a training medium for coalition operations. The 

training will involve the development of low-fidelity scenario-based exercises. Multiple, 

distributed participants will be required to make a decision or complete a task based on a 

cognitively authentic scenario-based challenge. 

Developing Training for Multinational Coalition Operations Teams 

This effort involved developing training requirements for multinational command and 

staff teams at the brigade and multinational division level, designing and implementing scenariobased training, and evaluating the distributed multinational training for feasibility and

effectiveness. Scenarios are based on real-world situations to provide participants access to the 

expertise required to address such challenges. Our goal was not to develop cultural training, per 

se, but to address the differences in how English-speaking nations1

 set goals and work to achieve 

them, as well as to highlight the differential strengths they bring to coalition operations. 

Scenario-based training has many advantages over factual, classroom training. It allows 

the students to assess situations, identify relevant facts, strategies, and procedures, generate 

possible solutions, make decisions, and receive natural and instructional feedback. Whereas 

classroom-based training is about gathering information, scenario-based training is about 

doing—making decisions, planning, and applying the knowledge and information for making 

decisions in an operational scenario. This training provides an opportunity to develop a more 

detailed mental model of how aspects of a particular domain actually work by working through 

the issues and interplay of factors through realistic problem scenarios. It is recognized that the 

Army uses tactical decision exercises and scenario-based training in the classroom on a 

consistent basis, however, the training approach presented here has the added benefit of allowing 

real time problem analysis, coordination and problem solving with multinational partners. 

 

1

 This research was supported by the Technical Cooperation Program (http://www.dtic.mil/ttcp/). 

3
Our research methodology is based on a combination of two training programs, both of 

which are grounded in the principles of Recognition Primed Decision (RPD) making and 

expertise development (Klein, 1989). The Think Like A Commandeer (TLAC) training program

is based on an understanding of how expertise is developed and embodies the concept of 

deliberate practice of thinking skills (Ross & Lussier, 1999). It aims to develop the learner along 

a path towards expertise by both helping him elaborate his mental models and by making the 

results of the recognitional decision making routines explicit so that they can be refined. A 

central component of the TLAC program is the use of context-rich training vignettes. The goal 

of TLAC is to teach officers “how to think” instead of “what to think” (Shadrick & Lussier, 

2002). 

The other training program informing our training methodology is the Decision Skills 

Training Program. This training program is based on an understanding of how expertise 

develops and functions in dynamic settings. The first task is to “unpack” the knowledge base of 

experts in a field so that the knowledge is available to training developers. Cognitive task 

analysis methods, primarily the Critical Decision Method (Crandall & Getchell-Reiter, 1993; 

Hoffman, Crandall, & Shadbolt, 1998), are used to leverage the critical events (stories) from

experts, and by so doing expose their domain expertise. Second, stories (vignettes) are carefully 

constructed to include realistic cues in the context. The incidents collected from experts are not 

used intact. Instead, elements are emphasized and tailored to sharpen the focus of desired 

features. All of the vignettes present a dilemma with some level of uncertainty and a 

requirement that forces the student to make a decision. Each participant has a limited amount of 

time to consider how he or she would react, which adds time pressure to the exercise. 

Facilitation ensures that the participants do not just have an experience, but that they focus on the 

cognitive aspects of that experience—that they consider the depth of their understanding and the 

likely consequences of their decisions. 

The training programs described above are somewhat different in execution, but they 

both support the development of mental models, the acquisition of knowledge in context, and the 

practice of cognitive activities with coaching. Both take the learner further along the path to 

“expert intuition.” Moreover, at the heart of both of these training methodologies is a carefully 

crafted story. What is to be learned and practiced has been made explicit and incorporated by the 

training developers. Stories represent a powerful means of transferring expertise and allowing 

the learner to activate that expertise in new situations. The central challenge is that of creating 

training scenarios that represent authentic and believable situations and represent accurately the 

task work, teamwork, and communication tasks that are needed to enable the trainee to practice 

the needed tasks and skills. 

Requirements Analysis and Training Development 

This effort was completed in four phases: Phase 1: Data Collection to Identify Training 

Requirements; Phase 2: Training Requirements Analysis and Scenario Development; Phase 3: 

Developing the Learning Strategy and Training Method; and Phase 4: Training Evaluation. 

4
Phase 1: Data Collection to Identify Training Requirements 

The data collection phase had two aspects to it: domain analysis and Critical Decision

Method (CDM) interviews. Each is briefly described below. 

Domain Analysis 

The purpose of the domain analysis process was to support a better understanding of the 

mission and tasks that are involved in multinational operations, particularly those that are 

cognitively challenging or in need of training. A secondary aim was to solicit the subject matter 

experts’ (SME) suggestions about fruitful avenues for scenario development. The SMEs were a 

British officer and a Canadian officer stationed at Fort Knox. Both were Foreign Liaison 

Officers and had extensive experience with coalition operations. The third SME was a U.S. 

Lieutenant Colonel (LTC), (Active Duty Reservist) with 25 years of military coalition 

experience. 

Critical Decision Method Interviews 

Critical Decision Method (CDM) interviews (Hoffman, Crandall, & Shadbolt, 1998; 

Militello & Hutton, 1998; Schraagen, Chipman, & Shalin, 2000) were conducted to identify the 

decision requirements in coalition operations in the contemporary operating environment. 

Critical Decision Method interviews were conducted with a total of 16 SMEs. Of those, seven 

interviews were conducted with SMEs who were Majors at the Command and General Staff 

College (CGSC), Fort Leavenworth, (six were U.S. Army officers, one was a Canadian Army

officer). The remaining nine CDM interviews were conducted at Joint Services Command Staff 

College (JSCSC) in England with UK Army, Air Force, and Navy officers. Officers ranks (or 

equivalents) were: Brigadier General (BG) (2), Colonel (COL) (2), Lieutenant Colonel (LTC) 

(2), and Major (MAJ) (3). 

The CDM interviews were organized around an initial, unstructured account of a specific 

incident experienced by the interviewee. In this case, participants were asked to recall an 

incident that involved collaboration with a coalition partner that they found to be particularly 

challenging. Once the participant identified a relevant incident, he or she recounted the episode 

in its entirety, with no interruptions from the interviewer. The initial incident account provided 

the focus and structure for the remainder of the interview. Once the report of the incident was 

completed, the CDM interviewer led the participant back over his or her incident account several 

times, using probes designed to focus attention on particular aspects of the incident and solicit 

information about them. The probes were designed to progressively deepen understanding of the 

interviewee’s account. The information obtained via these methods is concrete and specific, 

reflects the point of view of the decision maker, and is grounded in actual incidents. This 

method has been demonstrated to yield information richer in variety, specificity, and quantity 

than is typically available in experts’ unstructured verbal reports. Detailed descriptions of CDM

and the work surrounding it can be found in Klein and Hoffman (1993) and Klein, Calderwood, 

and MacGregor (1989). 

5
Phase 2: Training Requirements Analysis and Scenario Development 

This phase involved analysis of the interview data and the identification of High Level 

Focus Areas (HLFAs) and their associated Awareness Points (APs). A description of how these 

were determined is outlined in the next section. Training scenarios were subsequently developed 

which addressed the HLFAs identified in the CDM interviews. 

Training Requirements Analysis 

Interviews were coded in a thematic analysis according to the grounded theory approach 

(Glaser & Strauss, 1967). A process of classification and coding, then reclassification and recoding, led to the development of a classification system consisting of four themes or HLFAs 

and 27 underlying APs. We call the themes HLFAs because they guide the focus of the training 

objectives and scenario content. The HLFAs are the areas in which the coalition forces differ in 

operation. Coalition officers must develop understanding in how these issues can impact 

operations. Coalition officers without such an understanding may have less successful 

collaborations. This is evidenced by the experiences recounted by our interview participants. 

The four HLFAs are: organizational structure, work-style, stance, and level of integration. 

Organizational structure refers to the formal and informal structures that govern the 

organization of the force. This includes the degree of autonomy allowed to junior officers and 

their ability to influence decision making higher up the organizational hierarchy. Awareness 

points associated with this HLFA include formal chain of command, force structure, informal 

chain of command, and mission command. The HLFAs and APs are summarized in Table 1. 

Work-style refers to the culture and climate with each national force. It includes issues 

such as work tempo, task focus, and the preferred medium for transmitting information. It also 

includes factors which relate to the skill sets of the force (i.e., how adaptable and how flexible 

they are). Awareness points associated with this HLFA include: task structure, generalist versus 

specialist, task versus relationship, tempo, planning process, flexibility/adaptability, and 

exchange of information. 

Stance refers to attitudes toward force protection within a particular force. Related to this 

are attitudes toward the role of the military “in country,” the overriding objective of foreign 

missions, attitudes toward host nations and coalition partners, the problem solving approach 

typically adopted by a nation, and how fixed or flexible it is in using alternative methods to solve 

problems. Awareness points associated with this HLFA include: force protection, politics, rules 

of engagement (ROE), leverage points, problem solving, and manage presence. 

Level of integration refers to the level of awareness between coalition partners; how well 

they understand each other’s roles, functions, capabilities, responsibilities, approach and stance 

and how willing they are to collaborate to solve problems. Awareness Points associated with this 

HLFA include: understanding your coalition partner(s), understanding the host nation’s culture, 

understanding “coalition,” giving direction, information exchange, resources, ROE, priorities, 

trust, understanding NGOs, roles and functions, capabilities, commander’s intent, Title 10, and 

integration. 

6
7

Table 1 High Level Focus Areas and Awareness Points 

HLFA Description Awareness Point Description 

Formal chain of command • The formal chain of command as de

organizational chart.

Force Structure • The hierarchy within the command s

versus two units down). 

Informal chain of command • The level of decision making power 

ranks, degree of autonomy in one’s jmicromanagement. 

• Ability/willingness to speak up to se

senior officers, ability to influence d

ORGANIZATIONAL STRUCTURE 

This theme relates to the formal and informal structures that govern the organization of the force. 

Mission Command • The extent to which the command is 

command led.

Task Structure • Functional (stove-piped) versus resoGeneralist versus Specialist • Ability to multitask or diversify in re

situations. 

Task versus Relationship • Priority placed on task versus relatioTempo • Work pace and work hours. Planning Process • The extent to which the planning pro

timeline or project basis. 

Flexibility/Adaptability • The inclination to stay within the sta

openness to new processes.

WORK-STYLE The culture and climate 

within each national force. 

Exchange of Information • Formalized style (i.e., PowerPoint ve
8

HLFA Description Awareness Point Description

Force Protection • Emphasis on force protection versus Politics • Political awareness and concern with ROE • How rules of engagement are interpr

• How rules of engagement apply to th• Views and assumptions about others

Leverage Points • Are all potential resources and cours

and used?

Problem Solving • Approach to problem solving (e.g., f

affairs versus reconstruction versus ioperations).

STANCE Each nation’s views towards 

protecting its own forces, its views towards its role in other countries, its overridingobjective in foreign missions,and its approach to host nations and coalition partners. Its approach to problemsolving and how fixed or flexible it is in usingalternative methods to solve problems. 

Manage Presence • Willingness to use alternatives to mi

affairs/Force protection/Reconstruct

Understanding your coalitionpartner(s) 

• Motivation. • Objectives. • Reason for participating in mission. • Agenda and point of view. 

Understanding the host nation’s culture 

• Motivation. • Objectives. • Agenda and point of view. 

LEVEL OF INTEGRATION 

How willing are coalition partners to solve problemscollaboratively? How well do they understand each other’s roles functions, capabilities, responsibilities, approach and stance of partners?

Understanding “coalition” • How does each military unit conceiv

“coalition” (e.g., “partnership,” “U.Sonly peace keeping,” etc.)?
HLFA Description Awareness Point Description 

Giving Direction • Asking versus ordering. Information Exchange • Coordination/Establishing processes

• Ability to share information and inte

• Utilizing all of the available informa

intelligence resources. 

Resources • Integrating resources to accomplish ROE • Understanding partners’ ROE and im

mission. 

Priorities • Understanding the priorities of your 

determine whether or not certain thi

Trust • Trusting relationships versus consen

behaviors. 

Understanding NGOs • Motivations. 

• Goals.

• Agendas of NGOs and how they ma

another (e.g., not all NGOs are the s

Roles and Functions • Definitive or messy roles and functiCapabilities • Maximizing available capabilities su

contributing maximally to the opera

• Knowledge of partners’ capabilities 

mission success. 

Commander’s Intent • Understanding the intent of other coTitle 10 • Awareness of own and others’ respo

LEVEL OF INTEGRATION continued 

Integration • Willingness to share resources and i

• Willingness to fully engage a plan p

9
 10

Scenario Development 

The project team developed three scenarios. The scenarios were based on the lived

experiences of coalition forces recounted during the CDM interviews. Each scenario involves 

three individual, yet connected vignettes that build on each other to form evolving coalition 

operational challenges. Vignettes require the participants to interact and address situations from

the contemporary operational environment. Each vignette addresses one or more of the HLFAs 

so that issues that impact coalition operations can be worked through by participants, in context, 

with the support of a facilitator. The vignettes are designed to force students to engage in a 

problem solving task. The student will be required to explore his/her own (or a partner’s) 

assumptions, approach, goals, mission, objectives, structure, stance, and work-style, and to

examine the impacts of these on task and mission accomplishment.

The scenarios are summarized below. Full versions of the scenarios are available in the

accompanying compact disc (CD) attachment entitled Global Teams Scenarios and Facilitation 

Guides. They are also available online at http://www.361interactive.com/globalteams/index.html. 

Scenario 1: Quelling the Violence. This scenario is set in Iraq around the time of the

elections. The coalition includes United Kingdom (UK), United States (U.S.) Australia, and 

New Zealand forces, as well as several other allied nations. The theme of this group of three 

exercises is about dealing with requests for assistance from other coalition partners and how the 

coalition works together to calm an insurgency. The players take the role of staff officers at

Combined Joint Task Force Headquarters (CJTF HQ) in Multi-National Division South East 

(MND SE). As the country-wide elections approach, there is great political pressure on all 

coalition partners to ensure that the insurgent attacks are suppressed so that Iraqi citizens can go 

to the polls. The High Level Focus Areas (themes) being targeted in this scenario include: 

planning tempo, information exchange, integrating resources and capabilities, understanding host 

nation culture, understanding your coalition partner’s motivation, objectives, agenda and point of 

view, how ROE apply to the mission, political motivation for participating in the mission and 

ramifications of actions, identifying things in the environment to leverage to your advantage, 

different nations’ approach to solving a problem and understanding your coalition partners’ 

priorities. In Situation 1, the U.S. has asked the coalition partners to support ongoing peace 

enforcement operations. Players are required to list their concerns and list their information 

requirements. In Situation 2, coalition forces have been conducting counter-insurgency and 

peace enforcement operations. Players are required to list their concerns, what they plan to do, 

and how developments may change the way coalition operations are run in the future. Situation 

3 involves a hostage taking situation. Players are required to list their concerns and state how

they will respond.

Scenario 2: Security for the Refugee Camp. This scenario is about the establishment of a 

refugee camp and the requirement for collaboration between forces for maintaining camp

security. Coalition partners and NGOs are working together to set up refugee camps and deal 

with supporting the movement of large numbers of displaced persons. The players are members 

of a multinational coalition force engaged in combat and counter-insurgency operations. The 

coalition includes the UK, the U.S., Australia, Poland, and New Zealand, as well as several other 

allied nations. The High Level Focus Areas being targeted in this scenario include: understand
the planning process of your coalition partners, identify capabilities and limitations of your 

coalition partners, determine how to integrate assets and resources to accomplish the mission, 

determine roles and functions, and identify and understand potentially different perspectives on 

security. In Situation 1, the UK forces determine that they do not have sufficient resources to set 

up and secure a refugee camp without additional reinforcement from their coalition partners. 

Players are required to decide what security requirements are necessary for the refugee camp by 

describing their three top priorities for securing the camp and describing how they would want to 

allocate U.S. and UK security assets. In Situation 2, there is some unrest in the camp. Players 

are required to identify possible courses of action, identify considerations to take into account, 

and identify what roles various coalition members should take. In Situation 3, a convoy comes 

within a close distance to the group of hostile Iraqis. Players are required to decide what 

advice/order they would give to the convoy and determine their top three concerns about the 

situation. 

Scenario 3: Reconstruction and Compliance. This scenario is set in Afghanistan. 

Coalition forces include the U.S., the UK, Canada, Australia, New Zealand, and Germany. 

Reconstruction is the priority of this coalition effort. Players take the role of members of the 

Coalition HQ staff, part of the Coalition Joint Civil Military Operations Task Force (CJCMOTF) 

team. The team is responsible for monitoring and coordinating the activities of Provincial 

Reconstruction Teams (PRTs), as well as International Organizations and NGOs. High Level 

Focus Areas broached by this vignette include: identifying capabilities and limitations of your 

coalition partners; understanding your coalition partners’ motivation, objectives, reason for 

participating in mission; adaptability of skill sets; organizational structure; information 

exchange; integrating resources to accomplish the mission; understanding the intent of other 

coalition commanders; approach to problem solving; awareness and implications of actions, 

understanding NGO’s motivations, goals and agendas; force protection; and ROE. In Situation 

1, a large amount of money is available for distribution across the country for reconstruction. 

Players must describe ways to promote greater coordination between the PRTs, and identify 

failures in coordination which have taken place. In Situation 2, the World Health Organization, 

the Red Cross, and coalition partners are coordinating a medical distribution site. Players are 

required to identify critical issues in collaborations with NGOs and determine why the U.S. 

military and NGO coalition partners’ goals may conflict. In Situation 3, players are members of 

the HQ Operational Plans Group, responsible for developing a plan to manage security 

throughout the country so that the overall mission of security and nation building can be realized. 

Players must coordinate the activities of coalition partners and identify the priorities and 

appropriate posture of security forces. 

Phase 3: Developing the Learning Strategy and Training Method 

In this report, we use the term training method to refer to the training platform or tool 

itself; that is, the method or mechanism for delivering training. In this case, the training method 

is the use of a low-fidelity, synchronous, web-based, distributed training tool. The learning 

strategy refers to how the training content is structured and facilitated, the expectations and goals 

we have for learning, and the rationale for those factors. 

 11
Learning Strategy and Facilitation Guides 

The learning strategy employed in this project is designed to support participants’ mental 

model development regarding coalition challenges and how the challenges can be resolved to 

greater effectiveness in mission-specific circumstances. As noted above, the themes we found 

consistently in the experiences of our interviewee participants provide the challenges (as well as 

opportunities for success) for officers in coalition operation. These themes were incorporated 

into the scenarios. To build mental models about how coalitions can collaborate, the officers 

must accumulate experiences or surrogate experiences working through cognitive challenges. 

Surrogate experiences, facilitated to help the officers reflect on their assumptions and

performance, is the basis of the learning strategy. Barriers to coordination effectiveness in the 

coalition environment (other than the ubiquitous discussions of technology interoperability) 

include areas of expertise and capability and differences in assumptions about goals and how to 

achieve them. As awareness and understanding are developed, mental models of collaboration 

will develop. The mental models will include expectations for different situations and different 

allied forces, the nature of goals, and action plans and cues that indicate whether collaboration is 

proceeding successfully or not. Thus the basis for the initial learning strategy was 1) to build 

expertise in the decisions that need to be made in coalition operations through practice in 

working through challenges in the context of scenarios, and 2) to build mental models that 

support that expertise by having participants recognize the importance and meaning of aspects of 

staff collaboration within specific theme areas—our HLFAs. 

Facilitation of the training session is key to the learning strategy. Successful facilitation 

strategies include: 1) engaging in deliberate practice, so that each opportunity for practice has a 

goal and evaluation criteria; 2) obtaining feedback that is accurate and diagnostic; 3) building 

mental models; 4) developing metacognitive skills; and 5) becoming more mindful of 

opportunities for learning (Pliske, McCloskey, & Klein, 2001). Elaboration and comparison of 

each person’s decision making process brings out the assumptions, cues and factors, challenges, 

and strategies of everyone for comparison. These elements generally comprise the mental model 

and situational understanding of the training participants, and the process allows each person to 

develop more insight into situations, thus developing their own mental models. 

The role of the facilitator is an important topic in education today. Many different types 

of content are now delivered through web-based learning. Many of these training courses 

require facilitation because they involve concepts that cannot easily be represented and processed 

by the student without facilitation. Collaborative learning is also a growing area of research and 

development. To many educators and researchers, the use of technology is secondary to the 

development of learning goals, content, and process. Rather, learning needs to be facilitated for 

meaningful responses and insights to occur regardless of the technology (Berge, 1995). 

However, the training method or platform can support or hinder the processes as well as match 

or impede the needs of the learning audience. As we developed our learning strategy we 

attempted to synchronize the facilitation process, the content, and the technology supporting the 

facilitation process. 

A facilitation guide supports each of the three scenarios. Each guide includes: the 

teaching points targeted in the scenario, the requirements set for the students, and a process for 

 12
supporting the facilitator in leading the discussion. Potential student questions are also included, 

some with answers provided by SMEs. The facilitator must choose what questions to take, and 

when to say “that information is not available.” The guides for each scenario are integrated into 

the web-based tool to access online before and during training. The facilitation guides are not 

meant to fully train an instructor/facilitator in the art of facilitation; rather, experienced

facilitators are needed to capitalize on the training. (Full versions of the facilitation guides are 

available in the accompanying CD attachment entitled Global Teams Scenarios and Facilitation 

Guides. They are also available online at http://www.361interactive.com/globalteams/index. 

html). 

Training Method-The Global Teams Collaborative Training Tool 

The Global Teams tool is a collaborative, synchronous training tool developed to support 

representatives of military and non-military organizations of different countries in enhancing 

mutual understanding and communication skills. The tool allows facilitators to upload and 

conduct training vignettes with players distributed across the world, and can support up to 25 

concurrent users. Features such as shared drawing spaces, shared and private chat and response 

areas, and optional streaming audio support provide a truly interactive environment where 

facilitators can lead players through an engaging, interactive learning experience. 

The functionality of the tool and the interface design support the learning strategy, not 

any particular HLFA or AP. The tool was designed to be flexible and support a range of training 

options that depend on the strategy of facilitated, surrogate experiences. The tool does allow the 

users to address the HLFAs for each scenario, and supports both collaboration and facilitation, 

key elements of the strategy. How well the tool supports the HLFAs and APs is highly 

dependent on the ability of the facilitator. Descriptions of the functionalities and interface are 

presented below. Of those functionalities, the ability to manipulate graphics on the map allows 

for collaboration by the players as well as the ability of the facilitator to highlight information. 

The shared whitespace allows synchronous sketching of such things as initial concepts of 

operation or even assumptions about the area of operations. Private response forms provide the 

ability for the facilitator to gain an understanding of the different points of view and assumptions 

of the player participants prior to bringing them together to share those views. As the desired 

outcome is for players to develop their mental models around the HLFAs, this feature was 

designed to allow opportunities for the facilitator to assess changes over time in responses in 

both formative evaluations in real time and summative evaluations after training. The SplitScreen allows the facilitator to bring individually developed points of view together for all 

participants to compare to further support insight into how collaborations may be approached or 

even impeded. The chat features allow users to hold group discussions, and for researchers and 

facilitators to exchange views in real time as well as to direct the sessions. 

The training method is purposely designed to be low tech to allow access to the range of 

potential users who may have low levels of technology to support their training. The simple 

technology design attempts to minimize technology problems for a variety of users accessing the 

web-based tool. In addition, a goal for the tool was to support future researchers in their ability 

to easily add scenarios for continued or other research purposes. 

 

 13
The training platform is an ideal solution for the audience that is geographically 

separated. The simplicity of the method allows ad hoc teams to form easily, and without much 

preparation, to conduct training (if the facilitator is conversant in the learning strategy and the 

scenario forming the focus of the training). All the tools needed are present online and relieve 

the training audience from acquiring, loading software, carrying training products with them, and 

being in any certain location to train. 

The research team established the desired functionality for a training platform that would 

support web-based, synchronously facilitated training sessions, including some functions that 

were desired but not necessarily required in this effort, such as voice over internet protocol. The 

requirements at this point were based on existing training products, tools available for web-based 

collaboration in a low technology format, and the need to support scenario-based training. We

continued development of the basic design through iterative sessions. The primary supported 

functionalities are presented in Appendix A. 

Training Interface 

Figure 1 presents a screen capture of the Global Teams main page, to which all logged-in 

users will be directed after selecting a vignette from the library. A majority of this screen is 

common to all users. However, only registered facilitators have full access to the facilitator chat 

area and the facilitator guidance. Also, only facilitators can view each team’s response forms at 

any time. The functional areas of the main page are as follows: 

Shared Workspace Area. The largest area of the main page contains the shared 

workspace. Here, users can view and mark up the maps and graphics associated with a selected 

vignette. By default, the vignette’s background graphic is displayed initially. However, users 

can alternatively navigate to any of the vignette’s graphics (background, Situation 1, Situation 2, 

Situation 3), a shared whiteboard area, or the shared space text area at any time by clicking on 

the tabs on the top of the workspace. The currently selected graphic/workspace is designated by 

the tab color changing from black to red. Along the left side of the shared workspace is the 

workspace toolbar. Using these individual toolbar buttons, users can superimpose textboxes, 

free text, arrows, and a range of shapes onto the graphic for all other users to see. Users can also 

draw straight or freeform lines, can customize the colors of their additions, and can selectively 

move/delete individual items or clear the entire workspace. The toolbar can be hidden and reshown by clicking the small blue arrow. The background graphic is displayed initially. 

However, users can alternatively navigate to any of the vignette’s graphics (at the bottom of the 

bar). The shared space text area presents a dual text region where users can input and/or 

compare multiple answers simultaneously as a group. 

Username Login/Control Area. At the top right corner of the main screen, users can 

change their onscreen names by entering them in the textbox and clicking the Login button. This 

feature is especially useful if multiple parties are logged in as facilitators. Each facilitator can 

elect a unique name and that name will be associated with their cursors, audio messaging, and 

group chats. 

 14
Figure 1. Global Teams main page. 

Group Chat/Response Form Area. Directly below the login area is the group 

chat/response form area. By default, when the user arrives at this main page, the group chat 

functionality is enabled. Here, all users can freely communicate via instant text chatting. Text is 

entered by an individual user, and, upon clicking the Send button, the user’s text is displayed for 

all other users to instantly view. That user’s text is uniquely identified by both username and 

color (if that user has selected a unique color+see control area. To access the group chat, the 

user can click on the group chat tab at any time. To clear the chat window, a user can click on 

the clear chat button beneath the chat window. The second tab displays individual player 

response forms. When a registered player clicks on the response form tab, the group chat will be 

replaced onscreen by a text-based response form. Here, the player can enter responses to 

questions presented by the facilitator. While the other players cannot view this information, 

registered facilitators can view it at any time. When a facilitator clicks on the response form tab, 

the USA response form appears by default. In addition, five buttons (representing the countries 

of USA, UK, Canada, and Australia, and a generic non-government organization) appear below 

the form. These buttons are only visible to facilitators. By clicking on these buttons, the 

facilitator can observe the progress of each individual player in answering the questions. A help 

button in this area generates a new popup window with guidance in using these features as well 

as the name changing feature. 

Control Area. The bottom, right corner of the main page contains the control area. 

Several functionalities are located here. The box beneath the audio label displays all currently 

 15
logged-in users by username. It also serves a second, optional purpose. By pressing the talk 

button, users with microphones can speak to the other distributed users (if they have functional 

speakers). The user can either press and hold down the talk button to speak, ensuring privacy 

when they don’t want conversations overheard, or they can click the auto box, and the tool will 

detect when the user is speaking automatically. Lights to the left of each username in the box 

flash green when that particular user is speaking. 

The facilitator chat box to the right of the audio box is a private area, only 

accessible/visible to registered facilitators, where such users can conduct administrative chats 

that would otherwise distract players. 

The pointer color selector beneath the facilitator chat box allows any user to selectively 

customize his or her cursor and group chat color. This feature supports the distinction of 

distributed users. 

The buttons at the bottom of the control area generate new windows that access Wordformatted text documents. These documents provide the relevant rules of engagement, 

background and situational information associated with the selected vignette. By clicking on a 

button, the user’s browser will open a new window for the text-based information, allowing the 

user to simultaneously view the main page/graphics and supporting text. The facilitator tool

button is only visible to registered facilitators, and provides supporting documentation for 

conducting vignettes. A help button, visible to all users, generates a separate popup window 

with additional guidance on using the control area. 

Phase 4: Training Evaluation 

Training evaluation took place in two parts: a preliminary evaluation and a main 

evaluation. The purpose of the preliminary evaluation was to test the functionality of the training 

tool and the usability of the training strategy. The main evaluation was designed to test the 

effectiveness of the collaborative distributed web-based training methodology for supporting 

Soldiers’ understanding of coalition operational challenges. Four levels of evaluation were used 

to accomplish this goal. The measures are described below. 

Training evaluation was based on Kirkpatrick’s (1976) hierarchy for training evaluation. 

The hierarchy consists of four different levels of evaluation: reactions, learning, behavior, and 

organization. These levels represent a sequence of methods to evaluate a training program. 

Each level builds on the previous one, but also provides more valuable information. 

Level 1: Reactions: Survey and Discussion 

Reactions are concerned with how the participants react to the training. It is important to 

note that positive reaction does not ensure learning, although a negative reaction almost certainly

reduces the likelihood that learning has taken place (Kirkpatrick, 1998). To learn about 

participants’ reactions to the training, we developed: 1) a short survey to be filled out by each 

participant at the end of each training day (See Appendix B for Reactions Survey) and 2) a set of 

discussion questions to guide the elicitation of the participants’ reactions (See Appendix C for 

 16
Discussion Questions). This discussion took place in an open forum (or a focus group) at the end 

of the training session. The idea was to initiate a discussion about participants’ perceptions of 

the usefulness, value, and relevance of the training. 

Level 2: Learning: Situation Awareness (SA) Exercise 

Learning is the second level in the hierarchy, and refers to “the principles, facts, and 

skills which were understood and absorbed by the participants” (Kirkpatrick, 1976, p. 11). This 

level is concerned with whether the participants have acquired knowledge, or have modified their 

attitudes or beliefs as a result of attending the training course. It is important to measure 

learning, as no change in behavior can be expected if no new knowledge or change in attitudes 

has occurred. In this case, learning refers to whether participants have more knowledge about 

coalition operations after the training. In order to test the learning, three SA Exercise questions 

(Pliske, Militello, Phillips, & Battaglia, 2001) were embedded into the training tool. These three 

questions were asked after each of the three vignettes was presented. 

Level 3: Behavior: Behavioral Markers 

An evaluation at the behavior level is the assessment of whether knowledge learned in 

training actually transfers to behaviors on the job or a similar simulated environment. For each 

of our scenarios, three separate observations were made, one for each vignette. Observations 

were made using a behavioral marker system. Behavioral markers are descriptions of 

observable, non-technical behaviors that contribute to superior or substandard performance 

within a work environment. They describe specific, observable behaviors—not attitudes or 

personality traits—with clear definition (enactment of skills or knowledge is shown in behavior). 

Each behavior should have a demonstrated causal relationship to performance outcome. One of 

the main purposes of behavioral marker systems is to enable performance measurement for 

training evaluation. According to standard practice, observers made notes while observing, 

noting examples of positive and negative behaviors based on the dimensions highlighted in the 

system. At the end of the session the raters rated the team on each dimension. 

The behavioral marker system we used is adapted from the NOTECHS system

(Avermaete & Kruijsen, 1998). NOTECHS was developed by EC DGTREN (European 

Community Directorate for Transport and the Environment) and the Civil Aviation Authorities 

of France, Netherlands, Germany, and UK. The system was developed as a method for 

instructors to assess pilots’ non-technical skills during training. The system consists of a 

framework of desirable behaviors in a hierarchy of categories and elements. In the system we 

used, there are five primary categories and 18 elements as shown in Table 2. The categorical set 

was based on theoretical models identified from the teamworking literature (Cannon-Bowers & 

Salas, 1997; Zsambok, Klein, Kyne, & Klinger, 1993). Evaluators are required to indicate 

whether a particular element of behavior was observed during training and then make a judgment 

about the team’s performance on each category. 

 17
Table 2 

Framework of Desirable Behaviors 

Category Description Element Example 

Defining roles and 

functions 

• Are the team members actively 

assigning roles and 

responsibilities?

• Are they updating roles and 

responsibilities as the situation 

changes?

Engaging • Is the team capitalizing on team

member participation? 

• Has any member of the team

disengaged? Signs that may 

indicate a team member has 

disengaged: 

• Not paying attention during 

discussions. 

• Performing a different task during 

the discussion. 

• Not following up on questions or 

concerns. 

Compensating • Have members of the team

stepped outside their assigned 

roles or functions in order to help 

the team achieve its goals? 

Team identity The extent to 

which members 

conceive of the 

team as an 

interdependent unit 

and then operate 

from that

perspective. When 

members have 

weak team identity 

they tend to 

operate as 

individuals instead 

of as members of a 

team. 

Avoiding micromanagement 

• Are people, tasks, or information 

being managed at the wrong level 

of detail?

Time 

management/ 

workload 

management 

• Are the team members managing 

workload correctly? 

• Are they aware of the time 

limitation on tasks and planning 

work accordingly? 

Team self 

monitoring 

Ability of the team

to observe and 

monitor itself. 

Adjusting • Is the team recognizing and 

adjusting for problems?

 (Table Continues)

 18
Category Description Element Example

Teambuilding and

maintaining 

• Is there an atmosphere for open 

communication and participation?

• Is there input and feedback from

everyone on the team? 

Consideration of 

others 

• Do the team members take notice 

of the suggestions of all members 

even if they do not agree? 

Support of others • Do team members help each other 

in demanding situations? 

Cooperation The sense of 

‘team’ within the 

team and their 

willingness to 

support and help 

each other. 

Conflict solving • Do the team members concentrate 

on what is right rather than who is 

right? 

Envisioning goals 

and processes 

• Have the team members clearly 

communicated their goals? 

• Have they clearly communicated 

the plan? 

• Have team members asked for 

further clarification?

Forecasting: time 

horizons and 

range of factors 

• Have the team members

demonstrated the ability to focus 

decision making within an 

appropriate span of time? 

• Have the team members

demonstrated the ability to focus 

on a relevant breadth of concepts 

and information? 

• Have the team members

demonstrated the ability to project 

the results of their actions into the 

future?

Team 

conceptual level 

A high team 

conceptual level is 

achieved through 

the maintaining of

common ground, 

and a shared

understanding of 

goals, objectives, 

and situation 

assessment. 

Detecting gaps 

and ambiguities 

• Have the team members been able 

to easily recognize when they are 

missing certain information and 

when assumptions are inaccurate? 

• Have they had difficulty in 

determining that information is 

missing?

• Have they tended to ignore 

ambiguity?

 (Table Continues)

 19
Category Description Element Example

Team 

conceptual level 

continued 

Achieving 

situation 

assessment: 

diverging and 

converging

• Has the team sought alternate 

explanations and assessments from

team members?

• Has the team reached a shared 

understanding of the situation?

• Has the team employed mental 

simulation and perspective taking? 

Problem 

definition and 

diagnosis 

• Has the team actively gathered 

information and identified 

problems?

• Have they actively reviewed causal 

factors?

Option generation • Has the team stated an alternative 

Course of Action (CoA)?

• Have they asked members for 

options? 

Risk assessment 

and option 

selection 

• Has the team considered the risks 

of alternative CoAs? 

• Have they talked about possible 

risks for CoA?

• Has the team confirmed the CoA 

with all members? 

Team Decision 

Making 

The extent to 

which decision 

making is a 

sequential and 

deliberative 

process. In high 

performing teams 

diagnosis precedes 

option generation, 

risk assessment, 

and finally review. 

Outcome review • Has the team checked the outcome

against the plan? 

The system also includes an elements score form on which the observer notes whether the 

behavior was observed or not and a score form on which the observer rates how well the team

performs overall on that category. The evaluators first indicated whether a particular element of 

behavior was observed. Then, following the end of the exercise, the evaluator made a judgment 

about the team’s performance on each of the five dimensions (See Appendix D for elements and 

categories score forms). 

Level 4: Organization 

This is the highest level of evaluation in Kirkpatrick’s (1976) hierarchy. The ultimate 

aim of any training program is to produce tangible evidence at an organizational level. The 

problems with evaluating training at this level are that it can be difficult to establish discernible 

indicators and to be able to attribute these to the effects of a single training course. For obvious 

reasons, it is too early to try to measure organizational level change directly following the 

training exercise. However, we hope to be able to infer whether there is likely to be 

organizational level change from a comparison of the behavioral analysis across the three 

vignettes in each scenario. If we see consistent change in the right direction across the three 

 20
vignettes, over three scenarios, we will be able to infer with some level of certainty that 

participants are accommodating different points of view as a result of this training. However, 

within the limits of this project it was not possible to develop any specific measure of 

Organizational level change. 

Results of the Training Evaluation 

Preliminary Evaluation 

Two assessment methods were used in the preliminary evaluation. First the reactions 

survey was used to determine how participants reacted to the training. Second, the post training 

discussion was used to get participants’ feedback on the training and the tool. The outcome of 

this assessment process was used to guide our development process. Preliminary evaluations 

took place at the Joint Readiness Training Center (JRTC) at Fort Polk, Louisiana. Three U.S. 

Army JRTC Observer/Controllers (O/Cs) took part in the preliminary evaluation. There were 

two Captains (CPT) and one LTC who acted as the U.S. training participants. Two of the three 

participants had previous coalition operational experience in Iraq. An SME (retired Army

officer) with coalition experience played the roles of the Australian, Canadian, and UK 

contingent. The purpose of the preliminary evaluation was not to test the learning taking place, 

but rather to test the functionality of the training tool and the usability of the training strategy. 

Thus, preliminary evaluation data were limited to the Reactions Survey and Discussion 

Questions - as discussed in the training evaluation plan. 

Preliminary Evaluation Reactions Survey Results 

Participants were asked to rate on a scale of 1 (none at all) to 5 (a great deal) their 

answers to four questions about the training they had just been through. In all cases, the number 

of participants was three. Mean ratings are provided here as the consistency across participants’ 

responses was high. 

• How much value is there to training with coalition partners in general? Mean rating = 4.6 

• How much value was there in training with coalition partners in this exercise? Mean 

rating = 2 

• How would you rate the quality of the content presented in this exercise? Mean rating = 

2.33 

• How frequently should a staff member participate in similar training (On a scale of 1 

(never) to 5 (once a month)? Mean rating = 2.6 

Participants were also asked to list three things that made the training effective. All three 

participants stated that interaction with partners, working together to solve a problem, and the 

conversation with partnering nations were the factors that made the training effective. When 

asked to list the three things that would make the training more effective, the following are the 

responses in order of the number of times they were mentioned. 

• Greater use of graphic tools. 

• More detailed maps, more detailed information in the scenarios. 

 21
• More incorporation of doctrine. 

• The ability to interact to solve problems. 

A final question asked participants to provide any additional comments they would like 

to make about the training. Comments included: 1) The need for greater clarity about the 

objectives of the exercise before the training; and 2) the need to set the dilemmas at a level 

appropriate to the rank and role of the participants and the role they are being asked to play in the 

scenario. For example, battalion (BN) and brigade (BDE) operational level Multi-National 

Division (MND) staffs deal with tactical issues while CJTF staffs deal with strategic, political, 

and long term issues. 

Preliminary Evaluation Discussion Results 

Participants provided feedback on four main issues: the tool itself, the scenarios, the 

participant requirements, and the facilitation. A summary of the points they raised is presented 

below. 

The three participants felt that that there was a lack of connection between the graphic 

and the text; they would have liked more opportunity to put the graphic to use in solving the 

problem. In fact, the requirements (or challenges set for participants) were seen as not specific 

enough to require the use of the map. A related issue is the lack of a cause and effect dilemma 

with which participants could grapple. There was also consensus on the difficulties involved in 

using the chat function as a main means of communication. The “chat” became disjointed and 

difficult to follow; it was generally agreed that using the speaker phone was more useful than the 

chat function. 

The participants saw the scenarios as realistic and accurate, but needing to be pitched at 

the appropriate level (i.e., at the tactical level for BN and BDE level staffs, and at the strategic 

level for CJTF level staffs [these would be theater level issues]). They felt the scenarios should 

provide more information about capabilities, and the process should include giving different 

information to each team. Finally, the facilitation should provide the opportunity for more 

coordination during problem solving, and learning objectives should be given up front. 

In response to the issues raised by participants through the pre-evaluation reaction survey 

and discussion session, modifications were made to the scenarios, the participant questions and 

the facilitation guides. For example, participant questions were modified to allow for greater 

interactions and discussion over problem diagnosis as well as problem solving. The factual 

details in the scenarios were changed to make them more level appropriate, and potential 

participant questions and answers were modified to reflect the changes. 

Main Evaluation 

A total of 18 participants took part in the main evaluation (nine from the U.S. Army, 

three from the Canadian Army, three from the UK Army, and three from the Australian Army). 

Of the 18 participants, the ranks included five CPTs, four LTCs, two commanders (CMDRs), 

and seven MAJs. Fourteen participants had previous experience working with coalition partners. 

 22
In total, nine had coalition experience in Iraq, six had coalition experience in Bosnia, two had 

experience working with coalition partners in Germany, and one in East Timor. Four 

participants had no experience working with coalition partners. Most of the participants held 

positions related to training, development, and operational analysis including: doctrine writer; 

exercise planner; operational training developer; distributed learning director; leadership training 

planner; deputy director of training and development; program manager for training, doctrine 

and combat development; course tutors. 

Evaluation took place over three days. On each day three U.S. players and three players 

from either Australia, Canada, or the UK played through one scenario consisting of three 

vignettes. The teams were connected via telephone and through the web-based training tool. An 

observer accompanied each team. The observer provided participants with hard copies of the 

vignettes and distributed the survey at the end of the sessions. A remote facilitator experienced 

in military coalition operation facilitated the training sessions. Three training developers and one 

tool developer remotely observed the training and facilitated the survey and discussion sessions. 

Level 1: Reactions: Survey and Discussion 

Level 1 refers to how participants reacted to the training. While negative and positive 

reactions are not necessarily indicators of learning, the reactions were used to guide our 

development process. 

Survey Results. Participants rated on a scale of 1 (none at all) to 5 (a great deal) the 

following questions about the training they had just been through. In all cases, the number of 

participants was 18. 

• How much value is there to training with coalition partners in general? Mean rating = 4.9 

• How much value was there in training with coalition partners in this exercise? Mean 

rating = 4 

• How would you rate the quality of the content presented in this exercise? Mean rating = 

3.3 

• How frequently should a staff member participate in similar training? On a scale of 1 

(never) to 5 (once a month) Mean rating = 3.7 

When asked to list three things that made the training effective, three main factors were 

identified as key to the effectiveness of the training (responses are listed in order of frequency of 

mention by participants): 

• Conversation with partnering nations. 

• Realistic scenarios. 

• Experienced facilitator. 

All of the respondents agreed that conversation between the partners was the most 

effective aspect of the training. This provided them with the opportunity to exchange views, 

discuss doctrinal differences, and generate greater awareness of similarities and differences in 

operational activities. The fact that this discussion was facilitated by an experienced facilitator, 

 23
and that it was in connection with relevant operational issues was also acknowledged by a 

majority as being highly effective. It was acknowledged that the experienced facilitator provided 

probing questions and extra information that facilitated a broadening of perspective. A majority 

of respondents said that the scenarios were realistic, authentic, and factually accurate. 

Respondents were also asked to list three things that would make the training more 

effective. Below are the responses listed in the order of frequency that they were mentioned by 

participants. Six factors were identified: 

• Better use of graphics and graphic tools. 

• Greater interactivity. 

• Read-ahead materials. 

• More time to consider the problems. 

• Tailor the exercises to the target audience and pitch requirements at the correct level. 

• Greater closeness to Military Decision-Making Process (MDMP) and military doctrine. 

There was general consensus regarding the need for greater interactivity between the 

requirements set for participants and the graphic. In total five participants would have preferred 

having a requirement to use the graphic to answer the questions. In total nine participants would 

have liked the opportunity for even more interaction with partnering teams. Rather than simply 

sharing and discussing their responses to problems, they would have liked more opportunities to 

solve problems together. It was generally agreed that participants should be provided with readahead materials that outline the purpose of the exercise and provide background to the conditions 

of the scenarios and that participants should be given more time to consider the problems and 

come up with solutions. 

A total of seven participants commented that scenarios, roles and requirements should be 

pitched at the correct level for the participants, and moreover that the requirements should be 

matched to the participant’s role within the scenario. A somewhat related issue is the desire for 

the scenarios and requirements to match the MDMP and military doctrine more closely. 

Finally, participants were asked to provide any additional comments they would like to 

make about the training. The comments were focused around four main issues: 

• The need for greater clarity about the objectives of the exercise. 

• The need to set the dilemmas at a level appropriate to the rank and role of the participants 

and the role they are being asked to play in the scenario. 

• The need for a greater level of interactivity with partners in problem solving. 

• Usefulness of the training. 

Discussion Results. Participants provided feedback on six main issues: 

• The training tool. 

• The scenarios. 

• Participant requirements. 

• The facilitation process.

 24
• The relevance of the training. 

• Overall reactions. 

Regarding the tool, it was generally felt that there was limited interaction with it and 

particularly with the graphics presented. A majority of participants would have liked more 

opportunity to engage in problem solving processes that involved greater interactivity with the 

tool and the graphics. A majority of participants felt that the requirements supported the learning 

process. However a minority stated that they would have liked more time to consider the 

problems presented, that requirements could have been more specific, and that they should have 

had more interaction with partners in the problem solving process. 

The scenarios were generally perceived to be authentic and relevant; however, two 

participants preferred that they be closer to military doctrine. In total, three command and 

control (C2) anomalies were highlighted which were corrected in the final version of the 

scenarios. Two participants recommended the use of actual incidents in the scenarios. It was 

widely agreed that the experienced facilitator added value to the process by introducing other 

perspectives and broadening those perspectives. There was general agreement regarding the 

relevance of the training for Soldiers preparing for deployment to a coalition operational 

environment. However, there was a mix of beliefs about whether previous coalition experience 

was necessary or desirable for participation in the training. About half of the participants felt 

that this training is only appropriate for those with previous coalition experience, and that it is 

not suitable as baseline training. Conversely, half of the participants felt that anyone on a BDE 

or BN staff would benefit from this training. 

Overall, the training provided a good opportunity to problem solve and exchange ideas 

with coalition partners. A majority of participants stated that the exercise helped them to 

understand not only the partner’s point of view, but also how the partner might view their 

actions. 

Level 2: Learning: Situation Awareness (SA) Exercise 

The SA Exercise was used to assess whether learning was taking place as a result of the 

training. The first SA Exercise was completed before any discussion of collaborative problem

solving took place between participants. The second and third SA Exercises took place after 

participants engaged in discussions and collaborative problem solving activities. Below are 

participants’ responses from the SA Exercise from Situation 1 (collated across all three 

scenarios). 

Responses from SA calibration exercise for Situation 1 

1. Q What are your top three concerns and why? 

A Tactical, infrastructure, task constraints, resources, personnel, gaining control of 

situation, budget, priorities, reconnaissance, intelligence, situation report 

(SITREP), threat, timing, location. 

2. Q What are your assumptions about this situation? 

 25
A Capabilities, supplies, insurgent actions, security level, risk level, current state 

command and control, posture. 

3. Q What are your requests for information from the other team? 

A Capabilities, only logistical support, resources, none, enemy/insurgent activity, 

mission and tasks, command, control, communications, computers, intelligence, 

 surveillance, and reconnaissance (C4ISR) compatibility, adjacent commands. 

Below are participants’ responses from the SA Exercise from Situation 2 and 3 (collated 

across all three scenarios). 

1. Q What are your top three concerns and why? 

A Threat assessment, engaging leaders, what additional assets will other team

require, clarify responsibilities with partnering organization, what is the effect of 

own posture on climate, coalition cohesion, who is the lead agency, developing a 

joint plan, any competing priorities, who has primacy, consider complicating 

political/coalition factors.

2. Q What are your assumptions about this situation? 

A Climate, likely response to actions, likely Course of Action (CoA) of partners 

3. Q What are your requests for information from the other team? 

A Considering ROE, coordinating with other nations Information Operations (IO) 

and public affairs, requests for information about CoA, question other team’s 

priorities, question national caveats, requests for additional information, anticipate 

local reactions, anticipate partner reactions/implications, other actors/other 

involved parties. 

The results of the first SA Exercise suggests that respondents are already taking partners’ 

resources and capabilities into consideration in their planning and decision making. However, 

examination of the second and third SA Exercises suggests that participants are considering a 

broader range of issues and have greater awareness of partners’ perspective in the second and 

third vignettes – after discussion and collaborative problem solving activities have taken place. 

For example, in the second and third vignettes participants raised the following issues that were

not considered in the first vignette: estimating what assets partners may require, clarifying 

responsibilities with partnering organizations, coalition cohesion, coordinating with other 

nations, and anticipating partner reactions/implications. These findings suggest that the 

discussion sessions and collaborative problem solving process promoted a greater awareness of 

coalition issues and how to respond to them. 

Level 3: Behavior: Behavioral Markers 

Behavior, as defined here, refers to transfer of behaviors learned to the job or a simulated 

environment. While we were not assessing team behaviors in an environment separate from the 

training evaluation, we did examine teamwork performance during training and assessed the 

utility of a tool for assessing teamwork. Such a tool could be used to assess teamwork prior to 

and after training or across training scenarios. Evaluators at each site during the evaluation rated 

team behaviors using a behavioral marker system. The evaluators first indicated whether a 

particular element of behavior was observed. Then, following the end of the exercise, the 

 26
evaluator made a judgment about the team’s performance on each of the five dimensions. The 

mean ratings given by six observers across each category, on a scale of 1 (very poor) to 5 (very 

good), are shown in Table 3. The ratings table suggests that teams performed adequately on all 

dimensions of teamwork assessed in this exercise. The mean ratings for team conceptual level 

were given the highest ratings, cooperation within teams was rated the lowest, however, no 

statistically significant difference was found to exist. 

Table 3 

Mean Observer Ratings for Five Behavioral Categories 

Category Mean ratings across three scenarios 

Team identity 4.1 

Team conceptual level 4.3 

Team self monitoring 4.0 

Decision-making 4.1

Cooperation 3.6

It should be noted that this is a within team rating of performance. The results do not 

show whether there was an improvement in teamwork between teams. 

Level 4: Organization 

The ultimate aim of the highest level of training evaluation is to produce a difference at 

the organizational level. No specific measure of organizational level change was developed for 

this project. The small numbers of participants and limited scope of this evaluation prohibited 

any such endeavor, therefore, it is difficult to assess whether organization level change occurred 

as a result of this training. However, the results of the SA calibration exercise show that 

participants are considering a broader range of issues and have greater awareness of partners’ 

perspective following discussion and collaborative problem solving. Moreover, the results show 

consistent change in the right direction across the three vignettes for all three scenarios. These 

results show that organizational level change is possible with wider implementation of the 

training. However, specific measures to assess such a change would have to be developed. 

Lessons Learned 

The training effort was based on the theoretical foundation of previous training 

development in the area of advanced cognitive training for tactical thinking. The Commander 

TLAC program addresses tactical thinking skills, while much of the Decision Skills Training 

development has focused on tactical decision making. The TLAC is built on the notion of 

theme-based training. This was translated into the development of High Level Focus Areas in 

this project. Decision Skills Training provided a foundation of facilitation techniques for 

exploring and improving mental models of decision-making. This effort explored the application 

of these successful learning strategies, using a low technology, web-based, distributed, 

synchronous training method. A number of lessons learned from evaluation efforts set the stage 

 27
for the use and refinement of the Global Teams coalition training developed in this project both 

in terms of the learning strategy and the training method. 

Technology 

The technology supporting this training was purposely designed to be low tech so that 

participants with a low level of technology available would still be able to access the tool and 

participate in the training. Specifically, the tool was designed so that either a dial-up system or a 

DSL connection could be used to gain access. Nonetheless, significant problems were 

encountered with the technological side of this training. First, firewalls and blocks on streaming 

communications created considerable access problems for all participants, both national and 

international. A significant time investment was required to resolve these issues and involved 

the coordination of computing technicians from partnering nations. Even the telephone 

connection created some problems. International dialing codes and speaker phone problems

created delays in the training, and there were significant costs for international telephone charges 

as well as potential problems for users to obtain permission to make the calls to support training. 

A functionality which has the ability to overcome this latter problem is the introduction of voice 

over internet protocol (VOIP) to the tool. We originally did not add this function to conform to 

our design criteria of “low technology.” This functionality increases bandwidth issues with the 

technology and means that a DSL connection will be required to use the technology, but it also 

eliminates one communication channel problem. Moreover, participants appear to prefer hightech functionality, particularly those that improve the real-time interaction with partners. 

For the future, the tool should have an upload function which would allow an administrator or 

facilitator to upload new or existing scenarios. Our goal is to allow researchers to add scenarios 

for research, and the administrative rights of allowing various facilitators to upload scenarios will 

be the purview of the eventual user audience, should that function be desired. 

The Learning Strategy and the Training Audience 

There is a need to clarify the intent of the training and the learning objectives ahead of 

the training exercise. Participants need to have a context for the training so that they can situate 

themselves to that learning environment. A few participants in the evaluation were 

uncomfortable that this training did not correspond well with the doctrinal training they have 

received over many years and the fact that this training did not follow the same format as the 

Military Decision Making Process (MDMP). If training were to be considered on a continuum

of highly structured at one end to highly unstructured at the other end, the MDMP training 

process would sit at the highly structured end of the continuum whereas the Global Teams 

training would sit at the highly unstructured end. The aim of the Global Teams training is real 

time discovery and the development of understanding between participants. Each training 

session will uncover different aspects of the problem environment due to the different 

experiences and perspectives that participants bring to the training environment. Some of the 

discomfort felt by participants may be due to the fact that the training presented here did not 

conform to their previous experience of military training. This training forced the participants to 

consider problems without the guiding principles of doctrinal training, since doctrine is different 

across coalition nations. A minority of participants were uncomfortable with this level of

flexibility in the training process. Discomfort was also caused by the fact that there may have 

 28
been a mismatch between participants’ expectations for the training and the training they actually 

received. To this end, facilitators should clarify what the training is and what it is not. In 

particular, it should be emphasized that this training is not intended to be an extension of MDMP 

training, and it is not an exercise in detailed planning. Rather, the aim of the training is to 

introduce participants to coalition challenges, provide them with a safe environment to explore 

differences in culture, approach, stance, and style, and help them to de-center and take on 

another’s point of view when engaging in collaborative problem solving. 

Emphasis should be placed on the problem solving and coordination aspects of the 

exercise rather than on the accuracy of the facts or participants’ experiences of how things 

operate in theater. Facilitation can focus the participants away from disagreement with the 

scenario and onto the issues of collaboration by emphasizing that this is a notional situation 

based in a realistic part of the world. Changes should be made if the facts interfere with 

judgments and decisions the participants are making, or if the facts do not reflect the nature of 

the actual types of decisions and situations confronted by experts in the field. 

The evaluation process highlighted that the greatest benefit came from the interaction of

partners over problem solving and discussion of courses of action or plans for action. It is clear 

that this learning strategy and training method should capitalize on the opportunity for 

interaction as much as possible by allowing partners to interact not only over discussion but also 

over problem solving tasks. This could involve allowing participants the opportunity to set their 

own learning agenda by allowing them to figure out how to coordinate most effectively or to 

figure out what information or resources they need from each other or how the problem should 

be solved or tackled. For example, one suggestion during the evaluation was to give teams from

each country different information to bring to the problem (relevant to what they might actually 

know in the situation) to force collaboration. 

Essentially, the more collaboration and interaction participants engage in, the better. The 

problem solving process would allow them not only to work together on solving the problem at 

hand, but also provide the opportunity for participants to collaborate over defining the problem

and the role of each nation in resolving it. Deliberating courses of action and assumptions 

behind them follows the facilitation techniques of Decision Skills Training. Honing the 

challenges in the scenarios and getting the right mix of participants working together at the right 

level will allow researchers to better develop the facilitation techniques to increase interaction. 

There was a definite reluctance on the part of participants to disagree and contradict 

partners’ assumptions, plans, or goals. This may be due to the fact that military culture

transcends national cultural differences. It may be due to the fact that English-speaking partners 

are more similar than different, or it may be due to a mutual respect for partners of each others’ 

viewpoint. We identified a number of conflicting assumptions in our data collection, and we 

expect there is more difference among English-speaking partners than was shown in the 

evaluation. It is also likely that the participants in the evaluation were not equally matched in 

terms of experience. Some partners had more to offer to the training experience than others. 

Fruitful explorations of the concepts captured in the HLFA are not possible without some

background experience in these areas. Alternately, providing more specific information in the 

scenarios to highlight areas of difference and create more of a dilemma would probably benefit 

 29
less-experienced participants (such as those who soon will be going into coalition operations for 

the first time). Nonetheless, it is important to get at the differences and to get participants to 

state their stance clearly, even if it disagrees with a partner’s stance. 

An important lesson for the evaluation process is the need for the training to be set at the 

appropriate level. In other words, the scenario itself, the requirements, participants’ actual rank, 

and their assumed role in the scenario all should be in synch. If any of these elements are out of 

place, participants may be reluctant to make decisions at a level that is incongruous with their 

role or rank. Specifically, BN and BDE level staff operating at the Multi-National Division level 

should deal with tactical issues. Combined Joint Task Force staff should be tasked to deal with 

operational levels and strategic issues. While the tactical and operational levels of assessment 

and action are beginning to blend, and more complex decisions are being made at lower echelons 

than before, training participants still need to feel situated in their role and level of responsibility 

to support discussion and collaboration. 

An experienced military or former military facilitator is a crucial component of this 

training methodology. Facilitators with a military background provide an element of authenticity 

to the training experience. They can answer questions and field participant questions better than 

non-military facilitators; they can add to the discussion and suggest alternative courses of action; 

and they can feed off participant questions to suggest alternative interpretations of events. For 

greater success, these facilitators need to work with a researcher or trainer to understand the 

intent of the facilitating for advanced cognitive training, or possess some background in this 

level of training already. 

Online Facilitation of Collaborative Learning 

In facilitating the problem solving process, it is important for the facilitator to encourage 

participants to think (out loud) about how they are thinking about the problem, as opposed to 

simply thinking about solutions. In particular, facilitators should expose participants’ 

assumptions about the problem situation and how these may differ across nations. Bringing 

assumptions to light during the evaluation training sessions was perhaps the most effective part 

of facilitation in this project. The focus for continued development of facilitation is to get the 

participants into actual collaboration over problems while not letting the technical problems

become the focus. The participants must continue to examine their assumptions as the 

collaboration is ongoing. 

Tool functionality must be tied into the learning strategy. In other words, the scenarios, 

the graphics that support them, the participant requirements and how they are answered, and the 

facilitation process must all be synchronized with the functionality of the tool. In this way the 

functionality actually supports and enhances the learning. The training method and learning 

strategy, as we are using the terms, are linked in online collaboration. Tools should support 

facilitators at different levels of expertise, guiding the processes as needed. While such tools are 

available more readily for operational collaboration (such as meeting support), a similar 

technology development is not seen in education. However, attention to the role and processes 

of online facilitation is found in the educational research community. Technology for online 

collaborative learners should support the roles and processes being developed in the educational 

 30
literature. Current research into the roles and responsibilities of the online facilitator can serve as 

a basis for describing the roles and responsibilities of the facilitator for the Global Teams tool 

(Berge, 1995; Hootstein, 2002). This literature serves as a basis to clarify the facilitator’s role 

and need for technology support, but refinement for this particular level of cognitive learning and 

training audience will still be needed. 

Conclusions 

With the ever-increasing involvement in coalition operations, it is essential to begin 

looking at training and readiness. The training that currently exists is limited in scope, depth, 

and breadth, and tends to be mission-specific and region-specific. This report describes the 

research and development of a web-hosted, scenario-based training tool for coalition forces. The 

analysis of interviews with experienced coalition forces revealed a number of key focus areas 

and awareness points which were central to the training tool. Scenarios were developed to force 

participants to engage in problem solving in these areas, to discuss and grapple with them. The 

evaluations highlighted two key findings: first, that the scenario challenges need to support 

collaborative problem solving and second, that the training tool should support the collaborative 

process. One way to achieve the former is to provide participants with challenges that are as 

specific as possible so that participants are forced to coordinate and collaborate in solving them. 

A way to modify or enhance the existing process would be to give teams differing information 

and tasks so that they have to come together to achieve one larger task or challenge. For 

example, each team could be tasked with different information and area of responsibility towards 

a mission. The challenge should require them to come together to discuss and exchange 

recommendations based on prior knowledge and tasking. 

The evaluations highlighted a number of factors which can impact the effectiveness of 

the training including: the importance of clarifying the intent of the training and the learning 

objectives in advance, the need to emphasize the problem solving and coordination aspects of the 

exercise rather than the accuracy of the facts, the merit in capitalizing on the opportunity for 

interaction as much as possible by allowing partners to interact not only over discussion but also 

over problem solving, the need for the training to be set at the appropriate level, the value of an 

experienced facilitator, and the significance of tying tool functionality to the learning strategy. 

Overall, the training was shown to improve participants’ awareness of and ability to respond to 

the key themes of coalition coordination. The tool may provide an easy-to-implement and costefficient means for coalition partners to train in a distributed environment, on short notice, prior 

to deploying to the operating environment. 

This research effort was designed to test the viability of the collaborative distributed 

training method as a means to train distributed teams in coalition problem solving and 

coordination. The products from this effort include a web-based training tool, three predeveloped scenarios and three accompanying facilitation guides. The tool incorporates both a 

database and server-based filing structure that contains the pre-developed scenarios. Library 

scenarios can consist of up to four total graphics, four situations (including a background), rules 

of engagement, and a facilitator’s guide. This tool is available from the U.S. Army Research 

Institute for the Behavioral and Social Sciences (ARI) to parties interested in hosting such 

training. 

 31
 32
References 

The American-British-Canadian-Australian (ABCA) Program. (2001). Coalition Operations 

Handbook. Retrieved April 11, 2005 from http://www.apan-info.net/peace_operations/ 

uploads/coh.pdf. 

Avermaete, V., & Kruijsen, E. (Eds.). (1998). The evaluation of non-technical skills of multipilot aircrew in relation to the JAR-FCL requirements. Amsterdam, The Netherlands: 

NLR: EC NOTECHS. 

Bensahel, N. (2003). Preparing for coalition operations. In L. E. Davis & J. Shapiro (Eds.). The 

U.S. Army and the new national security strategy (Vol. MR+1657-A, pp. 111-128). 

Berge, Z. (1995). Computer mediated communication and the online classroom. Retrieved 

November 1, 2005 from http://www.december.com/cmc/mag/1995/feb/berge.html

Cannon-Bowers, J. A., & Salas, E. (1997). A framework for developing team performance 

measures in training. In M. T. Brannick, E. Salas & C. Prince (Eds.), Team performance 

assessment and measurement. Mahwah, NJ: Lawrence Erlbaum & Associates. 

Crandall, B., & Getchell-Reiter, K. (1993). Critical decision method: A technique for eliciting 

concrete assessment indicators from the "intuition" of NICU nurses. Advances in Nursing 

Sciences, 16(1), 42-51. 

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for 

qualitative research. Chicago: Aldine Pub. 

Hoffman, R. R., Crandall, B. W., & Shadbolt, N. R. (1998). Use of the critical decision method 

to elicit expert knowledge: A case study in cognitive task analysis methodology. Human 

Factors, 40(2), 254-276. 

Hootstein, E. (2002). Wearing four pairs of shoes: The roles of e-learning facilitators. 

Retrieved November 2, 2005 from http://www.learningcircuts.org/2002/oct2002elearn.html

Joint Chiefs of Staff. (2000). JP 3-16 Joint doctrine for multinational operations. Retrieved 

November 2, 2005 from http://www.dtic.mil/doctrine/jel/new_pubs/jp3_16.pdf 

Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig & L. R. Bittel (Eds.). Training 

and development handbook (pp. 18.11-18.27). New York, NY: McGraw Hill. 

Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San 

Francisco, CA: Berrett-Koehler Publishers. 

Klein, G. A. (1989). Recognition-primed decisions. In W. B. Rouse (Ed.). Advances in manmachine systems research: Vol. 5 (Vol. 5, pp. 47-92). Greenwich, CT: JAI Press, Inc. 

[also published as DTIC No. ADA240659]. 

 33
Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting 

knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462-472. 

Klein, G. A., & Hoffman, R. (1993). Seeing the invisible: Perceptual/cognitive aspects of 

expertise. In M. Rabinowitz (Ed.). Cognitive science foundations of instruction (pp. 203-

226). Mahwah, NJ: Lawrence Erlbaum & Associates. 

Militello, L. G., & Hutton, R. J. B. (1998). Applied Cognitive Task Analysis (ACTA): A 

practitioner’s toolkit for understanding cognitive task demands. Ergonomics, Special 

Issue: Task Analysis, 41(11), 1618-1641. 

Peters, J. E., Johnson, D., Bensahel, N., Liston, T., & Williams, T. (2001). European 

contributions to operation allied force: Implications for transatlantic cooperation, Rand 

Research Brief, MR-1391-AFhttp://www.rand.org/publications/RB/RB72/. San Jose, CA: 

Rand Corporation. 

Pliske, R. M., McCloskey, M. J., & Klein, G. (2001). Decision skills training: Facilitating 

learning from experience. In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic 

decision making (pp. 37-53). Mahwah, NJ: Lawrence Erlbaum Associates. 

Pliske, R. M., Militello, L. G., Phillips, J. K., & Battaglia, D. A. (2001). Evaluating an approach 

to MOUT decision skills training (Technical Report prepared under Contract #DASW01-

99-C-0002 for U.S. ARI (Infantry Research Unit Ft. Benning, GA). Fairborn, OH: Klein 

Associates [also published as DTIC No. ADA399392, http://www.dtic.mil]. 

Ross, K. G., & Lussier, J. W. (1999). A training solution for adaptive battlefield performance. 

Paper presented at the 1999 Interservice/Industry Training, Simulation, and Education 

Conference, Orlando, FL. 

Ryan, A. (2000). Achieving C3I Interoperability for Coalition operations: Lessons from 

INTERFET. Canberra, Australia: Land Warfare Studies Centre. 

Schraagen, J. M. C., Chipman, S. F., & Shalin, V. J. (Eds.). (2000). Cognitive task analysis. 

Mahwah, NJ: Lawrence Erlbaum & Associates. 

Shadrick, S. B., & Lussier, J. W. (2002). Think like a commander at the armor captain’s career 

course. Retrieved November 14, 2005 from http://www.iitsec.org 

U.S. Army Training and Doctrine Command. (2001). Statement of required capabilities future 

combat system of systems (FCS), p. 6-29. Fort Monroe, VA: Author. 

Zsambok, C. E., Klein, G., Kyne, M., & Klinger, D. W. (1993). Advanced team decision 

making: A model for high performance teams (No. Contract MDA903-90-C-0117 for U.S. 

Army Research Institute for the Behavioral and Social Sciences). Fairborn, OH: Klein 

Associates Inc. 

 34
Appendix A 

Global Teams Training Tool: Primary Supported Functionalities 

 A-1
Global Teams Training Tool: Primary Supported Functionalities 

• Login Restriction 

At the introduction screen, access is restricted to only those users who have recognized

Player IDs. 

• Dynamic Player Notes Area 

At the introduction screen, a dynamic note area allows facilitators to leave initial 

guidance to players, such as timing of the next session and specifics on which vignette is 

to be conducted. 

• Dynamic Library of Vignettes 

The tool incorporates both a database and server-based filing structure that contains three 

pre-developed vignettes. Library vignettes can consist of up to four total graphics, four 

situations (including a background), rules of engagement, and a facilitator’s guide. The 

tool also stores vignette identifiers to support the recognition/selection of desired 

vignettes. After initially logging in, the tool presents summary information (title, 

descriptive summary, creation date, and author) for each vignette in the database, and

allows the user to scroll through the summaries and select the vignette of interest. 

• Customized Interface/Functionality Based on User Login 

Users who log in as facilitators have access to all player responses as they are entered, 

whereas players representing the countries and organizations can only view their own 

responses, unless the facilitator shares information. Further, the facilitators/observers 

have access to an onscreen chat area only visible to them. This feature supports 

administrative chats (e.g., how to proceed, observations of a session, when to take breaks, 

etc.) that would otherwise distract the players. 

• Ability to Upload New Vignettes 

An innovative and essential feature of the tool is its ability to support the uploading of

new vignettes. Users who are logged in as administrators have the option of uploading 

their own vignettes, including text, graphics, supporting summary, and facilitation 

information. This allows for an ever-growing body of vignettes that, once uploaded, can 

be conducted by any experienced facilitator. 

• Collaborative, Synchronous Graphics Manipulation 

As the tool presents graphics associated with vignettes during a session, all users have the 

ability to superimpose images and text over the graphics. Users can choose to draw line 

segments, directional arrows, boxes, circles, and even freeform sketches for all other 

users see. Users can also enter freeform text and/or textboxes and customize the colors of 

any of their selections. The tool allows users to individually select, move, or delete 

individual drawing items, or clear the entire screen with the push of a button. 

• Collaborative, Synchronous Whitespace 

In addition to the graphics drawing, the Global Teams tool provides a shared white space 

where players and facilitators can sketch their own maps. This supports more freeform 

play of a vignette, where a facilitator can create a new situation and have players respond. 

• Private Response Forms 

Players have access, depending on their login, to specific response forms. These forms 

are not visible to other players and provide a semi-private area for a player/team to work 

out and record answers to vignette questions or requirements. Facilitators have access to 

all the response forms so they can: 1) assess progress online, 2) determine whether 

 A-2
individual players/teams require additional assistance or need to be redirected, 3) observe 

the response development process, and 4) evaluate relative responses as they are being 

formed. 

• Split-Screen Shared Space 

The Shared Space area of the tool consists of two vertically aligned shared text spaces 

where the facilitator can present information that all players can simultaneously view. 

The split-screen feature allows facilitators to present responses from different 

players/teams to the same questions side-by-side (if desired), to support the comparing 

and contrasting of responses. 

• Mouse/Chat Color and Name Identification 

Often in collaborative, online environments it is difficult to identify who is doing what

onscreen. For this reason, the tool provides name identifiers and attaches them to each 

user’s mouse icon. To do this, the player enters a screen name that he/she wishes to use, 

and then his/her pointer icon displays that name for other users to see. Further, individual 

players/facilitators can also customize their pointers by selecting their own color. These 

identifiers (color and name) distinguish individual users in the main chat area. 

• Chat/Facilitator Chats 

The tool provides an area where any/all users can have group chats. The tool also 

provides an additional area, only accessible to facilitators, where this subgroup can 

conduct private chats. Again, this feature supports administrative chats (e.g., how to 

proceed, observations of a session, when to take breaks, etc.) that would otherwise 

distract the players. 

• Optional Audio Conferencing 

Users have the option of utilizing the tool’s streaming audio functionality. If desired, 

players and facilitators can have their conversations streamed over the internet, 

eliminating the need for a phone connection. Players using this feature can either have 

the tool automatically detect when they are talking or manually trigger the sending of

audio (to reduce extraneous noise or to ensure privacy in offline discussions). Users can 

transfer between modes with ease. Onscreen display features are also presented to show 

which user is currently the active speaker. 

• Player List 

The screen names of all users who are logged onto the tool are displayed in a player list 

box. This feature prevents anonymous observations and also provides verification that all 

necessary players are present for the vignette session. 

• Customized Pop-up Help Screens 

To support novice users, the tool provides help files, customized to different functional 

areas within the tool. A help file is available to assist facilitators in uploading new 

vignettes, providing specific guidance in how to prepare the required graphic and 

MSWord files, and how to identify them through the tool’s upload interface. Additional 

help files provide guidance in implementing the main interface functionalities. 

 A-3
 A-4
Appendix B 

Reactions Survey 

 B-1 
Reactions Survey 

1. Your current duty position: 2. Rank: 

3. Briefly list any past experience working with coalitions and your job in those coalitions: 

4. How much value is there to training with coalition partners in general? 

None at all Very little Uncertain Some A great deal 

1 2 3 4 5

5. How much value was there in training with coalition partners in this exercise? 

None at all Very little Uncertain Some A great deal 

1 2 3 4 5

6. How would you rate the quality of the content presented in this exercise?

Very poor Poor Satisfactory Very good Excellent 

1 2 3 4 5

7. How frequently should a staff member participate in similar training? 

Never Once Once a year 2-3 times 

a year 

Once a month 

1 2 3 4 5

8. List three things that made the training more effective and explain why. 

a. 

b. 

c. 

 B-2 
9. List three things that would make the training more effective and explain how. 

a. 

b. 

c. 

10. Any additional comments: 

 B-3 
 B-4 
Appendix C 

Discussion Questions 

 C-1 
Discussion Questions 

Reactions 

• How useful did you find this training?

• Would you be willing to participate in a similar exercise six months from now?

Application

• How valuable was it to have a group from another country participating?

• What military positions would benefit most from this training? 

Further development 

• In your opinion, are the military details presented in the scenario and the related 

questions authentic? How could they be improved?

What did the facilitator do to support and reinforce learning during the exercise? How could 

facilitation be improved? 

 C-2 
Appendix D 

Behavioral Marker System – Score Forms

 D-1 
Behavioral Marker System (Elements Score Form) 

Team identity Y/N Team conceptual level Y/N

• Defining roles and functions 

• Engaging 

• Compensating 

• Avoiding micromanagement 

• Envisioning goals and processes 

• Forecasting: time horizons and range of 

factors 

• Detecting gaps and ambiguities 

• Achieving situation assessment: 

diverging and converging 

Team self monitoring Team decision making 

• Time management/workload 

management

• Adjusting 

• Problem definition and diagnosis 

• Option generation 

• Risk assessment and option selection 

• Outcome review 

Cooperation 

• Teambuilding and maintaining 

• Consideration of others 

• Support of others 

• Conflict solving 

Behavioral Marker System (Categories Score Form) 

Categories 

Very poor

Poor

Acceptable

Good

Very good

Not Observed

Comments 

Team identity 

Team self monitoring 

Cooperation 

Team conceptual level 

Decision making 

 D-2