Technical Report 1194

Army Excellence in Leadership (AXL):

A Multimedia Approach to Building

Tacit Knowledge and Cultural Reasoning

Michelle L. Zbylut and Kimberly A. Metcalf

U.S. Army Research Institute

Julia M. Kim, Randall W. Hill, Jr., and Scott Rocher

USC Institute for Creative Technologies

Christopher Vowels

Kansas State University

Consortium Research Fellows Program

January 2007 20070206260

United States Army Research Institute

for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the Department of the Army

Deputy Chief of Staff, G1

Authorized and approved for distribution:

SCOTT E. GRAHAM MICHELLE SAMS

Acting Technical Director Acting Director

Research accomplished under contract

for the U.S. Army Research institute

USC Institute for Creative Technologies

Technical review by

Allison Abbe, U. S. Army Research Institute

Daniel B. Horn, U. S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS,

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Technical Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

January 2007 Interim October 2005 - December 2006

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Army Excellence in Leadership (AXL): A Multimedia Approach W91 1 NF-04-D-0005

to Building Tacit Knowledge and Cultural Reasoning 5b. PROGRAM ELEMENT NUMBER

622785

6. AUTHOR(S): 5c. PROJECT NUMBER

Michelle L. Zbylut and Kimberly A. Metcalf (U.S. Army A790

Research Institute);

Julia M. Kim, Randall W. Hill, Jr., and Scott Rocher (USC 5d. TASK NUMBER

Institute for Creative Technologies); 269

Christopher Vowels (Kansas State University) 5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

U.S. Army Research Institute for the Behavioral and Social

Sciences

ATTN: DAPE-ARI-RK

851 McClellan Ave., Building 90

Fort Leavenworth, KS 66027-1360

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social ARI

Sciences

2511 Jefferson Davis Highway 11. MONITOR REPORT NUMBER

Arlington, VA 22202-3926 Technical Report 1194

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

Subject Matter POC and Contracting Officer's Representative: Michelle Zbylut

14. ABSTRACT (Maximum 200 words):

This report presents findings from a preliminary examination of the Army Excellence in Leadership (AXL) system,

a leader intervention that targets the development of tacit leadership knowledge and cultural awareness in junior

Army officers. Fifty-five junior officers interacted with a pilot version of a cultural awareness module from the AXL

system. Results indicated that the AXL approach resulted in improvements in leader judgment on a forcedchoice measure. Furthermore, results indicated that cultural issues were more salient to leaders after completion

of the cultural awareness module. Reactions to training were generally positive, with officers indicating that the

cultural awareness module was useful and stimulated thought. Additionally, this investigation explored the

relationship between affect and learning and found that emotional responses to the AXL system were related to

learning-relevant variables, such as judgment scores and officer reports that they could apply the training to their

activities as a leader.

15. SUBJECT TERMS

Leadership, TLAC-XL, AXL, Cultural Awareness, Tacit Knowledge, Training, Leader Development, Tripwire,

Power Hungry

SECURITY CLASSIFICATION OF 19. LIMITATION 20. NUMBER 21. RESPONSIBLE PERSON

OF ABSTRACT OF PAGES Ellen Kinzer

16. REPORT 17. ABSTRACT 18. THIS PAGE Technical Publication

Unclassified Unclassified Unclassified Unlimited 35 Specialist

703-602-8047
ii
Technical Report 1194

Army Excellence in Leadership (AXL):

A Multimedia Approach to Building

Tacit Knowledge and Cultural Reasoning

Michelle L. Zbylut and Kimberly A. Metcalf

U.S. Army Research Institute

Julia M. Kim, Randall W. Hill, and Scott Rocher

USC Institute for Creative Technologies

Christopher Vowels

Kansas State University

Consortium Research Fellows Program

Leader Development Research Unit

Stanley M. Halpin, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

January 2007

Army Project Number Personnel Performance

622785A790 and Training Technology

Approved for public release; distribution is unlimited.

iii
ACKNOWLEDGEMENTS

The Army Excellence in Leadership (AXL) project is a collaboration between the United States

Army Research Institute for the Behavioral and Social Sciences (ARI) and the University of

Southern California's Institute for Creative Technologies (ICT). Several individuals from ICT

helped to build the AXL.net system and its predecessor, Think Like a Commander-Excellence

in Leadership (TLAC-XL). These individuals include Andrew Gordon, Salvo Lavis, Sudeep

Gandhe, David Traum, Stewart King, Reid Swanson, Ashish Karnavat, Jay Douglas, Fred

Pighin, Paul Debevec, Martin van Velsen, and Tim Hawkins. Additionally, Richard Lindheim

provided advice and support for the project.

Kim LeMasters wrote and produced the Power Hungry and Tripwire films included in the

AXL.net system. Chuck Bowman directed Power Hungry and Kenny Johnson directed

Tripwire.

The authors would like to thank Stanley Halpin and Robert Solick of ARI for their continual

advice and support throughout the duration of this project. Several ARI consortium students also

provided assistance in various phases of the project, including Jeffery Mark, Jason Ward, and

Tuan Tran.

The Behavioral Sciences and Leadership Department at the United States Military Academy

(USMA) also has been a large source of support for this project, particularly COL Tom Kolditz,

COL Donna Brazil, Orly Ben-Yoav Nobel, LTC Sean Hannah, and the captains in the Tactical

Officer's Education Program (TOEP).

The authors also acknowledge the support provided by Jeffrey Wilkinson and Joseph Brennan of

the Research, Development, and Engineering Command (RDECOM).

Portions of this research were presented at the Academy of Management Conference and the

Army Science Conference in 2006.

iv
ARMY EXCELLENCE IN LEADERSHIP (AXL): A MULTIMEDIA APPROACH TO

BUILDING TACIT KNOWLEDGE AND CULTURAL REASONING

EXECUTIVE SUMMARY

Research Requirement:

The current operating environment requires junior officers to rely upon diverse skills.

Many junior leaders will encounter a wide spectrum of operations while deployed, and only a

portion of those operations will draw upon their tactical expertise. Junior leaders may be

required to engage in several non-tactical activities, such as interacting with the civilian

population, building marketplaces, and negotiating with tribal leaders. This report describes a

computer-administered leadership intervention called Army Excellence in Leadership (AXL),

which was designed to help prepare junior Army officers for the complex challenges that they

might encounter in the Middle East. This report describes preliminary research regarding the

effectiveness of a cultural awareness module from the AXL system.

Procedure:

Fifty-five junior officers from three United States Army installations completed a cultural

awareness module in the AXL system. The training module consisted of two primary

components. First, small groups of officers watched a short filmed case study about a fictional

mission in Iraq. Second, the AXL system facilitated small-group discussion of the cultural issues

and implications embedded in the case study. Officers completed research measures at three

time points during discussion: (1) before watching the film, (2) after watching the film, and (3)

after completing discussion. Measures assessed various types of emotional reactions, reactions

to the film, reactions to the training concept, and short-term indicators of learning.

Findings:

Results indicated that officers performed better on a judgment task after having

completed training. Additionally, cultural issues embedded in the scenario were more salient to

officers after completion of the cultural awareness module. With respect to different elements of

the AXL system, findings indicated that the film was emotionally evocative and the training

concept, in general, was well-received. Further, emotional responses to the film were related to

learning-relevant variables, such as judgment scores and officer reports that they could apply the

training to their activities as a leader.

Utilization and Dissemination of Findings:

This research provides preliminary evidence that the AXL system and the multimedia

design principles that it embodies can assist junior officers develop culturally aware thinking.

Instructors and trainers interested in helping officers acquire tacit knowledge can leverage some

of these multimedia concepts to their advantage. This investigation also suggests that affect may

play an important role in training and learning.

v
vi
ARMY EXCELLENCE IN LEADERSHIP (AXL): A MULTIMEDIA APPROACH TO

BUILDING TACIT KNOWLEDGE AND CULTURAL REASONING

CONTENTS

Page

OVERVIEW OF THE ARMY EXCELLENCE IN LEADERSHIP (AXL) SYSTEM ....... 1

AXL Case Studies ................................................................................................................. 2

Interactive AXL M odules ..................................................................................................... 5

Theoretical Underpinnings of AXL ...................................................................................... 7

PRELIMINARY EVALUATION OF A CULTURAL AWARENESS MODULE FOR

TRIP WIRE ............................................................................................................................. 9

M eth o d .................................................................................................................................. 9

Participants and Procedure ........................................................................................ 9

M easures ................................................................................................................... 10

R e su lts ................................................................................................................................... 14

Participant Reactions ................................................................................................ 14

Learning .................................................................................................................... 14

Em otional Response to the Training ......................................................................... 16

Relationships between Emotions and Learning ........................................................ 17

DISCUSSION ....................................................................................................................... 20

Challenges in Evaluating AXL Research ............................................................................. 22

Concluding Rem arks ............................................................................................................. 23

REFERENCES ..................................................................................................................... 25

vii
LIST OF TABLES

Page

TABLE 1. MEANS AND STANDARD DEVIATIONS OF VARIABLES (N= 55) ..... 15

TABLE 2. CORRELATIONS AMONG MEASURES (N= 55) ..................................... 18

LIST OF FIGURES

Page

FIGURE 1. IN POWER HUNGRY, AN AFGHAN WARLOD EXPRESSES HIS

DISPLEASURE TO AN AMERICAN SOLDIER ........................................ 4

FIGURE 2. IN TRIPWIRE, AMERICAN SOLDIERS INTERVIEW THEIR NEW

IN T E R PR E T E R ............................................................................................. . 4

FIGURE 3. A SCREEN SHOT FROM A CULTURAL AWARENESS MODULE OF

T R IP W IR E ..................................................................................................... . 6

FIGURE 4. EXAMPLE OF A LEADER INTERVIEWING ONE OF THE

CHARACTERS FROM TRIPW IRE .............................................................. 6

FIGURE 5. REPORTED AROUSAL OVER THE COURSE OF THE TRAINING

IN T E R V EN T IO N ........................................................................................... 16

viii
ARMY EXCELLENCE IN LEADERSHIP (AXL): A MULTIMEDIA APPROACH

TO BUILDING TACIT KNOWLEDGE AND CULTURAL REASONING

Today's military operating environment is demanding and complex, requiring Army

leaders to draw from skill sets that were unanticipated during the Cold War era. Indeed, it is not

unusual for junior leaders (i.e., lieutenants and captains) to deal with circumstances that would

have been handled by their senior leader counterparts twenty years ago. Many junior leaders will

encounter a wide spectrum of operations while deployed, and only a portion of those operations

will draw upon their tactical expertise. In addition to establishing local security and capturing

insurgents, junior leaders may be required to engage in non-tactical activities, such as interacting

with the civilian population, building marketplaces, and negotiating with tribal leaders. As

Major General Scales noted,

"Reflective senior officers returning from Iraq and Afghanistan are telling us that wars

are won by creating alliances, leveraging nonmilitary advantages, reading intentions,

building trust, converting opinions, and managing perceptions, all tasks that demand

an exceptional ability to understand people, their cultures, and their motivation. While

wars have become more complex, responsibility for those who fight them has

increasingly slipped down the chain of command to junior personnel. Yet these

young, inexperienced leaders have little time to prepare themselves to make strategic

decisions" (2006, pg. 38).

Because the current operating environment places many demands on junior leaders, the

United States Army has devoted substantial attention to the challenge of how to help junior

leaders become adaptive thinkers, culturally aware, and interpersonally competent (e.g., Brown,

2003; Burpo, 2006; Moilanen & Craig, 2000; Petraeus, 2006; Scales, 2006). This paper

describes a leadership intervention designed to prepare junior Army officers for the complex and

chaotic situations that they will encounter in the Middle East and presents preliminary research

regarding the effectiveness of the intervention prototype.

OVERVIEW OF THE ARMY EXCELLENCE IN LEADERSHIP (AXL) SYSTEM

The Army Excellence in Leadership (AXL) system is an online educational tool that

targets the development of tacit leadership knowledge and cultural awareness in junior leaders.

The AXL system is a variation of case method instruction in which junior leaders are presented

with fictional case studies about military leadership in a tactical context and then asked to

analyze, reflect on, and discuss the leadership issues embedded in the cases. The AXL approach

to case method instruction differs from traditional case method instruction in at least two ways.

First, traditional case method instruction often relies on paper-based cases. AXL cases are

Hollywood-style films. Second, in the traditional case method approach, an instructor plays an

essential role in facilitating discussion of the case. In AXL, the online system facilitates

discussion among leaders and serves as a proxy for a human instructor. These two features of

the AXL system are discussed in greater detail in the subsequent sections.

1
AXL Case Studies

The idea of using movies as case studies to demonstrate concepts or stimulate critical

thinking is not new or unique (e.g., Cavalier & Weber, 2002; Green & Cotlar, 1973a; Leeper,

1993; Richardson, 1994; Sprau, 2001). Filmed case studies offer several advantages over

traditional paper cases. First, paper cases require a certain level of reading proficiency, while

film does not.

Second, film provides a multimedia experience that allows students to see and hear all of

the events depicted in the case as if they were actually in the situation (Green & Cotlar, 1973b).

Paper cases rely primarily on the reader constructing a mental representation of events based on

written descriptions. When done correctly, film can achieve a higher level of fidelity to the

performance environment than can a paper case. Film, therefore, can accomplish a level of

immersion, complexity, and realism that might not be achieved by paper cases.

Third, a movie is an audiovisual experience containing dynamic visual images of

situational context, depictions of character activity, spoken character dialogue, and background

sounds. These attributes of film not only improve the fidelity of a student's experience; such

attributes also might help students process case information more effectively than they would

process the information contained in a paper case. Several lines of research on perception,

memory, and learning provide strong evidence that individuals process verbal and visual

information through two processing systems that operate somewhat independently of one another

(e.g., Baddeley, 1986, 2003; Baddeley & Hitch, 1991; Gehring & Toglia, 1988; Holmes, Brewin,

& Hennessy, 2004; Mayer, 2001; Mousavi, Low, & Sweller, 1995; Paivio, 1986, 1991). To

grossly oversimplify-sounds are processed through one system and visible images are

processed through another. Although limitations exist with respect to how much information

each system can process individually, cognitive load as a whole can be reduced by presenting

information in multiple modalities so that neither processing system is unduly overburdened.

Mousavi et al. (1995) indicated that:

"... [B]asic research into the characteristics of working memory has suggested that this

processing system is divided into at least two partially independent processors: an auditory

system devoted heavily to language and a visual system for handling images, including

writing. Because both systems can be used simultaneously, limited working memory capacity

might be effectively increased if information that must be stored or simultaneously processed

is presented in a manner that permits it to be divided between the two systems, rather than

processed in one system alone. As a consequence, informationally equivalent material that

may be difficult to process in a purely visual manner may be more easily handled if it can be

presented partially in both modalities" (pg. 331).

The two best known dual-processing theories are those of Paivio (1986) and Baddeley

(1986). More recently, Mayer (2001) proposed a dual-processing theory of multimedia learning,

which suggested that presentations using multiple modes of information transmission are

superior to single mode presentations. Mayer's theory proposed that individuals process

incoming information through two separate channels: a visual channel that registers images and

an auditory channel that registers sounds. These two channels operate as distinct processing

2
systems, and they result in two separate knowledge representations-a verbal representation and

a visual representation. In addition to the visual and verbal representations, individuals are able

to form connections between the two knowledge representations, resulting in a coherent and

elaborate representation of knowledge.

In sum, the accumulated body of research on dual-processing theories suggests that

individuals are able to process and retain more information when multiple modalities are used

than when only a single modality is used. A series of experiments conducted by Thompson and

Paivio (1994) exemplifies this idea well. They found that recall was significantly higher when

individuals were presented with both pictures of objects and their associated sounds rather than

presented with only pictures or only sounds.

Within the context of AXL, leaders who watch the filmed cases are exposed to rich

sources of both visual information and audio, and they may be more likely to retain this

information than if they are exposed to the case information presented in a single modality. This

improved probability of recalling relevant details from the multimedia scenario may, in turn,

help to foster more profound and informative discussion about the case among students during

the discussion phase of case method teaching.

The AXL system currently houses two filmed case studies, Power Hungry and Tripwire.

Both Tripwire and Power Hungry are short films, lasting no longer than 15 minutes. The films

were written, acted, directed, and edited by Hollywood professionals, resulting in high-quality

films similar to those viewed in a movie theater. The first film, Power Hungry, was created for

the AXL system in 2003. Power Hungry depicts a captain tasked with securing a site for a food

distribution operation in Afghanistan. In addition to poor interpersonal skills and a lack of

cultural awareness, the captain is confronted with warlords, terrain issues, time constraints, and

inexperienced subordinates. The mission ends in failure, with an Afghan warlord seizing the

food trucks. A still image from the Power Hungry film is presented in Figure 1.

Power Hungry centers around six teaching themes: understanding the mission, sharing

vision, command influence, establishing a model of command, cultural awareness, and

employing the expertise of non-commissioned officers (NCOs). Previous research on Power

Hungry indicated that watching the film heightened the emotional arousal of junior leaders. The

film also was more memorable and resulted in greater positive affect than the same case study

presented in a multimedia PowerPoint format (Zbylut & Ward, 2004a; Zbylut, Ward, & Mark,

2005). The potential for film to engage students both cognitively and emotionally may benefit

the learning process by helping students extract and retain important information from the lesson,

as well as keeping students motivated throughout the lesson.

The second filmed case study, Tripwire, was completed in 2005. The Tripwire film is set

in Iraq and the captain's mission is to arrange a meeting between two tribal leaders. In Tripwire,

the Army battery encounters several obstacles to accomplishing its mission, including a fatal

attack on an interpreter, insurgent activity, and improvised explosive devices (IEDs). Tripwire

incorporates many of the same communication, leadership, and cultural issues embedded in

Power Hungry, but includes additional themes, such as establishing trust with subordinates and

taking care of Soldiers. A still image from Tripwire is presented in Figure 2.

3
Figure 1. In Power Hungry, an Afghan warlord expresses his displeasure to an American Soldier.

Figure 2. In Tripwire, American Soldiers interview their new interpreter.

Although many case studies are based on historical accounts of events that occurred,

historical accuracy is not a necessary precondition for case study development. For example,

Maltby (2001) noted that, "There are two possibilities when designing cases-to invent a

situation, or to draw on a real one" (pg. 424). Similarly, Jennings (1996) indicated that "teaching

cases may not be 'true' accounts of business life" (pg. 5), and suggested that other important

criteria for determining the usefulness of a case study include how well the case stimulates

discussion among students and how well the case engages a student intellectually and

emotionally. What appears to be most important about a case is not necessarily that it is a true

4
account, but rather that it is realistic, complex, and built for accomplishing the teaching goals of

the lesson. Along these lines, the films embedded in the AXL system are not historical accounts.

Instead, the AXL films take an amalgam of real events and military issues to provide a rich,

layered, and multifaceted stimulus for prompting student discussion about teaching points

intentionally embedded in the stories. Both Power Hungry and Tripwire were built around

content extracted from interviews with captains who had returned from recent deployments

(Gordon, 2005; R. Hill, Douglas, Gordon, Pighin, & van Velsen, 2003; R. Hill, Gordon, & Kim,

2004). Challenges, problems, and situations that captains encountered while deployed were

interwoven into the storylines of the films. To ensure that the films were realistic, subject matter

experts (e.g., experts in military leadership, experts in IEDs) reviewed the film scripts prior to

filming. Department of Army personnel were on location during filming to ensure military

details were accurate.

Interactive AXL Modules

In traditional case-method teaching, an instructor guides student discussion and analysis

of the case. Unfortunately, preparing for a case study can be both labor and time-intensive

(Diamantes & Ovington, 2003), and many instructors may not have adequate time available to

prepare. Additionally, the Army is seeking innovative ways to improve distance learning, which

requires that appropriate online proxies for human instructors be developed. The AXL system

seeks to address both of these challenges by providing online multimedia exercises that will

prompt students to discuss and analyze the issues embedded in a case study without the aid of a

"live" instructor.

Each film in the AXL system is paired with several interactive modules that address

military leadership topics, such as cultural awareness, communicating with subordinates, and

working with non-commissioned officers (NCOs). The modules were intended to be completed

in small groups of two to four Soldiers. Junior leaders who wish to learn about a topic (or are

told to learn about a topic by an instructor) select a module related to that issue. After selecting a

topic, the AXL system presents the goals of the module to the group. The group then watches

the associated filmed case on the computer. Once the group has watched the film, the AXL

system automatically transitions into the discussion phase of instruction. A screen shot from an

AXL online module is presented in Figure 3.

AXL facilitates discussion of the filmed cases by using a combination of close-ended and

open-ended questions. The AXL system uses answers to close-ended questions to check for

understanding of basic concepts, provide feedback, and branch the discussion to compel leaders

to examine their thoughts and assumptions more closely. Open-ended questions are used to

stimulate reflection and analytical thinking, encourage the sharing and application of real-world

experiences, and generate solutions to problems in the case study. Another feature of the AXL

system is that leaders can interview the main characters from a case study to ascertain what the

characters knew, felt, and thought during the scenario. Leaders type questions into the AXL

system to reveal a character's perspective on the scenario and uncover information not included

in the film. Figure 4 depicts an example of a group querying a film character in the AXL system.

5
00

IOOS .. Noah tat pouVe thad the opportunity to remtewhome oftlae divershe Isses embededr In To901 0

er. 0,...let's focos specitically on Mle topic of cultural "oareness.

In Iri9re, the0 local situation attct the battery and the outcome 0f the scenarlo CPT Ho0lston

makes tactical decisions regarding lrnprncised exploS"m d"evS (IEDS) that affect the

perceptiont and genetal state oft oe total population He also encounters cnallenges in0 deln

ml.on a. ath intetprteres, first losing an Interpreter to guntire and thee acquiting a oew Interpreter Who

..h..r .ld has not been cleared thtough headquadters

010~~ We wril now considerthUe foli~olrg sine 35 they relate to cultural acaresefss

I.The loc al situation as a factor in the mission

2 Dealing wýth local residents

3 Diferences recin a slingle culture

Click Neat to begin

Figure 3. A screen shot from a Cultural Awareness module for Tripwire.

CO ... 0.. C~~5Wolo 

~C h J ~~,5 e wa e :

Fg r 4.E a ple. jofia leaderm inte.O~rview. oling one0 of th0hrcesfo rp ie T elae y e h usin

out nt -yu sygitwcharasprgn e I was4. It was.ll ~ecnould thn Nabout. ~jtoI me11an, 41here. I Cmamllo ie

Figuresod Exampleofadleaderp t. intuerviewaing onre of thechaactrs.Aso frometa mrinor Thealeadrtypes teros question

using "you" instead of "your," do not interfere with the relevance of the reply.

6
Theoretical Underpinnings of AXL

The AXL system design and pedagogical approach draws from a variety of research

domains. First, AXL is a form of case method instruction, which is widely hailed as an effective

approach for developing critical thinking, problem solving, and interpersonal skills (e.g.,

Crittenden, Crittenden, & Hawes, 1999; Hassall, Lewis, & Broadbent, 1998; Jennings, 1996;

Kreber, 2001; Richardson, 1994; Stewart & Dougherty, 1993). AXL instruction includes

realistic, multifaceted cases and focuses on helping leaders learn "how to think" rather than

"what to think," thereby preparing leaders to deal with complexity when they encounter it in

their work environment (Hannafin, J. Hill, & Land, 1997; Jonassen, 2002; Lesgold, 2001).

Moreover, AXL's use of multimedia cases in conjunction with interactive multimedia modules

can help leaders construct more elaborate and sophisticated knowledge representations than they

would if information were presented using a single modality (Mayer, 2001; Mayer, Dow, &

Mayer, 2003; Mayer & Sims, 1994; Moreno, 2006).

Second, AXL interactive modules build on several multimedia instructional design

principles thought to promote active learning. Open-ended questions in AXL are used to

encourage leaders to reflect, not only on various aspects of the case, but on how the case applies

to their own personal experiences. The combination of personal experience and reflection helps

to capture the leader's attention, as well as help the leader to integrate new information with past

knowledge (Moreno, 2006). Open-ended questions in AXL also are closely linked to the

pedagogical concepts of elaborative interrogation (e.g., Martin & Pressley, 1991; Pressley,

McDaniel, Turnure, Wood, & Ahmad, 1987; Willoughby, Wood, Desmarais, Sims, & Kalra,

1997) and self-explanations (e.g., Chi & VanLehn, 1991; Mayer et al., 2003). During an AXL

module, leaders are provided with multiple discussion questions that prompt them to examine the

relationships among different elements in the scenario, reflect on the causal factors that may

have influenced different events in the film, and consider the various implications of different

courses of action that could have been adopted by characters in the scenario. By answering and

discussing these types of questions (particularly questions about why certain events might have

occurred and how group-derived explanations correspond to personal experience), leaders are

better able to integrate new information they encounter during the module with previous

experience and knowledge. These processes of self-explanation and elaborative interrogation

can assist in retention of information over time, as well as generating inferences beyond the

original material contained in the case.

Like open-ended questions, close-ended questions play an important role in the AXL

system. While open-ended questions encourage leaders to think and actively construct

knowledge, close-ended questions allow leaders to receive feedback, guidance, and more

targeted lines of questioning. AXL uses close-ended questions to impose structure on an

otherwise ill-defined task (i.e., make sense of a chaotic situation) and this imposed structure may

be particularly helpful for novice leaders who do not possess well-developed schema that would

help them process the information and issues embedded in a complex case (Moreno, 2006).

The opportunity to interview characters also builds on two multimedia instructional

design principles: interactivity and control. The AXL capability to interview characters allows

leaders to test hypotheses about why characters performed particular actions, thereby

7
encouraging "the processing of new information by engaging students in an active search for

meaning" (Moreno, 2006, pg. 65). Moreover, the act of formulating a question to ask a character

requires leaders to process information at a deep level; question-asking obliges leaders to direct

their attention to specific content and monitor their level of comprehension with respect to that

content (Rosenshine, Meister, & Chapman, 1996). Additionally, the character interview feature

offers a certain degree of control to leaders. The discussion group must decide whether to

interview characters, what to ask characters, and when to pose questions of characters. Allowing

leaders to control when and if they want to interview characters may help to reduce cognitive

load, thus helping leaders to attend to and process new information more effectively (Mayer et

al., 2003).

A third domain of research that the AXL system draws from is research on tacit

knowledge. The work of Sternberg and colleagues (e.g., Matthew, Cianciolo, & Sternberg,

2005; Sternberg, Forsythe, Hedlund, Horvath, Wagner, et al., 2000) indicates that leaders can

acquire tacit (i.e., actionable) knowledge from reflecting on one's personal experiences, as well

as the stories and experiences of others. During AXL instruction, leaders have the opportunity to

reflect on multiple types of experiences. The first type of experience that leaders are exposed to

in the AXL system is the fictional situation depicted in the film, and AXL modules require

extensive reflection and analysis of the film. Leaders also examine a second type of experience

in the AXL system-specifically, leaders are asked to examine their personal leadership

experiences. In this instance, AXL presents leaders with the opportunity to think about their

personal experiences in a different way than they typically would, examining similarities and

differences between their personal experiences and the elements of the case. Leaders also have

an opportunity to hear how peers respond to their personal experiences. A third type of

experience that leaders are exposed to is the personal experiences of other discussion group

members, and these experiences may or may not be similar to their own personal experiences.

The comparison between one's personal experiences with the experiences of others and the

situation depicted in the film might be particularly beneficial to learning because research

suggests that comparing examples can assist in the development of problem-solving schema (see

Anderson, Greeno, Kline, & Neves, 1981; Gick, 1986).

The AXL system is still in development, with a finalized system to be completed in 2007.

The earliest version of AXL included only one film, Power Hungry, and one leadership module.

Early research in the AXL project focused on collecting leader reactions to the product interface

and content, and the primary goal of data collection was to examine media effects, gather

information that would inform system design choices, and refine the pedagogical approach

(Zbylut & Ward, 2004a; Zbylut et al., 2005). Based on those results a new version of AXL was

delivered in 2006, with substantial changes to the AXL interface and flow of instructional events.

The revised AXL system houses both the Tripwire and Power Hungry films, as well as several

new instructional modules for each film.

Before testing the new AXL modules, the research team first showed Tripwire to small

groups of lieutenants and captains (N = 27) at an Army installation in the Southwest.

Researchers conducted focus groups and administered questionnaires to the officers to gather

data about reactions to the film and to ascertain what leadership issues officers found to be

8
salient in the case study. Data from the questionnaires and focus group interviews were used to

create measures to assess learning from Tripwire modules.

ARI then pilot tested one of the new AXL modules (a cultural awareness module for

Tripwire) at three Army installations. The purpose of the investigation was to (a) collect

reactions to the module and instructional approach, (b) assess whether the module impacted how

leaders thought about the situation depicted in the film, (c) determine if learning occurred, and

(d) explore whether emotional reactions were related to learning. The next section presents the

method and results of that pilot investigation.

PRELIMINARY EVALUATION OF A CULTURAL AWARENESS

MODULE FOR TRIPWIRE

Method

Participants and Procedure

Participants were 44 captains (CPTs), six first lieutenants (1 LTs), and five second

lieutenants (2LTs) from three United States Army installations. Twelve of the officers had

deployed twice to the Middle East-i11 to Iraq and one to both Iraq and Afghanistan. Thirty

officers had deployed once to the Middle East-26 to Iraq and four to Afghanistan. The

remaining 13 officers had not deployed to either Iraq or Afghanistan.

Each officer participated in a small discussion group for the cultural awareness module of

Tripwire. Discussion group sizes tended to range from two to four junior officers, although one

training session had one officer, and another training session contained seven officers.I

Prior to watching the film, officers completed a brief questionnaire asking them to report

on their deployment experience and level of emotional arousal. Officers then watched Tripwire

as a group on a laptop computer. After watching the film, officers independently completed a

second set of measures. Once all of the officers completed the second set of measures, they

began the discussion phase of the cultural awareness module. The module consisted of a series

of open-ended and close-ended questions to prompt discussion about the cultural issues in

Tripwire, particularly concerning the treatment of locals, managing the perceptions of locals, and

dealing with IEDs in an urban environment. Discussion lasted approximately 45 to 90 minutes.

Upon completing the cultural awareness module, officers independently completed a final set of

measures. The entire process, including completion of measures, ranged from 90 to 120 minutes.

Eight officers were originally scheduled to participate in two discussion groups consisting of four officers each, but

three of the officers attended the wrong discussion group. This resulted in seven officers in one group, and one

officer in another "group." The officer who completed the cultural awareness module by himself was told to reflect

on the questions contained in the module. This individual had been deployed previously and presumably could draw

from his experience while reflecting on the scenario.

9
Measures

Demographic Items

Prior to training, officers indicated their rank, how many times they had been deployed to

Iraq, and how many times they had been deployed to Afghanistan.

Emotional Measures

Self-reported emotional arousal. Officers completed five self-report items that gauged

their level of emotional arousal. These five items were part of Mehrabian and Russell's (1974)

arousal scale and were anchored with bipolar adjectives on a nine-point scale ranging from -4 to

+4. The bipolar anchors were relaxed/stimulated, calm/excited, sluggish/neutral, dull/jittery, and

sleepy/wide awake. Arousal was operationalized as the mean of the five items. The arousal

measure was given at three time points: immediately prior to watching Tripwire (a = .66),

immediately after watching Tripwire ((x = .8 1), and immediately after completing the cultural

awareness training module (a = .74).

Positive and negative affect. After watching the film and completing a measure of

emotional arousal, officers completed the Positive and Negative Affect Schedule (PANAS;

Watson, Clark, & Tellegen, 1988). Officers indicated the extent to which they experienced 20

emotions while watching the film. The 20 items were anchored on a five-point scale ranging

from "very slightly or not at all" (1) to "extremely" (5). Positive affect (a = .91) was the mean of

10 positive emotions, such as active, excited, and proud. Negative affect (a = .85) was the mean

of 10 negative emotions, such as hostile, upset, and distressed.

Reactions to the Film

After completing the PANAS, officers completed 15 items that addressed reactions that

they had to the film. All of the items were anchored on a seven-point scale ranging from

"strongly disagree" (1) to "strongly agree" (7). These items were completed prior to participating

in discussion.

Character depth. Character depth was operationalized as the mean of six items that

assessed perceptions about how believable and complex the characters in the film were (a = .78).

Examples of items from the scale were, "The story did a good job of communicating the

thoughts, feelings, and emotions of the characters," "I can imagine meeting people similar to the

characters presented in the story," and "I could identify with at least one character in the story."

Film involvement. Officers answered four items that assessed the degree to which they

felt involved in the film's storyline. These items asked the extent to which officers wanted to

find out what happened next in the story, whether the story was involving, and whether the story

was suspenseful. An additional item, which was reverse-scored, asked officers if they were

bored while watching the movie. Film involvement was operationalized as the mean of the four

items (a = .79).

10
Confusion. Confusion was assessed using the item, "I was confused about what was

going on in the story."

Realism. Officers rated the extent to which they believed the story was realistic by using

the item, "The story was realistic."

Preference for film over traditional instructional methods. Officers completed two

items that assessed their preference for filmed case studies over traditional methods of

instruction found in the Army. The first item was, "In an educational context, I would rather

watch a film like this than read a scenario." The second item was, "In an educational context, I

would prefer to discuss a film like this than listen to a PowerPoint briefing."

Reactions to AXL Training

After finishing the discussion phase of the cultural awareness module and the final

measure of arousal, officers completed 16 items that addressed various reactions that they had to

the AXL system. Items were anchored on a seven-point scale ranging from "strongly disagree"

(1) to "strongly agree" (7).

Global reactions to training. Officers answered five items that assessed the perceived

value of the AXL tool, ease of use, how informative the training was, how responsive the

training was to the officer's educational needs, and how frustrating the training approach was

(reverse-scored). A score for global reactions to the training was formed by computing the mean

of the five items (a = .80).

Stimulated thought. Because one of the objectives of case method teaching is to

encourage critical thinking, four items assessed whether officers believed that the cultural

awareness module made them think. These items were "The questions included as part of the

training made me think critically about issues in the scenario," "This exercise made me think

about what makes a leader effective," "This exercise made me think about what makes a leader

ineffective," and "The training made me think about the scenario in a different way than I

usually would." A score for this scale was calculated by computing the mean of the four items

(a .75).

Character interactivity. Officers completed four items that addressed their reactions to

the character interview features of the AXL system. These items were "Interviews with the

characters gave me insight into why the mission failed," "I felt like the characters answered the

questions that they were asked," "Interviews with the characters helped me understand why there

was a breakdown in communication between different characters," and "I liked that the actors

stayed 'in character' when they answered my questions." A score for this scale was calculated

by computing the mean of the four items (a. = .81).

Applicability of training. Officers completed a single item, "I can apply some of the

things that I learned here to my activities as a leader," which was used to assess whether officers

believed they would transfer knowledge gained during the AXL module to their job.

11
Wantedfeedback. In order to assess whether officers believed the AXL system should

provide more feedback, officers indicated the extent to which they agreed with the following

item: "I wanted more feedback about how I was doing during the training modules."

Learning

One challenge to assessing whether learning occurred during AXL instruction is that the

AXL case method approach adopts a constructivist view of learning. According to the

constructivist paradigm, meaningful learning occurs when individuals construct their own

interpretations of the world. Moreover, multiple valid interpretations of the world are not only

possible, but expected (Dalgarno, 2001; Hannafin et al., 1997; Mayer, 1996). In this respect,

discussing a case study is largely a sensemaking process in which groups of leaders share

experiences and thoughts to derive a greater understanding about leadership than they had prior

to discussion. Given that the content and outcomes of different discussion groups may have

significant variability across groups, a primary challenge for the researcher is how to objectively

measure whether discussion was worthwhile. That is, if different leaders are learning different,

but valid things, how does one measure learning?

While leaders from any one discussion group may discuss different information and

ultimately arrive at different "wisdom" than leaders from another discussion group, the tenets of

constructivism clearly imply that individuals who engage in knowledge construction activities

(like discussion) should experience a change in their understanding or interpretation of events.

Additionally, if the researcher has a sense of what common themes or topics will emerge in

different discussion groups, the researcher can compile a heterogeneous set of learning items that

will likely tap at least a few of the topics brought up during any one discussion.

Learning in this investigation was assessed using two different approaches. The first

approach utilized a rank-ordering task to assess whether leaders experienced a change in their

understanding of the importance of cultural issues embedded in the case study. The second

approach used a series of forced-choice items to examine whether leaders demonstrated better

judgment after completing the cultural awareness module. While the first approach provided an

indicator of how a leader was framing issues and content embedded in Tripwire, the second

approach provided an indicator that the leader applied knowledge learned during discussion to

problems embedded in Tripwire.

Emphasis on cultural issues. As mentioned previously, an earlier data collection effort

used focus groups and written questionnaires to explore the leadership issues junior officers

believed were important in Tripwire. During that investigation, officers indicated what they

would have done differently if they were a Soldier in the Tripwire scenario. From this data, a list

of 21 issues was compiled.

2 The initial list consisted of 92 character actions discussed or written about during focus group sessions. Actions

having significant overlap were combined, resulting in a list of 47 items. Because rank ordering 47 issues would be

cognitively overwhelming for participants, the list was reduced to the 28 most commonly mentioned issues. A rankordering task with these 28 items was piloted at one of the three Army installations during data collection. Analysis

of the first administration of the measure indicated that some issues were selected infrequently, and seven issues

were removed from the measure to make the ranking task less cognitively demanding. Thus, the finalized ranking

task of 21 items was administered at only two of the three installations, resulting in n = 36 on this measure.

12
Eight of the 21 issues appeared as discussion topics in the cultural awareness module.

These issues not only reflected decisions and actions that could influence Iraqi behavior and

perceptions of the Army, but also included issues that required knowledge about cultural

antecedents and consequences surrounding events in the scenario. Cultural issues included

somewhat evident cultural topics such as, "CPT Holston should have refrained from shooting the

backpack with the Koran" and "I LT Porter should have refrained from striking the second

interpreter," as well as less apparent cultural topics such as, "CPT Holston should have

investigated why the first interpreter was shot."

The 21 issues were used in a rank-ordering task designed to measure a leader's

understanding of what issues he or she believed were most important in the film. In the rankordering task, officers were asked to select the top seven actions that Soldiers in the film should

have done differently. Officers were then asked to arrange those top seven actions from most

important to least important. Officers received one point for each cultural issue that they ranked

among their top seven most important issues.

Officers completed the rank ordering task twice-once after watching the film (but

before discussion) and once after discussion. The 21 issues were arranged in different orders for

the two time points. The ranking task was used to examine whether officers deemed the cultural

issues to be more important after discussing the scenario.

Behavioraljudgment. In addition to the ranking task, leaders completed eight items that

asked them to choose between two courses of action that could have been taken during the

Tripwire scenario. The forced-choice items represented a heterogeneous set of issues

intentionally embedded in the case study, including maintaining one's health versus self-sacrifice

for the mission, accomplishing the mission versus protecting one's Soldiers, listening to the

advice of Soldiers versus maintaining command authority, and neutralizing potential IED threats

versus offending Iraqi civilians. For each item, a leader received one point if he or she selected

the better course of action (i.e., consistent with the teaching goals built into Tripwire). A

behavioral judgment score was then computed by summing across the eight items. Items and

response options were randomly arranged for both the pretest and posttest. Additionally, one of

the "pretest" items was given prior to watching the film because it did not require watching the

film to answer the question. Two examples of forced-choice items are provided. The "correct"

answers are denoted by an asterisk.

During the planning meeting, the platoon leaders told CPT Holston that they should be

conducting "rolling Ts" in the town in order to keep things calm until the VIP's

arrival. After the first interpreter was killed, CPT Holston's Soldiers again indicated

that they believed a more aggressive stance should be taken in the town. Do you

believe that it is more important for CPT Holston to...

a. Maintain his initial restrained stance, especially given that his junior officers

are challenging him.

b. Act on the advice of his subordinates, even if they are challenging his

authority.*

13
In general, it is better for a leader to...

a. Put mission tasks ahead of physical well-being (e.g., skipping meals, reducing

sleep time) in order to get the job done.

b. Take time out to sleep, eat, and hydrate, even if it takes some time away from

important mission-related tasks.*

Results

Participant Reactions

The means and standard deviations for the measures used in this investigation are

presented in Table 1. Results indicated that reactions to the film were positive, with officers

rating the film as involving (M = 5.55, SD = 1.03) and characters as complex (M = 5.35, SD =

.93). In general, officers tended to view the film as realistic, although substantial variability

existed among leaders (M = 4.96, SD = 1.71). Regardless of whether the film was perceived as

realistic, officers overwhelmingly preferred watching a film to reading a scenario (M = 6.60, SD

= .81) or listening to a PowerPoint presentation (M = 6.76, SD = .58).

Reactions to the AXL training package overall also tended to be positive. Officers'

global reactions to the training were positive (M = 5.64, SD = .85), and officers indicated that the

cultural awareness module was thought provoking (M= 5.39, SD = .82). Likewise, officers

indicated they could apply something that they learned during the module to their activities as a

leader (M = 5.56, SD = 1.18). Officers were somewhat neutral with respect to whether they

believed the AXL system should provide more feedback (M = 3.45, SD = 1.71), with some

officers wanting substantially more feedback than others. The character interview features

received slightly positive ratings (M = 4.53, SD = 1.23), but ratings suggest that the character

interview functionality in the system needs improvement.

Learning

Emphasis on Cultural Issues

A paired-samples t-test was conducted to examine if officers placed stronger emphasis on

cultural issues after completing the cultural awareness module. Results indicated that officers

ranked more cultural issues as important after participating in discussion (M = 3.11 issues, SD

1.39) than before discussion (M= 2.64 issues, SD = 1.17), t(35) = 2.50,p < .05. This finding

provides evidence that the cultural awareness module resulted in a conceptual shift in officer

understanding of the situation depicted in the Tripwire scenario, such that officers viewed

cultural issues as significantly more important after completing the cultural awareness module.

14
Table 1

Means and Standard Deviations of Variables (N 55)

M SD

Reactions to the Film

Character Depth 5.35 .93

Involving Film 5.55 1.03

Confused 1.78 .92

Realism 4.96 1.71

Preference for Film over Text Scenario 6.60 .81

Preference for Film over PowerPoint 6.76 .58

Reactions to Training

Global Training Reactions 5.64 .84

Stimulated Thought 5.39 .82

Character Interactivity 4.53 1.23

Training Applicability 5.56 1.18

Wanted Feedback 3.45 1.71

Learning

Behavioral Judgment Pretest 4.82 1.22

Behavioral Judgment Posttest 5.08 1.30

Emphasized Cultural Issues (Post-film) a 2.64 1.17

Emphasized Cultural Issues (Post-training) a 3.11 1.39

Emotional Measures

Arousal (Pre-film) -.47 1.13

Arousal (Post-film) .80 1.30

Arousal (Post-training) .20 1.10

Positive Affect 3.03 .85

Negative Affect 1.82 .67

Note. Measures included in the Reactions to Film and Reactions to Training categories are based on a 7-point scale

ranging from "strongly disagree" to "strongly agree." Within the Learning category, the highest possible score that

could be achieved was 8 for the behavioral judgment test and 7 for the Emphasis on Cultural Issues task. With

respect to the Emotional Measures category, self-reports of arousal were on scale ranging from -4 to +4, with

positive scores indicating arousal and negative scores indicating lethargy. Positive and negative affect were

measured on a 5-point scale with higher scores indicating stronger affect.

a n=36

15
Changes in Behavioral Judgment

A paired-samples t-test also was used to examine whether discussing the film resulted in

better judgment by leaders. Results indicated that officers tended to exhibit better judgment

about behavioral courses of action after discussion (M-- 5.08, SD = 1.30) than they did before

discussion (M-- 4.82, SD = 1.22), t(54) = 2.0 8 , p < .05. Thus, the cultural awareness module

appears to have impacted leaders' judgment in a way consistent with the teaching goals of the

Tripwire case study.

Emotional Response to the Training

A repeated measures analysis of variance (ANOVA) indicated that self-reported levels of

arousal changed at different points during the cultural training, F(2, 108) = 25.99, p < .001.

Paired-samples t-tests were conducted to compare the three time points to examine the nature of

these differences. Results indicated that arousal was lowest prior to watching the film (M = -.47,

SD = 1.13), being significantly lower than arousal levels reported immediately after the film

(M .80, SD = 1.30), t(54) = -6.33, p < .001, and after the training (M = .20, SD = 1.10),

t(54) = -4.04, p < .001. In general, arousal was highest immediately after watching the film, with

arousal weakening slightly during the course of discussion, t(54) 3.76, p < .001. The pattern of

arousal throughout the training is depicted in Figure 5.

4 -

3, -2

-3

2 -1

cl) -2

-3

-4

Pre-film Post-film Post-discussion

Time

Figure 5. Reported Arousal over the Course of the Training Intervention

These results suggest that the training is emotionally engaging, with the film being

particularly arousing. However, while the arousal measure provides an indicator of the

energizing component of emotion, it does not provide an indicator of the valence (i.e.,

pleasantness) of emotion. Affect consists of two important elements: level of activation (i.e.,

16
arousal) and hedonic relevance (Feldman, 1995; Russell, 2003). With respect to Tripwire, it was

unclear whether the film would be received as a pleasant or an unpleasant experience. In one

respect, the film could generate positive affect, because the film might prompt the interest,

attention, and excitement of the viewer. Conversely, the unpleasant situation depicted in the film

could prompt a negative experience, generating feelings of anger, hostility, and distress. A

paired-samples t-test was conducted to examine whether the film evoked more positive versus

negative affect. The t-test indicated that the film evoked greater levels of positive affect (M =

3.03, SD = .85) than negative affect (M= 1.82, SD = .67), t(54) = 11.62,p < .001.

Relationships between Emotions and Learning

One of the purposes of this investigation was to explore the relationship between

emotions and learning. Of note is how emotional measures correlated with other measures (see

Table 2). While arousal levels were not significantly related to any learning variables, affect was

related to several learning-relevant variables. Positive affect was positively correlated with the

behavioral judgment posttest score (r = .33, p < .05) but was uncorrelated with the behavioral

judgment pretest score (r = 10, p = ns), suggesting that positive affect experienced during the

film may be related in some way to what an individual learns during the course of discussion.

Positive affect also was related to global reactions to the training (r = .43, p = .001), and both

positive affect and negative affect were related to how thought-provoking the training was

(positive affect: r = .31, p < .05; negative affect: r = .44, p = .001), and whether leaders believed

they learned anything that could be applied to their work (positive affect: r = .43, p = .001;

negative affect: r = .31, p < .05).

Results with respect to the relationship between affect and emphasis on cultural issues

were more difficult to interpret. Neither positive affect (r = .26, p = ns) or negative affect (r =

.21, p = ns) were significantly related to emphasis on cultural issues after the discussion.

However, positive affect was significantly correlated with how much officers emphasized

cultural issues prior to discussion (r = .44, p < .01). Thus, officers who experienced more

positive affect during the film appeared to focus their attention on culturally relevant courses of

action before discussion with other officers, but the relationship between positive affect and

emphasis on cultural issues was weaker after participating in discussion.

Several other correlations are worth noting. First, an officer's level of engagement with

the film might have bearing on how well that individual learns from the training. Officers who

found the characters to be complex and believable tended to score higher on the judgment

posttest than officers who did not rate the film characters highly (r= .30, p < .05). Additionally,

officers who rated the film as involving tended to score higher on the judgment posttest than

officers who did not rate the film as involving (r = .35,p < .01).

Second, unanticipated relationships were found between pre-film arousal levels and

reactions to both the film and training. Officers who reported higher levels of arousal prior to

watching the film also reported higher levels of confusion about what was happening in the film

(r = .

2 9, p < .05) and wanted to receive more feedback during training (r = .30, p < .05).

Officers with higher levels of arousal prior to watching the film also tended to indicate that they

did not prefer film to a PowerPoint briefing (r = -.36,p < .01).

17
e o -1 0- 0N 0 (N

00 C

m0 0t c C> t W) - W0 D0

En 00 '0ýoCý M 01 \OW

0 )

g C 00 - '0 C 000 )N C, (N I4 0c C

000

0 ~ ~ ~ ~ ~ N~ ~ r 1 'trN -- eN0

W 000 OD w In C> 0 m ONO ON WN '00

0a '0CN 0 )'t '

0 00

-l zmlU0) ~~U 0 00 '0 N '0 E~ m~ N 7E r~ 0 at ' (I W C / I aNco'1 r 't 0 't ( ~ ........................................

CL 4u - u C , EC4 U L)
0.

NN

U

~0 N0.) ~-~

w -0 -

ND -0 Cl ->

If) ) W) V) 

E 0E1.

V)

U, o - - 00

0)> 0

o 0o

0 cc

w) C's m

- cd 0 ca. r4 :

.2 -, : . CU .

0. uLL
DISCUSSION

The results of this investigation indicated that the case study approach adopted in the

AXL system holds promise in shaping the knowledge and judgment of Army leaders.

Specifically, after completing the cultural awareness module, leaders exhibited better judgment

about various behavioral courses of action that could have been adopted in the Tripwire scenario.

The cultural awareness module also appeared to play a role in influencing the importance leaders

placed on the cultural issues in the film, with leaders placing stronger emphasis on cultural issues

after having completed the module. These findings are consistent with research on tacit

knowledge that suggests individuals can develop tacit knowledge by reflecting on their

experiences and the experiences of others (Cianciolo, Anotonakis, & Sternberg, 2004; Matthew

et al., 2005; Sternberg et al., 2000).

Findings with respect to reactions to the Tripwire film were both encouraging and similar

to findings for the Power Hungry film (Zbylut & Ward, 2004a; Zbylut et al., 2005). Officers

rated Tripwire as involving and the characters as multi-layered. Moreover, officers found the

film to be fairly realistic and were not confused by the film. Indeed, officers overwhelmingly

indicated that they would prefer to watch a film rather than read a case study or listen to a

PowerPoint presentation, both of which are common instructional techniques used in Army

education. These results indicate that the two films are an appropriate way for depicting case

study material in the AXL system.

In addition to positive reactions to the film, officers responded positively to the AXL

cultural awareness module as a whole. Officers rated the AXL cultural module as valuable and

useful. More importantly, officers indicated that they would be able to transfer something that

they learned to their activities as a leader. Furthermore, officers reported that the module was

thought-provoking-not only compelling them think about what makes a leader effective versus

ineffective, but helping them to think about the Tripwire scenario in a different way than they

usually would. Taken as a whole, the results of this investigation support some of the basic

tenets of case method teaching; namely, case method teaching challenges the assumptions of

students and stimulates critical thinking (e.g., Crittenden et al., 1999; Jennings, 1996; Kreber,

2001; Richardson, 1994; Stewart & Dougherty, 1993).

This investigation also explored the relationship between emotions and learning. One of

the long-term goals of the AXL project is to examine if film is emotionally evocative and

whether this has any long term benefits for learning and behavior over time. Previous research

regarding AXL indicated that the first film, Power Hungry, led to heightened emotional arousal

and reports of positive affect (Zbylut & Ward, 2004a; Zbylut et al., 2005), and results found for

Tripwire replicate those findings. While the present investigation did not examine the long-term

impact of emotionality on learning and retention, results indicated a positive relationship

between positive affect and behavioral judgment posttest scores. Moreover, both positive affect

and negative affect were related to several self-report measures indicative of learningspecifically, how much leaders reported that the training stimulated thought and the extent to

which leaders could transfer knowledge to the work environment. Due to the design of this

research, it is unclear whether affect played a role in learning or was just a byproduct of the film

medium. However, a growing body of work on mood, affect, and arousal support the notion that

20
emotions play an integral role in learning. For example, Murray, Harish, Hirt, and Sujan (1990)

found that individuals in positive moods demonstrated enhanced cognitive flexibility and had

greater access to diverse and unusual information than individuals in neutral moods. Both

cognitive flexibility (i.e., being able to identify similarities and differences between stimuli) and

access to novel information would likely enhance case method discussion because they increase

the probability that leaders will (1) bring unique perspectives to group discussion, and (2) be able

to compare, contrast, and ultimately integrate those differing perspectives, resulting in new

knowledge. Like positive affect, negative affect might play a productive role in learning from

case studies because negative moods may trigger a desire to process information more deeply

and to think more analytically (George, 2000; Sinclair, Mark, & Clore, 1994). The results of the

present investigation are consistent with the proposition that negative affect is related to thinking

in that officers who reported experiencing negative affect during the film also tended to report

that the cultural awareness module was thought-provoking. However, negative affect did not

significantly correlate with the behavioral judgment posttest, which is another indicator of

critical thinking.

It is worth noting that the measure of positive affect used in this investigation included

several items that measured emotions that could be considered conducive to the learning process

(e.g., alert, attentive, enthusiastic, and interested). One explanation for why learning measures

and positive affect were correlated is that leaders who experienced positive affect might have

paid better attention to the details of the film, and leaders were then able to bring more film

content into discussion and analysis during their discussion groups. The unexpected correlation

between positive affect and emphasis on cultural issues prior to discussion of the film may serve

as evidence to support that claim. Prior to watching the film, leaders were informed that they

would be completing a module on cultural awareness, so it is possible that leaders who were

more alert and attentive tended to focus more on the cultural issues in the film. As a result,

individuals who experienced greater positive affect also tended to place greater weight on

cultural content during the ranking task. Unfortunately, the data collection design used in this

research does not merit such causal interpretations, and the correlation between positive affect

and emphasis on cultural issues is based on a relatively small sample size.

Another potential explanation for the relationship between positive affect and learning

could be that leaders carry over the positive affect that they experienced during the film and then

brought that energy and enthusiasm to the discussion. Further research is necessary to explore

the potential mechanisms by which positive affect, negative affect, and other emotional variables

may benefit or harm the learning process, particularly in the context of group discussion. One

line of inquiry that might stimulate research on how emotions impact group discussion is the

work on emotional contagion, which suggests that individuals can infect others with their mood

(Barsade, 2002; Gump & Kulik, 1997; Howard & Gengler, 2001; Pugh, 2001; Sy, Cote, &

Saavedra, 2005).

In sum, the findings of this research are especially encouraging given that many of the

leaders who participated have significant deployment experience, and yet still appeared to learn

something new from the AXL cultural awareness module. Moreover, the AXL system delivers

an online format for conducting effective case method instruction without requiring intervention

from an instructor. Such a format reduces demands on instructor time in the form of lesson

21
preparation and helps to deliver interesting educational content in an online learning

environment. The AXL system, however, is flexible with respect to allowing instructors to be as

involved in instruction as they wish to be. While instructor involvement is not required to

administer AXL modules, instructors can use AXL films to conduct their own class discussions

around whatever topics they choose (Zbylut & Ward, 2004a, 2004b). Additionally, the finalized

AXL system will allow instructors to create their own online modules around either of the two

existing films or upload new text-based or filmed case studies into the AXL system's library.

Challenges in Evaluating AXL Research

Research on the AXL system is ongoing, with some of the most important evaluation

events to occur in a longitudinal investigation scheduled for 2007. Although the results of this

pilot investigation were encouraging, this research examined only one of the several AXL

modules currently available. While findings provided evidence that the cultural awareness

module is a useful instructional tool, the AXL intervention as a whole consists of two case

studies and multiple leadership modules. Our expectation is that the most meaningful learning

occurs, not from a single module, but from discussion of multiple topics across the different

situations depicted in the two filmed case studies. Moreover, it is unclear from this investigation

how long the effects in learning will last over time, or whether leader behavior and judgment has

permanently changed as a result of the intervention.

The biggest obstacle to evaluating AXL instruction has been access to sufficient numbers

of Army officers for sufficient periods of time. Given the frequency of deployments to both Iraq

and Afghanistan, the typical pool of leaders available to participate in research is greatly

reduced. Moreover, when leaders are stateside, they are busy preparing their units for the next

deployment and do not have time to participate in research, particularly if the finalized training

intervention is not yet available. Additionally, administering the entire AXL intervention (both

films and a minimum of four modules) would require a large time commitment from Soldiers,

taking valuable time away from other important leadership activities.

The researchers have, therefore, adopted an iterative evaluation process to test various

components of the AXL system at a time. When the first beta-version of AXL was produced, the

researchers examined differences due to media effects (Zbylut & Ward, 2004a; Zbylut et al.,

2005). Testing for media effects allowed the researchers to conduct studies that required a

limited number of research participants for a few hours. This permitted the researchers to

observe how leaders discussed an AXL case study when an instructor was not present, as well as

gather reactions to different modes of presenting the material. Observations were then used to

refine the AXL interface and make modifications to the pedagogical approach. In sum, the first

wave of data collection primarily targeted usability issues, media effects, and reactions to the

AXL approach.

The results reported in this paper represent the second iteration of AXL evaluation. This

second wave of data collection focused on gathering reactions to the revised AXL approach,

assessing learning within the context of a single AXL module, and exploring the relationship

between emotions and learning. Similar to our previous experience, only a limited number of

leaders were available for participation and only two-and-a-half hours were allotted for data

22
collection. Given the time constraints and the desire to have a sample large enough to conduct

statistical tests, the researchers chose to focus evaluation efforts on a single AXL module and

evaluate learning within the constraints of a short-term context.

Evaluating learning within a short-term context presented a research challenge for this

particular project. Although a portion of AXL interactive modules is devoted solely to the

analysis of the film, the AXL system also requires that leaders reflect on and discuss their

personal experiences and make comparisons between their experiences and the film. Moreover,

it is likely that the sharing and discussion of these leadership experiences is one of the more

valuable aspects of the AXL system (Palus, Horth, Selvin, & Pulley; 2003; Zbylut & Ward,

2004a). Unfortunately, since each leader's experience is unique and group dynamics can take a

discussion in a multitude of different directions, it is difficult to predict precisely what "pearls of

wisdom" leaders will extract from AXL instruction.

We dealt with the challenge of assessing learning in a short-term context in a few

different ways. The simplest approach was to gather self-reports of learning. We asked leaders

to indicate whether the instructional approach was thought-provoking and also if they would be

able to transfer something that they learned to their activities as a leader. The other two

approaches used to assess learning were a rank-ordering task and a set of forced-choice items

targeting judgment. Both of these approaches required us to conduct a preliminary data

collection effort in order to ascertain what issues the target training audience found salient and

what courses of action the target training audience thought were appropriate in the Tripwire

scenario. While all of these short-term approaches taken together provide an indication that the

AXL system holds promise, the primary question that remains unanswered is how the approach

impacts general knowledge and leader behavior over time. Access to sufficient numbers of

officers in a longitudinal context is required to answer that question.

Concluding Remarks

Practitioners interested in the delivery of online leader development tools can easily

leverage many of the techniques applied in the AXL system. Specifically, using multimedia case

studies, particularly in film format, appears to capture leader attention and evoke emotion.

Moreover, a computer can be used to facilitate the group discussion process through the use of

open-ended questions that target analysis of the case study and reflection on personal

experiences. Close-ended questions can be used to branch discussion along different lines of

thought and provide feedback to individuals to exert quality control over the discussion. This

paper provides evidence that online instructional tools can be both interesting and educational.

23
24
REFERENCES

Anderson, J. R., Grenno, J. G., Kline, P. J., & Neves, D. M. (1981). Acquisition and problemsolving skill. In J. R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 191-230).

Hillsdale, NJ: Lawrence Erlbaum Associates.

Baddeley, A. D. (1986). Working memory. Oxford, England: Oxford University Press.

Baddeley, A. (2003). Working memory: Looking back and looking forward. Nature Reviews:

Neuroscience, 4, 829-839.

Baddeley, A. D., & Hitch, G. J. (1994). Developments in the concept of working memory.

Neuropsychology, 8, 485-493.

Barsade, S. G. (2002). The ripple effect: Emotional contagion and its influence on group

behavior. Administrative Science Quarterly, 47, 644-675.

Brown, F. J. (2003). Leaders for America's Army. Military Review, 83, 68-78.

Burpo, F. J. (2006, January-February). The great captains of chaos: Developing adaptive

leaders. Military Review, 64-70.

Cavalier, R., & Weber, K. (2002). Learning, media, and the case of Dax Cowart: A comparison

of text, film, and interactive multimedia. Interactive Learning Environments, 10, 243-

262.

Chi, M. T. H., & VanLehn, K. A. (1991). The content of physics self-explanations. The Journal

of the Learning Sciences, 1(1), 69-105.

Cianciolo, A. T., Antonakis, J., & Sternberg, R. J. (2004). Practical intelligence and leadership:

Using experience as a "mentor." In D. V. Day, S. J. Zaccaro, S. M. Halpin (Eds.), Leader

development for transforming organizations (pp. 211-236). Mahwah, NJ: Earlbaum.

Crittenden, V. L., Crittenden, W. F., & Hawes, J. M. (1999). The facilitation and use of student

teams in the case analysis process. Marketing Education Review, 9(3), 15-23.

Dalgarno, B. (2001). Interpretations of constructivism and consequences for computer assisted

learning. British Journal of Educational Technology, 32(2), 183-194.

Diamantes, T., & Ovington, J. (2003). Storytelling: Using a case method approach in

administration preparation programs. Education, 123(3), 465-469, 451.

Feldman, L. A. (1995). Valence focus and arousal focus: Individual differences in the structure

of affective experience. Journal of Personality and Social Psychology, 69, 153-166.

25
Gehring, R. E., & Toglia, M. P. (1988). Relative retention of verbal and audiovisual information

in a national training programme. Applied Cognitive Psychology, 2, 213-221.

George, J. M. (2000). Emotions and leadership: The role of emotional intelligence. Human

Relations, 53, 1027-1055.

Gick, M. L. (1986). Problem-solving strategies. Educational Psychologist, 21(1-2), 99-120.

Gordon, A. (2005). The fictionalization of lessons learned. IEEE Multimedia, Oct-Dec 2005, 12-

14.

Green, T. B., & Cotlar, M. (1973a). Do filmed cases improve the case method. Training and

Development Journal, 27(5), 28-31.

Green, T. B., & Cotlar, M. (1973b). An evaluation of a novel business simulation technique for

management development. Journal of Business, 46(2), 212-229.

Gump, B. B., & Kulik, J. A. (1997). Stress, affiliation, and emotional contagion. Journal of

Personality and Social Psychology, 72, 305-319.

Hannafin, M. J., Hill, J. R., & Land, S. M. (1997). Student-centered learning and interactive

multimedia: Status, issues, and implication. Contemporary Education, 68(2), 94-99.

Hassall, T., Lewis, S., & Broadbent, M. (1998). Teaching and learning using case studies: A

teaching note. Accounting Education, 7, 325-334.

Hill, R. W., Douglas, J., Gordon, A. S., Pighin, F., & van Velsen, M. (2003, August). Guided

conversations about leadership: Mentoring with movies and interactive characters.

Proceedings of the 15"' Innovative Applications of Artificial Intelligence Conference.

Acapulco, Mexico. Menlo Park, CA: AAAI Press.

Hill, R. W., Gordon, A. S., & Kim, J. M. (2004, December). Learning the lessons of leadership

experience: Tools for interactive case method analysis. Poster presented at the 24'h Army

Science Conference, Orlando, FL.

Holmes, E. A., Brewin, C. R., & Hennessy, R. G. (2004). Trauma films, information processing,

and intrusive memory development. Journal of Experimental Psychology: General, 133, 3-

22.

Howard, D. J., & Gengler, C. (2001). Emotional contagion effects on product attitudes. Journal

of Consumer Research, 28, 189-201.

Jennings, D. (1996). Strategic management and the case method. Journal of Management

Development, 15(9), 4-12.

26
Jonassen, D. H. (2002). Engaging and supporting problem solving in online learning. The

Quarterly Review of Distance Education, 3(1), 1-13.

Kreber, C. (2001). Learning experientially through case studies? A conceptual analysis.

Teaching in Higher Education, 6, 217-228.

Lesgold, A. M. (2001). The nature of learning by doing. American Psychologist, 56(11), 964-

973.

Leeper, R. V. (1993, April). Mutiny on the Bounty: A case study for leadership courses. Paper

presented at the annual convention of the Central States/Southern States Communication

Association, Lexington, Kentucky.

Maltby, J. (2001). Second thoughts about "Cases in Auditing." Accounting Education, 10, 421-

428.

Martin, V. L., & Pressley, M. (1991). Elaborative effects depend on the nature of the question.

Journal of Educational Psychology, 83, 113-119.

Matthew, C. T., Cianciolo, A. T., & Sternberg, R. J. (2005). Developing effective military

leaders: Facilitating the acquisition of experience-based tacit knowledge. (ARI Technical

Report 1161) Arlington, VA: U.S. Army Research Institute for the Behavioral and Social

Sciences.

Mayer, R. E. (1996). Learners as information processors: Legacies and limitations of educational

psychology's second metaphor. Educational Psychologist, 31 (3/4), 151-161.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.

Mayer, R. E., Dow, G. T., & Mayer, S. (2003). Multimedia learning in an interactive selfexplaining environment: What works in the design of agent based microworlds? Journal of

Educational Psychology, 95(4), 806-812.

Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions

of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86(3),

389-401.

Mehrabian, A., & and Russell, J. A. (1974). An approach to environmental psychology.

Cambridge, MA: MIT Press.

Moilanen, J. H., & Craig, D. M. (2000). Leader development in a transforming Army. Military

Review, 80, 12-16.

Moreno, R. (2006). Learning in high-tech and multimedia environments. Current Directions in

Psychological Science, 15(2), 63-67.

27
Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by missing auditory and

visual presentation modes. Journal of Educational Psychology, 87, 319-334.

Murray, N., Sujan, H., Hirt, E. R., & Sujan, M. (1990). The influence of mood on

categorization: A cognitive flexibility interpretation. Journal of Personality and Social

Psychology, 59, 411-425.

Paivio, A. (1986). Mental representations: A dual coding approach.. New York: Oxford

University Press.

Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of

Psychology, 45, 255-287.

Palus, C. J., Horth, D. M., Selvin, A. M., & Pulley, M. L. (2003). Exploration for development:

Developing leadership by making shared sense of complex challenges. Consulting

Psychology Journal: Practice and Research, 55, 26-40.

Petraeus, D. H. (2006, January-February). Learning counterinsurgency: Observations from

soldiering in Iraq. Military Review, 2-12.

Pugh, S. D. (2001). Service with a smile: Emotional contagion in the service encounter.

Academy of Management Journal, 44, 1018-1027.

Pressley, M., McDaniel, M. A., Turnure, J. E., Wood, E., & Ahmad, M. (1987). Generation and

precision of elaboration: Effects on intentional and incidental learning. Journal of

Experimental Psychology: Learning, Memory, and Cognition, 13(2), 291-300.

Richardson, B. (1994). Towards a comprehensive view of the case method in management

development. Industrial and Commercial Training, 26(9), 3-10.

Rosenshine, B., Meister, C., & Chapman, S. (1996). Teaching students to generate questions: A

review of the intervention studies. Review of Educational Research, 66(2), 181-221.

Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological

Review, 110, 145-172.

Scales, R. H. (2006, January-February). The second learning revolution. Military Review, 37-44.

Sinclair, R. C., Mark, M. M., & Clore, G. L. (1994) Mood-related persuasion depends on

(mis)attributions. Social Cognition, 12, 309-326.

Sprau, R. (2001). 1 saw it in the movies: Suggestions for incorporating film and experiential

learning in the college history survey course. College Student Journal, 35(1), 101-113.

28
Sternberg, R., Forsythe, G., Hedlund, J., Horvath, J., Wagner, R., Williams, W., Snook, S., &

Grigorenko, E., (2000). Practical intelligence in everyday life. New York, New York:

Cambridge University Press.

Stewart, J. P., & Dougherty, T. W. (1993). Using case studies in teaching accounting: A quasiexperimental study. Accounting Education, 2, 1-10.

Sy, T., Cote, S., & Saavedra, R. (2005). The contagious leader: Impact of the leader's mood on

the mood of group members, group affective tone, and group processes. Journal of Applied

Psychology, 90, 295-305.

Thompson, V. A., & Paivio, A. (1994). Memory for pictures and sounds: Independence of

auditory and visual codes. Canadian Journal of Experimental Psychology, 48, 380-396.

Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures

of positive and negative affect: The PANAS scales. Journal of Personality and Social

Psychology, 54, 1063-1070.

Willoughby, T., Wood, E., Desmarais, S., Sims, S., & Kalra, M. (1997). Mechanisms that

facilitate the effectiveness of elaboration strategies, Journal of Educational Psychology, 89,

682-685.

Zbylut, M. L., & Ward, J. N. (2004a, December). Developing interpersonal abilities with

interactive vignettes. Poster presented at the 24th Army Science Conference, Orlando, FL.

Zbylut, M. L., & Ward, J. N., (2004b). Think Like a Commander-Excellence in Leadership:

Educating Army leaders with the Power Hungryfilm. (ARI Research Product 2004-01).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Zbylut, M. L., Ward, J. N., & Mark, J. D. (2005, April). Constructivism in training: A

comparison of two interactive training tools. Poster presented at the 20th annual conference

for the Society of Industrial/Organizational Psychologists, Los Angeles, CA.

29