only two of the four tanks in a platoon fired their
weapons, whereas one could look at a table showing rounds
fired by individual tanks and immediately know who fired
and who did not.

Automated Aids Appended to the Simulation Network (SIMNET) .
To improve the automated performance measurement capabilities of
SIMNET, ARI guided the development of two generations of
automated aids for the training system: the Unit Performance
Assessment System (UPAS) and the Automated Training Analysis and
Feedback System (ATAFS).

In 1990, ARI initiated the development of the first
generation of automated performance measurement systems for
SIMNET: the UPAS, which was an inexpensive, DOS-based computer
system that collected and analyzed information from the SIMNET
data stream. The purpose of UPAS was to provide AAR leaders the
capability to create and experiment with new aids and displays,
which could be produced without special knowledge of computer
programming. According to White, McMeel, and Gross (1990), the
prototype UPAS provided for two types of AAR aids:

e A top-down Plan-View display that provided a bird’s-eye
view of the battlefield.

e Graphs and tables that summarized the results of the
battle.

The UPAS also provided graph and table editors that non-
programmers could use to modify existing graph and table
displays and/or to create new ones. Unfortunately, this useful
feature of UPAS was not incorporated into successor systems.

The variety of AAR aids that could be produced with the
UPAS increased with the continued development of this system,
and tools were added to help users manage the presentation of
AARs (Meliza and Tan, 1996). Many of the changes made in the
UPAS were in response to the results of user feedback (Shlecter,
Bessemer, Wade, and Nesselroade, 1994). UPAS aids were designed
to capture measures of unit performance that were difficult to
collect using SIMNET exercise replays. An example aid is the
“Exercise Timeline,” which displays a variety of battle events
on a timeline. Such displays might reveal when a unit crosses a
key battlefield location, such as phase lines. These data might
be used to show that a unit made many long halts during its
movement. This information could then be used to explain why
the unit failed to arrive at its attack position in time.

12



processes or techniques had been employed .. .” (p. 5). This
same emphasis on product measurement was also notable in the
early days of SIMNET development (e.g., Thorpe, 1987).

Early on, however, ARI researchers realized that process
measures were necessary in AARs to identify causes of outcomes
(i.e., to answer the question “Why did it happen?”). For
instance, significant differences between REALTRAIN and
conventionally trained units were observed for several process-
oriented performance measures (Meliza, Scott, & Epstein, 1979;
Scott, Meliza, Hardy, & Banks, 1979). To explain how light
infantry squads approached and crossed danger areas during
movement to contact missions, ARI placed a data collector on the
far side of a danger area to collect this information. In the
pre-training tests for both training conditions and in the post-
training test for conventionally trained units, the data
collectors observed most members of the squad before the first
soldier crossed the danger area and usually heard the unit
approaching the danger area before observing any soldiers. For
the squads trained using TES and AARs, very few soldiers were
observed before the first soldier crossed the danger area, and
the data collector usually heard nothing before seeing the first
soldier. The well-trained squads made contact with the enemy in
a manner that caused minimum risk to the squad as a whole,
helping to explain why these units demonstrated a higher rate of
mission success, inflicted more casualties on the enemy, and
sustained fewer casualties.

Training programs that were developed later have provided
for the collection of product and process performance measures.
For instance, the AARs in the VTP used outcome data from UPAS
and process measures of unit performance collected by 0/Cs
(Hoffman et al., 1995). The present consensus is that both
types of information are useful in AARs: product measures appear
most helpful in answering the first AAR question (What
happened?), whereas process measures aid participants in
answering the second AAR question (Why did it happen?).
Feedback can be particularly effective when those two types of
data are integrated to provide a more complete understanding of
battle events. An example of integrated AAR display is the
Exercise Timeline incorporated in the UPAS (Meliza, Bessemer, &
Tan, 1994). This display provides information on product
measures (e.g., enemy vehicles destroyed) along with crew
processes (e.g., radio reports) depicted on a common timeline.

34



O - Visual/audible indication of
impacting ordnance (direct hits only)
Visual/audible indication of ordnance effects:
0 - Fully operational

P. Visual means to [2 friendly ane enemy

0 - Catastrophic kill PB.

N - Mobility Kilt PB. Lasar dosh n capa

N - Firepower kill O- Location of impacting ordnance

N - Communications kill (may not be timely)

a MN Type combat damage Visual indication of ordnance effects:
aa “Ny O - Fully operational
\ 0 - Catastrophic kill
N- Mobility kilt
XN N - Firepower Kill

P.Gun dures C/TAF Feedback
T- Visuat/audible signature P = Player Hands-On Feadback
when weapon fires
Figure 3. Sources of intrinsic feedback needed by the
gun crew, fire direction center, laser designator
operator, and target vehicle crew relative to employment
of a 120-MM precision-guided mortar munition.
1. Monitor observer Call for Fire
procedures
2. Assess Call for Fire and Laser
designation procedures
3. If procedures valid call TAF to
FOICOLT OC * notify OC closest to victim(s), and
tell them to administratively kill
vehicte(s) depending on number
of rounds fired
4. Receive Fire Mission Data, from FDC personnel,
PLTOC forward target location for TAF analyst to enter
p 7 into TAF workstation
1. venue Section FDC OC 2. Receive PLT OC report on validity of Gun
2. If ammunition on hand Section procedures

and Gun section 3. Coordinate with FO/COLT OC on validity of FO

" procedures
procedures wai 4. If FO/COLT, FDC, and Gun section procedures

valid tell TAF analyst to fire mission
Figure 4. Examples of trainer exercise control and

feedback functions involved in providing intrinsic
feedback.

30



e Introducing an AAR Presentation Manager. This tool is
intended to integrate the various UPAS displays into a
smooth AAR presentation.

Automated Training Analysis and Feedback System (ATAFS) .
incorporated four display formats that were shown to be useful
in UPAS: the Plan View, Battle Snap Shot, Fire Fight, and Battle
Flow. (Figure 7 shows an example of the Fire Fight Display.)
ATAFS also provided graphs and tables similar to those used in
UPAS. However, ATAFS differed from UPAS in some critical ways.
First, ATAFS was implemented on a faster hardware platform and
was better able to meet the minimum acceptable delay in
preparing feedback for a SIMNET AAR. Second, the multi-tasking
system and audio capabilities of ATAFS allowed it to record and
play back radio communications synchronized with battle events.
Third, and most important, ATAFS incorporated a limited internal
knowledge base, which was designed to help the O/C choose and
create AAR displays.

Actions on Initial Contact 4st Plt, Co A, 3-70 AR
18 Nov 94
Exercise 012

Movement to Contact

Start Time: 00:05:45 Battle Drill Ordered

End Time: 00:06:40 Spot Report to Co/Tm Cdr

id
“Engaged and 4 Key points for
_ destroyed 4 AAR discussion:
_ two T-72s.

* What battle drill was
ordered? Why?

© What did the plt Idr
report to the Co/Tm
Cdr? Was the report
complete and accurate?

ae >.

Figure 7. Example of a fire fight display from ATAFS.

39



journalist and as a professional historian, Marshall developed
an oral history method, which he called the “interview after
combat,” where he assembled battle participants immediately
after the fighting had ended and conducted group interviews.
Marshall ‘and his associates would ask questions designed to lead
the participants through the battles. These interviews provided
a unique source of detailed information about what occurred
during the battle. Their efforts expanded, and, by the end of
the war, over 2,000 such interviews had been conducted in the
European theater alone (Everett, 1992). S. L. A. Marshall and
his colleagues extended their work to the Pacific theater and,
later, to the Korean and Vietnam wars. The combat interview
remains a primary method of military historical research. It
became a model for the AAR by providing a method for eliciting
objective and reliable information on combat performance.

Another influence on development of the AAR, albeit a
negative one, was the performance critique. This was the
dominant method for providing performance feedback before the
development of the AAR. Before the time when methods were
available to assess battle damage objectively, the outcomes of
simulated battles were determined subjectively by human umpires.
For instance, umpires determined the outcome of an assault on a
defended position by subjectively combining factors such as
numerical superiority, swiftness, and noise level of the
attacking unit (Gubler, 1997). Well after the battle had ended,
the senior umpire would determine an estimated outcome along
with his interpretation of tactical performance, which would be
provided in the performance critique. This feedback often
lacked credibility because it was based largely on subjective
opinion rather than on objectively measured performance and
battle outcomes. The lecture-formatted critique was often
negative in tone, thereby fomenting resentment among
participants and resistance to the umpire’s criticism.

By the early 1970s, a rapidly emerging consensus held that
the traditional performance critique was not an effective
performance feedback method and was actually counterproductive
to the goal of enhancing unit performance. Those who
implemented and participated in collective training realized
that some form of feedback was needed. This feedback
requirement fueled the development of the tactical engagement
simulation (TES), which was designed to provide more accurate
and objective feedback. Notably, in the early development
trials of TES, participants would spontaneously meet to discover
and discuss TES feedback to determine what happened during
simulated battles (Shriver, cited in Sulzen, 1986).



Meliza, L. L. & Tan, S.C. (1996). SIMNET Unit Performance

Assessment System (UPAS) Version 2.5 User’s Guide. (ARI
Research Product 96-05). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A318
046)

Meliza, L. L., & Vaden, E. A. (1995). Measuring entity and
group behaviors of semi-automated forces. Proceedings of the
Fifth Conference on Computer Generated Forces and Behavioral
Representation (pp. 181-192).

Mirabella, A., Siebold, G. L., & Love, J. F. (1998, June).
Assessment of command performance and feedback in multiforce
training. Paper presented at the NATO RTO Workshop, The
Human in Command, Kingston, Ontario, Canada.

Mirabella, A., Sticha, P., & Morrison, J. (1997). Assessment of
user reactions to the Multi-Service Distributed Training
Testbed (MDT2) System (ARI Technical Report 1061).
Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (ADA 328 473)

Mullen, B., & Copper, C. (1994). The relation between group
cohesiveness and performance: An integration. Psychological
Bulletin, 115, 215-227.

Nordyke, J., Gerlock, D., Montague, B., Huysoon, S., & Bucher,
J. (1998). CI Training Analysis and Feedback System (CTAFS)
software users manual (M67004-97-C-0014). Lawton, OK:
Advancia Corporation.

Pitts, D. (1997, July). Partnership for peace. [Interview with
General John J. Sheehan, Supreme Allied Commander for the
Atlantic]. Electronic Journal of the U.S. Information
Agency, 2(3).. Retrieved December 17, 1998, from the World
Wide Web:
http: //www.usemb.se/journals/itdhr/0797/idje/sheehan.htm

Root, R. T., Epstein, K. I., Steinheiser, F. H., Hayes, J. H.,
Wood, S. E., Sulzen, R. H., Burgess, G. G., Mirabella, A.,
Erwin, D. E., & Johnson, E. (1976). Initial validation of
REALTRAIN with Army combat units in Europe (ARI Research
Report 1191). Alexandria, VA: U.S. Army Research Institute
for the Behavioral and Social Sciences. (AD A034 610)

67



used in several different fields, such as management,
psychiatry, and law. For the AAR, Kaplan and Fallesen (1986)
described descriptive communications as methods that replace
abstractions with specific statements. In so doing, these
methods discourage judgmental comments in favor of more specific
comments about behavior. Using the concept of descriptive
communications, Downs, Johnson, and Fallesen (1987, p. B-3)
derived the following four prescriptions for the AAR leader:

1. Be specific—abstractions should be avoided. Statements
such as “I didn’t get a good feel for...” or “We
didn’t have our timing down” should be replaced with
more specific statements.

2. Be thorough—avoid the inclination “to make a long story
short.”

3. Focus on behaviors.

4. Refer to goals and how successfully or unsuccessfully
they were met.

Use of descriptive communications techniques should have
two effects on AAR processes. First, these techniques should
increase AAR learning by increasing the probability of
successful communication (i.e., being understood by others).
Second, these methods should minimize overly judgmental
comments, thereby decreasing the resentment that could result
from negative feedback. This latter affective consequence
increases the probability that participants will implement the
strategies at home station.

Questioning Techniques

The various sources for AAR guidance consistently agree on
the importance of the AAR leader's using “open-ended” questions.
One of the initial ARI sources (Bosley, Onoszko, Knerr, &
Sulzen, 1979) identifies asking open-ended questions as an
essential skill that AAR leaders must learn. The term “
refers to questions that are structured to allow for many
acceptable answers. . . . Open-ended questions are used to
stimulate thinking and to elicit discussion” (Cameron, et al.,
1997, pp. 35-36). A Leader’s Guide to After-Action Reviews
(CAC, 1993) advises the following:

[The AAR leader] should not ask yes or no questions,
but encourage participation and guide discussion by
using open-ended and leading questions. An open-ended

49



questions directly to those participants who do not appear to be
contributing to the discussion. In general, the best advice to
prevent social loafing may be to enforce the AAR rule that
requires everyone to participate actively in the process (e.g.,
Scott, 1983, 1984; Scott & Fobes, 1982; Bosley, Onoszko, Knerr,
& Sulzen, 1979).

Group Identification and Cohesiveness. The AAR
participants are not a group of unrelated individuals; rather,
they are members of a unit or team. In general, behavioral
research has shown that group performance improves to the extent
that members identify with and are committed to the group. The
document entitled A Leader’s Guide to After-Action Reviews( CAC,
1993) acknowledges this effect when it states that one purpose
of the AAR is to “ .. . promote bonding and esprit . . .” among
unit members (p. 1). Siebold and Kelly (1988) developed a short
questionnaire designed to assess the cohesiveness of combat
platoons. In a subsequent ARI study, Keesling, Ford, O’Mara,
McFann, and Holz (1992) examined the relationship of responses
to. this questionnaire and platoon performance at NTC. They
found a relatively small but positive correlation between
performance and a component of the instrument related to the
leader. characteristics—specifically, his perceived competency
and how well he trains with his platoon members.

A recent review of research has shown that the apparent
mechanism for the positive effects of group cohesiveness is the
group’s commitment to the task (Mullen & Copper, 1994).
Moreover, the relationship is bidirectional: cohesiveness
increases task performance by increasing commitment to the task
and the act of performing the task together increases
commitment, thereby increasing cohesiveness. The first
component of this bidirectional relationship implies that
increasing cohesiveness could promote better performance in the
AAR and in subsequent exercises. The second component implies
that cohesiveness is a result rather than a cause of good
performance in exercises—the interpretation favored by Keesling
et al. (1992).

Regardless of the direction of the relationship, increased
group cohesiveness is a valued outcome. In that regard, many
theories and findings in social psychology suggest that the
following factors increase group cohesiveness:

e Increasing intergroup interactions.
e Inducing agreement on group goals using consensus-

building.

47



A key feature of the experiential learning approach is its
emphasis on active learning through experimentation. In that
sense, it is similar to discovery learning. There are two
differences, however. First, experiential learning is more
structured. It is carefully designed by teachers, and learners
must critically reflect on their experiences (Fardouly, 1999).
Second, discovery learning evolved from concept learning
research on individuals, whereas experiential learning
emphasizes the role of peer group interaction, particularly in
providing feedback and helping others to reflect on the results
from experience (Cameron et al., 1997).

For the AAR process, the implications of experiential
learning are similar to those derived from discovery learning:
to achieve maximum benefits, unit members must actively
participate in the process. In addition, experiential learning
emphasizes the importance of feedback from others who share the
learning experience. Also, the cyclical nature of experiential
learning implies that the unit realizes the full benefit from
observation and conceptualization provided in the AAR only if it
follows that up with actively experimenting in the simulation
and experiencing the results. In many ways, the iterative
cycles of exercises and AARs used in the VIP provide an ideal
environment for experiential learning.

Cooperative Learning

Related to the concept of social facilitation is the
instructional concept of cooperative learning. This concept
stipulates that simply providing the opportunity for students to
work together enhances academic achievement. Although
cooperative learning is an old concept, modern research on the
topic dates back only to the early 1970s. Slavin (1995)
recently reviewed this work and characterized it as “ .. . one
of the greatest success stories in the history of educational
research” (p. 1). Not only is cooperative learning a popular
topic for educational research and theory, but also surveys by
Slavin and others show that it is a popular teaching method
across the United States.

Slavin (1995) maintained that the effects of cooperative
learning are usually explained using one of four theoretical
perspectives:

1. Motivation. Cooperative learning is a situation in
which group members are rewarded for group, as opposed
to individual, performance. This situation creates an
interpersonal reward structure wherein members give or

53



Research Opportunities

Our analysis of the principles that underlie the AAR
suggested several potential areas for research. Some of these
areas are described below.

The After Action Review (AAR) and Other Sources of Feedback

As suggested by Brown, Nordyke, et al. (1998), the AAR
should be viewed in the context. of all sources of feedback.
Current simulation-based training should be examined to identify
appropriate and inappropriate sources of feedback, including
that provided in an after-exercise session facilitated by a
leader. Hoffman et al. (1995) took a larger perspective in
their analysis of SIMUTA training materials, but their analysis
and conclusions focused narrowly on the methods and simulations
used in the RCVTP.

We propose that the requirements for feedback be examined
for the general case. The proposed method would be to review
the literature to answer basic questions about when and where
feedback should be presented. For instance, this research could
determine whether it would be better to present certain types of
feedback during the exercise (as coaching and mentoring) or
after the exercise. By viewing the AAR in a larger context, we
see how it fits in with other complementary feedback systems.

Compliance with After Action Review (AAR) Principles

The principles that underlie the AAR are relatively well
known. They are presented in Army documents (e.g., TC 25-20)
and are discussed in detail in the research reports cited in
this document. However, it would be useful to document the
extent to which the principles outlined in this report and other
Army documents are complied with in practice. Gubler (1997)
attempted to do this, but his attempt was based on only
seventeen videotaped platoon and company AARs from the NTC and
JRTC. This sort of work should be expanded to include examples
of AARs from higher echelons and from other training sites, such
as ARTEPs conducted at home station and collective exercises
performed at simulation centers.

A model of effective AARs should be synthesized from the
research literature and military guidance. The model would
suggest appropriate methods for research. For instance, one
part of the model might relate to methods for enhancing memory
for feedback. In that case, it would be useful to track,
through interview or systematic observation, the extent to which

57



analysis, whereas the latter model stresses experience in the
problem-solving process.

Mental Models

A mental model is an internal construct of a learner's
physical or social world to remember, think about, and explain
events and concepts. In the context of the AAR, mental models
help explain the events that occurred during the simulated
exercise. As participants acquire more diverse details about
exercise events, their mental models become more elaborate.
More elaborate mental models are easier to remember and
manipulate and, thus, are associated with enhanced performance.
Moreover, because AARs typically involve teams of individuals
(e.g., staffs, crews, platoons), they should share identical
(or, at least, very similar) mental models so that they have a
common understanding of events and their causes.

A particularly effective method for providing more
elaborate details about the simulated battle is to hear and
react to input from others. The AAR embodies this idea by
encouraging participants to provide their experiences and listen
to interpretations from others. This technique is particularly
effective in fostering shared mental models of task performance.
Another approach is to have members of other teams also discuss
the events from their points of view. OPFOR members provide a
valuable input because they offer a diametrically opposed point
of view on the exercise.

Group Dynamics

In addition to being a learning process, the AAR is a
social process in that participants come together to make
collective decisions about their performance. As collective
process, the facilitating and constraining effects of group
dynamics are relevant to AAR processes. Some of the better
known effects of group dynamics are discussed below.

Social Effects on Performance

Research indicates that the presence of others has
significant effects on performance. As described below, these
effects may be either positive or negative.

Social Facilitation. The AAR benefits from the positive
effects of social facilitation—which is the enhancement of
performance caused by the mere presence of other humans. This
phenomenon is widely observed and is usually attributed to

45



descriptions of training environments suggests that AAR methods
may differ as a function of the simulation (live, virtual, or
constructive) and echelon and whether the training is structured
or not structured. As the options for training environments
increase, the need for tailored guidance will also expand.

Various combinations of training environments and training
audiences appear to present certain unique challenges to the AAR
process; however, at the same time, many combinations also offer
certain unique benefits. For example, consider Word’s strategy
of having OPFOR take part in an AAR to provide units with
information about how the enemy perceived their actions. In
most virtual exercises, a computer-generated force represents
the enemy, so no human OPFOR take part in the AAR. Similarly,
feedback resources for live training may be limited when a
series of AARs are conducted, beginning, for example, at the
platoon, company, and then battalion level after a battalion
exercise. In the latter situation, bringing in that specific
part of the OPFOR with which a particular platoon or company
interacted would be difficult, so platoons and companies have to
look elsewhere for the information they need.

ARI continues to analyze the feedback and AAR requirements
of new developments in training and force modernization to
determine whether new methods or aids are required (Brown,
Nordyke, et al., 1998). ARI has focused recent efforts on
developments that have immediate impact on feedback
requirements: the new information sources provided by digitized
command, control, communications, computers, and intelligence
(c*I) systems. Before the advent of digital communications, an
0/C monitored voice communications and exercises while playing
the roles of higher headquarters, supporting units, and adjacent
units. With digital C*I systems, however, much of the
communication of tactical information is displayed visually on
computers located in combat vehicles.

Monitoring all sources of verbal and visual information is
difficult—if not impossible. The increased emphasis on
monitoring communications makes it more difficult for an O/C to
perform other duties, such as role playing, exercise control,
and so forth. Moreover, without the input from all digital
communications, the ability of the 0/C to prepare AARs on the CI
aspects of the mission has been reduced. To aid the 0/C in
developing AARs for digitized units, ARI is currently monitoring
the STRICOM-funded development of the C*I Training Analysis and
Feedback System (CTAFS) for implementation in the Close Combat

16



Data Stream Issues

The potential for automating performance measurement
functions was greatly enhanced with the introduction of
computer-based simulations, which provide rich data streams for
analysis and feedback. However, the technology for measuring
performance on such systems is complex and is constantly
evolving. ARI has continued to monitor this technological issue
through the Workshops on Standards for the Interoperability of
Defense Simulations, where standards are written and monitored
for the DIS protocols that replaced SIMNET protocols. These
standards are being developed for all U.S. Services, allied
military units, and industry with the intent of supporting
exercises in the live, virtual, constructive, and mixed
environments.

An example data stream issue that impacts AAR is the Change
of Status (COS) data packet. ARI requested a DIS data packet
that would serve the same function that the COS data packet had
served in SIMNET. A simulated entity sends this packet over the
network when the entity is tactically damaged or destroyed,
administratively destroyed, or administratively reincarnated.
This packet identifies the nature of the status change and the
cause of the status change. Under the original DIS protocols,
there was no direct replacement for the COS packet. Asa
result, the AAR system (or the human operator) was tasked with
deciding the cause of status changes when preparing many data
summary graphs and tables, thus slowing down the preparation of
BAR aids (Meliza, 1995).

Automated Performance Measurement Systems

Automating performance measurement systems have had a
positive effect on the AAR process in at least two ways. First,
they potentially have increased the amount and accuracy of
information available for the AAR. Second, they have relieved
0/Cs from many manual performance measurement functions and have
allowed them to focus on other important functions, such as
observing global aspects of performance and providing coaching
and mentoring as appropriate. As discussed in the Historical
Foundations section, ARI directed the development of two systems
for automating performance measurement functions in SIMNET and
other DIS systems: UPAS and ATAFS. The following sections
summarize the R&D efforts that supported the production of UPAS
and ATAFS. They also include research on STAARS, an evolving
system that is based on the lessons learned from these two
systems.

35



Meliza, L.L. (1996). Standardizing army after action review
systems (ARI Research Report 1702). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social
Sciences. (AD A322 044)

Meliza, L. L. (1999). A guide to standardizing after action
review (AAR) aids (ARI Research Product 99-01). Alexandria,
VA: U.S. Army Research Institute for the Behavioral and
Social Sciences. (AD A359 843)

Meliza, L. L., Bessemer, D. W., Burnside, B. L., & Schlechter,
T. M. (1992). Platoon-level After Action Review aids in the
SIMNET Unit Performance Assessment System (UPAS) user’s guide
(ARI Technical Report 956). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.
(AD A254 904)

Meliza, L. L., Bessemer, D. W., & Tan, S. C. (1994). . Unit
Performance Assessment System development. Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A285 805)

Meliza, L.L., & Brown, B. (1996). Increasing the speed
flexibility of feedback systems for DIS exercises.
Proceedings of the 18th Interservice/Industry Training
Systems and Education Conference (pp. 132-139).

Meliza, L. L., Scott, T. D., & Epstein, K. I. (1979). REALTRAIN
validation for rifle squads II: Tactical performance (ARI
Research Report 1203). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A069
957)

Meliza, L. L., Sulzen, R. H., Atwood, N. K.,.& Zimmerman, R.
(1987). National Training Center guidebook for force-on-
force company/platoon observer/controllers. Fort Irwin, CA:
National Training Center.

Meliza, L. L., & Paz, B. (1996). Integrating exercise control
and feedback systems in DIS. Proceedings of the 18th
Interservice/ Industry Training Systems and Education
Conference (pp. 333-340).

66



FOUNDATIONS OF THE AFTER ACTION REVIEW PROCESS

EXECUTIVE SUMMARY

Research Requirement:

The after action review (AAR) is the Army’s method for
providing performance feedback from a collective training
exercise. The U.S. Army Research Institute for the Behavioral
and Social Sciences (ARI) has performed much of the research and
development (R&D) for the AAR. The purpose of this report is to
review the evolution of the AAR process, identify some of the
major behavioral science principles that underlie the AAR, and
propose directions for future R&D.

Procedure:

The foundations of AAR were determined by conducting two
literature reviews. The first was a historical review to
examine the evolution of AAR concepts. This review described,
in chronological order, the major historical developments that
have shaped methods and concepts related to the AAR process.
The second was a conceptual review of research on the AAR
process. This review was focused on deriving the behavioral
science principles that underlie the AAR process and was
organized by general areas of theory and research in behavioral
sciences. Finally, the findings from the two reviews were
synthesized for implications for future research.

Findings:

The history of the AAR is relatively short, covering only
the last 25 years. Before that period, two influences set the
stage for the development of the AAR:

1. S. L. A. Marshall’s “interviews after combat.” This was
an oral history technique, first used in World War II.

2. “Performance critique.” This was a feedback technique
used to provide feedback from tactical exercises before
the 1970s.

The first AAR methods were implemented in the mid 1970s in
training programs developed by ARI for their optically based
Tactical Engagement System (TES) training programs. Soon
thereafter, AAR methods were modified for the next-generation
TES system, the Multiple Integrated Laser Engagement Simulation



Shlechter, T. M., Bessemer, D. W., Wade, C. R., & Nesselroade,
K. P., Jr. (1994). Evaluating the Unit Performance
Assessment System’s after action review displays (ARI
Technical Report 997). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A281
712)

Shriver, E. L., Mathers, B. L., Griffin, G. R., Jones, D. R.,

Word, L. E., Root, R. T., & Hayes, J. F. (1975). REALTRAIN:
A new method for tactical training of small units (ARI
Technical Report S-4). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A024
030)

Siebold, G. & Kelly, D. (1988). Development of the combat

platoon cohesion questionnaire (ARI Technical Report 817).
Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A204 917)

Sikora, J., & Coose, P. (1995). What in the world is ADS?
PHALANX, 28, 1, 6-8.

Slavin, R. E. (1995). Research on cooperative learning and
achievement: What we know, what we need to know, [Paper
available on line.] Retrieved January 7, 1999, from the

World Wide Web: http://scov.csos.jhu.edu/sfa/cooplear.html

Sullivan, G. R. (1995). Gordon R. Sullivan: The collected
works 1991-1995 (CMH Publication 70-44). Washington, DC:
U.S. Army Center of Military History. Retrieved November 17,
1998, from the World Wide Web:
http://www.army.mil/call/csa/sullivan.htm

Sullivan, G. R., & Harper, M. V. (1996). Hope is not a_ method:
What business leaders can learn from America's Army. New
York: Times Business, Random House.

Sulzen, R. H. (1986). Annotated bibliography of tactical
engagement simulation 1966-1984 (ARI Technical Report 725).
Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A178 671)

Thorpe, J. A. (1987). The new technology of large scale
simulator networking: Implications for mastering the art of
warfighting. Proceedings of the 9th Interservice Industry
Training Systems Conference.

69



Root, R. T., Hayes, J. F., Word, L. E., Shriver, E. L., &

Griffin, G. R. (1979). Field test of techniques for tactical
training of junior leaders in infantry units (Project
EFFTRAIN) (ARI Technical Report 79-A21). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A075 604)

Scott, T. D. (1983). Tactical engagement simulation after

action review guidebook (ARI Research Product 83-13).
Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences. (AD A143 014)

Scott, T. D. (1984). How to evaluate unit performance (ARI
Research Product 84-14). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A160
811)

Scott, T. D., & Fobes, J. L. (1982). After action review
guidebook I: National Training Center (ARI Research Product
83-11). Alexandria, VA: U.S. Army Research Institute for
the ‘Behavioral and Social Sciences. (AD A139 405)

Scott, T. D., Banks, J. H., Hardy, G. D., & Sulzen, R. H.
(1979). REALTRAIN validation for rifle squads III: Tactical
performance during movement-to-contact (ARI Research Report
1213). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A075 423)

Scott, T. D., Meliza, L. L., Hardy, G. D., & Banks, J. H.

(1979). Armor/anti-armor team tactical performance (ARI
Research Report 1218). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A075
354)

Scott, T. D., Meliza, L. L., Hardy, G. D., Banks, J. H., & Word,
L. E. REALTRAIN validation for armor/anti-armor teams (ARI

Research Report 1204). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A069
944)

Shlechter, T. M., Bessemer, D. W., Nesselroade, K. P., Jr., &
Anthony, J. (1995). An initial evaluation of a simulation-
based training program for Army National Guard units (ARI
Research Report 1679). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. (AD A297
271)

68



Witmer, B. G. (1990).

Conducting Precision Range and Integrated
Maneuver Exercise (PRIME) After Action Reviews (ARI Research

Product 90-21). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. (AD A225
130)

Word, L. E. (1987). Observations from three years at the
National Training Center (ARI Research Product 87-02).

Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A179 386)

Zajonc, R. B. (1965). Social facilitation. Science, 149,
269-274.
Zeidner, J

., & Drucker, A. J. (1988). Behavioral science in the
Army:

A corporate history of the Army Research Institute.

Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences.

Zsambok, C. E., & Klein, G. A. (Eds.) (1997).
decision making. Mahwah, Nd:

Naturalistic
Inc.

Lawrence Erlbaum Associates,

71



SIMNET: Modular Semi-automated Force (ModSAF) Version 1.2 and
Semi-automated Force (SAFOR) Version 4.1.3. One of the
comparisons involved using the UPAS to examine movement speeds
of tanks generated by each version of CGF as the tanks moved
along the same route to decide if one was more sensitive to
slope and other terrain variables that should influence speed.
The SAFOR-controlled tank drove at the same speed for 98 percent
of the test period, while the ModSAF-controlled tank drove at a
wide variety of speeds.

Exercise Control for Live Simulations. Trainers for live
force-on-force face a heavy workload. In addition to coaching
and mentoring units and conducting AARs, they must often help
simulate the effects of weapons and play the role of higher,
adjacent, and supporting units so that the units they are
training will receive appropriate intrinsic feedback. The
duties associated with providing this feedback can detract from
the time available to prepare for AARs.

The Army is fielding a large variety of new and innovative
operational systems over the next 10 years under the “force
modernization” and “Force XXI” rubrics, including new weapon
systems (smart and non-lethal varieties); reconnaissance,
surveillance, and target acquisition (RSTA) systems; and digital
communication and decision aid systems. Training developers
have suspected that the feedback requirements for these new
systems will be significantly greater than those for extant
systems and could overwhelm the trainers.

In response to a studies request from the TRADOC Army
Training Modernization Directorate, ARI initiated a program in
February 1997 to study instrumentation issues relevant to the
support of AARs in the live simulation training environment
(i.e., at the CTCs and at home station field exercises). The
result was a document entitled “The Training Analysis and
Feedback Aids (TAAF-Aids) Study” (Brown, Nordyke, et al., 1998).
The primary goal of this effort was to describe the impacts of
new weapons, RSTA, and digital C*I systems on exercise and
feedback functions performed by O/Cs and analysts who support
training. This work also set the stage for the longer term
objective of applying automation to support the training
feedback process. In that regard, the secondary objective was
to present high-level strategies for helping trainers address
the increasing workload expected to be required by the new
systems.

28



UPAS offered advantages over those provided by AAR aids
built into SIMNET. Although UPAS was improved continually
throughout its development, Meliza and Brown (1996) pointed out
some serious shortcomings that could not be overcome through
software modification.

The fundamental shortcoming of UPAS was that it was
implemented on an early-generation, DOS-based machine that could
only execute one task at a time. Three specific problems were
identified:

1. During exercises, UPAS was fully occupied collecting
exercise data; therefore, the task of selecting and
creating AAR aids could not begin until the end of the
exercise (ENDEX). As a result, the preparation of UPAS
feedback regularly exceeded the 10-20 minute time limit
that was generally accepted as the minimal acceptable
delay—despite software improvements designed to speed
up the system.

2. UPAS could not simultaneously collect voice data linked
to events while collecting network data. As a result,
it could not record and play back radio communications
data synchronized with visual data. The lack of voice
data was deemed a serious shortcoming in determining
what happened during an exercise.

3. UPAS did not provide trainers help in deciding what AAR
aids would be useful for a particular exercise and how
they might be used. This feature was crucial in
determining whether the system would be used as
designed.

With the limitations of UPAS in mind, the Army’s
Simulation, Training, and Instrumentation Command (STRICOM) and
the Defense Advanced Research Projects Agency (DARPA) funded and
ARI monitored the development of the next generation of
automated AAR aids: ATAFS. The ATAFS project’s goal was to
demonstrate the capability to provide AAR leaders a menu of
automatically and manually generated aids for use in the AAR
within 10 minutes after the end of a simulation exercise (Brown,
Wilkinson, et al., 1996). The ATAFS was able to meet this goal
by employing platforms with multi-tasking capabilities and by
using a knowledge database to automatically generate AAR aids
during exercises. As a result, the ATAFS was able to provide a
bin of candidate AAR aids available for use at ENDEX.

13



The concepts of structured training and focused AARs have
been adopted in subsequent training programs developed by ARI
and implemented at the RCVTP, which was subsequently renamed the
Virtual Training Program (VTP) to incorporate active and reserve
component units (Burnside, Leppert, & Myers, 1996). In addition
to SIMUTA, the additional major programs include the following:

¢ The Simulation-Based Mounted Brigade Training (SIMBART)
program developed in 1994 for training heavy combat
brigades in the ARNG (Koger et al., 1996).

e The series of efforts entitled Combined Arms Operations
at Brigade Level Realistically Achieved Through
Simulation (COBRAS) developed in 1995 for training
active component brigades (R. C. Campbell et al., 1999).

e The Staff Group Trainer (SGT) training support packages
for battalion and brigade staffs (Koger et al., 1998).

In addition, ARI has explored the impact of the digitization of
cI systems on structured training and on the AAR process
(Winsch, Garth, Ainslie, & Castleberry, 1996).

Structured training concepts have also been applied to the
CCTT, which is the successor to SIMNET (Deatz et al., 1998). In
particular, ARI is developing the Commander’s Integrated
Training Tool (CITT) to provide CCTT users a tool for selecting,
modifying, and developing structured training exercises for
platoons and company teams. The CITT will provide the
capability to access, modify, and print all required support
materials. Two versions of CITT are currently being designed:

a standalone version and a distributed version that will be
implemented on the Internet (Gossman, Dannemiller, & Bessemer,
1998).

Standard Army After Action Review System _(STAARS)

As discussed previously, several AAR methods have been
derived to fit particular training situations. Despite the
variety of authoring sources and training environments, Cameron,
Gentner, Schopper, and Mahaney (1997) observed that “ .
there is a solid core of agreement about what an AAR is, about
its purpose and structure, and about how it should be conducted

." (p. 5). Even with this tacit agreement, however, there
were concerns that the format and content of AAR data were not
commensurable across these and other applications.

19



such as those described in Table 2.

Table 2

BAR Guidance for Specific Training Environments

The column that provides

Program Name

Simulation Environment

ARI Publication(s)

Realistic Training
(REALTRAIN)

National Training
Center (NTC)

Army Training and
Evaluation Plan
(ARTEP)

Army Training Battle
Simulation System
(ARTBASS)

Precision Range and
Integrated Maneuver
Exercise (PRIME)

Multi-Service
Distributed Training
Testbed (MDT2)

Virtual Training
Program (VTP)

Force XXI Training
Program (Combined
Arms Operations at
Brigade Level
Realistically
Achieved Through
Simulation [COBRAS] )

Staff Group Trainer
(SGT)

Simulation-Based
Multiechelon
Training Program for
Armor Units-Digital
(SIMUTA-D)

Live, optically based simulation
for platoon- and company-level
training

Live, laser-based simulation and
instrumented range for
battalion- and brigade-level
training

Live, laser-based simulation for
squad-, platoon-, and company-
level training

Constructive computer-based
simulation for battalion staffs

Live, laser-based simulation and
instrumented range for crew- and
platoon-level tank training

Live, virtual, and constructive
simulation for training multi-
service personnel in close air
support

Structured programs of virtual
and constructive simulation
exercises from crew through
brigades

Structured programs of virtual
and constructive simulation for
brigade and battalion staffs

Constructive simulation training
programs for battalion and
brigade staff groups

Virtual and constructive
simulation for battalion staffs
of digitized forces

Bosley, Onoszko,
Knerr, & Sulzen
(1979)

Scott & Fobes
(1982); Meliza,
Sulzen, Atwood, &
Zimmerman (1987)

Scott (1984)

Kaplan & Fallesen
(1986)

Witmer (1990)

Bell et al. (1997);
Mirabella, Sticha,
Morrison (1997)

Koger et al. (1996)

R. C. Campbell et
al. (1999)

Koger et-al. (1998)

Winsch, Garth,
Ainslie, &
Castleberry (1996)

&

15



The TAAF-Aids Study examined the feedback requirements of
142 new systems in live, force-on-force training. Based on the
capabilities of current TES and instrumentation systems, this
study estimated whether specific elements of feedback would be
provided by either the TES-instrumentation system, by O/Cs and
the analysts who support them, or through soldier interaction
with the operational system. Of particular interest were those
cases where current TES-instrumentation systems fail to provide
appropriate intrinsic feedback, requiring the O/C-analyst team
to affect or control the exercise to provide the needed
feedback.4 On the basis of their findings, Brown, Nordyke, et
al. (1998) concluded that, in the absence of effective
interventions, the O/Cs, and analysts will be overwhelmed by the
need to monitor and control the simulation in order to provide
feedback.

Figure 3 shows how the intrinsic feedback needed to cue and
guide the employment of one of the new systems during unit
exercises can come from interaction with equipment, TES or
instrumentation systems, and trainers. In certain cases, there
may even be gaps in feedback. For cases where it appears that
trainers will be responsible for providing feedback, the
exercise control and functions required to provide feedback were
identified in the TAAF Aids Study, as shown in Figure 4. An
important Army goal is to apply automation (TES systems and
instrumentation) to free trainers from supporting simulations so
that they can spend more time coaching, mentoring, and preparing
for AARs. In deciding where to apply automation, the Army must
consider that some of the exercise control functions directly
support coaching, mentoring, and AARs, while others distract
trainers from these activities.

4 Brown et al. (1998) studied the effects of modernization on
extrinsic as well as intrinsic feedback. They determined that
the control tasks they identified as required for intrinsic
feedback also supported extrinsic feedback.

29



including home station field exercises (Scott, 1983; 1984) and
the Army Training Battle Simulation System (ARTBASS) (Kaplan &
Fallesen, 1986).

Computer Networked Simulation

The development of the Simulation Networking (SIMNET)
system also provided a great stimulus to the development and
refinement of AAR processes. Developed and fielded in the
1980s, SIMNET was the first networked system of virtual
simulators to be used for collective training of heavy combat
units (Alluisi, 1991). Compared with TES and other live
simulations, SIMNET has greatly increased the amount of
information that can be captured and used in the AAR. As
summarized by Meliza (1996), “ . . . we went from a situation
where vehicle status data were limited to periodic updates
regarding the location and damage status of vehicles to the
point of having near continuous updates on the locations of
vehicles, the speed at which vehicles were moving, the
orientation of vehicles, engine speeds, the orientation of gun
tubes, the elevation of gun tubes, fuel levels, and ammunition
levels . . .” (p. 13).

By providing the ability to reinitialize scenarios rapidly,
networked simulations also afford units repeated opportunities
to correct problems immediately and apply solutions identified
in previous AARs. At the same time, networked simulations
shorten the acceptable delay in providing the feedback from
tactical exercises (Meliza & Brown, 1996). In live simulations
(e.g., those employed at NTC), units typically perform logistic
and maintenance functions, such as vehicle maintenance and
resupply, immediately after an exercise. In SIMNET, these
activities are either not performed or are foreshortened.
Therefore, units have come to expect their AAR: soon after an
exercise ends.

Early Simulation Network (SIMNET) Feedback Aids and
Limitations. Despite the inherent potential of SIMNET to
provide detailed performance data, the initial training version
of SIMNET provided no features to support automated performance
measurement.? In fact, the initial concept for SIMNET called for

2 This section refers to the initial training version of
SIMNET, called SIMNET-T, as first implemented at Fort Knox,
Kentucky. Later SIMNET-T sites, particularly those designed to
support gunnery competition, included more extensive automated
performance measurement systems, as did developmental sites,

10



U.S. Army Armor School (1975, January). Tactical training for
combined arms elements—REALTRAIN (TC 71-5). Fort Knox, KY:
Author.

U.S. Army Combined Arms Center (1993). A leader’s guide to
after-action reviews (Training Circular 25-20). Fort
Leavenworth, KS: Author. Retrieved September 10, 1999, from
the World Wide Web: http://call.army.mil/call/trnggtr/tql-
98/table.htm

U.S. Army Infantry School (1973, February) . Squad Combat
Operations Exercise (Simulation)—SCOPES (TC 7-2). Fort
Benning, .GA: Author.

U.S. Army Research Institute for the Behavioral and Social
Sciences (1996, Summer). After action reviews: Lessons
learned from structured training. ARI Summer 1996
Newsletter. Alexandria, VA: Author. Retrieved December 16,
1998, from the World Wide Web: http://198.97.12/aars.htm

U.S. Army Training and Doctrine Command (1997). The Standard
Army After Action Review System (STAARS): After Action
Review (AAR) Handbook, Version 2.1, 9 May 1997. Fort
Leavenworth, National Simulations Center: Author. Retrieved
December 17, 1998, from the World Wide Web:
http: //www-leav.army.mil/nsc/stow/staars/handbook.htm

White, S., McMeel, K., & Gross, W. (1990). Unit performance
measurement and training program design for networked
simulators (PTR-4013-1-90)... Woodland Hills, CA:
Perceptronics.

Winsch, B. J., Garth, T. H., Ainslie, F. M., & Castleberry, J.D.

(1996). The development of structured simulation-based
training for digital forces: Initial battalion staff-level
efforts (ARI Research Product 96-08). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social
Sciences. (AD A318 289)

Winstein, C. J., & Schmidt, R. A. (1989). Sensorimotor

feedback. In D. Holding (Ed.), Human skills (2nd ed.).
Chichester, England: John Wiley & Sons.

70



information that directly supports the unit’s training
objectives and increased preparation of structured materials,
the AAR is now conceived as a process. This process view of the
AAR has encouraged the application of systems concepts to the
design, development, and delivery of the AAR. This
system-analytic interpretation of AAR processes is consistent
with the Army’s Systems Approach to Training (SAT), which is a
more general model for the design, development, and
implementation of training.

Analysis of Processes

One of the distinguishing features of systems concepts of
instruction is the analysis of training development and
implementation into sequential phases. For instance, the
original Interservice Procedures for Instructional Systems
Development (IPISD) divided training development into five
phases: analyze, design, develop, implement, and control
(Branson et al., 1975). In turn, each of these phases was
broken down into smaller units of activities called “blocks,”
which were broken into individual steps.

In similar fashion, Kaplan and Fallesen (1986) described
the AAR process for ARTBASS exercises as comprising four phases:
determine training objectives, observe the exercise, prepare the
AAR, and conduct the AAR. Each phase was analyzed further into
multiple sequential steps. Lockheed Martin Federal Systems and
Illusions, Inc. (1996) portrayed the AAR process as comprising
only three phases: preparation, execution, and review. The
latter phase encompassed both the preparation and conduct of the
AAR. These two analyses provide similar, but not identical,
views of the AAR process.

Evaluation Component

An essential element of any systems approach to
instructional development is the assessment or evaluation
process, which functions as feedback to the system. For
instance, the control phase of IPISD fulfills just this
function. In contrast, the process models of the AAR (Kaplan &
Fallesen, 1986; Lockheed Martin Federal Systems and Illusions,
Inc., 1996) lack this crucial process. We need an evaluation
procedure specifically designed for the AAR process. Such a
procedure would allow participants and third-party observers to
evaluate AAR sessions by established principles and practices.
Crissey (publication in preparation) identifies some AAR
objective performance measures that could be included: ratio of
unit to AAR leader comments, number of solutions

55



participants determine appropriate actions to solve
problems identified in their performance. Example
actions include changes to unit standard operating
procedures (SOPs) or increased training on basic drills.

These questions are addressed during AAR sessions, which
are conducted immediately after the end of a short exercise or
during logical breaks in longer exercises. The sessions can be
formal or informal. The critical difference is that the formal
AARs require more planning, coordination, and preparation than
the informal AARs.

AAR sessions are typically organized by echelon. Platoons
are scheduled first for about 30-45 minutes, followed by
companies for about 1 hour, and then by battalion and above for
about 2 hours (CAC, 1993). Discussion participants are unit
members who took part in the simulated exercise. The AAR
discussion leader is typically not a member of the unit being
trained; rather, he is a trainer who controlled and observed the
exercise. Throughout the discussion, the leader acts as
facilitator of—as opposed to a participant in—the discussion.
Unit members have to make their own decisions and reach their
own conclusions.

The leader spends considerable time preparing for the AAR—
before the exercise (e.g., preparing data collection
instruments) and after the exercise (e.g., collecting and
aggregating performance data from various sources) . Typically,
the AAR leader begins the session by reviewing the objectives of
the exercise and the unit’s mission. Then, by asking open-ended
and leading questions, the AAR leader gets the participants to
relate the significant events that happened during the exercise.
For each event, the AAR leader tries to solicit many
perspectives to determine what really happened. After the group
agrees on the facts of what happened, the AAR leader guides the
discussion to a closer examination of key outcomes to determine
why they happened and what could be done to improve performance.

The following fictional vignette of an AAR for an armor
company illustrates some possible outcomes from an AAR session.
An armor platoon fails to provide covering fire for a second
platoon, causing the second platoon to be destroyed by the enemy
when it is forced to cross an open area. At the beginning of
the AAR, company personnel establish that an implied task in the
unit's plan was that the first platoon maintain a position to
observe and fire on the enemy should the second platoon become
engaged. Further discussion of events reveals that the first



Information Feedback

Perhaps the most basic principle on which the AAR is based
is that learning and performance are enhanced when appropriate
feedback is provided. Feedback refers to information that
people receive during or after performance of an action to
control and learn the action. As discussed below, the feedback
can come from different sources. A fundamental distinction in
types of sources for information feedback is that between
intrinsic and extrinsic feedback. Intrinsic feedback refers to
information that is inherent to task performance, whereas
extrinsic feedback denotes information that augments or
supplements inherent feedback (Winstein & Schmidt, 1989).

Intrinsic Feedback

In the context of tactical training, Brown, Nordyke, et al.
(1998) defined intrinsic feedback as the "downrange" information
provided to collective training participants from both real and
simulated sources. The simulated outcomes from tactical
engagements are important sources of intrinsic feedback,
providing information necessary for learning during the battle.
This source of feedback is necessary to cue and guide
performance during exercises, providing an opportunity for
meaningful practice of task performance. For example, if a unit
is receiving no feedback during exercises about the location and
effects of its artillery missions, the unit cannot meaningfully
perform tasks associated with adjusting and terminating the use
of artillery. After the battle, exercise participants use their
memory of this feedback to answer the first and fundamental
issue of the AAR: to establish "what happened" during the
battle.

One of the most frequently asked questions in an AAR is why
a unit chose a particular course of action (e.g., Why did you
decide to open fire as soon as you saw the enemy?”) The
question sets the stage for ensuring that the unit does what it
should do to collect and interpret information about the
tactical situation and then select an appropriate course of

26



Brown, B., Nordyke, J., Gerlock, D., Begley I. J. II, & Meliza
(1998). Training analysis and feedback aids (TAAF Aids)
study for live training support. (ARI Study Report 98-04).
Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A351 107)

Brown, B., Wilkinson, S., Nordyke, J., Hawkins, R., Robideaux,

B., & Huyssoon, S. (1996). Demonstrating the concept of an
automated training analysis and feedback system (ARI
Technical Report 1043). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD B215
579)

Bruner, J. S. (1961). The act of discovery. Harvard

Educational Review, 31(1), 21-32.

Burnside, B. L. (1990). Assessing the capabilities of training
simulations: A method and Simulation Networking (SIMNET)
application (ARI Research Report 1565). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A226 354)

Burnside, B. L., Leppert, M. A., & Myers, W. E. (1996). Virtual
training program orientation guide (ARI Research Product 97-
02). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A322 045)

Cameron, J. A., Gentner, F. C., Schopper, A. W., & Mahaney, B.
K. (1997). Essential tasks, knowledges, and skills for
distributed simulation observer/controllers. Volume 1: Final
report (CSERIAC-RA-97-001A). Wright-Patterson AFB, Ohio:
Crew System Ergonomics Information Analysis Center.

Campbell, C. H., Deter, D. E., & Quinkert, K. A. (1997). Report
on the expanded methodology for development of structured
simulation-based training programs (ARI Research Report
1710). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A328 671)

Campbell, R. C., Deter, D. E., Ford, L. A., Graves, C. R.,
Campbell, C. H., Pratt, D. M., Jenkins, S. N., & Quinkert,
K. A. (1999). COBRAS III: Report of development and lessons
learned (RR 1736). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A359
920)

62



Goldberg, S. L., & Meliza, L. L. (1993). Assessing unit
performance in distributed interactive simulations: The Unit
Performance Assessment System (UPAS). In Proceedings of NATO
Defence Research Group Meeting, Panel 8 (Defence Applications
of Human and Bio-Medical Sciences, Training Strategies for
Networked Simulation and Gaming). Technical Proceedings
AC/243 (Panel 8) TN/5 (pp. 173-182).

Gossman, J. R., Dannemiller, M. A., & Bessemer, D. W. (1998,
Fall). CITT-A user tool for managing and developing
simulation exercises (Paper 98F-SIW-138). Simulation
Interoperability Workshop. Orlando, FL: Simulation
Interoperability Standards Organization. Retrieved February
17, 1999, from the World Wide Web:
http: //siso.sc.ist.ucf/siw/98fall/view-papers.htm

Gorman, P. F. (1992). The secret of future victories (IDA Paper
P-2653). Alexandria, VA: Institute for Defense Analyses.

Gubler, J. C. (1997). Unit simulation training system after
action reviews (AAR): A novel approach to achieve

effectiveness. Unpublished master’s thesis, University of
Central Florida, Orlando, FL.

Hoare, R. (1996). From debrief to After Action Review (AAR).
Modern Simulation & Training, 6, 13-17.

Hoffman, R. G., Graves, C. R., Koger, M. E., Flynn, M. R., &
Sever, R. S. (1995). Developing the Reserve Component
Virtual Training Program: History and lessons learned (ARI
Research Report 1675). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A296
153)

Kaplan, I. T., & Fallesen, J. J. (1986). After action review
(AAR) guide for the Army Training Battle Simulation System
(ARTBASS) (ARI Research Product 86-32). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social
Sciences. (AD A180 797)

Keeney, R. L. (1982). Decision analysis: A review. Operations
Research, 30, 803-833.

Keesling, J. W., Ford, J. P., O’Mara, F., McFann, H., & Holz, R.

(1992). The determinants of effective performance of combat
units at the National Training Center (Final Report, Contract
MDA903-86-R-0705). McLean, VA: PRC, Inc.

64



Koger, M. E., Long, D. L., Britt, D. B., Sanders, J. J.,

Broadwater, T., & Brewer, J. D. (1996). Simulation-Based
Mounted Brigade Training Program: History and lessons
learned (ARI Research Report 1689). Alexandria, VA: U.S.

Army Research Institute for the Behavioral and Social
Sciences. (AD A309 752)

Koger, M. E., Quensel, S. L., Sawyer, A. R., Sanders, J. J.,
Crumley, K. A., Brewer, J. D., & Sterling, B. S. (1998).
Staff Group Trainer: Development of a computer-driven,
structured, staff training environment (ARI Research Report

1718). Alexandria, VA: U.S. Army Research Institute for the
Behavioral and Social Sciences. (AD A347 074)
Kolb, D. (1983). Experiential learning: Experience as the

source of learning and development. Englewood Cliffs, NJ:
Prentice-Hall.

Koriat, A., & Goldsmith, M. (1996). Memory metaphors and the
real-life/laboratory controversy: Correspondence versus
storehouse conceptions of memory. Behavioral and Brain
Sciences, 19(2), 167-228.

Latane, B., Williams, K., & Harkins, S. (1979). Many hands make
light the work: The causes and consequences of social
loafing. Journal of Personality and Social Psychology, 37,
822-832.

Lockheed Martin Federal Systems, and Illusions, Inc. (1996).
Battlefield Distributed Simulation-Developmental (BDS-D)
After-Action Review (AAR) System functional requirements and
developmental strategy (ADST/LMRS/TR 96-W003561). Orlando,
FL: Authors.

MacDonald, B. (1998). The Joint Warfighting Center Training and
Exercise Division "cradle to grave" deliverable exercise
support. Joint Center for Lessons Learned (JCLL) Bulletin,
Summer. Retrieved December 17, 1998, from the World Wide
Web: http://www.jwfc.js.mil/PAGES/JCLL2/trex.htm

Meliza, L. L. (1995). Need for a change of status protocol data
unit (PDU) in the distributed interactive simulation protocol
standards. 12th Workshop on Standards for the
Interoperability of Distributed Simulations Volume I:
Position Papers, 637-638.

65



Conceptual Foundations

One of the defining characteristics of the AAR process is
that it is expressly based on behavioral science principles.
Because the AAR is a multi-faceted process, these principles
have been drawn from a variety of behavioral science
disciplines. The purpose of this section is to identify and
document the behavioral science theories and principles that
provide the foundations of the AAR process. We also review
recent AAR research that is relevant to those principles.

Identifying behavioral science principles was not always a
straightforward process. In some cases, these principles were
explicitly stated in the AAR research literature. In other
cases, these principles were inferred from the guidance
provided. Also, certain prescriptions were not unique to
particular theories. For instance, the AARs emphasis on active
learning can be derived from various points of view. The reader
should keep in mind that the purpose of the analysis was not to
provide a definitive and exhaustive listing of all behavioral
science principles that have guided the development of the AAR
process. Rather, the analysis was intended to provide an
appreciation for the range and variety of the theoretical
concepts that have been incorporated into the AAR.

Table 3 presents the results of this analysis. This table
lists the theories and techniques that have led to methods or
products that have been incorporated into the AAR process. For
expository purposes, these specific theories and techniques are
grouped into six general areas of theory and research:

1. Information feedback.

2. Performance measurement.

3. Memory and cognition.

4. Group processes and dynamics.

5. Communication theory and techniques.

6. Instructional science.

Each area is divided into several specific theories or methods
that have given rise to AAR methods, practices, or products.
These areas were then used as topics for the more detailed
analysis of current research issues.

23





FOUNDATIONS OF THE AFTER ACTION REVIEW PROCESS
CONTENTS
Page
INTRODUCTION . 2. 2 1 6 eee ee ee ee ee ee eee
HISTORICAL FOUNDATIONS 5
Antecedents of the After Action Review (AAR) . 5
Development of Simulation-Based Training Systems. . 7
Development of Specialized After Action Review
(AAR) Methods .. . sone soe ee ee 14
Standardization of Collective Training. woe ew ew we 17
Current Status and Future Extensions. ...... . 21
CONCEPTUAL FOUNDATIONS . ........ 2. 2... 4. +. 23
Information Feedback. . . ..... 2... ee ee. 26
Performance Measurement ........ +... +. 33
Memory and Cognition. ............4.4.. 42
Group Dynamics. ... rn 2)
Communication Theory and Techniques woe ee ee we 48
Instructional Science ...........:... 51
Systems Concepts. .....-....-..+.+ +... . 54
RESEARCH OPPORTUNITIES .. . eee ee ee ee ee BT
The After Action Review (AAR) and Other Sources
of Feedback... . woe oe ee BT
Compliance with After Action Review (ABR)
Principles. ... woe ewe BT
Utility of .Take- Home. packages (THPs) and) Related
After Action Review (AAR) Products ...... . 58
Problem-Solving and Decision-Making ....... . 58
| Group Cohesiveness Measures . . . - . 59
Computer-Aided Analysis of After action Review.
| (AAR) Content . . 2 2. / ee ee ee ee ee ee BD
REFERENCES . 2... ee ee ee ee eee ee ee eee GT
APPENDIX A: Acronym List. ..............A-1

vii



(MILES), and to support training at the Army’s National Training
Center (NTC). The next important influence on the AAR process
was the development of computer networked simulation systems in
the mid 1980s—in particular, the Simulation Networking (SIMNET)
system. ARI guided the development of two generations of low-
cost systems to provide SIMNET the capability to capture and
analyze electronic data for the AAR.

Since the initial AAR methods were developed for TES and
NTC, ARI has developed specialized AAR methods for application
to specific settings, including staff training in constructive
simulations, multiechelon training in virtual simulations, and
experimental programs for training digitized units. There have
also been recent efforts to standardize simulation-based
training programs, which include the AAR process (so-called
“structured training” programs). ARI also has made input to an
ongoing program to standardize and automate the AAR process:
the Standard Army After-Action Review System (STAARS). Over
this short history, the AAR has become a highly regarded process
that has been adopted by other Services and countries.

The review of research on the AAR process indicates that
AAR principles were derived from the information feedback,
performance measurement, cognition and memory, group processes,
communication theory, and instructional science areas of
behavioral science. For each of these areas, we identified
specific examples of AAR methods, practices, or products.

Five implications for future research were developed and
briefly described:

1. Integrate AAR with other sources of feedback.

2. Track adherence to AAR principles.

3. Determine the utility of AAR process and products.
4. Determine the nature of problem-solving in AAR.

5. Use new technologies for content analyses.

Utilization of Findings:

The information provided in this report is intended for
individuals and organizations performing R&D on AAR processes.
This report provides background for research on the AAR process
as well as suggestions for future research.

vi



using battle outcomes as the primary performance indicator. The
developers of SIMNET proscribed against the use of instructors,
controllers, or umpires. The AARs were to be performed solely
by the chain of command as they would in actual combat: “Just
as in combat, there are no overlords in this type of exercise
other than the chain of command” (Thorpe, 1987). Nevertheless,
two ad hoc capabilities to document and observe performance were
added soon after the system was developed:

1. The Plan-View display, which provided a top-down
(“bird’s eye”) view of the battlefield.

2. The Stealth-View display, which provided a virtual
(“out-the-window”) view of the battlefield from a
vehicle that was not visible to participants (hence,
“stealth”).

When coupled with a data logger that recorded network data
stream elements (or Protocol Data Units [PDUs]) associated with
a simulated exercise, both the Plan-View and Stealth-View
displays provided the capability to replay all or parts of the
battle at variable speed and perspective. Although these
features potentially provided a wealth of information for AARs,
the displays were initially used primarily to prepare and
control ongoing exercises. In addition, Goldberg and Meliza
(1993) noted several specific deficiencies in the design of the
initial SIMNET-T training facilities with regard to the AAR
process:

e The SIMNET training facility allowed many exercises to be
conducted concurrently but supported only one AAR at a
time. Further, the system could not be used for an.AAR
if it was employed to support exercise control functions
for an ongoing exercise.

e The system did not provide data summary tables and
graphics. Despite the fact that SIMNET could provide the
most extensive collection of objective data ever
available to examine unit performance, it did not include
tools for analyzing these data in a training environment.

e Replaying an exercise’s history was a slow and cumbersome
way to review unit performance. For example, one could
watch an engagement lasting 15 minutes to find out that

called SIMNET-D, at Fort Knox, Kentucky, and Fort Rucker,
Alabama.

i1



FOUNDATIONS OF THE AFTER ACTION REVIEW PROCESS

Introduction

Like combat, collective training exercises are complex
events where the causal connections between individual
performance, weapons effects, and mission outcomes are obscured
by the uncertainty, confusion, and stress of battle (i.e., “the
fog of war”). Thus, the answer to the question, “How did the
unit do?,” may not be immediately obvious to the exercise
participants or to those who control and observe collective
training exercises. However, to derive training value from
these exercises requires detailed feedback to the unit on their
individual and collective performance and their relation to
combat outcomes.

The Army developed a process, known as the after action
review (AAR), to provide such feedback. The U.S. Army Research
Institute for the Behavioral and Social Sciences (ARI) performed
much of the research and development (R&D) for the AAR. The
purpose of this report is to review the evolution of this
process, identify some of the major behavioral science
principles that underlie the AAR, and propose directions for
future R&D.

As defined in Training Circular (TC) 25-20, A Leader's
Guide to After-Action Reviews, (U.S. Army Combined Arms Center
[CAC], 1993), an AAR“ . . . is a professional discussion of an
event, focused on performance standards, that enables soldiers
to discover for themselves what happened, why it happened, and
how to sustain strengths and improve on weaknesses” (p. 1). In
other words, the units perform a collective self-examination in
which the more general question, “How did the unit do?,” is
broken down into three more specific questions:

1. “What happened during the collective training exercise?”
In other words, AAR participants attempt to specify the
facts (i.e., the important actions and outcomes) of the
simulated battle.

2. “Why did it happen?” Given the facts of the exercise,
the participants attempt to explain the causes of
particularly important actions and outcomes.

3. “How can units improve their performance?” Given that
the previous two questions are answered, the



Tactical Trainer (CCTT), the system that will eventually replace
SIMNET (Brown, Metzler, Riede, & Wonsewitz, 1996; Nordyke,
Gerlock, Montague, Huysoon, & Bucher, 1998). CTAFS is designed
to provide AAR aids and manage the AAR presentation for digital
units in a manner analogous to ATAFS, which assists the
development of AARs for conventional units.

Standardization of Collective Training

The so-called “combat model” that dominated thinking in the
early days of SIMNET development implied that training would be
conceived and conducted by units without benefit of observers,
controllers, or instructors. This model presented fundamental
problems for training. For one, the concept of the unit’s
nearly absolute autonomy in training matters ran counter to the
widely held principle that Army training should be standardized
across the force. Perhaps more fundamentally, this laissez
faire concept of simulation training had the potential to
undermine SIMNET and other simulations’ potential for improving
performance on tactical tasks. Partly as a reaction to these
specific problems, the Army started to standardize the methods
used to conduct simulation-based tactical training.

Structured Training

In the early 1990s, ARI developed a standardized approach
to simulation-based tactical training called “structured
training.” In general, structured training has been defined as
a coherent program of instruction and practical exercises for
learning and mastering specific learning objectives (ARI, 1996).
In the context of simulation-based tactical training, Campbell,
Deter, and Quinkert (1997, p. 6) identified the following key
features of structured training:

e A focus on the performance of selected critical tasks.

* Standardized exercise control to ensure practice of the
tasks.

e Standardized feedback to correct and reinforce
performance on the selected tasks.

° Exercise support by means of comprehensive training
material.

In structured training, the AAR is more than a mechanism
for delivering feedback. It provides the foundation for the

17



FOUNDATIONS OF THE

AETER ACTION

"DISTRIBUTION N STATEMENT A
Approved for Public Release
Distribution Unlimited

pecial Report #42
‘ohn E. Morrison f-
Larry L. Meliza j



generating a virtually unlimited number of “ground truth” aids,
which are best suited to answering the “what happened” and
perhaps the “why it happened” questions. The next-generation
system should include artificial intelligence components
designed to aid the trainer to select and generate a limited
number of aids for display during the AAR. In addition to
“ground truth” aids, this system should also display aids to
address the “how to improve” question. Typically, AAR leaders
identify a relevant diagram from a “How To Fight” manual, which
must then be displayed separately from the ATAFS displays (e.g.,
by using an overhead projector). The next-generation system
should also include a library of such slides and provide the
trainer help in selecting displays and inserting them in the
proper points in the AAR.

Self-Assessment Techniques

In examining the training requirements for multiforce
exercises, Mirabella, Siebold, and Love (1998) noted that the
standard approaches for measuring AAR performance present some
distinct disadvantages. First, the typical multiforce exercise
focuses on planning and executive management functions rather
than on the tactical control and performance outcomes that
characterize lower echelons. Thus, automated outcome measures
from simulations are not relevant to these kinds of exercises.
Second, the overhead costs for O/Cs in a complex multiforce
exercise would be prohibitive. Third, many of the processes
pertaining to a planning exercise are covert and, therefore, are
not subject to third-party observation.

Mirabella, Siebold, and Love’s (1998) solution for
providing feedback in multiforce exercises is to base the AAR
process on self-assessments of performance that are moderated
through a Delphi process. Abbreviated or “mini-” AARs are
interspersed throughout these extended exercises. Before the
mini-AAR, participants use standardized instruments to rate
independently the performance of the tactical cell to which they
belong. Then, the cell leader brings the individuals together,
reviews all ratings, and identifies major differences in
ratings. The cell leader then conducts a discussion among cell
members to promote consensus. This procedure results in a set
of group ratings on which members can agree. Note that the
processes of individuals independently rating performance,
identifying differences, and building a consensus are essential
elements of the Delphi process of problem-solving.

41



Figure 2. Participation of unit members in discussions
is key to the AAR process.

performed during the exercise and, for each, ask the
participants whether they think their unit is proficient (and
should sustain training practices) or deficient (and should
improve training). The commander, in his role as the unit’s
primary trainer, uses all the feedback from the AAR to assess
his unit’s performance.

The next two sections this report detail the foundations of
the AAR. These foundations were determined by conducting two
reviews of the literature. The first was a historical review to
determine the evolution of AAR concepts. The second was a
conceptual review of recent research on the AAR process. This
second review focused on deriving the behavioral science
principles that underlie the AAR process and was organized by
general areas of theory and research in behavioral sciences.

The final section synthesizes implications from the reviews into
implications for future research opportunities.



platoon arrived late at its battle position. Based on this
issue, the unit may decide to identify different methods that
can be used to ensure that covering fire is coordinated with
movement and to employ one or more of these methods in the

future.

To facilitate the AAR process, units can employ AAR aids.
For example, Figure 1 shows a situation where a unit is
observing a replay of a portion of an exercise as it discusses
mission execution. It is important to remember that the focus
of the AAR is on unit discussions of exercise events, and AAR
aids are one of many tools used to guide and encourage
participation. Figure 2 shows a case where the attention of
unit members is focused on the comments being made by one of

their own.

Figure 1. A unit observes a replay of an exercise
segment at an AAR station from an “out-the-window” view
as members discuss what happened, why it happened, and
how to improve or sustain performance.

In addition to a detailed analysis of combat outcomes and
unit performance, the AAR session often includes the
identification of areas that deserve special attention for
training. The AAR leader may identify the critical tasks



withhold social reinforcers in response to groupmates’
task-related efforts.

2. Social cohesion. Motivation is mediated by the
cohesiveness of the group. That is, students help one
another learn because they care about one another and
want one another to succeed.

3. Developmental. Interaction among students around
appropriate tasks increases their mastery of critical
concepts.

4. Cognitive. Peer tutoring increases the cognitive
“restructuring, or elaboration, of to-be-learned
material. The facilitation helps both tutor and tutee—
but especially the former.

Slavin’s review provided evidence suggesting that all four
theoretical perspectives may influence achievement. However,
the evidence for the motivational interpretation was most
convincing, because almost any cooperative learning manipulation
could be enhanced by the addition of group goals and individual
accountability. The exception is instruction on tasks that are
high in cognitive complexity—that is, tasks without a well-
defined path to the solution or a single solution. In this
case, it appears that participating in discussions and debates
or even listening to others do so may be sufficient to enhance
learning. This definition of a cognitively complex task fits
the task that the participants face in an AAR session.

Like experiential learning, the research on cooperative
learning suggests that AAR performance may be enhanced by active
interaction among participants. The research on cooperative
learning also suggests that this interaction may take the form
of group discussions and even arguments among participants.
Further, research suggests that the AAR leader does not have to
establish a complex system of rewards to reap the benefits of
cooperative learning.

Systems Concepts

Lockheed Martin Federal Systems and Illusions, Inc. (1996),
in their review of the AAR concept, made the point that the AAR
was originally conceived as a discrete event occurring at ENDEX,
where a leader mediates and facilitates discussion. This
concept was especially true in the early days, given that. most
AARs were informal in nature. However, with the advent of
advanced instrumentation and data processing, much more
information can now be integrated into the AAR. With more

54



SIMNET EXERCISE T IMEL INE

EXERCISE ID: @@5 DATE: 92-8-3 Company C LEVEL: PLT FORCE: RED
14:07 17 27 37 47 57 45:07 17 27 37
PLT 1 GOLDN BRONZE cAL
—_ SILUER CKPSINUM
; MOVE — ; T j T
—_—
k
SHOOT
| tT ° °
|
|
comm Ro 2

Move Legend:
@ Tine between first and last vehicle crossing the control measure -

Time during which no vehicle moved.

Figure 5. Example of an exercise timeline display from UPAS.

Shlechter, Bessemer, Wade, and Nesselroade (1994) had
potential users of UPAS rate how likely they were to use each of
the four display formats described by Meliza, Bessemer,
Burnside, and Shlechter (1992): the Plan View, Battle Flow,
Battle Snap Shot, and Exercise Timeline. They also had users
rate two additional display formats, Battle Scorecard and
Graphs, which provide summary data in tabular and graphical
forms, respectively. (Figure 6 presents an example of a UPAS
Graph.) Users indicated a preference for the Battle Snap Shot
Display and suggested that line-of-sight vectors be added to
indicate clear visual paths and blind spots in crewmembers’
fields of vision. Users also suggested the development of a
Fire Fight Display, which would show the results of selected
segments of the battle by indicating the hits, kills, and misses
for each tank by displaying color-coded vectors among all
vehicles (live or dead) at the end of a designated time segment.

37



Theories and

Research Areas Techniques AAR Methods, Practices, or Products
Group Social Exploiting the positive effects of group context
Processes and facilitation and
Dynamics social loafing Minimizing irrelevant distractions during AAR
Monitoring participation to prevent “social
loafing”
Group identity Establishing positive, non-threatening atmosphere
and cohesiveness
Minimizing “finger-pointing”
AAR leader acting as moderator, not discussant
Having unit identify problems and provide solutions
Communication Descriptive Use of specific statements instead of abstractions
Theory and communication
Techniques References to task goals and objectives
Questioning Emphasis on open-ended questioning
techniques
Form and content Prescriptive model of feedback that stresses
of feedback * Performance, not personal characteristics
¢ Importance of rationale
* References to goals and objectives
¢ Strategies for improving performance
Instructional Guided discovery Active identification and solution of problems by
Science learning unit members

Experiential
learning

Cooperative
learning

Systems concepts

Unit control of AAR content, with AAR leader
guiding the process

Active learning in a realistic group context

Importance of iterative cycles of exercises and
AARS

Encouragement of group participation in discussions

Analysis of AAR processes and subprocesses

25



Mirabella, Siebold, and Love (1998) also discuss two
enhancements to this basic procedure that can be used to
validate self-assessments. One enhancement specifies that those
cells receiving outputs from the cell being assessed rate the
timeliness and quality of those outputs. The other enhancement
is to have 0/Cs complete the same rating instruments that cell
members complete as a check for consistency and convergent
validity. Both enhancements are designed to be used as
resources allow.

Preliminary applications of this self-assessment procedure
have been encouraging. However, Mirabella et al. conclude that
more research is needed to validate the system formally for use
in higher echelon AARs.

Memory and Cognition

Researchers agree that the AAR is not simply an event where
exercise participants passively receive feedback; rather, it is
an instructional process where participants learn about the
causes of their actions and develop strategies for improving
their performance. Because the AAR is a learning process,
developers have recognized the relevance of memory and
cognition. The following sections discuss topics related to the
learning process in the AAR.

Transfer of Training

In the present context, transfer of training refers to
improvements in combat performance resulting from practice on a
training exercise. One of the most fundamental principles
underlying this concept is that the amount of transfer varies
directly with the similarity of conditions between training and
actual combat. In accord with that principle, Army doctrine
encourages leaders to “train as you fight” to maximize transfer
of training from simulations to combat.

Transfer of training principles have some important
implications for the AAR. One is that the discussion should
emphasize the conditions of combat, not those of the simulation.
Accordingly, one “rule of engagement” for conducting the AAR, as
stated in the STAARS Handbook, is to “ .. . discuss tactics,
not gaming” (TRADOC, 1997, p. 13). The AAR leader must also
point out when outcomes from a simulation do not correspond to
combat outcomes or to outcomes from other simulations. The
intent of the tactical focus is to increase the probability that

42



Table 3

AAR Methods, Practices, and Products Developed from Behavioral

Science Principles

Theories and

Research Areas Techniques AAR Methods, Practices, or Products
Information Intrinsic Development of live simulation systems (SCOPES,
Feedback feedback REALTRAIN, MILES) to provide realistic BDA
Use of AAR to determine fidelity requirements
Live feedback requirements for emerging weapon
systems
Extrinsic Design that minimizes delay of feedback
feedback
Improvements to take-home package (THP)
Intervention guidelines for providing coaching and
mentoring
Performance Process vs. Incorporation of process with product measures to
Measurement. product aid diagnosis of performance problems
measurement
Automated Unit Performance Assessment System (UPAS)
performance
measurement Automated Training Analysis and Feedback System
technology (ATAFS)

Self-assessment

techniques
Memory and Transfer of
Cognition training

Memory aiding

Problem-solving/

decision-making -

Mental models

Standardization of AAR through input to STAARS

Experimental methods for providing feedback for
multiforce exercises

Focus on tactics, not gaming

Initial review of “what happened” to refresh
participants’ memories

Use of summaries to reinforce AAR points
THPs to refresh memory after AAR

Determination of causes of performance problems
(why)

Determination of training solutions (how to
improve)

Benefits of multiple points of view

Participation of OPFOR

(table continues)

24



References

Alluisi, E. A. (1991). The development of technology for
collective training: SIMNET, a case history. Human Factors,
33, 343-362.

Banks, J. H., Hardy, G. D., Scott, T. D., Kress, G., & Word, L.
E. (1977). REALTRAIN validation for rifle squads: Mission
accomplishment (ARI Research Report 1192). Alexandria, VA:
U.S. Army Research Institute for the Behavioral and Social
Sciences. (AD A034 515)

Bell, H. H., Dwyer, D. J., Love, J. F., Meliza, L. L.,
Mirabella, A., & Moses, F. L. (1997). Recommendations for
planning and conducting multi-service tactical training with
distributed interactive simulation technology (Research

Product 97-03). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences. (AD A328
480)

Bessemer, D. W., & Myers, W. E. (1998). Sustaining and
improving structured simulator-based training (Research
Report 1722). Alexandria, VA: U.S. Army Research Institute
for the Behavioral and Social Sciences. (AD A344 895)

Bosley, J. J., Onoszko, P. W. J., & Sevilla, E. R., Jr. (1979).
The role of the after action review leader in REALTRAIN:
Research on training needs. McLean, VA: Human Sciences
Research. (AD A066 717)

Bosley, J. J., Onoszko, P. W. J., Knerr, C. S., & Sulzen, R. H.

(1979). Tactical engagement simulation training techniques:
Two training programs for the conduct of after action review
(ARI Research Product 79-2). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences.
(AD A073 724)

Branson, R. K., Raynor, G. T., Cox, J. L., Furman, J. P., King,
F. J., & Hannum, W. H. (1975). Interservice procedures for
instructional systems development. Tallahassee, FL: The
Center for Educational Technology, Florida State University.
(ADA 019 486)

Brown, B. R., Metzler, T., Riede, D., & Wonsewitz, R. (1996).
Exploiting artificial intelligence to improve after action
reviews for the digitized company team (M67007-96-C-0014).
Lawton, OK: Advancia Corporation.

61



CONTENTS

(CONTINUED)

Table 1.

Figure 1.

Page

LIST OF TABLES

Contrast of Performance Critiques and AARs........... 8
AAR Guidance for Specific Training Environments.... 15

AAR Methods, Practices, and Products Developed
from Behavioral Science Principles.................. 24

LIST OF FIGURES

A unit observes a replay of an exercise segment

at an AAR station from an “out-the-window” view as
members discuss what happened, why it happened, and
how to improve or sustain performance............... 3

Participation of unit members in discussions is key
to the AAR process

Sources of intrinsic feedback needed by the gun
crew, fire direction center, laser designator
operator and target vehicle crew relative to
employment of a 120 MM precision guided mortar
munition............

Examples of trainer exercise control and feedback
functions involved in providing intrinsic feedback.30

Example of an Exercise Timeline Display
ELOM UPAS. 1... eee eee cece cece cece ete e ee eeceee 2-237

Example of a Battle Graphics Display from UPAS.....38
Example of a Fire Fight display from ATAFS...... 22-39

Example of display showing items in the ATAFS
aid bin.......

Bee cece e eee e eee eee eee eee see cee ee 40



Development of Specialized After Action Review (AAR) Methods

The unique needs of particular training environments have
demanded the development of specialized AAR methods. As
summarized in Table 2, ARI has developed specialized methods and
guidance for delivering AARs in a variety of specific training
and simulation environments. In Table 2, the descriptions of
the training environments are suggestive of the situational
variables (echelon, type of simulation, and so forth) that
necessitated the development of specialized guidance.

Bosley, Onoszko, Knerr, and Sulzen (1979) developed the
first ARI document to guide Army leaders in conducting AARs. It
was based upon observations of AARs in association with the use
of REALTRAIN. Scott and Fobes (1982) provided a guidebook for
conducting AARs at the Army’s NTC. This guidebook was based, in
part, on observations of AARs in association with the use of
REALTRAIN’s replacement, the MILES system. It was modified
twice to fit new audiences. As part of ARI’s efforts to support
the fielding of MILES within units across the Army, Scott (1983)
modified the guidebook for use by trainers at home station.
Kaplan and Fallesen (1986) tailored the guidebook to fit AARs
for the ARTBASS constructive training simulation.

In a 1986 ARI report, L. E. Word provided a new source of
information for conducting AARs, based on his tour of duty as a
senior O/C at the Army’s NTC. Among other things, Word stressed
the importance of having the OPFOR members participate in the
AAR. These OPFOR participants bring two sources of information
designed to improve the perspective of the members of the unit
being trained:

1. An additional viewpoint on exercise events, as viewed
from the perspective of the enemy.

2. Casualty exchange data (casualties inflicted and
sustained) that would not normally be known to units
because of the “fog of war.”

Certain strategies presented in Word’s (1987) report for
improving the quality of AARs are applicable across training
environments, but others have to be modified to fit different
environments. This observation mirrors the trend demonstrated
by ARI in 1982 and beyond to tailor AAR guidance for specific
situations. Since 1982, the guidance developed by ARI for
conducting AARs tends to address specific training situations,

14



own knowledge and experience and with little or no intervention
from an instructor. In his model of instruction, the teacher
does not lecture; rather, the teacher facilitates the learning
process. The AAR process is consistent with this model since
the leader facilitates in AAR discussions rather than directly
participating in these discussions. Through the process of
self-discovery, moreover, unit members achieve a better
understanding of events, causes of those events, and solutions
for performance problems.

Early in the development of the AAR, discovery learning was
recognized as an important part of the process (e.g.,- Shriver et
al., 1975) and continues to be acknowledged in more recent
publications (e.g., Cameron et al., 1997; Gubler, 1997; TRADOC,
1997). In particular, Gubler (1997) used the term “guided
discovery learning” to imply that AAR leaders, although they
avoid direct participation in the AAR discussion, provide
direction and structure to the process. For instance, the AAR
leader is typically advised to guide the discussion and to keep
it focused on the key issues. The dilemma is this: whereas
some guidance may be needed to keep the AAR on track, too much
guidance could turn the free-play exercise into a lecture.
Research should be addressed to determine the amount and kind of
guidance that is actually needed in the AAR.

Experiential Learning

Cameron et al. (1997) point out that, in addition to
discovery learning, the AAR is based on experiential learning
concepts. Experiential learning has roots in the “project
method” of instruction that dates to the early part of this
century. This approach stipulates that learning is facilitated
by real-world experiences and is therefore often associated with
the phrase “learn by doing.” This is not to deny the role of
more abstract forms of learning. In fact, starting in the early
1980s, formal theories of experiential learning have
increasingly specified the conditions under which concrete
experience should facilitate more abstract forms of learning.
For instance, Kolb (1983) has described a sequential and cyclic
model of learning that includes the four stages of (1) concrete
experience, (2) reflective observation, (3) abstract
conceptualization, and (4) active experimentation. Furthermore,
active experimentation (stage 4) restarts this cycle by causing
the apprehension of a new experience (stage 1). Presumably, the
AAR process corresponds to stages 2 and 3, whereas the
simulation exercise corresponds to stages 1 and 4.

52



could be used to improve the AAR process. If, on the other
hand, the research confirmed that successful AAR sessions are
characterized by a recognition-based process as opposed to an
analytic one, the instructional approach may need to center on
drawing from or building up the participants’ base of
experience. For instance, AAR leaders may want to construct
prompts to get participants to use their experience in their
discussions. Alternatively, if the experience base of the
participants were limited, the leader could supply appropriate
cases as appropriate. Clearly, both models have implications
for the conduct of AARs.

Group Cohesiveness Measures

The research literature implies that group cohesiveness can
have two relations with AAR effectiveness. Cohesiveness can
contribute to AAR effectiveness, and it can also be an outcome
of the AAR. Empirical research should be performed to determine
whether one or both of these implications are true.

ARI has developed a questionnaire for assessing the
cohesiveness of combat platoons (Siebold & Kelly, 1988). The
methods and items used to develop this instrument could be used
to assess the cohesiveness of higher echelons. To differentiate
causes from effects, experiments could be devised that vary the
time of administration and/or conditions that affect
cohesiveness.

Depending on the findings of the research, there should be
important implications for cohesiveness and the AAR process. If
cohesiveness is shown to be a positive factor in the AAR
process, the AAR designers and leaders should actively promote
the development of group cohesiveness before and during the AAR
session. If cohesiveness is an outcome from an effective AAR,
measures of cohesiveness should be incorporated as an important
index of AAR effectiveness.

Computer-Aided Analysis of After Action Review (AAR) Content

Downs, Johnson, and Fallesen (1987) provided much insight
into the nature of AAR feedback through detailed content
analysis of videotaped AARs. However, the number of AAR samples
was greatly limited because of the labor-intensive nature of
content analysis. Since that time, software has been created to
speed the content analysis process. This improved technology
could greatly increase the potential number and range of AARs

59



Take-Home Packages (THPs). Important sources of feedback
during the AAR are the discussions on how to sustain performance
strengths and improve on weaknesses observed during the
simulated exercise. The purpose of the THP is to document these
lessons learned so that they can be implemented at home station.

ARI investigated the extent to which THP feedback is
actually used by interviewing the leaders of units that had
recently trained at NTC (Fobes & Meliza, 1989). Fobes and
Meliza found leaders to be generally receptive to NTC feedback.
In particular, unit leaders felt that information from the NTC

should be used “ . . . to improve unit SOPs, refine their Tables
of Organization and Equipment, and address gaps in unit training
plans” (p. 13). However, these same leaders criticized the

print-based parts of the THP for being too lengthy and for being
filled with unnecessary, irrelevant, and sometimes inconsistent
details. Most important, they thought that the paper-based
parts of the THP failed to provide concrete solutions for their
training problems. On the other hand, the interviewees found
that the videotaped AARs, which were part of the THP, provided
focused and specific information that could be used to implement
positive innovations. Fobes and Meliza used information from
these interviews to provide specific suggestions on the
structure and content of THPs.

Fober, Dyer, and Salter (1994) found similar results in
their analysis of THPs from units that rotated through the Joint
Readiness Training Center (JRTC). The JRTC is similar in
concept and purpose to the NTC but focuses on light (as opposed
to heavy) combat units. They performed a content analysis of
THPs from 45 companies and 15 infantry battalions that rotated
through the JRTC in Fiscal Year (FY) 1990. The purpose of their
analysis was to determine the value of THPs as a source of
performance measures. Overall, they found that THPs provided
detailed information on unit strengths and weaknesses that is
valuable both as feedback to the unit and as performance data
for the researcher. However, they also noted some of the
weaknesses of THPs as performance measures. In particular, they
examined the training recommendations in THPs as a method to
compare unit performance. Their analysis indicated that the
training recommendations in the THPs tended be boiler plated
information (i.e., that similar training advice is provided for
all THPs). For example, many THPs included the recommendation
conduct more “force-on-force, multi-echelon training.” Although
such information is based on actual perceived weaknesses, the
generic nature of the feedback makes it difficult to implement
in an actual training plan.

32



Appendix A: Acronym List

BAR .... eee eee After Action Review

ACR... . eee eee Advanced Concepts and Requirements

ARI ..........-- U.S. Army Research Institute for the Behavioral
and Social Sciences

ARNG ..........- Army National Guard

ARTBASS ....... Army Training Battle Simulation System

ARTEP ......... Army Training and Evaluation Program

ATAFS ......... Automated Training Analysis and Feedback System

BDA .........-- Battle Damage Assessment

BDS-D......... Battlefield Distributed Simulation-Developmental

CI wwe. eee eee Command, Control, Communications, Computers, and
Intelligence

Combined Arms Center

Close Combat Tactical Trainer

. Computer-generated Forces

. Commander's Integrated Training Tool
Combined Arms Operations at Brigade Level
Realistically Achieved Through Simulation
Change of Status

. C’I Training Analysis and Feedback System
. Combat Training Center

. Defense Advanced Research Projects Agency
oe . Distributed Interactive Simulation

DOS .........-.-. Disk Operating System

ENDEX........- End of the Exercise

FY 2.2... eee eee Fiscal Year

IPISD......... Interservice Procedures for Instructional Systems
Development

JCOS .......... Joint Countermine Operational Simulation

JRTC ........-. Joint Readiness Training Center

JSIMS ......... Joint Simulation System

OWFC .......... Joint Warfighting Center

KR ev ee eee eee Knowledge of Results

MDT2 .......... Multi-Service Distributed Training Testbed

METT-T........ Mission, Enemy, Terrain, Time, and Troops

MILES ......... Multiple Integrated Laser Engagement System

ModSAF Modular Semi-automated Force

MTP... . Mission Training Plan

NSC.. . National Simulation Center

NTC... . National Training Center

o/c... . Observer/Controller

OPFOR . . Opposing Force

ORD ....... . Operational Requirements Document

PDU ........--- Protocol Data Unit



Performance Measurement

The AAR process is based on the objective measurement of
performance, by either a trained observer or an automated
system. In this section, we discuss four performance
measurement issues that have a direct impact on the AAR process:

1. Product and process performance measurement.
2. Data stream issues.
3. Automated performance measurement systems.

4. Self-assessment techniques.

The first issue is primarily theoretical, concerning the types
of performance measures that can and should be obtained for
feedback purposes. The next two issues are essentially
technological, relating to the nature of the data stream in
Distributed Interactive Simulation (DIS) -based simulation
systems and to the automation of performance measurement
functions. Finally, the last issue addresses a different
approach to performance measurement—an approach that is based
on self-assessment.

Product and Process Performance Measures

A fundamental distinction in performance measurement is
that between product and process measures. Product measures
refer to the outcomes or end results of performance, such as
number of friendly and threat vehicles lost during an
engagement. Process measures refer to the procedure or
technique used to achieve (or attempt to achieve) an outcome,
such as percent of tasks correctly performed to standard: A
similar distinction is drawn in the test and evaluation (T&E)
community between measures of effectiveness (battle outcomes)
and measures of performance (process for achieving those
outcomes) .

A primary rationale for developing TES methods in 1970 was
to provide objective indicators and measures of battle outcomes.
Some felt that these types of product measures were more
relevant than process measures and provided all the information
that soldiers needed. For example, Shriver et al. (1975)

maintained that “ .. . the measurement objective was to record
data that indicated whether the defined job had been
successfully performed . . . rather than whether the correct

33



e Increasing intergroup competition.
e Reducing intragroup competition and discord.
e Emphasizing group success.

e Maintaining a pleasant, positive atmosphere.

Many factors that increase cohesiveness are operative
during the AAR or are actively promoted by the AAR leader. For
instance, the fact that the unit has just fought against a
common enemy (intergroup competition) should increase unit
cohesiveness. Kaplan and Fallesen (1986) stressed the
importance for the AAR leader to maintain a positive atmosphere
in which participants can disagree without fear of retribution.
A Leader's Guide to After-Action Reviews (CAC, 1993) echoes this
idea, recommending that leaders should “ . . . create a positive
atmosphere conducive to maximum participation” (p. 4-2). In
addition, the leader is advised not to allow blame (“finger-
pointing”) to focus on any member or group of members. This
reduces the threat of intragroup competition, which could reduce
cohesiveness.

Another AAR strategy is also designed to increase group
cohesiveness: the advice to AAR leaders that they facilitate
but not take part in the discussion concerning group performance
problems and solutions. Staying out of this process promotes
unit integrity. It also encourages the unit to develop their
own training solutions, thereby increasing group commitment
toward any solution that they derive. As Scott (1983, 1984)
suggested, having the unit members discover and solve their own
problems promotes “ownership” of the problem and the solution,
and increases the likelihood that the solutions are actually
implemented. An additional implication is that the process of
agreeing on these common goals should also provide feedback to
promote more group cohesion.

Communication Theory and Techniques

AAR success is highly dependent on the quantity and quality
of communication among participants and communication between
participants and the AAR leader. Thus, implications for the AAR
have been drawn from the field of communication theory and
techniques.

Descriptive Communications

The term “descriptive communications” refers to a set of
techniques that are regarded as being highly effective and are

48



The Army National Simulation Center (NSC) initiated STAARS
in 1995 to develop a structured and standard AAR process.
STAARS was intended to produce AAR methods, data, and displays
that are standardized across simulation environments (live,
virtual, and constructive) and across application domains
(training exercises and military operations [TEMO]; research,
development, and acquisition [RDA]; and advanced concepts and
requirements [ACR]). In 1996, STAARS came under the aegis of
the Warfighter XXI (WF XXI) program. As such, it became one of
the five major components of the WF XXI program, providing the
needed performance feedback loop.3

Given ARI’s long history of adapting the AAR process to
specific training situations, its support of standardization
efforts may seem odd, unless one considers the benefits to be
gained by appropriate forms of AAR standardization (Meliza,
1999). When correctly applied, the concept of standardizing AAR
aids can reduce the amount of work trainers must perform in
planning, preparing and conducting effective AARs.
Standardization of AAR aids can also (a) reduce substantially
the costs of developing and maintaining automated AAR systems
across training environments and audiences and (b) make it
easier to link training conducted within a wide variety of
settings with capstone training events, like rotations to a
combat training center CTC.

ARI has been actively involved in the development of
STAARS, including providing input to the STAARS Operational
Requirements Document (ORD) and STAARS Action Plan (Meliza,
1996). Many of ARI’s recommendations concern the capabilities
that an AAR system must have to support the timely preparation
and efficient delivery of AAR aids in a manner that keeps
trainer workloads at a reasonable level and provides users with

3. The other four components of WF XXI are (1) the Standard
Army Training System (SATS), a computer-based software system
that integrates and automates the Army's training management
doctrine; (2) Training Support Packages (TSPs), an integrated
and stand-alone set of products, materials, and instructions
needed to plan, prepare and, execute simulation-based training;
(3) Training Aids, Devices, Simulations, and Simulators (TADSS),
technology-based tools for training soldiers and units to
prescribed standards in a resource-effective manner; and (4) the
General Dennis J. Reimer Army Training and Doctrine Digital
Library, a globally accessible, digital repository of training
knowledge sets and interactive applications to support training
of individuals and units.

20



2. Rationale. Feedback should provide a rationale for
participants’ performance to explain why they did what
they did.

3. Goals. Feedback information that references task goals
or objectives should be provided along with specific
information about performance.

4. Corrective actions. Feedback should provide strategies
for continuing effective behaviors and changing
ineffective ones.

Results indicated that most comments came from the
battalion commander, the ARTBASS leader, and company commanders.
Fewer comments came from battalion staff officers (intelligence,
operations, logistics, and fire support staff officers) and from
the OPFOR. The content of most messages concerned the adequacy
of information provided during the battle (e.g., failure to
request or provide the appropriate vital information) and
coordination issues (e.g., expending resources too quickly and
failure to understand battle objectives). With regard to their
prescriptive model, the analysts deemed that adequate feedback
was provided in three of the four categories. The exception was
the relative infrequency of requests for and submissions of
information on the rationales of performance. The most serious
problem noted by the authors was the AAR leaders’ failure to use
questions to guide the AAR process.

Instructional Science

Instructional science refers to the practical science and
technology related to the design, development, implementation,
and evaluation of training. Because the AAR is a pedagogical
process, the principles of instructional science should be used
to enhance the effectiveness of the AAR. A variety of
instructional science principles have been applied to the AAR
process, some of which are discussed below.

Guided Discovery Learning

The concept of “discovery learning” is generally attributed
to Jerome Bruner as a component of his learner-based theory of
instruction. According to Bruner (1961), learning is a
rearrangement and transformation of existing knowledge. This
implies that learning is a personal and active process. The
best method for achieving true insight is for learners
themselves to “discover” the solution to problems based on their

51



included in a sample and reduce concerns about reliability of
measurement.

60



FOREWORD

The U.S. Army has adopted the After Action Review (AAR) as the primary
method for delivering feedback after unit training exercises. The AAR is also being
adopted by proponents within other Services and by proponents for non-military group
training feedback applications. The U.S. Army Research Institute (ARI) has supported
the development and implementation of AAR procedures for over 20 years.

This report documents the history of the AAR. It also describes the research and
development (R&D) efforts conducted to support AAR implementation across the many
training situations that have emerged over the years as a result of new training
technologies and force modernization.

EDGAR M. JOHNSON
Director



The ATAFS knowledge base could be used to generate a sample
set of prototype displays that were automatically created based
on performance elements of the platoon task entitled “Execute
Actions on Contact.” In addition, 0/Cs could manually generate
other aids to help illustrate points in the AAR. All aids were
‘stored in a “bin” from which the user selected and sequenced the
aids for presentation. Figure 8 provides a display of items in
a typical ATAFS bin. As indicated by the VCR button in the
bottom right of this figure, ATAFS also has the capability to
load display images onto videotape. This feature frees up the
ATAFS workstation for subsequent data collection and creates a
tangible product that AAR participants can replay at home
station.

ALL AIDS
No. Start End Type Title
00:01:16 00:04:06 PVA Movement to the LD
00:04:06 00:06:43 PVA Movement from LD to First Enemy Contact
00:06:43 00:07:13 Fire Fight Actions on Initial Contact
00:06:52 00:07:22 PVA Call for Fire
00:07:15 00:07:30 PVA Report to Co Tm Cdr
00:04:27 00:04:27 PVA FRAGO from Co Tm Cdr
00:05:20 00:07:37 Fire Fight _ Attack by Fire

Add

Actions on Initial Contact
Attack by Fire

Figure 8. Example of display showing items in the ATAFS
AAR aid bin.

Meliza and Brown (1996) discussed improvements to the ATAFS

system that potentially increase its effectiveness in providing
relevant feedback to units. The current ATAFS is capable of

40



Firing Accuracy over Range by Platoon
<In 500 meter Intervals>

HM Platoon 2

7a-

6e5

405

30-

Percent Hits
1

20-7

18-7

r T r T T 1
1.0K 2.0K 3.0K

Range (in kilometers >

Figure 6. Example of a battle graphics display from UPAS.

Meliza, Bessemer, and Tan (1994) also documented a variety
of improvements to the UPAS. In addition to display
improvements described in an earlier report (Meliza, Bessemer,
Burnside, & Shlechter, 1992), Meliza, Bessemer, and Tan
discussed the following enhancements to UPAS:

¢ Improving methods to select, filter, and process PDUs
from the network. The purpose was to prevent UPAS from
becoming overloaded with data.

¢ Increasing the rate at which network data are loaded
into a relational database. The goal was to have data
analyzed 10 minutes after ENDEX.

© Adding a time indexing file for PDUs for use with all
map displays. This facility permits users to move
quickly from one point in the exercise to another.

38



Crissey, M. (in preparation). ARI support of the after action
review process (AAR). Orlando, FL: U.S. Army Simulation,
Training, and Instrumentation Command.

Davenport, T. H., & Prusak, L. (1997). Working knowledge: How
organizations manage what they know. Cambridge, MA: Harvard
Business School Press.

Deatz, R. C., Forrest, D., Holden, W. T., Jr., Sawyer, A.,
Britt, D. B., & Gray, R. (1998). Follow-on development of
structured training for the Close Combat Tactical Trainer
(ARI Research Report 1725). Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social Sciences.
(AD A350 545)

Downs, C. W., Johnson, K. M., & Fallesen, J. J. (1987).
Analysis of feedback in after action reviews (ARI Technical

Report 745). Alexandria, VA: U.S. Army Research Institute
for the Behavioral and Social Sciences. (AD A188 366)
Everett, S. E. (1992). Oral history: Techniques and

procedures. Washington, DC: Center of Military History,
U.S. Army. Retrieved September 10, 1999, from the World Wide
Web: http://www.army.mil/cmh-pg/books/oral .htm

Fardouly, N. (1999). Learning styles and experiential learning.
Sydney, Australia: The University of New South Wales.
Retrieved April 1, 1999, from the World Wide Web:
http: //unsw.edu.au/learning/instructionaldesign/styles.htm

Fober, G. W., Dyer, J. L., & Salter, M. S. (1994). Measurement
of performance at the Joint Readiness Training Center: Tools
of assessment. In R. F. Holz, J. H. Hiller, & H. H. McFann

(Eds.), Determinants of effective unit performance: Research
on measuring and managing unit training readiness (pp. 39-
67). Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences.

Fobes, J. L., & Meliza, L. L. (1989). Integrating National
Training Center feedback into home station training
management (ARI Research Report 1523). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and Social
Sciences. (AD A213 436)

63



To realize the benefits of new simulation techniques, the
Army determined that a new approach for providing performance
feedback was needed. ARI researchers led the effort to develop
and implement this new method, which became the AAR. The AAR
was designed to incorporate the objectivity and non-punitive
atmosphere that was characteristic of S. L. A. Marshall’s
interviews after combat (Bosley, Osnoszko, Knerr, & Sulzen,
1979). At the same time, the AAR design avoided some of the
more egregious deficiencies of the performance critique.

Table 1 compares some of the features of this new approach
with corresponding features of the traditional performance
critique. As can be seen, the new AAR approach for providing
feedback differed from the old performance critique in both form
and content. The final entry in Table 1 identifies a critical
difference: whereas the performance critique is based on
subjective judgment, the AAR is based on objective performance
indicators. The technologies that enabled more objective
methods for obtaining performance data are discussed in the next
section.

Development of Simulation-Based Training Systems

As implied in the definition of the AAR, an initial and
fundamental requirement is to determine what happened during a
simulated battle. Thus, a significant impetus to the evolution
of systematic AAR procedures was the development of objective
methods for credible casualty and battle damage assessments
(BDAs). The following sections describe modern simulation
methods that were developed in the last 25-30 years to provide
this capability and that contributed to the development of
systematic AAR processes.

Tactical Engagement Simulation (TES)

In 1971, General W. C. Westmoreland, then Chief of Staff of
the Army, directed a board of officers to consider new
collective training approaches that were motivating and
effective (Gorman, 1992). The board initially investigated a
system of laser-based emitters and detectors for simulating
infantry and armor close weapons effects. This system was later
named the Multiple Integrated Laser Engagement System (MILES) .
Because the MILES technology would require at least 12 years to
develop and field, the board also tasked ARI to develop rapidly
an alternative, optically based TES system to help bridge the
gap (Gorman, 1992).



question has no specific answer and allows the person
answering to reply based on what was significant to
him. Open-ended questions are also much less likely
to put him on the defensive . . . (p. 4-3).

Because open-ended questions purportedly stimulate
discussion and thought about battle incidents, they seem
particularly well suited to addressing the causes of particular
performance problems (i.e., the “why” question). However,
recent research suggests that open-ended questions may also be
helpful for the initial question (the “what happened” question) .
This research, which is based on naturalistic or “real-life”
conception of memory, has shown that open-ended questions can
improve the recall of facts and incidents (Koriat & Goldsmith,
1996). The facilitation is thought to be caused by the
learners’ increased control of their own memory processes.

To the extent that open-ended questions are abstract, their
use appears to contradict guidance from descriptive
communication that specific comments are better than abstract
ones. In fact, results from ARI research (Downs, Johnson, and
Fallesen, 1987) suggest that, during the AAR, specific questions
tended to elicit specific comments, whereas abstract questions
elicited abstract comments. Downs, Johnson, and Fallesen (1987)
suggest that perhaps the best advice is to mix abstract and
specific comments. The issue here appears to be one of timing:
open-ended questioning may be good to initiate a discussion, but
specific questions and comments may be needed later to focus on
particular incidents and causes.

Form and Content of Feedback

Downs, Johnson, and Fallesen (1987) used message content
analysis methods to determine the form and content of feedback
provided in AARs. They examined videotaped AARs of six
battalion-level exercises on the ARTBASS. Transcripts of AAR
sessions identified individual utterances, which were defined as
uninterrupted verbal strings. Trained analysts then classified
each utterance with respect to certain characteristics,
including those derived from a prescriptive model of performance
feedback. According to their prescriptive model, feedback
should provide four types of information:

1. Performance vs. personal characteristics. Feedback
should be directed toward correctable behaviors rather
than toward personal characteristics.

50



developed/planned, whether or not problem solution is reinforced
through retraining, and time between training and the AAR and
between the AAR and retraining. The instrument should also
include subjective but structured items on the perceived
effectiveness and efficiency of the AAR process. This proposed
instrument would provide feedback to AAR leaders, who would use
the information to improve their performance, and it would
provide data for research on the AAR process.

Embedded System

Another hallmark of systems models is embeddedness: small
systems are often embedded in larger, more encompassing ones.
Accordingly, the method for developing structured training
(C. H. Campbell et al., 1997) depicts the AAR process as
subsystems within a larger system describing development and
delivery of structured simulation-based training. In this
model, the AAR process is divided into separate development and
implementation subprocesses, which are embedded into two of the
four components of the overall model as described below:

1. Needs analysis. Determine the critical tasks and the
resources needed to train them.

2. Development of training support package. Develop all
materials, including scenario and tactical materials,
guides for all support personnel, and training
materials, which contain observation guides and AAR
materials.

3. Training processes. Identify and provide cues from the
simulation, scripted messages, and role players.

4. Learning processes. Monitor performance and feedback on
critical tasks, and execute the AAR.

This embedded systems model makes the point that AARs do not
exist in isolation. AARs have implications for the development and
evaluation of simulation exercises, and the proposed systems model

of AAR processes must include inputs from and outputs to larger,
more inclusive instructional systems.

56



objective representation of “ground truth” from tactical
engagement outcomes. As such, SCOPES and REALTRAIN provided an
appropriate context for the first reported implementations of
the AAR process (Bosley, Onoszko, Knerr, & Sulzen, 1979; Bosley,
Onoszko, & Sevilla, 1979). The training programs that
incorporated these TES technologies and the AAR methods were
evaluated in a series of ARI research projects (e.g., Banks,
Hardy, Scott, Kress, & Word, 1977; Root, Epstein, et al., 1976;
Root, Hayes, Word, Shriver, & Griffin, 1979; Scott, Banks,
Hardy, & Sulzen, 1979). These studies confirmed that the SCOPES
and REALTRAIN training programs were effective and
motivationally engaging methods for training tactical skills;
however, the AAR components of the programs were not evaluated
separately. The demonstrable successes of these training
packages led to their rapid adoption by the U.S. Army’s Training
and Doctrine Command (TRADOC). ARI helped to write the TCs
published by the Infantry School for SCOPES in 1973 and by the
Armor School for REALTRAIN in 1975.

MILES was introduced in 1980, but it was not put into
general use until mid-decade (Gorman, 1992). Once fielded,
however, its impact was immediate and significant. Zeidner and
Drucker (1988) characterized MILES as “ . . . one of the most
revolutionary changes in combat training ever introduced”

(p. 186). Compared with SCOPES and REALTRAIN, the laser-based
MILES system significantly increased the accuracy of determining
hits and misses. Moreover, MILES greatly reduced the number of
controllers required to support a TES exercise and provided more
immediate feedback to participants. Although MILES equipment
was fielded to units on an Army-wide basis, its initial
implementation was at the National Training Center (NTC), where
force-on-force maneuvers were first conducted in 1982.

As the NTC rapidly became the Army’s premier combat
training setting, ARI adapted the AAR process first developed
for SCOPES and REALTRAIN to the innovations at NTC, which
offered several new resources and technologies for collecting |
and documenting objective performance data. Most notably, MILES
extended ARI’s earlier TES methods for providing objective
performance data. In addition to MILES, other components of the
NTC had a significant impact on the AAR process, including its
sophisticated instrumentation system for exercise control and
data collection, a highly trained OPFOR, and expert
observer/controllers (0/Cs). The NTC’s reliance on the AAR as
their primary feedback method increased the prominence of the
AAR as an effective training method. Accordingly, ARI has
continued to adapt the NTC methods to other training contexts,





Extrinsic Feedback

In collective training, extrinsic feedback refers to
information provided in the AAR, coaching and mentoring during
the exercise, and information in the take-home package (THP)
given to units after a series of exercises. Each of these
sources of extrinsic feedback is discussed below.

Information Provided in the After Action Review (AAR). The
information provided in the AAR is a form of extrinsic feedback
that is sometimes referred to as "knowledge of results" or KR.
KR has been repeatedly demonstrated to be one of the most
powerful variables in the acquisition and retention of simple
and complex tasks. In general, any manipulation that increases
the quantity or quality of KR has a positive effect on learning
and performance. According to this principle, then, AARs should
be scheduled as often as possible, usually immediately after
each exercise or simulated mission. Because much of the
discussion during the AAR depends on the participants' memories
for events and outcomes that occurred during the exercise, the
AAR is usually scheduled as soon as possible after ENDEX to
mitigate the effects of forgetting. To derive maximum benefit
from the AAR, subsequent exercises must be scheduled soon after
the end of the AAR so participants can implement lessons learned
from their feedback. Accordingly, the tactical training tables
and exercises developed for the VTP are designed to minimize two
intervals: the interval between the original exercise and the
AAR session and the one between the AAR session and the next
exercise (Hoffman et al., 1995).

Coaching and Mentoring. For the most part, O/Cs interfere
little in simulation exercises and allow participants to
experience the consequences of their own decisions and
responses. On occasion, however, O/Cs may need to intervene for
reasons such as training an inexperienced crew, advancing the
action, or preventing counterproductive actions. Research
suggests that providing feedback during these interventions can
be effective under certain circumstances; however, it must be
used judiciously to prevent performers from developing a
dependence on extrinsic feedback and to encourage the processing
of more appropriate intrinsic feedback sources. The VTP,
recognizing the need for occasional coaching while also knowing
its potential problems, drafted and tested “intervention
guidelines” for instructing O/Cs on when and how to intervene
during a SIMUTA exercise (Hoffman et al., 1995).

31



PRIME ......... Precision Range and Integrated Maneuver Exercise

RED ........00- Research and Development

RCVTP......... Reserve Component Virtual Training Program

RDA ........... Research, Development, and Acquisition

REALTRAIN ..... Realistic Training

RSTA .......... Reconnaissance, Surveillance, and Target
Acquisition

SAFOR ......... Semi-Automated Force

SAT ........... Systems Approach To Training

SATS .......... Standard Army Training System

SCOPES ........ Squad Combat Operations Exercises (Simulation)

SGT .........0. Staff Group Trainer

SIMBART ....... Simulation-Based Mounted Brigade Training

SIMNET........ Simulation Networking :

SIMUTA ........ Simulation-Based Multiechelon Training for

Armor Units

Simulation Unit Training Assembly-Digital
Standard Operating Procedure

Standard Army After Action Review System
Simulation, Training, and Instrumentation Command

T&E ...... ee Test and Evaluation

TADSS ......... Training Aids, Devices, Simulations, and
Simulators

TAFF-Aids ..... Training Analysis and Feedback Aids

OR Training Circular

TEMO .......... Training Exercises and Military Operations

TES .........0. Tactical Engagement Simulation

THP .........2. Take-Home Package

TRADOC ........ U.S. Army Training and Doctrine Command

TSP ..... ee. Training Support Package

UPAS .......... Unit Performance Assessment System

VTP... eee eee Virtual Training Program

WF XXI........ Warfighter XxI



development of structured training scenarios. The design of
structured training begins with the specification of training
objectives (tasks and standards) to be covered in the AAR. The
simulation exercises are then carefully designed to ensure that
they provide the opportunity for units to perform those tasks

and for the O0/Cs to observe task performance (ARI, 1996). Long
exercises are carefully divided into short segments to allow for
frequent AARs (Béssemer & Myers, 1998). The fact that

structured training exercises are more controlled allows the
AARs to be more “focused” (i.e., limited to the specific
training objectives that define the exercise). With this
structure, the AAR leader can better anticipate whether and when
key exercise events will occur and more quickly prepare
appropriate AAR materials (Meliza, 1996). The standardization
of scenarios and supporting materials also makes possible the
development of AAR materials that can be stored on-line and
reused as appropriate (Meliza & Paz, 1996).

In 1993, ARI initiated the development of materials and
methods for the first such structured training program. The
original training materials were named Simulation-Based
Multiechelon Training for Armor Units (SIMUTA). SIMUTA
comprised an integrated set of platoon and company exercises
(called "tables") for implementation on SIMNET and battalion
staff exercises for implementation on Janus, which is a
constructive simulation for battle staff training. SIMUTA
materials, developed specifically for Army National Guard (ARNG)
battalions, provided a "turn-key" package of items required to
execute training, including orders, overlays, and videotape
demonstrations of successful performance (Hoffman, Graves,
Koger, Flynn, & Sever, 1995).

SIMUTA methods and materials were implemented in a program
managed by the U.S. Army Armor School. This program was called
the Reserve Component Virtual Training Program (RCVTP). A key
component of this program was a dedicated team of O/Cs whose
role was to guide performance during the exercise and provide
feedback during the AAR. According to SIMUTA guidance, AARs
should occur frequently (about one every 2 hours or at the end
of every table) and should be conducted about 10-15 minutes
after exercise completion (ARI, 1996). AARs should include
replays from the stealth (i.e., virtual) display of SIMNET and
recordings of selected radio messages. Shlechter, Bessemer,
Nesselroade, and Anthony (1995) investigated the effectiveness
of the RCVTP and concluded that it was an effective program for
training ARNG units.

18



July 1999

Foundations of the
After Action Review Process

John E. Morrison
Institute for Defense Analyses

Larry L. Meliza
U.S. Army Research Institute

Special Report 42

United States Army Research Institute
for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.

i
DTIC QUALITY INSPECTED 4



competencies acquired in the AAR will, in fact, transfer to
combat situations.

Memory Aiding

The AAR learning process depends heavily on memory.
Although technologies such as TES, virtual simulation, and
automated performance measurement may help substantiate what
happened, participants’ memories of an exercise remain an
essential source of information for understanding what events
occurred and why they happened. Further, participants depend on
their memories to remember the suggested changes to unit
training identified during the AAR so that they can implement
these changes in subsequent performance or at home station.

Several aspects of the AAR are specifically designed to aid
the participants’ memories for these questions. The AAR session
starts with a review of “what happened” to refresh the
participants’ memories before discussing the events. The
discussions provide additional details to broaden the
understanding of events. The AAR concludes with a summary to
reinforce the AAR points. Finally, the THPs provide cues for
remembering the performance events and training solutions
discussed during the AAR. As such, the THP is designed to.
provide a bridge between the AAR and home station training.

The potential effectiveness of these AAR memory aids is
beyond question. The question is whether they are, in fact,
used as designed. For instance, Fobes and Meliza (1989) found
that the print-based portions of the THP were not being used to
develop unit training plans because:

e The THPs were too large and complex (sometimes over 1,000
pages in length).

* Some of the detailed information was unnecessary and
irrelevant, and important details (e.g., nature of
deficiencies and personnel/unit involved) were often
omitted.

e There were inconsistencies within THPs and between THPs
and AARs.

e Training solutions are typically not provided or are
buried in the details.

43



action. If there are problems with the quality of intrinsic
feedback provided to units, artificial aspects of the training
situation may guide unit decisions or the training environment
may not allow the unit to employ an appropriate course of
action. The following sections provide a discussion of ARI’s
involvement in issues related to intrinsic feedback, first with
regard to the fidelity of virtual simulations and then with
regard to requirements of live simulations.

Simulator Fidelity. With regard to intrinsic feedback, one
issue that continues to interest researchers concerns simulator
fidelity (i.e., the correspondence of stimuli and responses
between simulators and the actual equipment that they emulate).
Many simulator designers have implicitly endorsed what may be
termed a “full fidelity” model, wherein the simulator is
designed to emulate as much of the physical system as possible.
In contrast, SIMNET developers introduced the concept of
"selective fidelity,” which stipulates that the simulation
should replicate only those aspects of the physical system that
contribute directly to training. Those aspects of the physical
system that do not contribute directly are represented in low
fidelity or are not represented at all (Thorpe, 1987).

ARI developed a method for rationally determining the
fidelity of SIMNET after it had been fielded (Burnside, 1990).
Burnside had subject matter experts rate—according to explicit
criteria—the degree to which tasks could be performed within
the SIMNET environment, using a 3-point scale: fully trained,
partially trained, and not trained. Assuming that tasks must be
at least partially performable to be trainable in SIMNET, his
results indicated that 35 percent of tasks from the Army
Training and Evaluation Program (ARTEP) Mission Training Plan
(MTP) could be trained with SIMNET. The platoon echelon had the
highest percentage of trainable tasks (41 percent) and of tasks
not supported by the simulation (46 percent). On the basis of
these findings, Burnside (1990) concluded that this method was
an efficient analytic approach to determining the fidelity of
any training aid, device, simulation, or simulator.
Subsequently, these methods were used to identify appropriate
training objectives for, and feedback from, the structured
training exercises in SIMUTA (Hoffman et al., 1995).

Behavior of Computer-Generated Forces (CGF). Another major
fidelity issue with.which ARI has been involved is the realism
of the behavior of friendly and enemy computer generated forces
(CGF). For example, Meliza and Vaden (1995) compared two
versions of software that controlled the behavior of the CGF in

27



Unit Performance Assessment System (UPAS). Meliza,

Bessemer, Burnside, and Shlechter (1992) examined the ARTEP MTP,
which describes the requirements for tactical training. From
these requirements, these researchers recommended that the UPAS
Plan View Display be improved by including the major terrain
features and tactical control measures. Some additional
recommendations included the ability to search rapidly through
the exercise file (as opposed to a slower visual search) and the
ability to magnify the battlefield depictions to sections as
small as 200 x 200 m. In addition, they suggested that UPAS
include the following display formats:

Battle Flow. This is an animated figure that traces the
movements of vehicles and units throughout the mission
or mission segment. Symbols are used to mark positions
at specified time periods (e.g., every minute). Unlike
the Plan View Display, the symbols do not indicate
vehicle and weapons system orientation. This display
was primarily designed to provide feedback on movement
formations and techniques.

Battle Snap Shot. This provides still shots of vehicle
positions in relation to terrain and control measures.
This display includes icons that indicate vehicle and
weapon system orientation.

Exercise Timeline. This displays events on three
timelines corresponding to movement, direct and indirect
firing, and communication. The purpose is to provide
feedback on the coordination of these three functions.
Figure 5 presents a typical Exercise Timeline Display
that can be used at an AAR.

36



the flexibility to refine products in response to lessons
learned regarding their utility. These recommendations were
largely based on lessons learned from the UPAS and ATAFS
projects.

Current Status and Future Extensions

Over the last decade and a half, the AAR process has become
institutionalized in Army training and provides the focus for
virtually all collective training (Lockheed Martin Federal
Systems, and Illusions, Inc., 1996). The high regard for the
AAR process is attested by the following recent sample of
testimonials:

After Action Reviews (AARs) have proven to be the
single most important event in collective training.
This trend will continue as we progress into the 21°
Century (U.S. Army Training and Doctrine Command,
1997, p. 1).

The Army’s institutionalization of the AAR as an
essential part of training is one of the most
important training interventions ever. . . . At all
levels, the AAR provides us an honest appraisal of our
performance and directs our efforts to correct
shortcomings (Sullivan, 1995, pp. 70, 163).

Excellent analytical AARs during well organised and
supported force-on-force scenarios against a highly
capable OPFOR is the secret of the U.S. Army recovery
since Vietnam (Hoare, 1996, p. 17).

Another testimony to the usefulness and generality of the
AAR concept is that institutions outside of the U.S. Army have

adopted it.

Experts from the Training and Exercise Division of the

Joint Warfighting Center (JWFC) conduct AARs for U.S. Joint Service
Elements )MacDonald, 1998). AAR capabilities are also incorporated
into two Joint Service training simulations: the Joint Countermine
Operational Simulation (JCOS) and the Joint Simulation System
(JSIMS). Foreign militaries, including the militaries of the
former Soviet Union, have also used the AAR (Pitts, 1997).

Finally, the

AAR has been cited as a military innovation that

commercial organizations should adopt to provide feedback and
effect positive change (Davenport & Prusak, 1997; Sullivan &

Harper, 1996.

21



Historical Foundations

The concept of the AAR has evolved over the last 25 years—
a period that has seen dramatic improvements in collective
training technology. At the beginning of this period, Army
units conducted relatively free-play, force-on-force exercises
that were evaluated in field settings by human umpires who were
ill-equipped and poorly positioned to observe performance.
Performance feedback, therefore, was based on the umpires’
subjective estimates of mission outcomes (e.g., “hits” and
‘kills” of soldiers and systems) and judgments of overall
mission success.

At present, Army units train in live, virtual, or
constructive simulation environments! that allow automated
measurement of a wide array of performance indicators, including
tactical movement, engagement outcomes, and communications.
Moreover, unit performance is often assessed in the context of a
highly structured exercise based on a standardized training
support package that specifies training objectives, training
events, and performance measures. This marked progress in
collective training and evaluation methods parallels the
development of methods and technologies related to the AAR
process. This section discusses some of the major historical
developments that have shaped methods and concepts related to
the AAR process.

Antecedents of the After Action Review (AAR)

An important historical antecedent of the AAR is the combat
interview developed by military historian Samuel Lyman Atwood
(S. L. A.) Marshall (1900-1977) (Gubler, 1997; Bosley, Onoszko,
Knerr, & Sulzen, 1979; Word, 1987). S. L. A. Marshall was a
member of a team of Army historians tasked to document the
events of World War II as they unfolded (Everett, 1992).
Initially assigned to the Pacific theater, Marshall was
frustrated in his attempts to reconstruct a battle’s sequence of
events from normal historical data sources, such as official
records and documents. Drawing on his experience as a

1 These three types of simulation vary primarily with respect
to realism. Constructive simulations rely on simulated
operators, equipment, and situations; virtual simulations rely
on real operators in simulated equipment and simulated
situations; and live simulation rely on real operators and real
equipment in simulated situations (Sikora & Coose, 1995).



AAR practices enhance memory. For example, do participants
engage in active discussions that elaborate on memory content?
When such discussions occur, do all participants remember or
understand the events? For AAR summaries, one could similarly
ask the following questions: Do AAR leaders provide a summary?
Is the content of AAR summaries sufficient to aid memory and
understanding? Do participants remember these summaries?

The model of effective AARs may provide a potentially
useful by-product of this research because it could be used to
derive a checklist of activities that should occur during an
AAR. This checklist could serve as an instrument for evaluating
AARS. Ironically, AAR leaders, who provide feedback to units,
do not have a standardized instrument for receiving feedback on
their own performance.

Utility of Take-Home Packages (THPs) and
Related After Action Review (AAR) Products

This issue was last addressed when Fobes and Meliza (1989)
conducted interviews in March and April of 1987 on the utility
of THPs from NTC. As documented in this present study, much has
been learned and developed in the 10 years since their research.
It would be useful to compare and contrast THPs from NTC with
the automated THPs that are used at the VIP. It would also be
useful to revisit this issue in a wider context than NTC,
including various instances of live, virtual, and constructive
simulations.

Problem-Solving and Decision-Making

The consensus among researchers and military writers is
that AAR sessions involve problem-solving and decision-making
processes. However, writers have not specified the nature of
these internal processes as they apply to the AAR. To enhance
these key processes, a more detailed concept of these processes
has to be developed.

One research approach might be to compare and contrast the
processes that are characteristic of successful and unsuccessful
AAR sessions. Findings may help determine whether decision-
making follows the formal model (e.g., Keeney, 1982) or a
recognition-based process (e.g., Zsambok & Klein, 1997).
Findings from such exploratory research might lead to subsequent
research on appropriate prompting and/or aids for AAR sessions.
If the formal model of decision-making were demonstrated to be
applicable to the AAR, the literature on applied decision-making

58



REPORT DOCUMENTATION PAGE

1, REPORT DATE (dd-mm-yy)

July 1999

2. REPORT TYPE
Final

3. DATES COVERED (from. . . to)
October 1998 — May 1999

4. TITLE AND SUBTITLE
FOUNDATIONS OF THE AFTER ACTION REVIEW

PROCESS

Sa CONTRACT OR GRANT NUMBER
DASW01-98-C-0067

5b. PROGRAM ELEMENT NUMBER
20363007A793

6. AUTHOR(S) John E. Morrison (Institute for Defense
Analyses) & Larry L. Meliza (U.S. Army Research

Institute)

5c. PROJECTNUMBER A793

5d. TASK NUMBER
2114

Se. WORK UNITNUMBER =.CO2

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Institute for Defense Analyses

1801 N. Beauregard Street

Alexandria, VA 22311-1772

8. PERFORMING ORGANIZATION REPORT NUMBER

D-2332

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U.S. Army Research Institute for the Behavioral and

Social Sciences. Attn: TAPC-AR-IF

5001 Eisenhower Avenue
Alexandria, VA 22333-5600

10. MONITOR ACRONYM
ARI

11, MONITOR REPORT NUMBER
Special Report 42

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES:

14, ABSTRACT (Maximum 200 words):

The U.S. Army has adopted the After Action Review (AAR) as its primary method of providing
feedback after unit collective training exercises. The AAR is an interactive discussion in which unit
members decide what happened, why it happened, and how to improve or sustain collective
performance in future exercises. Other services and organizations outside the military are also
beginning to employ the AAR as a feedback tool. This report describes the twenty-five year history
of AAR research and development and the major behavioral research areas contributing to AAR
development and refinement. In addition, this report defines goals for future AAR research.

15. SUBJECT TERMS After Action Review (AAR)

intrinsic Feedback Extrinsic Feedback

Force-on-Force Training

Collective Training — Training Feedback
National Training Center (NTC) Combat Training Centers

Take Home Packages Tactical Engagement Simulation

19. LIMITATION OF

16. REPORT
Unclassified

17. ABSTRACT
Unclassified

ABSTRACT
18. THIS PAGE

Unclassified | Unlimited

20. NUMBER 21. RESPONSIBLE PERSON
OF PAGES (Name and Telephone Number)

Larry L. Meliza
80 (407) 384-3992



It would be interesting to find out if present-day THPs suffer
from some of the same problems.

Problem-Solving

The authors of A Leader's Guide to After-Action Reviews
(CAC, 1993) explicitly recognized that AAR is a “problem-solving
process” (p. 4-3). As has been described previously, exercise
participants are faced with a three-part problem, which
corresponds to the three questions that comprise the focus for
the AAR. The problem posed by the first AAR question (What
happened?) is relatively simple in that it can be answered by
studying the recorded portions of the battle and recalling
events from memory. Answers to the remaining two questions (Why
did it happen? and How can performance be sustained or
improved?) are clearly more challenging since they require
participants to make inferences about the causes and outcomes of
their actions.

Many theories describe problem-solving and decision-making
processes, and prescribe procedures and training techniques for
enhancing these interrelated processes. Most of these theories
fall on a continuum—with formal, analytic models on one end and
naturalistic, perception-based models on the other end.
According to the formal model (e.g., Keeney, 1982), decision-
making is a linear process. Participants solve a problem by
breaking it down into a linear sequence of steps. An example
series of steps might be described as follows:

e Determine goals and objectives.
e Develop alternative courses of action.

¢ Compare and evaluate courses of action on criteria
consistent with goals and objectives.

* Choose course that best matches criteria.

e Implement the decision.

The naturalistic model of decision-making (e.g., Zsambok
& Klein, 1997) assumes that most real-world decisions are not—
and cannot be—made in such a slow and deliberate fashion.
Instead, this model maintains that decisions are based on a
recognition process that matches the present problem with one or
more similar problems that the problem solvers remember having
faced in the past. Thus, the former model emphasizes formal

44



increased motivation or arousal (e.g., Zajonc, 1965). However,
the presence of others may depress performance when a new and/or
difficult task is being performed or when this presence causes
distractions. These exceptions do not seem to apply to the AAR
for two reasons:

1. Factors such as evaluation apprehension, fatigue, and
exercise complexity may increase difficulty of the AAR;
however, participating in AARs is normally a routine
activity and is not particularly difficult.

2. AAR leaders are trained to keep the discussion focused
on the key points of the AAR. While leaders encourage
participants to speak freely, they request that
participants speak one at a time to minimize
distractions.

Another limitation to social facilitation is that the
function relating number of participants to motivation and/or
performance is often depicted as being non-linear in form. That
is, the effect is positive up to small or moderately sized
groups but decreases and perhaps turns negative as group size
increases beyond some given number. This has been related to an
oft-cited generalization, sometimes referred to as the Yerkes-
Dodson law, that describes the effects of arousal on performance
as an inverted U-shaped function: the effects are increasing up
to some optimal point and decreasing beyond that. In addition
to this theoretical constraint, there is a practical constraint
to social facilitation: it is simply more difficult to control
the interpersonal distractions in larger crowds than in smaller
crowds. For both reasons, some optimal size probably exists for
an AAR. However, the optimum is likely to vary with other
variables, such as the format of AAR (formal or informal), the
echelon (platoon, company, battalion, or higher), and the type
of simulation (live, virtual, or constructive).

Social Loafing. In contrast to social facilitation, the
related concept of social loafing has a negative effect on ~
performance. Social loafing is the effect of decreased
individual performance observed in larger groups, where
individuals let others in the group do their work (Latane,
Williams, & Harkins, 1979). The phenomenon of social loafing
appears to occur when the group produces a single product and
individuals are not held accountable for their own performance.
Social loafing is minimized in AAR sessions because leaders
encourage everyone to participate actively in the discussion.
In this regard, one technique that AAR leaders use is to address

46



Table 1
Contrast of Performance Critiques and AARs

Characteristics Performance Feedback Method
of Feedback
Sessions Traditional Performance

Critique AAR
Soldier Soldiers are passive Soldiers are active
participation members of an audience. participants ina

discussion.

Main topic of Errors committed. Sequence of events.
discussion
Direction of One-way (from leader to Two-way.
communication participants) .
Atmosphere Defensive. Open to suggestion.
Instructional Traditional lecture. Guided discovery learning.
style
Source of Exercise leader and Participants and members
information: controllers. of the opposing force
why it happened? (OPFOR) and exercise

leaders and controllers.

Source of Subjective judgment. Objective performance
Information: indicators.
what happened?

ARI’s work on an optically based TES system began in 1972,
and the first product was fielded in 1973. This initial effort
produced a method for training small,’ dismounted infantry units
called Squad Combat Operations Exercises (Simulation) or SCOPES.
The basic hardware for SCOPES was a 6X telescope mounted on
every participant’s rifle and helmets with numbers that uniquely
identified each soldier in the exercise. To engage the
opposition, a participant would identify an opponent’s number,
fire a blank round, and report the number to a controller. The
controller would report to a counterpart controller of the
opposition forces, who would then assess the participant whose
number was identified as a casualty. The SCOPES approach was
extended to combined arms, armor and antiarmor units in a
product called REALTRAIN (Realistic Training) .

SCOPES and REALTRAIN were significant because they were the
first TES technologies to provide trainers and participants an