Technical Report 1086

Self-Assessment Based Mini-After Action Review

(SAMAAR) Methodology: Developmental

Application to Division Artillery Staff Training

Angelo Mirabella

U.S. Army Research Institute 00C=0

James F. Love

Consultant

July 1998

United States Army Research Institute

for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON

Director

Technical Review by

J. Douglas Dressel

Richard E. Christ

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001

Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research for the Behavioral and Social

Sciences.

NOTE: The findings in this Technical Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED Oct 1997 to April 1998 July 1998 Final

4. TITLE AND SUBTITLE: Self-Assessment Based Mini-After Action 5a. CONTRACT OR GRANT NUMBER

Review (SAMAAR) Methodology: Developmental

Application to Division Artillery Staff Training 5b. PROGRAM ELEMENT NUMBER 0603007A

6. AUTHOR(S) Angelo Mirabella (ARI) and 5c. PROJECT NUMBER A792

James F. Love (Consultant) 5d. TASK NUMBER 2129

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

U.S. Army Research Institute for the Behavioral and Social Sciences

ATTN: TAPC-ARI-II

5001 Eisenhower Avenue

Alexandria, VA 22333-5600

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM ARI

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Ave. 11. MONITOR REPORT NUMBER

Alexandria, VA 22333-5600

Technical Report 1086

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words): This research was conducted as a pilot effort in preparation for anticipated future

research applications of the self assessment based mini-AAR (SAMAAR) approach. The approach combines the Army's

Socratic based after action review with the Delphi technique. Delphi elicits independent judgments from experts before

bringing them together to solve problems as a group. In the SAMAAR approach, trainees, at the end of an exercise day or

shift fill out rating forms to assess the units training progress and then convene with their completed forms to participate in

a mini-AAR. The mini-AAR is a preliminary training review carried out by small groups prior to an end of exercise after

action review. SAMAAR was developmentally applied to Division Artillery Staff training at Fort Hood. The approach was

judged by training participants to be a feasible and timely way to support training feedback.

15. SUBJECT TERMS Training, Learning, Instruction, Joint Training, Measurement, Assessment, Psychometrics,

Psychological Testing

~~ 9.. LMTTINO 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number) 16. REPORT

Unclassified 17. ABSTRACT 18. THIS PAGE 64 Angelo Mirabella

Unclassified Unclassified Unlimited DSN 767-0325

i
Technical Report 1086

Self-Assessment Based Mini-After Action Review

(SAMAAR) Methodology: Developmental

Application to Division Artillery Staff Training

Angelo Mirabella

U.S. Army Research Institute

James F. Love

Consultant

Advanced Training Methods Research Unit

Robert J. Seidel, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

July 1998

Army Project Number Manpower and Personnel

20363007A792

Approved for public release; distribution is unlimited.

iii
FOREWORD

The U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) conducts

research on how to design unit training strategies. Within the past few years, its mission has been

extended to include the Army's role in inter service training. This document is one of a series of

reports that have been produced under the Joint and Multi-Service Distributed Training Testbed

(JMDT2) program. The purpose of the program has been to apply the basic concepts of

instructional systems design (ISD) to the development and management of inter service training.

A critical feature of ISD is assessment of training progress and remedial feedback.

This report summarizes the beginning of innovative efforts to introduce ground breaking

new ideas about training feedback. These ideas are extensions of classical ISD concepts,

especially tailored to the unstructured, non-linear, often fuzzy conditions of high echelon inter

service mission planning and management. The work extends the Army's Socratic after action

review (AAR) method of feedback by adding the philosophy and methods of the Delphi

Approach. The trainee is recognized as an untapped source of a wealth of information about

training deficiencies and training progress, particularly in inter service training. Methods for

extracting that information according to Delphi procedures were developed and then pilot tested

in a division artillery staff training exercise at Fort Hood, in February, 1998. Preliminary results of

an assessment of user reactions indicate that the methodology can substantially leverage the

effectiveness of future staff training at high echelons, whether single or inter service.

Z A M. SIMUTIS

T 'chnical Director
ACKNOWLEDGMENTS

We are indebted to Dr. Linda Pierce, Chief, Army Research Laboratory (ARL) Field

Element, Fort Sill for allowing us to join her project team at the February, 1998 division artillery

staff training exercise, Fort Hood, to conduct the research described in this report. We thank Ken

Hallion of the ARL project team for his assistance in helping us to work along with that team.

We also appreciate the generous cooperation of the 1st Cavalry Division Artillery staff. Special

thanks are extended to Major Matt Merrick, S3, and Major Ken Gantt, Assistant S3.

vi
SELF-ASSESSMENT BASED MINI-AAR (SAMAAR) METHODOLOGY:

DEVLOPMENTAL RESEARCH IN DIVISION ARTILLERY STAFF TRAINING

EXECUTIVE SUMMARY

Research Requirement:

The overall requirement was to pilot test the Self-Assessment Based Mini-After Action Review

(SAMAAR) methodology. Specific goals included the following: (1) Assess the value and

features of SAMAAR tools; (2) Assess problems in research Implementation of SAMAAR; and

(3) Compile "Lessons Learned" for follow-up Corps-level Research Application of SAMAAR.

Procedure:

The research test bed was provided by a regularly scheduled field training exercise (FIFX) of the

V• Cavalry Division Artillery Staff at Fort Hood, TX. Research participants included members of

three staff elements: Operations (Ops), Fire Control Element (FCE), and the Division Tactical

Center Fire Support Element (DTAC FSE). A self-assessment checklist was distributed to staff

members at the beginning of each of two days of a field training exercise, along with some job

aids describing the tasks for each staff element. It was anticipated that filling out the checklist

forms would be a preliminary step in preparation for an end of shift mini-AAR on each day. For a

variety of administrative reasons the mini-AARs were not conducted, but the checklists were

completed by staff members. At the end of two exercise days, a questionnaire to assess the value

of the self-assessment approach and tools was administered.

Findings:

The SAMAAR approach was judged to be useful and timely for surfacing training problems that

otherwise might be overlooked. The self-assessment checklist was judged to be clear and easy to

complete. It was demonstrated how checklist data under standardized conditions can provide a

way to track training progress across unit elements and mission days. The value and appropriate

use of the job aids - task flow charts and tables - are inconclusive. The job aids require further

development and test.

Use of Findings:

The findings provide a technical base for further research and development of the self-assessment

approach in a joint training environment. Lessons were learned about how to improve the

implementation of the self-assessment methodology and how to conduct research on its

effectiveness.

vii
SELF-ASSESSMENT BASED MINI-AAR (SAMAAR) METHODOLOGY:

DEVLOPMENTAL RESEARCH IN DIVISION ARTILLERY STAFF TRAINING

CONTENTS Page

INTRODUCTION ........................................................................................... 1

Program Background ............................................................................... 1

The SAMAAR Approach ...................................................................... 2

The SAMAAR Procedures ...................................................................... 5

Research Goals ................................................................................... 6

METHOD ................................................................................................. 7

R E SU L T S ..................................................................................................... 9

Opinions of Value and Features of SAMAAR .............................................. 9

Assessment Checklist Data ....................................................................... 15

DISCUSSION ............................................................................................ 17

Evaluation of SAMAAR ...................................................................... 17

Self-Assessment Checklist Data ............................................................... 17

Implementation Shortfalls ...................................................................... 18

CONCLUSIONS ......................................................................................... 19

REFERENCES .............................................................................................. 20

APPENDIX A: TARGET AUDIENCE FOR SELF-ASSESSMENT PILOT TEST ............ A-1

B: COMPONENTS OF THE SELF-ASSESSMENT PACKAGE .................. B-1

C: EVALUATION OF SELF-ASSESSMENT PACKAGE AND MIN-AARS ... C-1

LIST OF TABLES

Table 1. Evaluation of SAMAAR ..................................................................... 9

2. Percent of "Needs Improvement" Responses .............................................. 15

LIST OF FIGURES

Figure 1. Future enhancement of the SAMAAR approach ............................................ 18

ix
Self-Assessment Based Mini-After Action Review (SAMAAR) Methodology:

Developmental Application to Division Artillery Staff Training

INTRODUCTION

Program Background

This report describes research conducted as part of an ARI program to develop measures

and methods for supporting after action reviews (AARs) in joint and multi-service training. The

purpose of the report is to summarize developmental research on a Self-Assessment Based, MiniAfter Action Review (SAMAAR) methodology that has evolved from earlier efforts by and

under the auspices of the US Army Research Institute (ARI) (Love, 1998a, 1998b, 1998c). The

current research was conducted as a supplement to a Division Artillery (DivArty) Staff Training

Event at Fort Hood, in January 1998. It resulted in "lessons learned" for follow-on research

dealing with joint training.

The SAMAAR methodology is designed to increase the efficiency and effectiveness of

training feedback and, in turn, training effectiveness. Training feedback is defined here as

diagnostic information regarding the effectiveness of task performance. The ARI program

focuses on feedback generated by the trainees themselves (i.e., "self-assessment") and

incorporated into daily staff cell AARs (i.e., "mini" AARs) conducted prior to unit-wide AARs

at the end of exercises (EndExs).

The Army uses a standardized form of AAR which draws heavily on the trainees'

assessments of their units' strengths and needs for improvement (Brown, Wilkinson, Nordyke,

Riede, Huyssoon, Aguilar, & Wonsewitz, & Meliza, '1997, p. 3; Department of the Army, 1993;

Dyer, 1994; Keene, 1994; Meliza, 1996; & Scott, 1984). This form is, in fact, an application of

the Socratic method. In a typical Army AAR, a senior observer/controller (O/C) moderates a

discussion of what happened, how and why it happened, and how unit performance can be

improved in subsequent exercises or exercise phases. Consistent with the Socratic method, the

Senior O/C is likely to steer the discussion towards particular problems which he and his cadre of

O/Cs have noted and documented during the training exercise. But he will through questioning

draw out from the trainees descriptions of their own experiences and thought processes.

The AAR methodology is based on the assumption that active involvement of the trainee

yields motivational and learning benefits. Furthermore, the trainee provides critical information

and judgments about training effectiveness, judgments which supplement information and

assessments made by O/Cs (Scott, 1983, pp. 2-5).

This AAR methodology has been exploited and institutionalized extensively for tactical

"training at the combat training centers (CTCs) where complex instrumentation and trained cadres

of O/Cs are available (Dyer, 1994; Sulzen, 1997; U.S. Army Research Institute, 1992). But the

methodology is less thoroughly used in the joint and multi-service arenas, even though AAR

principles and practices are explicitly and thoroughly detailed in the Joint Training Manual

(Joint Chiefs of Staff, 1996a). "Hot washes" (i.e., short summary briefings about exercise pluses
and minuses in key areas) and after-action reports do provide good training feedback in Theater

and Joint Task Force (JTF) level training events. However, these occur at the end of two to three

week rotations, are aimed primarily at senior leadership, and deal as much with training system

and training management problems as with staff performance problems. Some observer/trainers

may hold an AAR following an exercise, but even this is not done after every exercise. The

foregoing is not a criticism, but simply an observation about an opportunity to leverage joint

training effectiveness at relatively little cost.

According to psychological principles, training feedback needs to be done frequently and

close to the targeted performance to be effective (Fedor & Buckley, 1987; Weinstein, Goetz, &

Alexander, 1988, p. 122;). Its effectiveness has been shown to decrease in an inverse exponential

manner, i.e., effectiveness drops sharply at first, then levels off (Klein, 1991, p. 137; Wolfle,

1951). "Theory" also recognizes that trainees have valid and valuable insights into training

progress and effectiveness (Dominick, Reilly, & McGourty, 1997; Keesling, Ford, & Harrison,

1994; Mirabella, Sticha, & Morrison, 1997; Zazanis & Lappin, 1998). For example, evidence

from National Training Center (NTC) exercises supports the validity of self-assessment for

battalion task force training. In these exercises effectiveness ratings by battalion staff personnel

agreed with ratings by O/Cs (Keesling, Ford, & Harrison, 1994, p. 145). Zazanis and Lappin

(1998) studied the use of trainees as sources of evaluation data for special forces training. Their

research suggested that such data "... may provide the most critical and irreplaceable information

in situations that have a high requirement for interpersonal skills, or that have difficult or

ambiguous tasks requiring high levels of effort or persistence." Therefore, a wealth of

knowledge may remain untapped in the experiences of trainees, knowledge that can be extracted

through enhanced self-assessment tools and methods. The characteristics of that methodology

are described below.

The SAMAAR Approach

Overview

The SAMAAR approach expands the philosophy and techniques of the Army's Socratic

AAR method by adding the philosophy and techniques of the Delphi method. The Army's

approach has already been described. The Delphi method adds a mechanism for trainees to

individually think about and document their perceptions of unit training progress at their level of

experience, so that they are well prepared to contribute to a follow-up mini-AAR. The

mechanism is designed to draw out participant experience, maximally and efficiently, while

correcting limitations in a given trainee's situation awareness.

In its original applications Delphi was used to facilitate the efforts of experts to solve

fuzzy, complex national policy problems. The experts were asked to generate and document

solutions individually and then convene for group discussion to reconcile differences and arrive

at an efficient consensus. In getting to the consensus, differences among the experts were

discussed. These might be the result of different premises or experiences. They were factored

into the consensus solution. Thus Delphi was a mechanism for increasing the reliability of

subjective "measurement."

2
An adaptation of Delphi philosophy and techniques appears well suited to assessing staff

training performance, especially the very "soft" skills inherent in operations planning and

concept development. These soft skills become increasingly important at increasing echelon

levels and are prevalent in joint training environments.

Measurement Characteristics

Trainee as the Data Generator. Collective training effectiveness measurement has

traditionally employed O/Cs, electronic data streams, or both as data sources as at the Army's

Combat Training Centers. O/C cadres are employed as data generators for: (a) ARI s Army

Command and Control Evaluation System (ACCES), (b) the Naval Air Warfare Center's

Tactically Relevant Assessment of Combat Teams (TRACT), and (c) The Tactically Relevant

Assessment of Combat Events (TRACE) methodologies, as well as for most joint training

exercises. The SAMAAR approach is somewhat unique in viewing the trainee as a surrogate

O/C and as a key source of training assessment data. Therefore, where an O/C cadre is available

it would be leveraged by the trainees. Where a cadre is not available, the trainees would act a

surrogate cadre.

Task as Unit of Analysis. The SAMAAR methodology is task oriented rather than

events-based. A similar orientation can be seen for the measurement approach in the Universal

Joint Task List (UJTL). But SAMAAR contrasts with TRACE, which is used to verify the

actual occurrence of fine-grained, sequenced combat events against a list of expected events.

SAMAAR is more consistent with a view of battle as dynamically nonlinear and complexly

interactive. TRACE is more consistent with a view of battle as a predictable sequence of events.

Each perspective is a simplifying assumption since combat clearly has both linear and nonlinear

characteristics. Nonetheless, the perspectives are not necessarily equally useful in generating

measurement methodology for supporting training feedback. A forerunner of TRACE was found

to be only marginally useful in supporting AARs in a battalion task force, simulation-based

training event (Mirabella, Sticha, & Morrison, 1997). SAMAAR, on the other hand, has evolved

specifically with AARs in mind. SAMAAR also contrasts with approaches that attempt to

measure universal (psychological) teamwork characteristics, e.g., the TRACT methodology. A

forerunner of TRACT was also examined in the battalion task force exercises. Exercise O/Cs

were unanimous in reporting that it was not useful for preparing or conducting AARs (Mirabella,

Sticha, & Morrison, 1997).

Process and Product Measurement. SAMAAR measures behavioral processes as well as

products and directly assesses perceived training deficiencies. It contrasts with ACCES (Halpin,

1996) and with the UJTL (Joint Chiefs of Staff, 1995, 1996a,b) measurement scheme. ACCES

and UJTL employ "statistical" dimensions to measure the outputs of process, e.g., percentage,

duration, quantity, and distance. This "statistical" approach for feedback has received some

negative comment (e.g., at the 1997 Universal Joint Task List Conference, Joint Warfighting

Center, Fort Monroe, VA) as being too rigid, restrictive, and fine-grained for the "fuzzy"

character of training at high echelons. SAMAAR is deliberately more discursive; it avoids the

emphasis on mathematical precision implicit in ACCES and UJTL.

3
Easy Application. SAMAAR is designed for frequent application (e.g., immediately

following the end of each shift). It calls for a 15-minute period at the end of a day or shift for

trainees to fill out an assessment of their cell's performance, using a short checklist about key

products and processes. This activity is followed by a mini-AAR which is led by the senior

member of a staff cell (or tactical sub unit). Availability of an O/C cadre is not assumed for use

of the methodology. AARs that rely on O/C or electronic data require extensive preparation time

and personnel resources (Brown et al, 1997; Meliza, 1996)

Multiple Uses. The SAMAAR approach is designed to maximize involvement of all

players in the training process and learning experience. First it provides an unconstrained

opportunity for each member of a staff cell to record his or her reactions to training progress.

Then it provides for a meeting in which each staff member's voice has equal weight in arriving at

a consensus (or disagreements) on where training improvements are needed.

The method can also facilitate hand-off to the next shift. Problems identified by staff

cells in one shift can be highlighted for cells in the following shift. Similarly inter-cell problems,

e.g., communication shortfalls, can be addressed for the benefit of that and subsequent shifts.

Finally, results of the SAMAAR process can be made available for and integrated into

unit-wide AARs at the end of the training event or rotation. As part of this, the results of the

SAMAAR process could be used to do a trend analysis across days within the training event or

across training events.

Anticipated Training Benefits

Based upon principles of learning psychology, we can anticipate the following positive

effects of the SAMAAR approach on training progress:

1. Timely Feedback. It was mentioned earlier that the effectiveness of training feedback

increases the closer it is given to when the targeted performance occurs. The logistic simplicity

of SAMAAR makes frequent application of the approach feasible.

2. Increased Time on Task (TOT). The SAMAAR approach is a very efficient way to

amplify the effective amount of practice time on mission-essential tasks, because in half an hour

or so, attention can be focused on tasks most in need of improvement. This can be viewed

almost as a form of compressed training. In turn, the increased TOT can substantially raise the

unit's skill level and decrease the rate of skill decay (Loftus, 1985; Mirabella, Macpherson, &

Patterson, 1989, p. 40).

3. Active Involvement. The method increases active involvement by encouraging inputs

to the feedback process from each member of a staff cell or other sub-unit involved in miniAARs. Active participation is critical to effective training (Mirabella, Macpherson, & Patterson,

1989; Travers, Van Wagenen, Haygood, & McCormick, 1964).

4
SAMAAR Procedures

Orientation

The purpose of SAMAAR and its procedures need to be clearly explained to all the

trainees who will be asked to fill in the self-assessment checklists and take part in the miniAARs. Orientation should not be limited to the staff cell leaders. The orientation should stress

that SAMAAR is designed to encourage all the trainees to participate in and contribute to the

AAR process. But it's meant to provide a set of tools, not a "straightjacket." It also needs to be

clear that filling out the checklist is a way for the trainees to record their opinions about training

progress so that they can contribute to the follow-up mini-AAR. Finally, the SAMAAR

orientation should be included with other pre-exercise preparations, so that it becomes an integral

part of the training, rather than a fringe activity.

Administration of Self-Assessment Checklist

At the end of a shift, trainees should be gathered in a setting as comfortable as

circumstances will permit. Provide a table if possible and offer pencils. Go over the highlights

of the initial orientation. Then explain the different parts of the checklist and the rating

procedure, e.g., explain what the checklist categories mean. Pick out a few items and illustrate

some possible reactions to those items. Encourage the participants to add comments, especially

where trainees indicate a need for improvement. Finally, ask for questions and have the trainees

complete the self-assessment checklist.

Conduct of the Mini-AAR

"House Rules"

"* Everyone has an equal voice in discussion; no one has all the answers. Listening and talking

are equally important to good training feedback.

"* Focus on what happened, what needs improvement, and how to improve.

"* Use the mini-AAR to spot and clear up misunderstandings about mission process or product,

especially for new staff members.

Procedures for Conducting "Self-Assessment" Mini-AARs

"* Staff cell gets together at the end of each shift for a mini-AAR. Staff members bring their

filled-out checklists to help them contribute to the mini-AAR.

"* Senior cell member leads the mini-AAR, starting with a blank checklist

0 Steps for conducting the mini-AAR.

a. Address each checklist item in turn.

b. If everyone agrees to a satisfactory rating, go to the next item.

5
c. Otherwise, discuss the item and make notes on training problems and solutions.

d. If a member of the cell (e.g., new replacement) doesn't understand a mission product

or process, use this opportunity to clear up questions. Then:

"> AAR leader records a consensus response.

"> If no consensus, leader picks a response, but makes notes on disagreements.

e. Keep the consensus sheet for use in shift change over, to check progress over

subsequent shifts or mission phases, and as an input to a larger AAR at the end of the training

rotation.

Research Goals

Overall Goal

Pilot Research Application of SAMAAR Approach. The primary audience for SAMAAR

is the joint training community. However, the division artillery exercises at Fort Hood provided

a target of opportunity and a credible surrogate environment to conduct pilot research in

preparation for an anticipated joint training research application. But, in addition, the results of

the pilot application were expected to extend the Army's technical base on training feedback.

Specific Objectives

1. Assessment of Value and Features of SAMAAR Tools. A major goal was to assess

user reactions to the following questions: Does the SAMAAR approach and its specific tools

add value to staff training exercises? What improvements or revisions are needed?

2. Assessment of Problems in Research Implementation of SAMAAR. Since this was a

pilot effort to support follow-on research planning, an important objective was to document

problems in scheduling and administering SAMAAR and research procedures. How well could

these be incorporated into the on-going exercises, given the constraint of non-interference? What

difficulties would the units experience in using the self-assessment checklist to conduct miniAARs?

3. Compilation of "Lessons Learned" for Corps-Level Research Application of

SAMAAR. This objective was to draw on our division Artillery experience to design SAMAAR

and write a research plan for follow-on research in a joint, corps-level exercise test bed.

6
METHOD

Test Bed

The research test bed was provided by a regularly scheduled field training exercise (FTX)

of the 1st Cavalry Division Artillery Staff at Fort Hood, TX. This was a three-day exercise from

20 to 22 January 1998. One shift of trainees participated on Days 1 and 3; a second shift

participated on Day 2. The FTX differed in one important respect from other rotations. It

provided a primary test bed for research by the Fort Sill Field Unit of the Army Research

Laboratory (ARL). ARI worked under the auspices of ARL, on the basis of non-interference

with either the exercise or the ARL research activities. ARL's purpose was to examine the

training effectiveness of a simulator/stimulator developed for operational testing, the Fire

Support Automated Test System (FSATS). A related purpose was to test combat team

measurement instruments adapted from the Mult-Service Distributed Training Test Bed (MDT2)

program (Bell, Dwyer, Love, Meliza, Mirabella, & Moses, 1997).

Participants

Research participants included members of four staff elements: Operations (OPS), Fire

Control Element (FCE), Targeting Element (TGTG), and the Division Tactical Operations

Center Fire Support Element (DTAC FSE). Appendix A lists the positions in each of the

elements which did participate. Two shifts of staff members went through training. The first

shift trained on Days 1 and 3, the second on Day 2. But some staff members participated in both

shifts.

Research Materials

SAMAAR tools: Package with introduction, self-assessment checklist for the staff

element, flow chart showing inputs from and outputs to other staff elements, table listing inputs,

outputs, and processes that link/convert inputs to outputs. These packages were compiled for

each staff element from the set of materials shown in Appendix B.

Research tools

SAMAAR opinion questionnaire. A questionnaire was designed to assess the

participants' reactions to the value and features of the SAMAAR approach and its tools

(Appendix C). The questionnaire contained five items on value and five items on the features of

the SAMAAR approach and tools. A six point response scale ranging from strongly agree to

strongly disagree was included with each questionnaire item. The scale was patterned after one

used by Mirabella, Sticha, and Morrison (1997). The survey form also provides space for

comments.

7
Tape recorders. Tape recorders were brought to the test site to record the mini-AARs.

We intended to content-analyze the recordings for problems with use of the self-assessment

forms during the mini-AARs. These recordings were to supplement observations of the AARs

by the research staff following a practice employed by Mirabella, Sticha, and Morrison (1997)

in the MDT2 program. The tape recorders were not used for reasons indicated below.

Procedures

Orientation to SAMAAR. On the morning of Day 1 a series of orientation briefings were

given to the entire DivArty staff by the ARL research staff. Included among the briefings was a

short (approximately 5-minute) overview of and introduction by ARI to the SAMAAR approach.

Administration of self assessment checklists. On Days 1 and 2, sets of SAMAAR

packages (Appendix B) were given to each of the DivArty staff element non-commissioned

officers in charge (NCOICs). These were to be distributed to each member of each of the four

staff elements. The assessment checklist in the package was to be completed at the end of the

day (shift), prior to a mini-AARs. Checklists were not distributed on Day 3 because we learned

that the exercise was to be terminated after a few hours and time for administering either the selfassessment checklists or the mini-AARs was not anticipated.

Mini-AARs. For administrative reasons beyond our control, the expected mini-AARs on

Days 1 and 2 were not conducted. Consequently, we were able to apply only the first segment of

the SAMAAR approach.

Administration of opinion survey on SAMAAR. The opinion survey (Appendix C) was

administered on the morning of Day 3, as part of a larger administration of questionnaires by the

ARL staff.

8
RESULTS

Opinions on Value and Features of SAMAAR

Responses to the evaluation questionnaire were received from 8 participants. A summary

of responses to a reduced set of items is presented in Table 1. A number of the items on the

original questionnaire dealt with mini-AARs, which never took place. As a result checklist

responses to those items were considered uninterpretable. Table 1 lists the remaining items and

their percentages of agreement ("Agree" to "Strongly Agree"). Positive reaction is uniformly

high for value and features (75% to 100% agreement).

Table 1. Evaluation of SAMAAR

Evaluation Item Agree

4. A self-assessment package like this is not currently available to my unit 100

5. A self-assessment package like this or an improved version would be useful in 88

future exercises

6. The instruction sheet was clear, well organized, and had about the right 75

amount of information.

7. Assessment checklist items were useful as memory joggers and could be 88

answered.

10. I felt that I could respond accurately to the self-assessment tools 75

Comments from the questionnaires were compiled and organized from the entire original

questionnaire. For developmental purposes, it was considered useful to examine all the

comments. These are clustered, with annotations, under a set of categories derived from the

original questionnaire. Included are positive and negative comments on the value of the miniAAR methodology. Also included are constructive comments on technical aspects of the

methodology. Each category of comment provides useful information and "lessons learned" for

further research and development, and eventual implementation of the methodology.

Positive Comments

The positive comments indicate that the self-assessment approach can be useful and

timely in surfacing training problems that might have otherwise been overlooked. It is also a

way to ensure that all levels, enlisted personnel and officers, participate in improving the

training, and that communication and coordination problems among cells not "fall through the

cracks." This conclusion is based on the following comments:

"More detailed AAR package would be a great tool to help improve our performance."

"Although not a replacement for the formal AAR, the mini-AAR enhances training

9
through self-evaluation."

"It is a good tool for leaders to look at the way they do things and make quick

improvements."

"If utilized correctly, the mini-AAR has value. It prompted me to look at areas I might

have overlooked."

"It made us aware of what to give a little more attention to as we went along."

"It helped identify areas where improvement is needed as well as being able to coordinate

with the other sections."

"Gets input from all levels."

"Have them at the end of the exercise."

The positive comments suggest that in our future explanations and training we should

point out that the mini-AAR can provide "just-in-time" remedial feedback -- feedback which is

given at least at the end of each training day, in small enough groups, so that all members of a

staff cell can contribute. A number of comments from an ARL survey indicate interest in having

AARs several times a day (Linda Pierce, personal communication, February, 1998). One trainee

in the ARL survey suggested AARs at the end of one-hour blocks, a second trainee suggested

AARs at the end of four-hour blocks. We also need to point out that the mini-AAR can help to

point up problems in inter-cell communication and coordination. These can then be worked out

either informally among cells or as part of a larger AAR that brings the cells together.

An interesting observation by ARL personnel is that trainees often have incomplete or

inaccurate information about the consequences of their actions and therefore may not be in a

position to self-evaluate (Linda Pierce, personal communication, February, 1998). However, to

the contrary, the mini-AAR process can counteract these inaccurate perceptions by allowing

various points of view to surface during the cell mini-AAR meetings. If one cell member has

limited or inaccurate perceptions, contrary opinions by other cell members can serve as "midcourse" corrections. The possibility still exists that all cell members may have inaccurate

information about the consequences of their performance. Here is where other sources of

information may be brought to bear: expectations vs. actual performance; the electronic data

stream which records actual events; observations and judgments of O/Cs, where an O/C cadre is

available; and finally the perspectives of senior leadership and training managers. The challenge

here is how to incorporate these other sources into the "brainstorming" of the staff cells as they

go through their mini-AARs.

10
Negative Comments

A number of comments question the need for and the value of the mini-AAR

methodology under research investigation by ARE The comments communicate the message

that we shouldn't be trying to fix something that's not broken. They maintain that Army policy

and practice on after action review is sufficient and requires no innovation. The comments

further suggest that command leadership has the tools necessary to provide training feedback and

requires no additional training tools or innovations.

"I don't feel we need a supplement to the guidelines given by the Army on how to conduct

AARs."

"We rely on officer or NCO leadership to improve performance. We know our strengths

and weaknesses w/o being prompted by a form."

"I think the sections are good at finding & fixing problems. We are always able to find

problems with training. It's never hunky-dory."

"DivArty could easily develop one on their own. We conduct AARs after every training

event, as the results are used to tailor future training."

"If leaders are focused then important areas won't be overlooked."

"I think that a leader-led AAR focused on the METL tasks and their training objectives is

sufficient. Officers and NCOs can put comments on paper if necessary, but a long form is

unnecessary and usually ineffective."

"N/A - Didn't see any noticeable improvement."

Some thoughtful comments and food for thought are provided above. There may be a

communication problem here, a misunderstanding about what we're trying to achieve. The

misunderstanding is not surprising, since we were not able to actually get the daily mini-AARs

implemented. Trainees did not experience the complete feedback process. We are in fact trying

to provide some tools to support cell leader-led AARs focused on the METL tasks and training

objectives. Our checklist was derived from METL tasks and training objectives. We're also

trying to develop a methodology to promote timely and frequent feedback -- at least daily if not

more often, as suggested in comments from the ARL/NAWCTSD survey. The current AARs

would be sufficient if, in fact, they were done in a systematic, persistent, and timely fashion. At

the January Fort Hood DivArty exercises there were no section AARs. Whether this is the usual

case, or was a special case because of the ARL research, we don't know. We do know from

scientific research that training feedback is most effective when given close to where the training

deficiencies occur. The methodology we are examining is designed to encourage more timely

feedback as well as greater participation from all the players.

11
Some concern was expressed about the value of filling out "self-assessment" checklists.

Again I think we didn't communicate our methodology adequately. We need to explain more

effectively that we are developing and testing an assessment/feedback system, with the checklist

as a component of the system. The purpose of the checklist is to give the staff cell participants a

chance to document their own perspectives and judgments about training effectiveness and

problems, and then use the checklists to contribute when the cell convenes for the actual miniAAR. The checklist is designed to extract maximal benefit from each participant's training

experience. We agree with the concerns about excessive paper work. We're trying to keep the

paper work to a minimum. A research goal is to design short, simple forms that get the job done.

Finally, the view was expressed that trainees and their leaders are fully capable of

identifying and fixing training problems with current SOP and capabilities. We don't doubt this

in the least. In fact those capabilities provide a key premise underlying the prototype selfassessment/mini-AAR methodology. It is because the trainees "know" their "strengths and

weaknesses" that the self-assessment/mini-AAR can add value to the training. The methodology

allows each member of the cell to document his or her views of training problems and helps

bring out differences of opinion for constructive discussion when the cell meets.

Our preliminary data show frequent same-task disagreements ranging from "needs

improvement" to "commendable." So, not everyone in the cell will see strengths and weaknesses

in the same way. Psychological theory indicates that well qualified personnel observing the

same events may focus on different aspects of those events and arrive at different conclusions.

The value of our methodology is that it encourages discussion to discover and reconcile the

different perceptions and cognitive conclusions. It is a mechanism for approaching ground truth

and providing feedback based on that truth.

Constructive Critiques-General

Though few in number, a set of comments surfaced some interesting methodological

problems that we need to attend to: (1) carefully tailoring the self assessment checklists to the

needs of specific staff cells; (2) considering the roll of other information sources such as the

major sequence event list (MSEL), and information from O/Cs; (3) AAR facilities, i.e., the

logistics of implementing AARs; and (4) providing adequate training in the use of the

methodology. These issues were suggested by the following comments:

"The self-assessments should be tailored more towards battle operations than the orders

process."

"It dealt more with planning than execution."

"Must be aligned with duty position."

"There was no section AAR conducted. If each section was given a copy of the MEL which

affected their section, they could have a good AAR."

12
"Use desks, not chairs, separate officers and enlisted. Give the soldiers an accurate

understanding of what they are to do."

The first two comments above are from the Fire Support Element (FSE) and merit

special discussion. The comments show a concern about tasks that FSE personnel didn't think

they were supposed to do. The Fire Support Element (FSE) self-assessment tool was designed

for both the FSE at the DMAIN location (involved with planning/orders preparation) and the

FSE at the DTAC location (involved with current battle execution). Since the exercise at Fort

Hood played the FSE only at the DTAC, the comments referenced above are understandable.

For future exercises, the FSE self-assessment tool could be written in two versions: one for FSE

at DMAIN and one for FSE at DTAC. The broader lesson learned is that resources must be

allocated and arrangements made to staff the self-assessment materials with the types of

personnel who will be asked to use them.

The comment referencing the MEL (Major Event List) raises an interesting question.

Would it be useful to work the MEL into the mini-AAR/self assessment process and how would

it be best to do so? Our self-assessment checklists are task and training objectives oriented,

rather than event-sequenced like the MEL. How can we combine the two? Maybe the cell can

begin with the larger task judgments and then go to the MEL to get specific examples of

problems. Is there some way to "job-aid" this linkage? ARL/NAWCTSD tried to get a crosswalk between the MEL (i.e., TRACE) and more generic judgments (TRACT) and didn't seem to

have much luck doing so. But it might be easier to link MEL events with the process and

product statements in the self-assessment forms.

The last comment above raises three issues: (1) logistics of administering the selfassessment/min-AAR; (2) composition of the mini-AAR group; and (3) orientation and training

in the purposes and procedures of the self-assessment/mini-AAR. We need to try to arrange

things more comfortably and make it easier for the troops to carry out the various parts of the

self-assessment/mini-AAR process. Desks may not be feasible, especially in field environments,

but maybe a table can be set up, or at the very least, clip boards provided as minimal writing

surfaces.

Regarding the separation of officers and enlisted personnel, arguments exist on both

sides. Enlisted personnel may be intimidated by officers. On the other hand, if they are

separated, each segment will not have the benefit of the other segments viewpoints. The value of

diversity of perspectives and ideas is a key premise of the self-assessment/mini-AAR process.

The final clause in the last comment is especially important. In future tests and

applications, significant time and effort should be expended to explain the self-assessment/miniAAR methodology. A cursory, five-minute overview isn't going to get the job done. This is a

bit tricky. We don't want to burden the players with additional training requirements, but we

have to insure that "the soldiers [have] an accurate understanding of what they are to do."

13
Constructive Comments on the Self-Assessment Package -- Instructions and Self-Assessment

Checklist

This package included an instruction sheet, a self-assessment checklist, and several job

aids: a flow chart showing inputs and outputs from each cell to every other cell and a table

describing inputs, related processes, and outputs for each staff cell. Survey responses and

comments indicated that the instruction sheet and the self-assessment checklist were positively

received. One respondent, however, did indicate that he was unaware of an instruction sheet.

His comment reinforces the need to take greater care and expend more effort in explaining the

self-assessment package.

"Very concise." [Instruction Sheet]

"What instruction sheet?"

"They [Self-Assessment Checklist] were good reminders."

"I got an extra one [Self-Assessment Checklist]."

Constructive Comments on the Self-Assessment Package -- Job Aids

The comments below indicate a range of opinions on the usefulness of the job aids. The

tables and flow charts we provided may be more useful as training materials, than as AAR

guides. But without having observed mini-AARs, we don't know how they might have been

used during the discussions. Cross-references from the checklist to the job aids may increase the

usefulness of the job aids during the mini-AAR.

"Couldn't see relevance of flow chart to AAR."

"They [flow charts] provided good details."

"My section is very talkative."

"They [table of processes] served as a useful or useful guides."

"I know how to do an AAR."

14
Assessment Checklist Data

Table 2 summarizes responses filled out on assessment checklists by members of three

staff cells, on the first two days of a three-day Division Artillery Staff Training exercise. A

different shift of personnel participated each day, but a few people participated on both days.

Table 2. Percent of "Needs Improvement" Responses

Day Staff Element Percent "Opportunity for Improvement"

I Product Process

1 Fire Support Element (FSE) [3] 50% (12/24) 30% (16/54)

2 Fire Support Element [5] 40% (16/40) 49% (44/90)

1 Fire Control Element (FCE) [4] 4% (1/24) 16% (12/77)

2 Fire Control Element [3] 11% (2/18) 28% (15/54)

1 Operations (OPS) [7] 29% (7/24) 44% (65/148)

2 Operations [5] 10% (2/20) 11% (11/104)

] number of staff cell members who completed the checklist

The numbers of personnel responding to the checklists are given in square brackets under "Staff

Element." The data under "Product" and "Process" reflect percentages of "Opportunity for

Improvement" responses. The baseline for the percentages includes responses to all the checklist

response categories.

Sensitivity to Training Problems. Note in Table 2 the large numbers and percentages of

processes and products across cells and days judged to be in need of improvement. These

quantities underscore the potential value of the players as data sources and the related value of

the self-assessment scales for extracting that data. That is, the self-assessment instrument

appears to be sensitive to perceived training deficiencies.

Trend Analysis and Comparisons. It is not meaningful to do trend analysis or comparison

with the current data because too many confounding variables were operating across days. For

example, different shifts, with different S3s participated on Days 1 and 2, though some personnel

did participate on both days. And the need to accommodate a test of a new simulation facility

added its own complications, not normally present in scheduled training events for division

artillery staff. Finally, the reliability and validity of the checklist itself are unknown.

But the results illustrate how the checklist data might be used to do a "temperature check"

of training progress across days and across staff cell elements under more controlled, comparable

conditions. By extension these same analyses might be used for comparing alternative training

systems. Imagine then that the checklist data were obtained under circumstances that justified

meaningful comparisons across days and staff cell elements. What might the data tell us?

15
a. Process and product measures appear to co-vary. A non-significant chi-square test for

product vs. process across days and staff elements supports this.

b. The OPS cell "judged" itself much in need of improvement at the end of Day 1, but

"made substantial progress" by the end of Day 2.

c. The FSE "experienced" a large (the largest) proportion of perceived training

deficiencies overall.

d. The FCE "reported" the smallest amount of training deficiency, especially for the

"Product" measure, but relatively more training deficiencies on Day 2 than on Day 1.

Again, it needs to be emphasized that the above comments are a hypothetical illustration

of how self-assessment data might be summarized and used to get a "temperature check" on

training progress and needs. The comments are not intended to characterize the staff that

participated in the training event.

16
DISCUSSION

Evaluation of SAMAAR

Value/Need. The survey results indicate the value of and the need for a self-assessment

approach as a timely and feasible way to support training feedback. Reactions to the checklist

generally were favorable. A few negative reactions were aimed at use of the checklist in

isolation of a mini-AAR. But since mini-AARs were not carried out, the purpose of the checklist

as a precursor to the mini-AAR was perhaps not understood. Some participants did question the

need to fix something that "wasn't broken." Their reactions suggested that a clearer explanation

of the SAMAAR approach needs to be included in future trial applications.

Job Aids. Reactions to the job aids included in the self-assessment package were mixed.

Some respondents felt that the flow chart and input-process-output table were useful; others did

not. The role of such aids needs further thought and evaluation. Perhaps we need to find a more

explicit way to link these job aids to the checklist and mini-AAR and then explain how to use

them. Cross-references from checklist items to specific sections of the job aids might be useful.

If so we need to create the references and then explain their use during the orientation phase of

the test bed exercise. If we were to distribute similar job aids in follow-up joint training

research, we should do so as part of the pre-exercise orientation and then assess their utility for

both exercise preparation and as an aid in the SAMAAR process.

Self-Assessment Checklist Data

The checklist data and evaluation results indicated that the checklist was easy to complete

and sensitive to training problems. Comments and checklist responses from the FSE cell did

indicate that many of its items were not well tailored to its role in this particular exercise, e.g.,

"The self assessments should be tailored more towards battle operations than the orders process."

"It dealt more with planning than execution." But these results only underscore the importance

of very careful staffing of the checklist with its target population and the need for a capability to

make revisions in checklist items "at the last minute."

In the Results Section we presented summaries of checklist responses across trainees

within staff cells and days. The purpose for presenting these summaries was to illustrate how

such data might be used to do trend analyses of training progress across days and comparisons of

training progress among staff cells under appropriately standardized conditions. Appropriate

conditions for trend analysis and comparisons did not exist here. There were too many artificial

and confounding factors. For example, the exercise was a test bed for evaluating a stimulatorsimulator, normally used for operational training. Its unconventional use resulted in technical

problems that delayed or interrupted the exercises several times. But under appropriate

conditions the checklist data might be useful in helping to monitor training progress. They might

also provide a mechanism to "red tag" recurring, persistent training problems across rotations,

problems that may benefit from special attention. This latter use, however, may be problematic

if exercise scenarios are not standardized.

17
Implementation Shortfalls

The trial implementation of SAMAAR fell short of expectations. The shortfalls were, in

part, the result of limited control of scheduling the SAMAAR events and administration of

research materials. For example, in the interest of "non-interference" the checklist forms were

given to staff personnel for distribution to and collection from staff cell members. A number of

consequences resulted. Completed checklist forms were received from only three of the four

staff elements. Heading information, e.g., date, time, cell, duty position was missing on many

otherwise completed forms. Only eight participants filled out SAMAAR evaluation forms. Most

significant, mini-AARs that were anticipated at the end of each of the three mission days did not

occur. But the shortfalls did provide constructive "lessons-learned" for follow on research.

To be worth the effort, future SAMAAR trial applications need to be conducted under a

direct, pre-arranged understanding of research requirements and agreement of support by the host

unit. Such research can still be carried out on a relatively non-interfering basis. However, it's

essential to maintain a roster of participants in order to track application of SAMAAR and other

research materials. It would still be possible to keep completed forms anonymous, if desired.

Future enhancements of the SAMAAR approach could include assessment of cell

performance by those receiving cell outputs. A research application might include assessments

by observer/trainers (O/Ts). A comparison of their assessments with those of the trainees could

provide a basis for validity and reliability analyses. Figure 1 outlines these enhancements (Guy

Siebold, personal communication, February 1998). In addition the role of electronic data from

digital force training exercises needs to be examined for its usefulness in assessing the training of

mission planning and management.

2. Cell as a group

assesses performance.

1. Cell members each 3. Those receiving

assess group performance. Staff cell outputs assess

ICell ) the outputs.

Mem Other Other

Cell Cell

Me Other

Cell

Observer!

Trainer

4. Observer/Trainer assesses

group performance of cell

and of cell's mini-AAR.

Figure 1. Future enhancement of the SAMAAR approach.

18
CONCLUSIONS

1. The SAMAAR approach was judged to be useful and timely for providing feedback and

surfacing training problems that otherwise might be overlooked.

2. The self-assessment checklist was judged to be clear and easy to complete.

3. The checklist data under standardized conditions can provide a way to track training progress

across unit elements and mission days

4. The value and appropriate use of the job aids, task flow charts and tables, are inconclusive.

They require further development and test.

19
REFERENCES

Bell, H. H., Dwyer, D. J., Love, J. F., Meliza, L. L., Mirabella, A., & Moses, F. L. (1997).

Recommendations for planning and conducting multi-service tactical training with

distributed interactive simulation technology: a four-service project (Research Product).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

(AD-A32 8480)

Brown, B., Wilkinson, S., Nordyke, J., Riede, D., Huyssoon, S., Aguilar, D.,

Wonsewitz, R., & Meliza, L. L. (1997). Developing an automated training analysis and

feedback system for tank platoons (Research Report 1708). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. (AD-A32 8445)

Dominick, P. G., Reilly, R. R., & McGourty, J. W. (1997). The effects of peer feedback on team

member behavior. Organization Management, 22(4), 508-520.

Dyer, J. L. (1994). A comparison of information in the Joint Readiness Training Center

archival records (Research Report 1659). Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences. (AD-A27 7676)

Fedor, D. B. & Buckley, M. R. (1987). Providing feedback to organizational members: a

reconsideration. Journal of Business Psychology, 2(2), 171-180.

Halpin, S. M. (1996). The Army command and control evaluation system (ACCES)

(Research Product 96-04). Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences. (AD-A31 7213)

Joint Chiefs of Staff (1996a, June). Joint training manual for the armed forces of the

United States (CJCSM 3500.03). Washington, DC: Author.

Joint Chiefs of Staff (1996b, Sept). Universal joint task list (CJCSM 3500.04A).

Washington, DC: Author.

Keesling, W., Ford, P., & Harrison, K. (1994). Application of the principles of training

in armor and mechanized infantry unit. In R. F. Holz, J.H. Hiller, & H.H. McFann

(Eds.), Determinants of effective unit performance: research on measuring and managing

unit training readiness (pp. 137-178). Alexandria, VA: U.S. Army Research Institute for

the Behavioral and Social Sciences.

Klein, S.B. (1991). Learning principles and applications. New York: McGraw-Hill.

20
Love, J. F. (1 998a). Joint targeting planning training guide (Research Product).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Manuscript in preparation.

Love, J. F. (1998b). How to produce training guides from battlefield functions

(Research Product). Alexandria, VA: U.S. Army Research Institute for the Behavioral

and Social Sciences. Manuscript in preparation.

Love, J. F. (1998c). Joint fires training guide for a corps JTF (Research Product). Alexandria,

VA: U.S. Army Research Institute for the Behavioral and Social Sciences. Manuscript

in preparation.

Loftus, G. (1985). Evaluating forgetting curves. Journal of Experimental Psychology (Learning,

Memory, and Cognition), 11(2), 397-406.

Meliza, L. L. & Tan, S. C. (1996). SiMNET Unit Performance Assessment System (UPAS)

version 2.5 user's Guide (Research Product 96-05). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences.

(AD-A28 5805)

Mirabella, A., Macpherson, D.H., & Patterson, C.A. (1989). Application of training research

literature to maintenance performance training. (ARI Research Report 1527).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Mirabella, A., Sticha, P., & Morrison, J. (1997). Assessment of user reactions to the

multi-service distributed training testbed (MDT2) system (Technical Report 1061).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Scott, T.D. (1983). Tactical engagement simulation after action review guidebook. (ARI

Research Product 83-13). Alexandria, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences

Scott, T.D. (1984). How to evaluate unit performance (ARI Research Product 84-14).

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Sulzen, R. H. (1997). National Training Center research element Fort Irwin, California

1986-1996. (Research Special Report 30). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. (AD-A32 6056)

Travers, R.M.W., Van Wagenen, R.K., Haygood, D.H., & McCormick, M. (1964). Learning as a

consequence of the learner's task involvement under different conditions of feedback.

Journal of Educational Psychology, 55, 167-173.

21
U.S. Army Research Institute (1992). Determinants of effective unit performance, 'focus issue"

(ARI Newsletter, October 1992, Volume 10). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences.

Weinstein, C. E., Goetz, E. T., & Alexander, P. A. (1988). Learning and study strategies.

San Diego, CA: Academic Press.

Wolfle, D. (1951). Training. In Stevens, S. S., (Ed.), Handbook of experimental psychology.

New York: John Wiley.

Zazanis, M. M. & Lappin, M. S. (1998). Predicting perfbrmance ratings using motivational

antecedents. (Technical Report 1077). Alexandria, VA: U.S. Army Research Institute

for the Behavioral and Social Sciences.

22
APPENDIX A

TARGET AUDIENCE FOR SELF-ASSESSMENT PILOT TEST

OPERATIONS (OPS) RANK NOTES

Operations Officer (S-3) MAJ

Counter fire Officer CPT

Operations Sgt SGM

Chemical Officer CPT

Clerk/RTO SPC

Reconnaissance Survey Officer CPT

Surveyor SFC

FIRE CONTROL ELEMENT (FCE) RANK NOTES

Fire Control Officer CPT

Fire Control NCO SFC

Computer Operator SSG

Tacfire Equipment Spec PFC

Tacfire Equipment Spec PFC

Tacfire Equipment Spec PFC

TARGETING ELEMENT (TGTG) RANK NOTES

Intel Officer CPT

Processing Chief SSG

Target Processing NCO SGT

Target Processing Spec PFC

Target Processing Spec SPC

Intel SGT (NCOIC) MSG

Intel Analyst SSG

Intel Analyst SGT

Intel Analyst PFC

DTAC FIRE SUPPORT ELEMENT RANK NOTES

(FSE)

Assistant FS Coordinator CPT

Fire Support SGT SFC

Fire Support NCO SGT

Clerk SPC

Fire Support Specialist SPC

A-1
APPENDIX B

COMPONENTS OF THE SELF-ASSESSMENT PACKAGE

Package Page

Background and Purpose of Tools ..................................... B-2

Operations Section ............................................. B-OPS-1

Fire Control Element ........................................... B-FCE-1

Targeting Section .............................................. B-TGT-1

DTAC Fire Support Element ..................................... B-FSE-l

B-1
Self-Assessment Tools for

Division Artillery Staff Sections

1. Background. U.S. Army Research Institute (ARI) has been developing a concept for

measuring performance during training of high-level (Joint Task Force and above) joint staffs.

For staff sections at these levels, it is difficult to obtain outside observer/

controllers to provide meaningful feedback during the training of these sections. ARI's basic

premise is that staff sections possess the expertise to assess their own performance, and that if

provided the proper self-assessment tools, they can conduct effective mini-AARs to improve

their training. This concept was tested during a major CINCCENT exercise in 1997, with very

favorable results.

2. Purpose. ARI would like to try out a set of self-assessment tools developed for use by the

key staff sections involved with the fire support operations of a division artillery. This is

completely voluntary on the part of the unit.

a. Enclosed are four sets of tools each with an introduction to the self-assessment tools

themselves, instructions for their use, and diagrams of inputs, outputs, and processes. There is a

set for the operations section, the targeting section, the fire control element, and the fire support

element. Many of the functions these sections perform are process oriented, as opposed to

outcome oriented. To perform the processes involves, to a large degree, the performance of

military "art" based on collective judgment. This is something difficult to measure objectively.

It was, therefore, necessary to derive mainly subjective performance measurement tools.

b. The self-assessment tools are for the use of the senior officer in charge of each staff section

to conduct a 10-15 minute mini-AAR (after action review) at the conclusion of each iteration or

phase of the section's fire support processes. This allows him to give and receive feedback on

how well the process was performed and on what needs to be fixed or improved for the next

iteration. The self-assessment tools are in the form of questions to ask oneself and the section

members. Their purpose is to:

1) Provide a reminder and a format to conduct on-going, semi-structured reviews of

what's being done, for what purpose, and how well it's being done.

2) Provide a record of performance progress throughout the duration of a training

exercise or during actual combat.

3) Provide a record of lessons learned and critical events for inclusion into the overall

DivArty AAR.

4) Identify tasks which require additional training or need added emphasis in future

iterations of the process being performed and in future training exercises.

3. At the conclusion of the training exercise, ARI would like to receive feedback from the staff

sections on the self-assessment process in general and on the individual tools in particular.

B-2
Training Objective Self-Assessment Tools for

Division Artillery Operations Section

Mini-AARs

1. Training Objective. Train the Operations Section in planning and controlling the

employment of FA unit operations. Major tasks involved are:

a. Prepare the FA support plan.

b. Direct and coordinate FA unit operations.

c. Plan employment of FA assets.

d. Collect and manage FA tactical information.

2. Self-Assessment and Feedback (Background/Purpose).

a. One of the major challenges to assessing training performance and giving meaningful

feedback is finding the right people with the right expertise to conduct the assessment. At lower

echelons, those personnel can be found in like units or in higher echelon units and can be tasked

to perform assessment and feedback for other units' training. But for a division, it is often times

impractical to task like or higher units outside of the division for personnel. Additionally, during

actual combat or for training at times other than during a major exercise, expert

observers/controllers/trainers will not be present to provide the necessary feedback for

improvement. So where does one find the required expertise? It can be found within the unit's

own staff personnel. Therefore, the training performance measures and feedback methods

contained in this guide were designed for self-assessment by personnel in those division artillery

staff elements involved with fire support operations.

b. Many of the tasks performed by DivArty staff elements in fire support operations are

process oriented, as opposed to outcome oriented (see page OPS-7). To perform the processes

involves, to a large degree, the performance of military "art" based on collective judgment. This

is something difficult to measure objectively. It was, therefore, necessary to derive mainly

subjective performance measurement tools.

c. The Self-Assessment Tools are to be used by the senior officer in charge of each shift of

the operations section to conduct a 10-15 minute mini-AAR (after action review) for his shift at

the conclusion of each iteration or phase of its fire support processes. This allows him to give

and receive feedback on how well the process was performed and on what needs to be fixed or

improved for the next iteration. The Self-Assessment Tools:

1) Provide a reminder and a format to conduct on-going, semi-structured reviews of

what's being done, for what purpose, and how well it's being done.

2) Provide a record of performance progress throughout the duration of a training

exercise or during actual combat.

B-OPS-1
3) Provide a record of lessons learned for inclusion into the overall DivArty AAR.

4) Identify tasks which, require additional training or need added emphasis in future

iterations of the process being performed and in future training exercises.

3. Instructions. The attached Self-Assessment Tool (pages OPS-3 and OPS-4) for the

operations section (OPS) is in the form of questions for the senior officer in charge of each OPS

shift to ask himself and the members of his shift while conducting a mini-AAR prior to the

conclusion of each shift or after a major process has been accomplished. Pages OPS-5 and OPS6 are provided as a ready reference and show input/output flow for the DivArty as a whole and

for the OPS section in particular.

1) The date/time and person conducting the mini-AAR should be filled in, so that a

record of progress can be maintained, and any future questions can be directed to the

appropriate person.

2) Any of the "Products/Outputs" which were developed should be rated as to how

effective they are.

3) Each of the questions under "Assessment of Processes" should be discussed and

rated. Any examples of significant or critical events arising from these discussions

should be written in the "Comments" section on the back or attached on separate sheets.

4) Lessons learned, fixes to problems, or other relevant information discovered during

the mini-AAR should be addressed with the on-coming shift.

B-OPS-2
DivArty OPS Section

Date, Time, Prepared by:

_ _ _/

Self-Assessment Checklist

0

Assessment of ProductslOutputs

Field Artillery Support Plan

DivArty commander's guidance

Movement orders

Mission changes

Assessment of Processes

How well were FA assets allocated and prioritized?

Were changes in allocation projected over time lAW div OPORD?

Were instructions for timely detection/attack of HPTs given?

Were requirements for positioning of all FA assets determined?

Were make-up of basic loads & the RSR determined?

Were locations of ASPs and ATPs disseminated?

Were cmdr's attack guidance & tgt select stnd determined?

Was effective control of fire control operations maintained?

How well was the coordination of target acquisition directed?

How well were the planning and execution of survey conducted?

How well was the movement of units managed?

Was DivArty cmdr kept informed and guidance sought from him?

How effective was commo w/higher, lower, and adjacent units?

B-OPS-3
DivArty OPS Section

(Continued)

4,

Assessment of Processes (Continued)

How well were appropriate inputs and documents used?

Were appropriate personnel (external/internal) consulted?

How well did key personnel provide relevant input?

Were doctrinal principles and cmdr's philosophies adhered to?

Were matters sufficiently discussed? Alternatives addressed?

Was each output consistent with cmdr's guidance/intent?

Were outputs produced and distributed in a timely manner?

Was a "mini-AAR" conducted w/in the S-3 Ops after each activity?

Comments (fixes/lessons learnedlcritical incidents/examples):

B-OPS-4
0119 C4

4-0 Eu ___ __

~~ CL. 0 .

0 Pm 0

-Cc: -g

0~~~~ u________ 04______

~~i2

~~UC

-- u ~ um E-

~F

E4I - ________cc 0 o U ..
lp L

0*

& E

1 72 < 63 uý .2 0 o'-

ElE

0Cu

CCCCo

04 E c.o

-41=0t 2 C) 

w5.
-r. -"A

0. 0 )) & 

0 dco E 0 0 

u o +

0-E

~uu)

4-4e

-4 -op.- '.. u~~4 "t 0 -) 0

rj d CZ 7 C) -

c 0 C41 0-

C/) LgCs C

03~

o ~ C~b-o 0 (1

V) InCi

C)0 rJ

0 tQ-o P 

En C, 0 4, 

C)~ 1- 0 c1 *. CN 

;.T0 S C -c 

Cý3 -~ C 4-

.0* 0 r.) *C4

~ cz O Un~) C 0

C)l

-- 4

u Cd -o I I I
Training Objective Self-Assessment Tools for

Division Artillery Fire Control Element

Mini-AARs

1. Training Objective. Train the FCE Section in providing timely, accurate, and

effective technical fire solutions and tactical fire control of FA fires. Major tasks

involved are:

a. Control and coordinate FA fire missions.

b. Prepare schedules of fire.

c. Control target attack.

d. Provide counterfire.

e. Perform and provide target value analysis.

2. Self-Assessment and Feedback (Background/Purpose).

a. One of the major challenges to assessing training performance and giving

meaningful feedback is finding the right people with the right expertise to conduct the

assessment. At lower echelons, those personnel can be found in like units or in higher

echelon units and can be tasked to perform assessment and feedback for other units'

training. But for a division, it is often times impractical to task like or higher units

outside of the division for personnel. Additionally, during actual combat or for training

at times other than during a major joint exercise, expert observers/controllers/trainers will

not be present to provide the necessary feedback for improvement. So where does one

find the required expertise? It can be found within the unit's own staff personnel.

Therefore, the training performance measures and feedback methods contained in this

guide were designed for self-assessment by personnel in those division artillery staff

elements involved with fire support operations.

b. Many of the tasks performed by DivArty staff elements in fire support operations

are process oriented, as opposed to outcome oriented (see page FCE-7). To perform the

processes involves, to a large degree, the performance of military "art" based on

collective judgment. This is something difficult to measure objectively. It was, therefore,

necessary to derive mainly subjective performance measurement tools.

c. The Self-Assessment Tools are to be used by the senior officer in charge of each

FCE shift to conduct a 10-15 minute mini-AAR (after action review) for his shift at the

conclusion of each iteration or phase of its fire support processes. This allows him to

give and receive feedback on how well the process was performed and on what needs to

be fixed or improved for the next iteration. The Self-Assessment Tools:

1) Provide a reminder and a format to conduct on-going, semi-structured reviews

of what's being done, for what purpose, and how well it's being done.

B-FCE-1
2) Provide a record of performance progress throughout the duration of a training

exercise or during actual combat.

3) Provide a record of lessons learned for inclusion into the overall DivArty

AAR.

4) Identify tasks which require additional training or need added emphasis in

future iterations of the process being performed and in future training exercises.

3. Instructions. The attached Self-Assessment Tool (pages FCE-3 and FCE-4) for the

fire control element (FCE) is in the form of questions for the senior officer in charge of

each FCE shift to ask himself and the members of his shift while conducting a mini-AAR

prior to the conclusion of each shift or after a major process has been accomplished.

Pages FCE-5 and FCE-6 are provided as a ready reference and show input/output flow for

the DivArty as a whole and for the FCE section in particular.

1) The date/time and person conducting the mini-AAR should be filled in, so that

a record of progress can be maintained, and any future questions can be directed

to the appropriate person.

2) Any of the "Products/Outputs" which were developed should be rated as to

how effective they are.

3) Each of the questions under "Assessment of Processes" should be discussed

and rated. Any examples of significant or critical events arising from these

discussions should be written in the "Comments" section on the back or attached

on separate sheets.

4) Lessons learned, fixes to problems, or other relevant information discovered

during the mini-AAR should be addressed with the on-coming shift.

B-FCE-2
DivArty Fire Control Element (FCE)

Date. Time, Prepare b:

Self-Assessment Checklist o

Assessment of Prod ucts/Outputs

Schedules of fire

Target lists

Fire plans

Requests for clearance of fires

Nominations of targets to be engaged by non-DivArty assets

Fire missions

Assessment of Processes

Were fire msns, tgt lists, & schedules sent to appropriate units?

Were tgts which could not be engaged passed to FSE?

Were fires coordinated with adjacent units?

Were ammo files updated upon completion of fire msns?

Were clearances requested from FSE when necessary?

Were JAAT fire support msns coordinated w/FSE?

Was fire unit status monitored, updated, and disseminated?

Was FLOT continuously updated?

Were fire msn tactical solutions disseminated to appropriate units?

Were counterfire templates input to counterfire schedules?

Were requests for EW, CAS, and NGF support passed to FSE?

Were reinforcing/attached FA brigades kept informed/updated?

B-FCE-3
DivArty Fire Control Element (FCE)

(Continued)

Assessment of Processes (Continued)

How well were appropriate inputs and documents used?

Were appropriate personnel (external/internal) consulted?

How well did key personnel provide relevant input?

Were doctrinal principles and cmdr's philosophies adhered to?

Were matters sufficiently discussed? Alternatives addressed?

Was each output consistent with cmdr's guidance/intent?

Were outputs produced and distributed in a timely manner?

Was a "mini-AAR" conducted w/in the FCE after each activity?

Comments (fixes/lessons learned/critical incidents/examples):

B-FCE-4
E~ 0 0 0 EI

V E 4-4. 0.O2 - 0 0 0

D~u 

m A~

U. E .

_ _ _ _ I!IE _ _ _ 

m E = 0o E m M4

0 1

066 

0(40

0 -f. 0 1,

I~ ~ ~ t - tE iZ F- a

jooC , W- J.u .. 0 4cI,
1 0 ) 0 

+10j

4) 0 4

-400)0004)

4)0~ !3~

E~ Z

iT 4)0.

iz)~) 0 4)- '~~~FC-4)) '0,)..
OD 0

oo >

00

C~ 0

z bl)

14-44-

IS 0

'0 0.

4- 0

0

PC

0 _

+, C)

> 0un 0 M ~ 1

04 uc ý cu

0r

C/)

O,

C)

- ) >
Training Objective Self-Assessment Tools for

Division Artillery Targeting Section

Mini-AARs

1. Training Objective. Train the Targeting Section in how to accomplish the decidedetect-deliver-assess process. Major tasks involved are:

a. Conduct intelligence preparation of the battlefield.

b. Develop targets and potential targets.

c. Predict and produce targets.

d. Coordinate target acquisition and counterfire.

2. Self-Assessment and Feedback (Background/Purpose).

a. One of the major challenges to assessing training performance and giving

meaningful feedback is finding the right people with the right expertise to conduct the

assessment. At lower echelons, those personnel can be found in like units or in higher

echelon units and can be tasked to perform assessment and feedback for other units'

training. But for a division, it is often times impractical to task like or higher units

outside of the division for personnel. Additionally, during actual combat or for training

at times other than during a major joint exercise, expert observers/controllers/trainers will

not be present to provide the necessary feedback for improvement. So where does one

find the required expertise? It can be found within the unit's own staff personnel.

Therefore, the training performance measures and feedback methods contained in this

guide were designed for self-assessment by personnel in those division artillery staff

elements involved with fire support operations.

b. Many of the tasks performed by DivArty staff elements in fire support operations

are process oriented, as opposed to outcome oriented (see page TGT-7). To perform the

processes involves, to a large degree, the performance of military "art" based on

collective judgment. This is something difficult to measure objectively. It was, therefore,

necessary to derive mainly subjective performance measurement tools.

c. The Self-Assessment Tools are to be used by the senior officer in charge of each

shift of the targeting section to conduct a 10-15 minute mini-AAR (after action review)

for his shift at the conclusion of each iteration or phase of its fire support processes. This

allows him to give and receive feedback on how well the process was performed and on

what needs to be fixed or improved for the next iteration. The Self-Assessment Tools:

1) Provide a reminder and a format to conduct on-going, semi-structured reviews

of what's being done, for what purpose, and how well it's being done.

2) Provide a record of performance progress throughout the duration of a training

exercise or during actual combat.

B-TGT-1
3) Provide a record of lessons learned for inclusion into the overall DivArty

AAR.

4) Identify tasks which require additional training or need added emphasis in

future iterations of the process being performed and in future training exercises.

3. Instructions. The attached Self-Assessment Tool (pages TGT-3 and TGT-4) for the

targeting section (TGT) is in the form of questions for the senior officer in charge of each

TGT shift to ask himself and the members of his shift while conducting a mini-AAR prior

to the conclusion of each shift or after a major process has been accomplished. Pages

TGT-5 and TGT-6 are provided as a ready reference and show input/output flow for the

DivArty as a whole and for the TGT section in particular.

1) The date/time and person conducting the mini-AAR should be filled in, so that

a record of progress can be maintained, and any future questions can be directed

to the appropriate person.

2) Any of the "Products/Outputs" which were developed should be rated as to

how effective they are.

3) Each of the questions under "Assessment of Processes" should be discussed

and rated. Any examples of significant or critical events arising from these

discussions should be written in the "Comments" section on the back or attached

on separate sheets.

4) Lessons learned, fixes to problems, or other relevant information discovered

during the mini-AAR should be addressed with the on-coming shift.

B-TGT-2
DivArty Targeting Section

Date. Time, Prepared by:

Self-Assessment Checklist

Assessment of Products/Outputs

Counterfire templates

Intel data base

High Value Targets (HVTs)

Critical nodes

NAIs & TAIs

Intel estimates/annex

Target acquisition missions

Radar Deployment Orders (RDO)

Requests for additional intel collection assets

Assessment of Processes

How well were critical nodes, NAIs, TAIs, & HV-Is identified?

Were G2 & FSE consulted during FA IPB process?

How effective was development of targets and potential targets?

Were positioning and relocation of TA assets recommended to $3?

Did target value analysis determine high value targets (HVTs)?

Were high payoff targets (HPTs) recommended to S3 and FSE?

Were radar deployment orders effectively prepared?

Were counterfire templates prepared and passed to FCE?

Was BDA requested and BDA analysis conducted for enemy FA?

B-TGT-3
"DivArty Targeting Section

(Continued)

Assessment of Processes (Continued)

How well were appropriate inputs and documents used?

Were appropriate personnel (external/internal) consulted?

How well did key personnel provide relevant input?

Were doctrinal principles and cmdr's philosophies adhered to?

Were matters sufficiently discussed? Alternatives addressed?

Was each output consistent with cmdr's guidance/intent?

Were outputs produced and distributed in a timely manner?

Was a "mini-AAR" conducted w/in S2 Tgt'ing after each activity?

Comments (fixes/lessons learned/critical incidents/examples):

B-TGT-4
I - .

2. 

o•_ !n

42 424 t. ... . -_

2. U20E! 8

<42~~4242 ~ E ~22*24

E -0

S - "" 42 

j1I o -1 :• li

40 c0 4

S 

2. 

.. 

..

""o 

i_

0 ME

lz u u

e 2 I _m~

oz F2. 42 u) j-

.__ •o24. 

242

• <4 

- 4

- -. IEr E ~ •>b = 2 

* 2I
4 15, 

4) 0, 4C)

4)C4

a, LA

CC,0

TGT-6
'*0p

0 j VA -8 

ow*- = U- - 0

Ee F 0o

>~ *0

0d +ý 0.S Cu 4:; b

U) 0

C) cu 

4)0 0 -- Pt

-t4e 'eCcS g- a cd En C;

Cd0000

4.4

S S 0 4- 0L 0 - o En0
Training Objective Self-Assessment Tools for

Division Fire Support Element

Mini-AARs

1. Training Objective. Train the FSE Section in the planning, coordination,

integration, and synchronization of fire support operations. Major tasks involved are:

a. Conduct fire support coordination in support of ground operations.

b. Synchronize fire support operations.

c. Conduct fire support planning.

d. Coordinate target attack.

e. Process target attack.

2. Self-Assessment and Feedback (Background/Purpose).

a. One of the major challenges to assessing training performance and giving

meaningful feedback is finding the right people with the right expertise to conduct the

assessment. At lower echelons, those personnel can be found in like units or in higher

echelon units and can be tasked to perform assessment and feedback for other units'

training. But for a division, it is often times impractical to task like or higher units

outside of the division for personnel. Additionally, during actual combat or for training

at times other than during a major joint exercise, expert observers/controllers/trainers will

not be present to provide the necessary feedback for improvement. So where does one

find the required expertise? It can be found within the unit's own staff personnel.

Therefore, the training performance measures and feedback methods contained in this

guide were designed for self-assessment by personnel in those division artillery staff

elements involved with fire support operations.

b. Many of the tasks performed by DivArty staff elements in fire support operations

are process oriented, as opposed to outcome oriented (see page FSE-7). To perform the

processes involves, to a large degree, the performance of military "art" based on

collective judgment. This is something difficult to measure objectively. It was, therefore,

necessary to derive mainly subjective performance measurement tools.

c. The Self-Assessment Tools are to be used by the senior officer in charge of each

FSE shift at both the DMAIN and DTAC to conduct a 10-15 minute mini-AAR (after

action review) for his shift at the conclusion of each iteration or phase of its fire support

processes. This allows him to give and receive feedback on how well the process was

performed and on what needs to be fixed or improved for the next iteration. The SelfAssessment Tools:

1) Provide a reminder and a format to conduct on-going, semi-structured reviews

of what's being done, for what purpose, and how well it's being done.

B-FSE-1
2) Provide a record of performance progress throughout the duration of a training

exercise or during actual combat.

3) Provide a record of lessons learned for inclusion into the overall DivArty

AAR.

4) Identify tasks which require additional training or need added emphasis in

future iterations of the process being performed and in future training exercises.

3. Instructions. The attached Self-Assessment Tool (pages FSE-3 and FSE-4) for the

fire support element (FSE) is in the form of questions for the senior officer in charge of

each FSE shift at both the DMAIN and DTAC to ask himself and the members of his

shift while conducting a mini-AAR prior to the conclusion of each shift or after a major

process has been accomplished. Pages FSE-5 and FSE-6 are provided as a ready

reference and show input/output flow for the DivArty as a whole and for the FSE section

in particular.

1) The date/time and person conducting the mini-AAR should be filled in, so that

a record of progress can be maintained, and any future questions can be directed

to the appropriate person.

2) Any of the "Products/Outputs" which were developed should be rated as to

how effective they are.

3) Each of the questions under "Assessment of Processes" should be discussed

and rated. Any examples of significant or critical events arising from these

discussions should be written in the "Comments" section on the back or attached

on separate sheets.

4) Lessons learned, fixes to problems, or other relevant information discovered

during the mini-AAR should be addressed with the on-coming shift.

B-FSE-2
DivArty Fire Support Element (FSE)

Date, Time, Prepared by:

Self-Assessment Checklist

Assessment of Products/Outputs

Fire support portion of division OPLAN/OPORD

Fire Support Annex to division OPLAN/OPORD

Fire missions

Close air support requests

Air interdiction requests

Cross boundary fires clearances

Changes to fire support coordination measures (FSCM)

Joint air attack team (JAAT) fire support

Assessment of Processes

Was FS planning conducted simultaneously w/maneuver planning?

Were all FS portions of Corps OPORD considered?

Did ALO provide necessary sortie information?

Were G-2's IPB products considered and passed to DivArty?

Was DivArty Cmdr's guidance solicited? ... used?

Did FS coordination support cmdr's intent/concept of operations?

Was FS planning and execution synchronized w/other forces?

Were attacks on tgt's coordinated and processed effectively?

Was coordination with higher & adjacent units maintained?

Was execution of FS matrix monitored and refined?

B-FSE-3
DivArty Fire Support Element (FSE)

(Continued)

10

Assessment of Processes (Continued)

How well were appropriate inputs and documents used?

Were appropriate personnel (external/internal) consulted?

How well did key personnel provide relevant input?

Were doctrinal principles and cmdr's philosophies adhered to?

Were matters sufficiently discussed? Alternatives addressed?

Was each output consistent with cmdres guidancelintent?

Were outputs produced and distributed in a timely manner?

Was a "mini-AAR" conducted w/in the FSE after each activity?

Comments (fixesllessons learned/critical incidents/examples):

B-FSE-4
30 -43ý 6 6' ~ 3-3o. 3. .3 3

00 0 E- 43z.E.E

V3 U~ E~4J4 i .

- 0 0 343 3 3

00 ~4340. 4to

PUI ) i >~ 

.~~.-. 4 Si> 

____ la

o~~6 c 430

cn 43 

o

0043

034 0 "I 43E

E 3 ot0 10 ~ 43 3

43434 04

0~~43H '~E 

0'cc4 E Z3

0~ I=3 >-4
40 Eo

0 t 2 E 

*00

.a i z -u F E -
04C1

o 0

0 ~ Q 

E 0n

Co"

0

~0

C,3 0

-ý u

0

00

ruJ;

u C t
APPENDIX C

EVALUATION OF SELF ASSESSMENT PACKAGE AND MINI-AARS

Directions: Please check one of the categories of agreement or disagreement for each item

below. Abbreviate the name of your cell (staff section): OPS, TGTG, FSE, FCE.

Need/value of mini-AAR/self-assessment

Date: Time: Cell (staff section): SSN (last 4 digits)

1. The mini-AAR/self-assessment approach provides useful trainhingfeedback to supplement the

AARs currently given (check one of the boxes in the following line). __________

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

2. The mini-AAR/self-assessment approach helped improve petrformance for me or my cell over

the 3-day exercise (check one of the following boxes ),.

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

3. My cell was able to pick out training problems that we might have o*verlookd without the

sei s assessment package (check one of the following boxes).

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

C-1
4. A self-assessment package similar to this is not currently available to my unit (check below).

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

5. A self-assessment package similar to this or an improved version would be useful in future

exercises (check one of the following boxes).

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

Features of self-assessment methods and tools (check one of the boxes below each item)

SSN (last 4 digits)

6. The instruction sheet iwas clear, :well organized, and had about the right amount of

information.

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

* 7. Assessment checklist items were useful as memory joggers and could be answered.

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

8. The input/output flow charts helped us to fill out the forms and conduct mini-AARs.

C-2
Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

9. The table of inputs, processes, and outputs helped us tofill out the forms and conduct miniAARs

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

10. Ifelt that I could respond accurately to the seif-assessmient tools

Strongly Moderately Agree Disagree Moderately Strongly Disagree

Agree Agree Disagree

Comments:

C-3