1 'Research Note 85-33 /

aO)/ (Vb

DEVELOPMENT OF GUIDELINES FOR REVISING

1%TRA INING PROGRAM DEFICIENCIES IN

CO AIR DEFENSE TRAINING

Calspan Corporation

for

Contracting Officer's Representative

Richard J. Carter

ARI Field Unit at Fort Bliss, Texas

Michael H. Strub, Chief

a..- SYSTEMS RESEARCH LABORATORY

LA]

Jerrold M. Levine, Director DT1LC -

I'- ELECTE MAY 16 1986

U. S. Army

Research Institute for the Behavioral and Social Sciences

March 1985

Anproved for public release; distribintion unlimited.

86 5 16 086.

.. . .. .. . -j ,, ., ., _,-a :.. r . ,. . , i .. . . . ..- . . . - , _. ,, , *,- - ,. *. ,. - *
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

L. NEALE COSBY

EIDGAR M. JOHNSON Colonel. IN

Technical Director Commander

I 'T1S GRA&I

T, TAB : ,"

i tification -___.__

.. .....

This ,epo., as submitted by the contractor, has been cleared for release to Defense Technical Information Center

(DTIC) to comply with regulatory requirements. It has been given no primary distribution other than to DTIC

and will be available only through DTIC or other reference services such as the National Technical Information

Service (NTIS). The views. opiniens, and/or findings contained in this report are those of the author(s) and

* should not be construej as an official opstmeni of the Army position, policy, or decision. unless so designated

* by other official documentation.

.-

. . . . .. . . . . . . . .. . . . i - i -- . -... - . 4*....., .4. ,
-vmJ t* * &4 mmAL_ C U 

SECURITY CLASSIFICATION OF THIS PAGE (1Rhon Data Entered)

REPORT DOCMENTATION PAGE READ INSTRUCTiONS

BEFORE COMPLETWG FORM

t. REPORT NUMBER 2. GOVT ACCESSION NO 3. RECIPIENT'S CATALOG NUMBER

ARI Research Note 85-33

4. TITLE (and Sublild) S. TYPE OF REPORT & PERIOD COVERED

DEVELOPMENT OF GUIDELINES FOR REVISING Final Report

TRAINING PROGRAM DEFICIENCIES IN Nov. 81- Jan. 83

AIR DEFENSE TRAINING 6. PERFORMING ORG. REPORT NUMBER

7. AUTOR(.) S. CONTRACT OR GRANT NUMBERMa

MDA 903-80-C-0579

Peter S. Winne

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK

AREA & WORK UNIT NUMBERS

Calspan Corporation, 2Q263739A793

Advanced Technology Center, 3227 52A

P.O. Box 400 . Buffalo NY 14225 11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

US Army Research Institute for the Behavioral March 1985

and Social Sciences, 5001 Eisenhower Avenue, ,S. NUMBEROPPAGES

Alexandria, VA 2233-5600 31 14. MONITORING AGENCY NAME & ADORESS(lf diffrent frm Ccnmrollbn Office) IS. SECURITY CLASS. (of this report)

Unclassified

IS&. DECLASSI FICATION/ DOWNGRADING SCHEDULE n/a

16. DISTRIBUTION STATEMENT (of thla Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abtracl atered In Block 20, If dlfferent fro i Report)

II. SUPPLEMENTARY NOTES

This project was monitored technically by Gary G. Sarli and Richard J. Carter,

ARI Field Unit at Fort Bliss, Texas.

Richard J. Carter, contracting officer's representative.

19. KEY WORDS (Conlnue on rovere* aide It necessary mad Identify by block number)

Tra- ng Air Defense)' Tests

Training Effectiveness Instructions; T-*Operational Tests;

"3 Training evaluation, Instructors; Instructional Sstems

Training Effectiveness Evaluatien Product Evaluation. "Idevelopment.,

Process Lvaluation, Air Dffene Training. 2& ANTRACT' (Cat a 0 revere st N R0 m and Identify by block number)

-The development of a User's Guide for Revising Training Program Deficiencies

for Army Air Defense Training is described. The guidelines were designed to

complement the Training Effectiveness Evaluation (TEE) methodology which was

developed during the first year of this research effort. The report consists of

a description of the TEE system and of the literature in training program

evaluation which was used as a baseline for this project.>The content of the

revision guideline is outlined.

(over)

D ,O 1473 EMT100 OFP NOV OSOLETEII" '05 

SECURITY CLASSIFICATION OF THIS PAG E (When Data Entered9

• . .1
SECURITY CLASSIFICATION OF THIS PAGEIWlmm Date Ent.'.d)-

ARI Research Note 85-33

20. (continued)

The User's Guide for Revising Training Program Deficiencies presents a

detailed set of 20 problems which could be discovered during a TEE. Each

problem is associated with specific worksheets and questions for the TEE

system. Specific solutions for correcting each deficiency are discussed.

UNLSSFE

SECUITY LASSFICAION F 'IS PGE("n Do Ent.od
DEVELOPMENT OF GUIDELINES FOR REVISING

, TRAINING PROGRAM DEFICIENCIES IN

AIR DEFENSE TRAINLNG

,I BRIEF

I Requirement:

To develop a methodology and user's guide for correcting deficiencies

in Air Defense training packages that are discovered as a result of training

effectiveness evaluations (TEEs).""

.1 Procedure:

The guidelines for revising training program deficiencies were

designed in order to provide training analysts with solutions for correcting

problems in training identified during a TEE. Using prior developmental work

by the Army Research Institute Field Unit Fort Knox in the area of Training

Program Evaluation as the baseline, this effort involved three steps:

(1) Identify discrepancies in training program processes and

*f I products on the basis of ratings to the questions included on the TEE;

(2) Trace discrepancies to their underlying causes; and

(3) Recommend specific modifications to processes and products

which would increase training effectiveness.

The issues addressed by the Revision Guidelines directly correspond

to the training program components incorporated in the TEE. The TEE methodology

provides a mechanism for analyzing training processes (e.g., how training is

conducted), which was the focus of the earlier Harless Guidelines, and also

for evaluating training products (e.g., the design of instruction), which had

not been a part of training evaluation methodology. The Revision Guidelines

':1 provide specific solutions for correcting problems in these areas.

In format, the guidelines follow the general scheme used by the

job aid for modifying training programs(Research Product 81-17; Iristiansen,

1981). A total of 29 problem headings were categorized under four subdomains

of training, which were (1) Training Environment, (2) Presentation, (3) Practice/Demonstration, and (4) Testing. Each problem was defined, related to

specific TEE worksheets and questions, and discussed in terms of the specific

actions needed to correct it.

:................. .........
Findings:

The TEE methodology and companion revision guidelines represent a

broader and more complete evaluation methodology than was previously available

for identifying and correcting shortcomings in Army Air Defense packages.

The TEE system is fully documented in the TEE User's Guide, and was shown to

be applicable to Air Defense training processes and products during a preliminary validation. The Revision Guidelines, while they have not been tested

empirically, correspond closely to the TEE in subject content matter.

Utilization of Findings:

The methodologies developed for evaluating and revising training

program components can potentially contribute to the more effective use of

training by specifying deficient areas and the means for correcting discrepancies. When applied to ongoing Air Defense training packages or training for

operational tests (OTs), they provide an important input to design considerations

and the conduct of training.

I :

I

iv

........ ..... ,
I,

TABLE OF CONTENTS

Section Page

I INTRODUCTION . ....... ... 1

[Baunk gr...o u .... nd.... 1

Purposeand Scope ,, ........... . . .. . . . 2

II DEVELOPMENT OF GUIDELINES FOR TRAINING

PROGRAM REVISION ...................... . 4

IThe TEE Methodology ............ .-. . . . . .. 5

The Baseline for the Revision Guidelines ...... 6

III DESCRIPTION OF THE REVISION GUIDELINES. ...... 12

IV SUMMARY AND CONCLUSIONS .......... ........ 16

V REFERENCES ... ............... ........ 18

APPENDIX Short Forms of the Master List Questions

I :11

;:4.t

V '-

:1- . ", ', S % ,'"'"'.,.-,,-'.-,",. . ."" . . ."."- "-" . ,'-" •"-+- • "-• . . - ... ' ... ' . ' " . " . . . . ."•'• " -"

. r._,r_ t'.<.]<' '.."q" ' t'..'." .................................................................................... "..."...-."."....."..."-.
Section I

INTRODUCTION

,,L Background

I aCurrently the Army is engaged in the development and/or fielding of

four new Air Defense systems - STINGER, ROLAND, SGT YORK, and PATRIOT. The

design and implementation of these systems offers great potential for increasing

the Army's capability to provide effective Air Defense. Recognizing that

the new weapons will only be as effective as the soldiers who man them, p

[i however, the Army requires that programs and material to train or retrain

soldiers to these system be implemented concurrently with the fielding of

each system. TRADOC Reg 350-7 [Not referenced] provides Army policy governing

a systems approach to training. It prescribes in generic terms the minimum

requirements of what must be done to insure the systematic evaluation, analysis,

I design, development, and implementation of training programs and training

support materials. The TRADOC model for a systems approach to training

incorporates the same concept as the Interservice Procedures for Instructional

Systems Development model, commonly known as the ISD model, but is purposely

expressed in a non-linear fashion to demonstrate the continuing nature of a

systems approach and the interdependence of its process. It also allows for

entry at any point as the result of evaluation.

The Air Defense community has expressed the need for methodologies

II

to evaluate the training programs for these systems during the operational

test (OT) phase of the developmental cycle. These needs focused on three

specific concerns.

1. Methodologies for conducting training effectiveness evaluations

(TEE) of Air Defense training packages during the OT phase;

2. Materials and Methods for performing training effectiveness

testing (TET) of Air Defense training devices during the validation, verification and system integration phases; and

3. Methodologies for applying feedback information from TEE's and

TET to improve training packages and devices.

The Army Research Institute has undertaken to address each of these

needs through support of research in the development of materials, methodologies

and procedures for conducting TEEs and TETs. Specifically, the development

of a TEE methodology was addressed by research performed by Calspan Corporation's

%" "~ .i_+... ~ .. ._.......-i,.."...".-."..S> '. l "".- -". -,. .. ' .
II

Advanced Technology Center under the technical direction of ARI's Field Unit

at Fort Bliss (ARIFUFB). This effort, performed under Contract Number

lMDA903-80-C-0579 during the period August 1980 - October 1981, is documented *[ in the Year 1 final report Development of Guidelines for Conducting Training Effectiveness Evaluation of Air Defense Training (Fishburne, Rolnick, & Larsen,

1981).

The first year's effort entailed development of methodologies for

evaluating training packages during OT that was consistent with Instructional

Systems Development (ISD) procedures and designing procedures for the structural observation and assessment of the training process. The development

of the TEE and resultant outputs (e.g., TEE materials and User's Guide) are

discussed below in greater detail.

1The second year of the research focused on the third need identified

by the Air Defense community noted above, namely, the development of a methodology for applying the results of a TEE in the improvement of training programs.

In addition, the user's guidelines for conducting TEE's were revised and refined.

The efforts and products of Calspan's research are documented in this report.

Purpose and Scope

:1 The overall objectives of the research were (1) to update and revise

existing methodologies for evaluating Army Air Defense packages and (2) to

I develop a set of guidelines for revising training discrepancies identified by

a TEE. The purpose of the first year's research was to validate and refine

- [-the Harless Guidelines (1980) for training program evaluation. As reported in

the first year's report, the researchers conducted a preliminary validation of

1' the Harless Guidelines. Based on the findings, the TEE Guidelines were

Ideveloped to broaden the earlier evaluation job aid by incorporating product

and courseware evaluation.

The purpose of the second year research was to extend the TEE

methodology to include the revision process. The primary assumption underlying the effort was, simply, that evaluation is pointless unless action is

1 2

. S" *. .. 9. .".. . .
taken to correct the deficiencies and discrepancies revealed by that process.

Based on earlier work in training program revisions, notably, the Seville

methodology (Spears, Maxey, and Roush, 1980) and Kristiansen's (1981) job aid

for modifying training, the approach was taken to trace symptoms of discrepancies

in training products and procedures directly to their causes and to specify

jdetailed recommendations for modifying them. The effort is broader than

earlier methodologies with respect to the detail of specific solutions and

the scope of deficiencies addressed, especially in the areas of product and

courseware revision. The major deliverable from this effort is a set of

guidelines for applying feedback information from a TEE in order to improve

training packages.

I3

t

:o

4 3.

".°'-

".,-I -.- .) '- . . .-" -"-"," , -. -.,-,i.,-" ' "."- -" .-.,.-,-' - 2 " - . . . .". , "" -"" "-"."1 ."- ."•"- -' "
. -. p. - . -. . L " . - . - .- %

0.

, Section II

DEVELOPMENT OF GUIDELINES FOR TRAINING PROGRAM REVISIONS

A number of sources were used in the development of the Guidelines

1for Revising Training Program Deficiencies. Since the revision guidelines

* were designed as a companion manual to guidelines for conducting TEEs for

TEE analysts, to be initiated in response to a TEE, the detailed problems

and associated solutions directly correspond to the domain covered by the

Guidelines for conducting a TEE. | - Many of the documents referenced in TEE development were useful :1 in both the conceptual and technical aspects of the developmental process

during the TEE development phase. These were used to provide standards for

rating the adequacy of training products and procedures. During the development of the Revision Guidelines, these sources provided the basic material

for specifying the solutions.

In addition, the issues addressed by the Seville Revision Methodology

(Spears, Maxey & Roush, 1980), Kristiansen's (1980) working paper in training

program evaluation (TPE) modification, and especially the content and structure

of Kristiansen's (1981) job aid for modifying ineffective or inefficient

training programs, directly influenced the final form of the guidelines.

I.

' . ". . '%- " -' .f ,.. ... '''-. € ". - . ., - .. . - * . , . . . ** .*. . .. -' "'
I

The TEE Methodology 0

As background for the development of the Revision Methodology,

it is important to understand the basic components and procedures involved

£ in a TEE. A complete description and rationale for TEE is contained in

:1 the final report for the first year (1981).

The design of TEE methodology involved the identification of the

purposes and scope of evaluating training effectiveness, and the development

of a methodology and a set of job aids for conducting the TEE. The Calspan

effort focused on building an assessment tool on the baseline established

by the Harless Guidelines (1980). Examination of the Guidelines indicated

that they were adequate with respect to the process evaluation component, the

conduct of training, but did not address several components considered

important in a broader approach to training evaluation. Foremost was the

exclusion of a product evaluation methodology, involving an analysis of

training design based on training documentation. Therefore, the Instructional

[,1 Quality Inventory (IQI) (Wulfeck et al., 1978; Ellis et al., 1978, 1979;

Fredericks, 1980) was used as a basis for developing the TEE product evaluation

component.

Development of the TEE procedures also followed the general strucI ture of the Harless Guidelines, but with several important differences. A

master list of 94 evaluation questions was developed to identify performance

discrepancies and to enable comparison of training products to specified

standards. The major advantage of the list of questions is its flexibility

for allowing a TEE analyst to select applicable areas for evaluation. The

procedure for identifying discrepancies by comparing training components to

specified standards also facilitates the revision process by providing a

direct approach for tracing performance discrepancies to their probable causes.

*I The Appendix of this report contains the short form of the Master Questions.

Other procedural considerations involved design of a systematic

method for carrying out a TEE from its initial request and planning phase

to the documentation of the TEE effort and preparation of the final report.

1 5

t*....... • '- .. -Z.Z.'.','' _'# .'' , .'2 2._ ' '.' 2 '2 " ,""€ ."' > ". " ". " " " ". " " " " "" "• " '" " " . ." ," ". '.
TEE training materials and job aids were designed to assist the TEE analyst

and other data collectors. A user's manual for the conduct of a TEE (1981)

and a separate data collector's handbook (1981) for assisting personnel were

developed to facilitate the TEE process.

Baseline for the Revision Guidelines

For the most part, the conceptual and procedural approach for

designing the Guidelines for Revising Training Program Deficiencies was

defined in the process of developing the TEE system. Relevant issues,

decision points, and coverage followed directly from the TEE itself, and

indirectly from the materials used to design the TEE. However, as stated

earlier, three sources contributed substantially to the concepts and procedures

related to the Revision Methodology. These are (1) the Seville Project Report

80-8 Methodology for Correcting Deficiencies in Training Programs (Spears,

Maxey & Roush, 1980); (2) Kristiansen's (1980) Training Program Evaluation.

A Job Aid for Modifyins Ineffective or Inefficient Training (Working Paper

FKFU 80-8); (3) Kristiansen's (1981) A Job Aid for Modifying Ineffective or

Inefficient Training Programs (Research Product 81-17).

The Seville project sought to extend the Harless Guidelines by

developing procedures for correcting training deficiencies discovered in a

TPE. Their major objective was to provide a set of guidelines or principles

which could be used by training evaluators to direct the revision process.

-.1 Although the focus was on the 200 tank (as was the Harless Guidelines'), the

researchers sought to make the guidance general enough to have applicability

to any training program where skilled performance was involved. The scope

of the project was to specify principles for correcting (a) deficiencies

identified by the Harless Guidelines and traceable to specifiable instructional

sources through systematic observation; and (b) those attributable to

administration of training described in the Harless Guidelines as "non-skill

related."

The methodology focused on systematic observation and analysis of

instructional processes and their effects on the daily achievements of trainees.

I '6

*x*-. -' . *.*
Practice - the amount, type, and schedule - was postulated to be the underi lying determinant of skill development and to provide opportunity for the

constructive application of guidance and feedback. Fifteen principles concerned

with practice, guidance and feedback, provide the conceptual framework for

describing a good training program. These principles are as follows:

1. A person learns to do whatever he practices in the learning

situation.

2. The total amount of practice should be sufficient to Jb I

ensure " i

stable criterion performance. 1 3. The length and frequency of practice sessions should be

adapted to the task being taught, the experience of the learner, and the

opportunities for performing the task in the future.

4. Progress should be made during practice.

5. The learner should be intensively involved during practice.

6. Guidance is needed when the learner cannot determine a correct

action on his own.

7. Guidance should direct the learner's attention to individual

cues and specific performance requirements.

8. Verbal knowledge from classroom instruction, text material

memory aids, and job aids should be presented and their use practiced in

ways that capitalize on their guiding value.

9. Guidance should not be used when it is not needed.

10. Self-guidance should be a specific goal in all training.

11. Feedback should be specific to the response.

12. Feedback should occur while the learner still remembers clearly

what he did.

13. When the ongoing coordination of an action depends on conI tinuous feedback from each part of the action, no delay in feedback can be

tolerated.

14. Feedback should be provided in ways that help the learner see

when he is making progress. 1 15. When errors are made, or when there is little or no progress in

skill acquisition, the feedback should focus on correcting the difficulties,

not punishing the learner.

1. 7

"o-'. ". , * * ....................- -"" ' " " " " , ." ' , . . , , , . .. -- .
--- i - -- , T.-' V %-ji-, -- - -,..y, _ -j.,k %Kr. ril 7

Instructional deficiencies are evidenced through the systematic

observation of the training process from the Harless Guidelines. These

guidelines list more than twenty specific deficiencies in the Seville Report.

These deficiencies are characterized by five aspects of trainer behavior ,,.1

which signal problems in the instructional system. These "symptoms" of

instructional discrepancies are:

1 . No hands-on practice is provided.

2. Trainees do not know what to do in the presence of

relevant cues.

3. Performance shows undue variability from one trial to

another.

4. Trainees are unduly slow in performing a task.

5. There is too much dependence on job or memory aids.

Evidence of the presence for one or more of these symptoms suggests

that the particular component of training under observation fails to adhere

to one or more of the training principles listed above, and that the procedures

need to be modified in line with the underlying principle.

The Seville Report also addresses deficiencies in training programs

that are determined primarily by the training system design or administrative

management practices. Five categories of shortcomings are reviewed: (1) TndiJ cations of Need to Redesign Training Programs; (2) Training Support and

Facilitation; (3) Instructor Characteristics and Qualifications; (4) Management of Attitudes; and (5) Problems in Performance Testing.

e Indications of Need to Redesign Training Programs discusses 15

problems in training outcomes identified by the Harless report that involve

general shortcomings in the design of a program. For example, the deficiency

"Pertinent tasks are omitted from trainingr suggests that task analysis

procedures were not initially followed, and therefore, imply that general

revisions of the program are needed from top to bottom, requiring substantial "' "_-"

administrative support and resources.

8

..................- ....
I I.I .

e Training Support and Facilitation covers 12 factors which signal I an administrative neglect of training program quality, for example, through

adverse ratios of trainees to resources (instructors, equipment, space), lack

of facility communication or quality assurance practices, etc. Correction

of problems requires administrative attention and resolve, and some may in

addition necessitate additional resources.

* Instructor Characteristics and Qualifications addresses three

If problems, namely skill of the content area, skill in teaching, and attitude.

Judicious use of human resource management principles can resolve specific

problems.

0 Management of Attitudes lists four problems involving negative

attitudes or their underlying causes. Understanding attitude formation and

incorporating positive motivational techniques and reward systems are discussed as solutions.

9 Problems in Performance Testing deals with 16 Harless-identified

problems in the testing component. Representative examples are "Mismatch

between training and test conditions" and "Poor test administration." Careful

test design and standardization procedures, and consideration of the systematic

relations between training and testing are called for.

The Seville report provided an approach to training program modification which has widely influenced later researchers in the area. Either

* implicitly or explicitly, the principles and specific solutions documented

I in the methodology provided a conceptual baseline for guiding thought on

correcting deficient training processes. The report emphasized practice as Il the critical component of training, reliance on identifying and correcting

underlying causes of performance discrepancies, and most importantly, support

of the systematic approach to training development. Its principal limitations,

I. addressed by later research, include (i) lack of a clearcut methodology for

the evaluator to directly bridge between the discovered deficiency and its I correction, and (2) relative neglect of courseware deficiencies.

[ Kristiansen (1980) addressed the first of these constraints by

developing a job aid for correcting training problem deficiencies which

I ii 9 .
7 .

4 6 -I- V ,V Is ,%-7 77VITV .% VVTVITQVVL Mr

!N.

listed 35 problems corresponding to specific documented deficiencies from a

I TPE; each problem was associated directly with one or more modifications

to bring the training program in line with ISO specifications. This working

paper served as the prototype in terms of its problem-solution format for

his 1981 job aid and for the present guidelines.

Kristiansen's (1981) A Job Aid for Modifying Ineffective or %

Inefficient Training Programs (Research Product 18-17) is referenced by

ARI as the baseline for the present project. The job aid is one of four to

perform training program evaluations; its specific purpose is to assist

training analysts in identifying solutions for problems in training processes

discovered during a TPE.

Although Kristiansen (1981) follows the same general methodology as

the earlier working paper, the scope of problems and solutions was greatly

expanded in comparison to the working paper. Underlying the job aid is a

generic approach to evaluation which assumes that training programs should

incorporate a sequence of five activities:

1. Enabling knowledge

2. Demonstration

3. Part-Task Practice

4. Whole-Task Practice

5. Testing

The heart of the job aid is 21 problem areas which could potentially

be encountered in training environments. Each problem describes one or more

specific deficiencies identified by items from the Job Aid for the Structured

Observation of Training (Research Product 81-16). In addition, specific fixes

for the discrepancies are identified. Further, the problems are grouped

under five major headings of training. The five categories and the problems

I discussed are as follows:

1. Training Environment I a. Number of Instructors

1 10 J, l

,I

.. * * * '.*-* .. . . . -
b. Training Equipment

c. Training Site

d. Distractions

e. Training Duration

2. Lecture Events [ a. Training objectives and purpose

b. Terminology

3. Demonstration/Practice

a. Demonstration

b. Practice

c. Feedback

d. Job Aids

e. Progression of Training Events

4. General Observations

a. Implementation of Training Plan

b. Training Aids

c. Instructor Performance

d. Training Duration

5. Testing

a. Test Instructions

b. Test-training Match

c. Test Realism

d. Contamination

Ie. Feedback

The Kristiansen (1981) job aid provided a strong basis for the

present set of revision guidelines and much of the material has been incorporated in the present handbook. However, the job aid dealt explicitly only

with training processes and needed to be extended to both the design of

training and the quality of instructional courseware, following the efforts

of TEE development. In addition, team and collective training was not a .

major consideration in this job aid. Although the guidelines are for correcting

individual components, they are also relevant to collective training components.

Specific consideration of team practice needs to be addressed. Further

discussion of these points is contained in the first year's report.

.*'...I *** , . .I *~"* 

. . .: i i * ..... 
Section III

DESCRIPTION OF THE REVISION GUIDELINES

Problems and solutions are identified in the Guidelines for

Revising Training Deficiencies by categorizing them under four broad training

system components. This organization is largely similar to that used by

Kristiansen (1981) and generally represents distinguishable, interdependent

domains for assessing training. The broad topical areas are: (1) Training

Environment; (2) Presentation; (3) Practice/Demonstration; and (4) Testing.

Kristiansen also used a "General Observations" category; problems clustered

under that heading which were incorporated into the TEE were reclassified

under one of the four categories used here.

Within each of the broad headings, specific problems and their

solutions are detailed in three steps that follow the methodology of

Kristiansen:

Problem: the nature of the deficiency is defined in terms of the

observable discrepancies of the training event.

Evidence: sources for problem identification are listed. These

include the TEE worksheets which were used to document the deficiency

and the specific TEE items related to the defined problem. j Recommendation for modifying deficiency. For each of the specific

elements (TEE items), standards of adequate conditions and ways of achieving

41I standards through program modification are addressed.

Many of the solutions are straightforward fixes (e.g., if the

environment is too noisy, the solution involves reducing the noise level

or taking other steps to insure that the noise does not interfere with

accomplishing the objective). However, as noted by Kristiansen and others

(Spears, et al. 1980), the transparency of many training problems and their

solutions has apparently had little impact on making appropriate corrections.

The guidelines are organized as follows:

12 €.*
I.°

I. Training Environment covers training deficiencies which appear to

have a pervasive influence on many more specific aspects of the training

system. Generally, the problems grouped under this category are symptomatic

of deficiencies in administrative responsibility (e.g., Spears et al., 1980)

and may require more global administrative intervention to be modified.

Problem areas (with their related TEE Questions) include the following:

a. Training site (TEE Questions 85, 86 also under lg, 87, 88) addresses environmental conditions including space, noise, lighting and temperature.

b. Implementation of Training Plan (Questions 82, 83, 84) details

deficiencies and contamination of the actual content of training vis-a-vis

lesson plans and instructor guidelines.

c. Number of Instructors ( Question 81) considers the effect of

instructor/trainee ratio on adequate guidance and feedback activities.

d. Training duration (Questions 90, 92) deals with the length of

time devoted to the training course and with rest periods.

e. Attitudes and Motivational Techniques (Questions 27, 28, 89)

address potential solutions to attitudinal and motivational deficiencies.

f. Equipment (Question 93) relates to proper functioning of

training devices.

g. Distractions (Question 86) address interruptions in training

events such as equipment breakdowns, instructor absences, and visitors.

2. Presentation encompasses a number of issues that are related to

the adequacy of the plan for training, the lecture format, and the details of

getting information across to trainees. Many of the problems addressed in

this category have been labelled instructional deficiencies (Spears, et al.,

1980) and are similar to those grouped in Kristiansen's Lecture category. [ The problems under Presentation include the following: -

a. Prerequisites (Questions 29, 30) address incoming skills and

knowledges required of trainees.

b. Training objectives (Question 31) considers whether the

objectives of training were presented to the student at the beginning of

each lesson.

.Ne

1 13

I . .-.. : .. . .. . . .. . . . . . .. . . . . . . . .. . . .... t ' '. _ ,, .¢ ;. .", . - , ,i " . """ "' : - . '"-".". - '-.-. -"'"". . , -"- ', -" .'- -
Ii

c. Course Administration Directions (Questions 79,80) deals with

the completeness and realistic demands of conducting lesson plans.

d. Adequacy (Questions 32, 33, 34, 35) refers to the completeness I of the basic presentation components presented during lectures for facts,

concepts, procedures and rules.

e. Sequencing (Questions 75, 76) deals with the order of objectives within lesson plans and also the order of lesson plans in a course.

f. Clarity and comprehensibility (Questions 69, 70, 71, 72, 91, 94)

'8considers the problems and solutions related to spoken and written instrucI tional materials, including the identification, technical quality, and ease

• =of understanding.

g. Examples (Questions 37, 38, 39, 40, 41) addresses the deficiencies

L and corrections for examples and includes their adequacy, sequence, and the

use of non-examples.

h. Media (Questions 77, 78) covers the appropriateness and

adequacy of training devices such as equipment, slides, and text for conveying

the information in lesson plans.

i. Visuals (Questions 73, 74) refers to the use and adequacy

* of pictures in enhancing instruction.

3. Practice/Demonstration includes eight potential problems which

I could adversely affect training effectiveness. These problems address performance oriented activities requiring hands-on practice and associated

elements. Since soldiers will not learn to perform the tasks required in

operational settings without hands-on practice, the identification of

problems in this category and their successful resolution is essential to

optimal training outcomes. The following problems are defined and addressed:

- a. Practice (General)(Questions 58, 59, 60, 61, 62) refers to 6-

motor aspects of practice, including objective-practice consistency and com-

~ Ipleteness, evidence that trainees practice, and final practice.

b. Practice Remembering (Questions 45, 46) details the use of ,- 3: memory aids and practice-test consistency.

14

11,

." ." ". " - 2 '" -. " " .,""" " " " ". "" - "" . "' ." ". _' ." " "' " ," ". . " ' '% '.-'- N". ," ". " " -'- "" , " " .. " - . "'''"• " ,
Lr T

I

c. Practice Using (Questions 47, 48, 49, 50) denotes problems

related to USE or performance training, including their sequence, adequacy,

and integration. 4,

d. Demonstrations (Questions 42, 43, 44) addresses adequacy and

effectiveness of instructive demonstrations.

e. Team Practice (Questions 65, 66, 67, 68) encompasses potential

shortcomings and modifications for training team functions, including its

provision, realism, and adequacy of feedback.

f. Feedback (Questions 63, 64) deals with the knowledge of results

provided to trainees during practice.

g. Job Performance Aids (Questions 36, 51, 52) considers the use,

adequacy and explanation of training aids.

h. Practice-test Consistency (Questions 53, 54, 55, 56, 57)

examines the relationship of items practiced to those tested, including level,

type, format, condition, and standards.

4. Testing addresses the problems and recommended methods for increasing

effectiveness of tests. The deficiencies covered under this category include

those related to the instructional design of testing materials and the way

in which testing is carried out. Five generic problem areas are discussed

along with recommended solutions. These are:

a. Objectives-test consistency (Questions 1, 2, 3, 4, 5, 6, 7)

covers the match between learning objectives and corresponding test questions.

"I The consistency of test item format, task level, content, conditions, and

standards to learning objectives is discussed.

b. Test instructions (Questions 21, 22, 23) deals with the completeness and clarity of test directions and the degree to which instructors 4 adhered to them during test administration.

c. Test adequacy (Questions 8, 9, 11, 12, 13, 14, 15, 16, 17, 18,

19, 20) involves the design of test items, answer keys, and test formats and

* recommendations for modifications.

d. Test realism (Questions 10, 24, 25) considers the relationship

Iof the test to the conditions and requirements of the job being trained.

e. Contamination (Question 26) addresses the amount of help and

1 external cues available during testing which may invalidate test results.

15

,II . -' ..'.. .... -. .' -, .-. .. .
" Section IV

-

SUMMARY AND CONCLUSIONS

Training Effectiveness Evaluation (TEE) offers one means of fulfilling

many evaluation requirements within the context of a systems approach to ,

training. The scope of a TEE may vary according to management needs and

available resources, but the main purpose is always to identify trainingrelated problems and to suggest underlying causes. A TEE may be conducted in %

support of an operational test of a new system, or it may address on-going

training in a review capacity or in response to complaints. Within a specific

TEE, data collection may encompass product evaluation, process evaluation, or II&L

both. Product evaluation involves an analysis of training design based on

training documentation. Process evaluation examines the conduct of training.

If the TEE addresses both product and process issues, performance discrepancies can be identified and corresponding training deficiencies can be isolated

from the perspective of both design and implementation. In this case, much

of the diagnostic work necessary to specify revision requirements is accomplished for the training manager and ISD support personnel. If only a product

I evaluation is conducted, only design deficiencies can be identified. This

can be useful, however, as a quality control mechanism prior to investigating

new discrepancies and deficiencies in the conduct of training, provided that

the adequacy of objectives and the adequacy and consistency (with objectives)

j of the tests can be verified. Thus, a product evaluation component is a

necessary component of a process evaluation.

Evaluation is not to be considered simply as the process following

implementation. The term "evaluate" is used in the general judgmental sense

of the continuous monitoring of a program or of the training function as a

whole and involves both verification and validation. The process consists of

internally evaluating the training program during each phase of its preparation

(to the degree that resources permit) while concurrently externally evaluating

the overall training function. Thus, following implementation, feedback is

used to evaluate the program, assess the quality of soldiers' performances

and check the organization's responsiveness to training needs. If evaluation

of the program indicates a need for change, the development cycle is reentered

16

. .. . . . . . . .

l% % _A,
at the appropriate point. In a broader sense evaluation is a continual,

empirically based process that involves the verification, validation, and

assessment of data used by the training system, to insure that the needs of the

II Army are met in producing trained soldiers and combat ready units. It is important, therefore, to obtain a variety of complementary data from different

sources; evaluation based on the evidence of only one type of feedback may be

unreliable and lack validity. It should..be remembered that evaluation is

pointless unless action is taken to correct deficiencies and discrepancies

revealed by that process, and that the management function of providing for

quality control is equally important during each phase of the training process.

The Guidelines for Revising Training Program Deficiencies described

in this report provides, we believe, a broader and more complete method for

-I revising training program curriculum and procedural discrepancies than has :4 heretofore been available to Army Air Defense. As a part of the TEE system,

the guidelines provide the specific means for increasing training effectiveness

by correcting the major discrepancies revealed by the assessment. Together

with the TEE, the guidelines provide extension of the evaluation methodology

initiated by the Harless Guidelines and developed further by Kristiansen and

his coworkers at the Army Research Institute. The present results and documents

are an additional step in refining instructional systems evaluation technology

to increase training effectiveness.

.17

-:.. * .-

- . * . * * ., 
Section V

9REFERENCES

Author, Classifying instructional objectives. San Diego, CA:

Courseware, Inc., 1977. ,

Author, Designing segments for concepts. San Diego, CA: Courseware, Inc., 1977.

i Author, Courseware, Designing Inc., segments 1977. for memory level instruction. San Diego, CA: 1,or

Author, Designing segments for procedures. San Diego, CA: Courseware, Inc.,

-1977

Author, Designing segments for rules. CA: Courseware, Inc., 1977.

Author, identifying technically correct test items. San Diego, CA:

Courseware, Inc., 1977.

Ellis, J.A., and Wulfeck, W.H., II. The instructional quality inventory:

IV. Job performance aid (NPRDC Special Report 79-5). San Diego, CA: Navy

Personnel Research and Development Center, 1978.

Ellis, J.A., Wulfeck, W.H., II, and Fredericks, P.S. The instructional

quality inventory: I. User's manual (NPRDC Special Report 79-24). San Diego,

CA: Navy Personnel Research and Development Center, 1979.

Fishburne, R.P., Jr., Rolnick, S.J., and Larsen, J.Y., III. Development of

guidelines for conducting training effectiveness evaluations of air defense

training (Calspan Report No. 6757-K-3). Buffalo, NY: Calspan, 1981.

Fredericks, P.S. The instructional quality inventory: III. Training workbook ,g..

(NPRDC Special Report 80-25). San Diego, CA: Navy Personnel Research and

Development Center, 1980.

Harless Performance Guild. Guidelines for conducting a training program

evaluation (Working Paper FKFU 80-1). Fort Knox, KY: U.S. Army Research

Insti-tue for the Behavioral Social Sciences, Fort Knox Field Unit, 1979.

Kristiansen, D.M. Training program evaluation. A job aid for modifying

ineffective or inefficient training (Working Paper FKFU 80-8). Fort Knox, KY:

U.S. Army Research Institute for the Behavioral and Social Sciences, 1980.

Kristiansen, D.M. A Job aid for modifying ineffective and inefficient training

programs (Research Product 81-17). Fort Knox, KY: U.S. Army Research Institute

for the Behavioral and Social Sciences, Fort Knox Field Unit, 1981.

Kristiansen, D.M. and Witmer, B.G. A job aid for the systematic evaluation

of lesson plans (Research Product 81-15). Fort Knox, KY: U.S. Army Research

Institute for the Behavioral and Social Sciences, Fort Knox Field Unit, 1981

18

.•- . .. . . .. . . .. ... .-
Kristiansen, D.M. and Witmer, B.G. Guidelines for conducting a training program evaluation (TPE) (Research Product 81-18). Fort Knox, KY: U.S. Army

Research Institute for the Behavioral and Social Sciences, Fort Knox Field

Unit, 1981.

Larsen, III, J.Y., Rolnick, S.J., and Fishburne, R.P., Jr. Guidelines for

conducting a training effectiveness evaluation Volume I: TEE evaluator's

handbook (Calspan Report No. 6757-K-i). Buffalo, NY: Calspan, 1981.

Larsen, III, J.Y., Rolnick, S.J., and Fishburne, R.P., Jr. Guidelines for

conducting a training effectiveness evaluation. Volume II: Data collector's

-manual (Calspan Report 6757-K-2). Buffalo, NY: Calspan, 1981.

Roid, G., & Haladyna, T. A handbook on item writing for criterion referenced

1tests (NPRDC Technical Note 80-8). San Diego, CA: Navy Personnel Research and

Development Center, 1980. (Preliminary document, limited distribution)

Schulz, R.E., and Farrell, J.R. Job aid manuals for Phase II - Design of

the instructional systems development model (Research Product 80-16).

Alexandria, VA: Army Research Institute for the Behavioral and Social I Sciences, 1980.

Schulz, R.E., and Farrell, J.R. Job aid manuals for Phase III - Develop the

instructional systems model (Research Product 80-18). Alexandria, VA: Army

Research Institute for the Behavioral and Social Sciences, 1980.

Spears, W.D., Maxey, J.L., and Roush, M.D. Methodology for correcting

deficiencies in training programs (Seville Project Report PR80-08).

Pensacola, FL: Seville Research Corporation, 1980.

Witmer, B.G. A job aid for the structured observation of training (Research

* Product 81-16). Fort Knox, KY: U.S. Army Research Institute for the

I Behavioral and Social Sciences, Fort Knox Field Unit, 1981.

Wulfeck, W.H., II, Ellis, J.A., Richards, R.E., Wood, N.D., and Merrill, M.D.

The instructional quality inventory: I. Introduction and overview (NPRDC

Special Report 79-3). San Diego, CA: Navy Personnel Research and Development Center, 1978.

119 I,

I19J

:::
, 

lf ' * '" ' = ' 'I ' f' I Z ' ' d ' - T 

- --'7 ' '''r 'C,- r'-* ) ' T '' * :'r ,

i-SO - -- T 1F-M O S "E ---.- -. - -UE' ONS- - - -

I :

I C.

"I

I-°.

• :.'
I,. W 7V

1. Is the TEST FORMAT appropriate (See table in job aid.)

for the OBJECTIVE?

2. Are there test items for the TLO 1 = Test items for TLO or all

or all of its critical parts/LOs9 parts/LOs

(See job aid for critical parts.) 2 = No test items for TLO and

for some parts/LOs

3 = No test items for TLO and

for most parts/LOs

3. Is there a test item for each I = Items for all parts

critical part of each LO? 2 = Items for many, but not all parts

(See job aid for critical parts.) 3 = Items for only a few parts or

for no parts

4. Does the TASK LEVEL of the (See table in job aid.)

test item match the TASK LEVEL

of its OB3ECTIVE?

5. Does the content of the test item I = Same match the content of its 2 = Slightly different OBJECTIVE? 3 = Very different

6. Do the CONDITIONS of the test I = Exact match

item match the CONDITIONS 2 = Minor mismatch

of its OBJECTIVE? 3 = Severe mismatch

7. Do the STANDARDS of the test I = Exact match

item match the STANDARDS of 2 = Minor mismatch

its OBJECTIVE? 3 = Severe mismatch

8. For true-false, multiple choice, I = Only one answer is correct

and matching items is only one 3 = More than one answer can be

answer correct? correct

9. For short answer, fill-in, listing, I = All correct answers are in

and performance items are all answer key ':

acceptable answers in the answer 3 = Some correct answers are not

key? in answer key

10. Does the test item provide I = Yes

opportunities for COMMON 3 = No

ERRORS to be made?

II. Is the language of the test item I = Easy

easy for students to understand? 2 = Somewhat difficult

3 = Very difficult

12. Is the test item different 12* from 1 = Different :V.

previous PRACTICE and 2 = Presented before, USE-UNAIDED 0

EXAMPLES? USE-RULE, or (USE-CONCEPT, 3 = Presented before, USE-AIDED i

USE-PRINCIPLE only)

i. i
13. Is 

given 

the 

away 

answer 

by 

to 

other 

the 

item(s)? 

test item 

2 

I = 

= 

Answer 

Other items 

not given 

give clues

away U

3 = Answer can be found in other

item(s)

14. Is the answer to the test item I = Answer not dependent on 

I

dependent on answering previous other items

item(s) correctly? 3 = Previous items must be correctly

answered

1 15. Are sketches and diagrams used 1 = Easy to understand

in the test item easy to 2 = Somewhat confusing

understand? 3 = Very confusing

16. Is the test Item tricky or 1 = Not misleading

misleading? 2 = Somewhat misleading

3 = Very misleading

17. Is the test item well constructed? I = Meets all criteria

(See job aid for criteria list for 2 = Deficient on noncritical criteria

the test format used.) 3 = Deficient on critical criteria

18. When performance steps are I = Fills in completely

scored, does the instructor use 2 = Uses as a reference or fills

a checklist? In partially

3 = Does not use

19. Is each correct answer position I = Yes

used about the same number of 3 = No

times? (true-false, multiple

choice, or matching items only)

20. Are specific patterns of correct I = No patterns easily seen

answer positions repeated across 3 = Patterns can be seen

test items or are single positions

repeated in blocks? (true-false,

multiple choice, or matching

Items only)

21. Are test administration I = Directions are complete

directions complete? 2 = Directions provided, but

incomplete or unclear

3 = Directions are not provided

22. Do instructors follow the I = Yes

directions when administering 2 = Some variations from directions

the test? 3 = Significant variations from

directions

23. Are adequate test Instructions I = Yes

provided to the student? 2 = Instructions provided, but

unclear 

-

3 = No instructions provided

'Z:"
I

24. Does the FINAL TEST integrate I = Yes

tasks as they are integrated in 2 = Partially

the "real world?" 3 = No, tasks are tested separately

25. Are tasks and task steps tested I = Yes

in the same sequence as they are 2 = Slightly out of sequence

performed in the "real world?" 3 = Very different sequence

26. In the test free of external cues I = Yes

or help? 2 = Hints given

3 = Answers are given away

27. Are motivational techniques I = Yes

employed? 3 = No

28. Is the trainee attitude positive? I = Positive 12 = Indifferent

3 = Hostile or frustrated

29. Are course ENTRY SKILLS I = Review with practice

. reviewed? 2 = Review with no practice

3 = No review

, 30. Is mastery of prerequisite skills I = Yes .'

verified prior to new instructions? 3 = No

31. Are OB3ECTIVES presented to I = Yes

the student? 3 = No

32. Are the basic PRESENTATION (See guidance and tables in

COMPONENTS present? handbook.)

33. Are STATEMENTS complete? 1 = STATEMENT complete I2 = Few parts missing 3 = Many parts missing

34. Are STATEMENTS for CONCEPTS, I = Completely adequate

PROCEDURES, or RULES 3 = Some or all features omitted

adequate? (See job aid criteria.)

35. Does STATEMETI HELP provide I = Help provides sufficient

sufficient explanation? explanation

2 = Help gives insufficient

explanation

3 = Help is confusing

36. Does training include instruction I = Yes

, on the use of required job 3 = No

performance aids?

37. Are EXAMPLES and 1 = Yes

NON-EXAMPLES adequate? 3 = No

:1 ...

• "I '.' ' " J " .,. "." ," ." ..." .o ... ., .', .--.. ...-. ..". .' ". .,. ." ' .
38. Is EXAMPLE HELP adequate? I = Help provides sufficient ,i

• .explanation

2 = Help gives insufficient

" 3 explanation

3 = Help is confusing

39. Are EXAMPLES sequences from I = Yes '

easy to hard? (CONCEPTS only) 3 = No

40. Are there enough EXAMPLES? I = Yes

(See job aid for criteria.) 3 = No

41. Are NON-EXAMPLES included? I = Yes

(CONCEPTS only) 3 = No

42. Do DEMONSTRATIONS show I = Yes

2 how to correct/avoid common 3 = No

errors?

. 43. Are steps in a DEMONSTRATION I = Yes

the appropriate size? (See job 2 = Step size is too small

aid.) 3 = Step size is too large

44. Are tasks and task steps I = Yes

DEMONSTRATED in the same 2 = Slightly out of sequence

sequences as they are performed 3 = Very different sequence I in the "real world?"

45. Are memory aids used? I = Used

(PRACTICE REMEMBERING 3 = Not Used

only)

46. Does each PRACTICE I = Same

REMEMBERING item have the same 2 = Same content, different

content and format as the test format

item? 3 = Different content

47. Are PRACTICE USING items I Yes

sequenced from easy to hard? 3= No

48. Do PRACTICE USING items I = Yes

provide opportunities for 3 = No

COMMON ERRORS to be made?

49. Are PRACTICE items different I = Different

from EXAMPLES? (USE-CONCEPT, 3 =Presented before

USE-RULE, or USE-PRINCIPLE -

only)

50. Does PRACTICE USING integrate I Yes t tasks as they are integrated in the 2 = Partially

"real world?" 3 = No, tasks are practiced

separately

I- ',"'.".-e-.- e "- ."" ." " ' " -.""--..-""" ""''' " '' " .' '"\ ..L. .,''
%-1 -0* .7

;J' '

5 1. Are job performance aids I = Easy to use

(3PAs) usable? (See criteria 2 = Hard to use ( in job aid.) 3= Unusable

52. Do all students use the job 1 = Yes

performance aid (3PA)? 2 = Up to 20% do not use 3PA

3 = More than 20% do not use JPA

53. Does the TASK LEVEL of the I = Yes

PRACTICE item match that of the (2 and 3 -See table in job aid.)

test item(s)?

54. Does the CONTENT TYPE of I- Yes

the PRACTICE item match that of 3 = No

i1 the test item(s)?

55. Does the FORMAT of the I = Yes

PRACTICE item match that of (2 and 3 - See table in job aid.) jthe test item(s)?

56. Do the CONDITIONS of each I = Yes

FINAL PRACTICE item match 2 = Slightly different I those of the test item(s)? 3 = Very different

57. Do the STANDARD of each I = Yes

FINAL PRACTICE item match 2 = Slightly different

those of the test item(s)? 3 = Very different

58. Is FINAL PRACTICE free of I = Yes

external cues or help? 2 = Hints given 3 = Answers are given away

59. Are there PRACTICE items for I = PRACTICE items for the

each TLO or all of its critical TLO or all parts/LOs

partsILOs? 2 = No PRACTICE items* for the

TLO and for some parts/LOs

3 = No PRACTICE items for the

TLO and for most partsLOs

60. Is there a PRACTICE items for I = PRACTICE for all parts

each critical part of each LO? 2 = PRACTICE for many, but not

(See job aid for critical parts.) all parts

3 = PRACTICE for only a few

parts or for no parts

61. Do all students PRACTICE? I = Yes

2 = Up to 20% of students do not

PRACTICE

3 =More than 20% of the students

do not PRACTICE

-|

I,
I 62. Do all students meet the I = Yes

required STANDARDS in FINAL 2 = Up to 20% of students do not

PRACTICE? 3 = More than 20% of students

do not

63. Is FEEDBACK provided for I = FEEDBACK HELP is given

PRACTICE? 2 = Correct answer only is given

3 = No feedback is given

64. Is FEEDBACK HELP adequate? I = Help gives enough explanation .

2 = Help gives insufficient

I ~explanation ,

3 = Help is confusing

65. Is TEAM PRACTICE provided? I = Yes

3 = No

66. Are TEAM PRACTICE I = Yes

CONDITIONS the same as (or as 2 = Slightly different

Sclose as possible to) those of 3 = Very different

the real task?

67. Is TEAM PRACTICE FEEDBACK I = FEEDBACK HELP is given

provided? 2 = Success/Failure feedback

only is given

It 3 = No feedback is given

68. Is FEEDBACK HELP for TEAM 1 = Help gives enough explanation

PRACTICE adequate? 2 = Help gives insufficient explanation 3 = Help Is confusing

69. Are all PRESENTATION I = Yes

COMPONENTS separated and 2 = Some are not

identified? 3 = Most or all are not

. 70. Is the technical quality of written I = Most criteria met

or spoken material adequate? 2 = Several criteria not met

(See job aid for criteria. Make 3 = Few criteria met

K! notes on specific problems.)

71. Is the wording of written or 1 = Yes, few hard words and

spoken material easy for the long sentences

students to understand? 2 = Some hard words and long

sentences I 3 = Many hard words and long

sentences

72. Is the instructor's presentation I = Yes

or the narration easy to listen to? 2 = Dull and monotonous

3 = Hard to listen to

"- p-

- .. .
73. Is the instructor's presentation I = Completely

or the narration supported by 2 = Partially

visuals? 3 = Not at all

74. Are visuals easily understood? I = Yes ,

2 = Understandable with effort

3 = Very hard to understand

75. Are the OBJECTIVES (TLOs and 1 = Yes

LOs) within each LESSON 3 = No

sequenced properly?

(Prerequisites taught first.)

76. Are the LESSONS sequenced I = Yes

properly within the course? 3 = No

77. Are the media appropriate for the I = Yes

objectives? (See table in job aid.) 3 = No (note key words,

underlined in table, on j1 worksheet.)

78. Can the media used provide all I = Yes

necesary stimuli? 3 = No

79. Are the course administration 1 = Yes

directions complete? 2 = Partially incomplete

3 = Incomplete or non-existent

80. Do course administration I = All demands are realistic

directions make realistic demands 3 = Some demands are unrealistic

of students and instructors? (Note what they are.)

81. Is the instructor/trainee ratio I1= Yes

such that all students can see, 2 = A few students cannot see,

hear, and receive feedback? hear, and rcive feedback

3 = Many students cannot. see,

hear, and receive feedback

82. Does the instructor follow the = Yes

methods in the Instructor Guide? 2 = Follows to some extent

3 = Follows very little or

not at all

83. Does the instructor teach all of I = Yes

the content in the LESSON 2 = Much of the content

materials? 3 = Very little of the content Li (If 2 or 3, note what was left out.)

84. Did the instructor limit his 1 = Yes

teaching to the content in the 3 = No (Please note what other

LESSON materials? things he taught.)

•.I: %
85. Is there enough space for all 1 = Yes

of the trainees? 2 = A little crowded

3 = Very cramped or some

students can't fit in the

space at all

86. Is instruction free of 1 = Yes

distractions? 2 = Distractions are annoying

3 = Distractions seriously

interfere with the

instruction

87. Is the lighting appropriate for 1 = Yes

the training situation? 2 = Students have trouble -

reading or seeing displays

& equipment

3 = Students cannot read or see

displays & equipment

88. Is the temperature appropriate 1 = Yes

for the training situation? 2 = Temperature makes students

uncomfortable

3 = Temperature seriously

interferes with learning

89. Is the instructor's attitude I = Yes Ipositive? 3 =No

90. Are frequent breaks provided? I = Yes

(5-10 minute breaks every hour) 2 = Breaks too short or.. ii infrequent 3 = Breaks not provided

91. Is the speed of presentation l= Yes 1 appropriate? 2 = Too slow

3 = Too fast

92. Was the allotted training time I = Appropriate length

too long or too short? 2 = Too long

3 = Too short

93. Does the training device! I = yes ,

equipment used in training 2 = Minor malfunctions, little

function properly? change from intended task

performance

3 = Major malfunctions, substantial

change from intended task -:IL performance "

94. Is there anything else unusual about the LESSON materials, or do any other !critical incidents occur during training that would interfere with learning?

(Describe each one below. Rating 3)

:I

[ , .., '." ,,' ,r..",,.. .",.",',',',',,' .- .'-".. . .* *. . " '.'a . " "," ' " '. . .. ", .. ."...* •", ". .. ". .. .'