Technical Report 527 ( I) LEVEL' ;

rilalDEVELOPMENT OF JOB AIDS FOR

INSTRUCTIONAL SYSTEMS DEVELOPMENT

Russel E. Schultz and Harold Wagner

9Z5 Human Resources Research Organization

SYSTEMS MANNING TECHNICAL AREA

DTIC

' S~ELLCTLJAIN 2 0J82

•~B ....

R,.. U.S. Army

Research Institute for the Behavioral and Social Sciences

February 1981 ( A....pulcrl 20 82 0 5t

j• •,,-Apprioved for public release; distribution unlimited,
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

ý"

L. NEALE COSBY

"JOSEPH ZEiDTER Colonel, IN

Technical D7irector Commander

Research accomplished under contract

to the Department of the Army

Hum.an Resources Research Organization

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S,

Armv Research Institute for the Behavioral and Social Sciences, ATTN:

PERI-TST, 5001 Eisenhower Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when It Is no longer

needed. Please do not return It to the U.S. Army Research Institute for

the Behavioral and Social Sciences.

NOTE: The findings In this report are not to be construed as an official

Department of the Army position, unless so designated by other authorized

documents.
UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (ohen Data Entered)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

REPORT__ DOCUMENTATIONPAGE_ BEFORE COMPLETING FORM

"I. REPORT NUMSER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Technical Report 527A -A 4. TITLE (and Subtitle) S. TYPE OF REPORT & PERIOD COVERED

DEVELOPMENT OF JOB AIDS FOR INSTRUCTIONAL

SYSTEMS DEVELOPMENT Final Report

6. PERFORMING ORG. REPORT NUMSER

FR-ETSD-81-2 7. AUTHOR(a) 8. CONTRACT OR GRANT NUMBER(&)

Russel E. Schulz & Harold Wagner DAHC 19-78-C-0100

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASK

ARUA & WORK UNIT NUMBERS

Human Resources Research Organization

300 North Washington Street 2Q263743A794

Alexandria, VA 22314

|, CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

US Army Research Institute for the Behavioral February 1981

and Social Sciences 13, NUMBER OF PAGES

5001 Eisenhower Avenue, Alexandria, VA 22333 58

14. MONITORING AGENCY NAME & ADDRESS(it dillfoteat from Controlling Office) IS. SECURITY CLASS. (of tOi report)

UNCLASSIFIED

ISm. DECL ASSIFICAT1ON/DOWNGRADING

,SCHEDULE

16. DISTRIBUTION STATEMENT (of thle Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abstract entered in Block 20, It different tram Report)

IS. SUPPLEMENTARY NOTES

This project was monitored technically by Dr. Melissa Berkowitz and

Dr. Bruce W. Knerr

II. KEY WORDS (Continue on reverae aide It neo*eaay and Identify by block numbe")

Author Aids

Instructional Systems Development ISD •

Job Aids

Training Development

20, A5ST ACT (Continue on reveres aide It n.ec.sary and Identity by block number) 'Lte pA -pose of tne researcn

was to develop and evaluate 18 Job Aids for use in the analysis, design,

development, implementation, and control of military inst'ructional systems.

This report discusses the history of the research effort, the conduct of the

needs Assessment, and the development and formative evaluation of the Job Aids

The evaluation findings strongly support the usefulness of the Job Aids to

military instructional development personnel

DO I JAN I OV 1 OBSOLETEARTT

CSICURITY CLAWIFICATION OF TNIS PAGE (ften Dat Anter4od)

; y( '

,•%/•'.?.• : , •,:.'..'•.". ,. . ' #,• .W *41 . .... . . .. ,,, .'.. .. ... . . ~ ~ .&t t _ 'A
Technical Report 527

DEVELOPMENT OF JOB AIDS FOR

INSTRUCTIONAL SYSTEMS DEVELOPMENT

Russel E. Schultz and Harold Wagner

Human Resources Research Organization

Submitted by:

James D. Baker, Chief

SYSTEMS MANNING TECHNICAL AREA

Approved by:

Edgar M. Johnson, Director

SYSTEMS RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Offioe. Deputy Chief of Staff for Personrel

Department of the Army

February 1981

acot Number Education and

AIO743A794 Training

Apptowd for puWbic Us2sw driwidtow unlifwiwd

i" Ili
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies, Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv

II
FOREWORD

The Computer Based Instructional Systems Team of the US Army Research

Institute for the Behavioral and Social Sciences (ARI) performs research and

development in the area of educational technology that app/lies to military training. Of interest are methods for training individuals to develop and utilize

instructional courseware in reasonable time, at an accept-able cost.

This Technical Report describes the development and formative evaluation

of eighteen Job Aids designed to support users of the 1nstructional Systems

Development Model (ISD, TRADOC Pamphlet 350-30). The ASD Model is a step-by-step

procedure for the analysis, design, development, impl~mentation, and control of

military course materials. To accomplish this research, ARI's resources were

augmented by contract DAHCI9-78-C-OI00 with the Humo Resources Research Organization.

Personnel at the US Army Engineer School, Foit Belvoir, Virginia, provided

guidance during the developmental phase of the ploject: Dr. Everett Rompf,

Mr. Jack Ainsworth and Ms. Lois Passman. In addition, the following individuals

at the US Army Signal Center and Fort Gordon gave valuable assistance during the

needs assessment and evaluation phases: Dr. William Ketner, Mr. Earl Duncan,

Mr. Richard Arnold, Mr. John Meyers, Ms. Doris Lakeman, Mr, Dan Danilovich,

Ms. Yvonne Waldin and Mr. Richard Eberl. Ms. Jean Farrell and Mr. Michael

Hillelsohn of HumRRO also contributed to this effort.

The entire research project is responsive to the requirements of Army

Project 2Q263743A794, FY 80 Work Program.

JosHE DN R

Techi ical '9~'rector

L) I

V/Vi
PREFACE

This document reports on a research project in which 18 Job Aids were developed to permit implementation of all phases (analyze, design, develop, implement, and control)

of the Instxuctional Systems Development (ISD) model.

The research effort was performed in HumRRO's Educational and Training Systems

Division, Dr. Robert J. Seidel, Director. Mr. Russel E. Schulz was the Project Director.

This research was supported by the U.S. Army Research Institute for the Behavioral

and Social Sciences under Contract No. DAHC19-78-C-0010, "Computer Interactive

Author Aiding." The project was monitored technically by Dr. Melissa Berkowitz, Dr. Bruce

Knerr, Dr. Harold F. O'Neil, Jr., and Dr. Leon Nawrocki of the U.S. Army Research Institute. Their support and assistance in this research effort is gratefully appreciated. The

authors also wish to express gratitude to all personnel at the U.S. Army Engineer School,

Fort Belvoir, Virginia and the U.S. Army Signal Center and Fort Gordon, Fort Gordon,

Georgia who provided assistance in the research project. Special appreciation is extended

to the following individuals who contributed to the success of the project:

U.S. Army Engineer School

Dr. Everett Rompf, Education Specialist, Individual Training Division

Mr. Jack Ainsworth, Education Specialist, Individual Training Division

Ms. Lois Passman, Education Specialist, Individual Training Division

U.S. Army Signal Center and Fort Gordon

Dr. William D. Ketner, Deputy Assistant Commandant for Educational Technology

Mr. Earl Duncan, Education Specialist, Educational Technology Division

Mr. Richard Arnold, Education Specialist, Educational Technology Division

Mr. John Meyers, Former Chief, CE Operations Branch, Training Analysis Division

Ms. Doris D. Lakeman, Chief, CE Operations Branch, Training Analysis Division

Mr. Dan Danilovich, Chief, CE Maintenance Branch, Design and Development Division

Ms. Yvonne S. Walden, Education Specialist, Design and Development Division

Mr. Richard Eberl, Trabi4ig Specialist, Design and Development Division

The assistance provided by Ms. Jean Farrell, formerly of HumRRO, who co-authored

ij• many of the Job Aids is gratefully acknowledged as is the work of Mr. Michael Hfillelsoin

who authored the Job Aid for ISD V.3 Revise System. Finally, a special thanks to

Mrs. Alice Thompson whose excellent work contributed to the design and overall

appearance of Ute Job Aids.

Av I
DEVELOPMENT OF JOB AIDS FOR INSTRUCTIONAL SYSTEMS DEVELOPMENT

BRIEF

Requirement:

The purpose of the research was to develop and evaluate Job Aids ("how to

do it" guidance) for the activities identified in the Instructional Systems

Development Model (ISD, TRADOC Pamphlet 350-30). The usefulness of the ISD Model

depends upon Job Aids which will enable training personnel to translate the ISD

procedures into instructional products.

Procedure:

A series of Job Aids was designed and developed for each of the five phases

of the ISD Model based on needs assessments conducted at the US Army Engineer

School (USAES), Ft Belvoir, VA and the US Army Signal Center and Ft Gordon

(USASC&FG), Ft Gordon, GA. Baseline data were collected from USAES and USASC&FG

personnel as they performed their ISD functions without the assistance of the

Job Aids. The Job Aids were then provided to individuals who were engaged in

the activity covered by the Job Aid. Data were collected relating to problems

alleviated by using tle Job Aid, problems encountered in using the Job Aid, and

user acceptance of the Job Aid.

Findings:

Comparisons were made with the baseline data to assess the value of the

Job Aids as used in the ISD process. The findings strongly supported the usefulness of the Job Aids.

Utilization of Findings:

A summative evaluation of the Job Aids is planned to assess their value to

training personnel in the preparation and delivet-y of a complete course of

instruction.

I"/X
TABLE OF CONTENTS

Page

Introduction ....................................................... 1

A System s Approach ............................................... 1

The Need for Job Aids ............................................. 2

Project H istory ................................................... 3

Research Activities . ... ................................ ............. 6

Task 1: Analysis and Selection of Job Aids ................................ 6

Step 1. Perform Job Aid Needs Assessment .......................... 6

Step 2. Analyze Existing Aids ................................... 9

Step 3. Identify Candidate Aids .................................. 9

Step 4. Refine, Modify or Complete Development of Aids ............... 9

Task 2: Development of Job Aids ...................................... 9

Phase I - Analyze .................................................. 14

ISO 1.2 Select Tasks/Functions .................................... 14

ISO 1.3 Construct Job Performance Measures ................ ......... 14

ISO 1.4 Analyze Existing Courses .................................. 14

ISO 1.5 Select Instructional Setting ................................. 14

Phase II - Design ................................................... 15

ISO 11.1 Dovelop Objectives ..................................... 15

ISO 11.2 Develop Tests ........................................ 15

ISO 11.3 Describe Entry Behavior .................................. 15

ISO 11.4 Determine Sequence and ltructwt ........................... 15

Phase III . Develop ................................................. 16

ISO 111.1 Specify Learning Events/Acivities ......................... 16

ISO 111.2 Specify Instructional Managenwnt Plan and Delivery System ....... 16

ISO 111.3 Review/Select Existing Materials ........................... 16

ISO 111.4 Develop Instruction ................................... 16

IWS 111.6 Validate Instruction ................................... 17

Pthase IV Implement ............................................... 17

ISO IV.1 Implement Instructional Management Plan ................... 17

ISO IV.2 Conduct Intstruction ...................................... 17

Phase V - Control .................................................. 17

ISD V.1 Conduct Internal Evwluation .............................. 17

IS V.2 Conduct External Evaluation ............................. 18

ISO V.3 Revise System ....................................... 18

Xi

-
Page

Task 3: Evaluation of Job Aids ....................................... 18

Evaluation of Job Aids - Level I (Informal Review of Known Aids) ............. 18

Evaluation of Job Aids - Level 2 (Formative Evaluation of New Aids) ......... 18

Evaluation of Job Aids - Level 3 (Impact on Instruction) ................. 19

Findings ..................................................... 19

Recommendations ...................................................... 21

References ............................................................ 22

Appendices

A Needs Assessment Inventory for ISD Phase I, II, and Ili Job Aid Development ....... 24

B Needs Assessment Review Form for ISD Phase I, !1. anc Hll .job Aid Development ... 31

C Needs Assessment Inventory for ISD Phase IV and V Job Aid Development ........ 36

D Background Information Form and ISD Activity Checklit ................... 42

E Job Aid Follow.Up Interview Form ..................................... 45

Figutes

1 The ISO Blocks in Each of the Five IS. Phass ...... .................... 4

2 Flowchart of Block 11.1: Develop Objectives .............................. 7

3 Major Topics Included in the Introduction to the Use of Job Aids and

Job Aids Resource. Manual .......................................... 11

4 Representative Page From Descriptive Authoring Flowrcharts .................... 12

5 Representative Page From Job Aid Manual ............................... .3

: xii
INTRODUCTION

The purpose of this section is to provide a brief description of research and development in this area, a discussion of the rationale for this research, and an overview of the

project history.

A SYSTEMS APPROACH TO TRAINING

The systems approach defines a process which focuses on the job that is ultimately

to be performed and upon the individual who is to learn to perform that job. The systems

approach is just what the name implies: a systematic process for specifying the desired

products of training and selecting what will be taught, how it will be taught, what the

presentation mechanism will be, and evaluating the effects of each phase of the process.

it focuses on student performance its a determinant of content. Its proper application can

hardly fail to improve instruction where only incidental attention has been given to these

functions. However, after more than two decades of research and development in this

area, the full benefits of applying the systems approach to instruction have yet to be

realized.

i~arly efforts to standardize a definitive technology in this area included those by

IlumRRO Crawford. 1962; Finan, 1962; Vallance & Crawford. 1962) on behalf of the

U.S. Army in the early 1960s and the development of the U.S. ('ontinental Army ('ommand Regulation 350-100-1, Systems Engineering of Tiuning, (19681 and the U.S. Air

* Force Manual 5W02. Instructional Systems Development t1970).

These attempts have culminated in the Instructional Systems Development e ISD)

Model (TRADOC Parm 350.30, 1975). This model was prepaxred by the (*enter for

Edtucational T'e-hnology at Florida State University under contract v'ith thie Interservice

('onunittee for InstruCtional Systems and Development. involving tOe Navy, Air Force.

Ma'ine Corps, and the Army. Trhle IS) contains standardized ratiotws,% terminology, and

basic concepts of instructional systems.

[)ifficultito arose in apliying tLite 11) approach to military training. Much of the early rmsearch in attempting to overcome these probleits involved attention to thi, tech.

nical difficultles, One of the plrimry difftcuities of the 181) model and it,% anuals i-s

that it ernithasirvs ','hiat to do." not "how to do it." 1St) noulual|.s are intendted to have

ge-neral applicaility. |owever the saunme nivlhoologi-es cannot b6 applidt to the univeis,

of training problems. "Hlow to do it" may vary widely even though the sp-,cific provesss

or "what to do- may rvnaa n fairly constant across taining problemst. Iinttially, til' "whlat

to do" nature of the 181) nuuals was not a $eAe litolileil inasJmuch as early users of

the mtanuals Werv "expiert" training developers. However. later, the p•osibility• of l|tving

taypetsons use the ISO) approactl to achieve the succtxi of the experts was expilorx.d, The Use of an ISD manual by military pWersonnel with little or no experi)tice In tU-ai111g

program design may cost a fracton of thei cVist of hiring or contracting oiut to experts to

do the development. However. specific guidance, tuols, and pcedures are it''iedtl by

such laypwrsons for ISD to ý, aplwroprtialy and ewt-vffectively inmletment-ed within the

military traning esablistmt,
THE NEED FOR JOB AIDS

In this report "how to do it" guidance/tools/procedures will be referred to as ,Job

Aids. These are designed to help all members of an instructional staff charged with meeting the objectives of one or more components of the instructional systems development

process.

Job Aids are needed in all Phases of the ISD process if significant improvements are

to occur in applying this systematic approach toward producing instruction. This approach

is expensive, time consuming, critical, and requires specialized capabilities.

A wide variety of aids do exist, but the majority are more of the "what to do" type

than the "how to do it" variety. Rather than providing actual help in performing the

work, or even providing detailed how-to-do-it guidance, most of the existing aids may serve

to reinforce or broaden the guidance provided in the ISD manuals. Aagard and Braby

(1976) note the practical limitations in using their algorithms and guidelines for general

purposes, "The task categories and related guidelines are not at a level that will accommodate any training setting." Briggs' handbook (1970) may also serve such a broad guidancte

function.

Care should be exercised in the selection of aids to be integrated with ISD, however,

so that authors are not confused by different yet similar models, sets of jargon, procedures of forms, Existing general manuals differ from one another in that they:

* Include different steps or different names for steps.

* hiclude different methods of accomplishing each step.

e Provide different levels of spwcificit]l in the details included under each

step.

P9rovide different formats for reporting the work accomplished under

each step.

11us, to provide the most efficient aids to authors. the guidance found in some of these

nuiuals and guidebooks nteds to he translatod vid ititegrated into the ISD framework,

rauter than referred to in its source form,

Recently. increasing emphasis has been placwt on the development of "how to do

it" aids. A course which guides trainee in specifying learning activities and in developing

t sts and instructional materials is re|-krted by O'Neml and O'Neal (1970). Dyer, Matthews,

Wright and Yudowitcl (1976) preiared guidance for instructional technologists who are

(asked to develop or administer questionnaires ws purt of Army field tests and evaluations.

In an effort to a.mure the quality of inmtuctional developnent, Ellis, Wulfeck. anid

Fl'Ydericks (1979) developed the Instructional Quality Inventory, Conoley and O'Neil

(1979) presented a detailed and pragmatic set of guidelines for multiple--hioice item

omeration and review.

In the project described in this document, Job Aids were developed to pwovide the

specific guidancl ("how to do it") needed for iimplenwthig the IS) model.

r- --
PROJECT HISTORY

On-line Job Aids developed in a previous ARI project (Schulz, Hibbits, Wagner &

Seidel, 1979) demonstrated the utility of assisting authors to implement the ISD process.

The major characteristics which accounted for the usefulness of the Job Aids are:

e ihe Aids are query-based and interactive.

* •hey are based upon a realistic authoring needs assessment at an operational

site.

* The Job Aids were developed and evaluated according to ISD guidance and

recommended procedures.

o Detailed authoring flowcharts accompanied on-line Job Aids.

* The Job Aids were designed to be system independent.

* The Job Aids concentrate on assisting users to deal with structural and administrative details permitting them to concentrate their efforts on the subject

matter content of the materials.

The research project described in this report is an extension of the previous ARI

project. In the present research, Job Aids were developed to assist Army instructional

systems development personnel in the analysis, design, development, implementation and

control of instructional materials.

The research began in November 1977. During the research period Job Aids were

developed and formatively evaluated. The ISD model was used to define the general

activities to be covered in the Job Aids. (Figure 1 shows the ISD model.) Each Job Aid

consists of two volumes-Descriptive Authoring Flowcharts and a Job Aid Manual. The

latter contains prescriptive descriptions of the activities to be performed, along with

critical references. They were designed to be stand-alone, step-by-step procedural guides,

equally useful to individuals at all experience levels of the Instructional Systems Development process. The Job Aids were based on an assessment of user needs at the U.S. Army

Engineer School (USAES), Fort Belvoir, Virginia, and the U.S. Army Signal Center and

Fort Gordon (USASC&FG), Fort Gordon, Georgia. They were implemented and evaluated

at the USASC&FG.

Initially, 13 Job Aids wera developed and evaluated, one for each block of the ISD

model except for Block 1.1, Analyze Job. A Job Aid was not developed for this block

for the following reasons:

(1) The research plan called for the formative evaluation of the Job Aids at the

USAES. Discussions with USAES personnel indicated that the majority of jobs at the

USAES had undergone job analysis. Since personnel at USAES would not be performing

this activity during the research period, one could not evaluate a Job Aid developed for

this activity.

(2) Job analysis is a highly complex activity that requires input from many sources.

Many of these sources are outside of the UiSAES. Therefore, it would have been difficult

to obta .' the input in a form needed for the research effort.

S. (3) Because of the complexity of performing job analysis, and the multiplicity of

Job Ai& needed for doing so, development of an adequate Job Aid for this block would

require too much of the total research resources.
THE BLOCKS IN EACH PHASE ARE:

1 2 1.3 141 .

SELECT CONSTRUCT ANALYZE •SELECT

PHASE I OANAL IZE SEB TASKS/ LENCTA S JO PERFORMANCE EXISTING COURSES ýINSTRUCTIONAL SETTING oFUNCTIONS MEASURES

11.2 [1.3 11.4

DE VELOP DEVELOP BE DETERMINE PHASE II 'DEVELOP DEVL ENTRY SEQUENCE&

OBJECTIVES r TESTS J BEHAVIOR STRUCTURE

.. 2

n .1 SPECIFY

SPECIFY INSTRUCTION 1 4

PHASE III - LEARNING MANAGEMENT REVIEWSELECT DEVELOP VALIDATE EVENTS/ PLAN & MERIS / INSTRUCTION INSTRUCTION

ACTIVITIES DELIVERY MATERIAL I

SYSTE

SIMPLEMENT Ml 2 2

PHASE IV INSTRUCTIONAL CONDUCT

MANAGEMENT PLMANAGMN INSTRUCTION

P CONDUCT CONDUCT RV 3 PHASEV -- INTERNAL !tE XTE RNAl, REVISE

EVALUATION EVALUATION SYSTEM

Figure 1. THE ISD BLOCKS IN EACH OF THE FIVE ISD PHASES

4
The evaluaLtm plan for tl initial 13 Job Aids was to have USAES instru_;tional

development personnel use the Job Aids in an actual job setting as they were performing

the activities required by the ISD model. However, shortly after beginning work it was

discovered that all Phase I ISD activities for courses at USAES had been completed.

Previous information indicated that all ISD Phase I, II, and III activities were being continuously performed at USAES. Because no new courses were to be established during

the research period it would not be possible to conduct a formative evaluation of the Job

Aids for Blocks 1.2 (Select Tasks/Functions), 1.3 (Construct Job Performance Measures),

1.4 (Analyze Existing Courses), and 1.5 (Select Instructional Setting) at USAES. In order

to accomplish the work, a revision in the research plan was made to change the location

at which the aids wvere to be evaluated. The U.. Army Signal Center and Fort Gordon

(USASC&FG), Fort Gordon, Georgia was selected as a substitute for USAES for the

following reasons:

(1) Activities in all phases of the ISD model were being actively worked on at

USASC&FG.

(2) The avail-ability of more complete data at USASC&FG would permit more

reliable and useful Job Aids to result from the research effort.

(3) Due to the considerable number of personnel available at USASC&FG who

were engaged in the ISD process, it would be feasible to collect the data required for

the formative evaluation in a timely manner requring a minimal delay in project activities.

Upon completion of the development and formative evaluation of the Job Aids for

the first three phases of the ISD model another project effort was initiated. It involved

the development and evaluation of a prototype system-independent Programming Design

Guide for an interactive Job Aid. The r,-.s-aroh was performed to evaluate the feasibility

of translating the Job Aids into a computer-based instruction format.

The Programming Design Guide provided all of the guidance necessary for a user to

implement the Job Aid on any of a large number of computer systems. The specific

Job Aid selected for the effort was the Job Aid for ISD Block 1.5, Select Instructional

Setting. This Job Aid was selected because it was representative of all of the Job Aids

in the first three phases of the ISD model in terms of level of difficulty of translation

into computer-based instruction format.

The Programming Design Guide format used led to a successful implementation of

the Job Aid for selecting instructional settings. As this Job Aid is representative of the

other Job Aids, it is expected that the approach used in the development of the Programming Design Guide will be equally successful for implementing other Job Aids

on-line. The Programming Design Guide research is described in a report by Schulz,

Underhill, Hargan, and Wagner (1980).

The successful use of th.. 13 Job Aids developed earlier in this project for ISD

Phases I, II, and III, indicated chat the interactive job aid approach would also meet

the requirements for Phases IV and V of the ISD model. Therefore, five additional

Job Aids covering the activities in Phases IV and V were developed. Implementation

and evaluation of these Job Aids occurred at the USASC&FG.

The next sect-ion of this report provides a detailed description of the analyses

selection, development, and formative evaluation of the 18 Job Aids developed during

the research effort.

-:B
RESEARCH ACTIVITIES

In this section the research activities involved in the analysis, selection, development,

ard formative evaluation of the 18 Job Aids will be discussed. Where the research activities differed between the development of the first 13 Job Aids (Phases I, II, and III of

the ISD model) and the last 5 Job Aids (Phases IV and V) the research activities will be

described separately. (The first 13 Job Aids will hereafter be referred to as Phase I, II

and III Job Aids. Similarly the 5 Job Aids for Phases IV and V will hereafter be referred

to as Phase IV and V Job Aids.) The development of the Programming Design Guide for

on-line implementation of the ISD 1.5 Select Instructional Setting Job Aid will not be

discussed. The reader is referred to the report devoted to this Guide (Schulz, Underhill,

Hargan, and Wagner, 1980).

The research effort was divided into three mvor tasks:

* Task 1: Analysis and Selection of Job Aids

* Task 2 Development of Job Aids

* Task 3: Evaluation of Job Aids

The activities involved in each of the three tasks are discussed below.

T1\SK 1: ANALYSIS AN!D SELECTION OF .lOB AIDS

Tas1" 1 activiuies were conducted in four steps. These steps are (1) periorm Job Aids

needs assessment, (2) analyze existing job aids, (3) identify candidate Job Aids, and (4)

refine, modify o: complete development of job aids.

Szep 1. Perform Job Aids Needs Assassment

As shown in Figure 1, the -SD Model consists of 19 major ISD activities (blocks).

In the Model, each of these activitieM is expanded into sub-element activities. For example.

ISD Block II.1 Develop Objectives has L .an expanded into the sub-element activities

shown in Figure 2.

In Step 1 a detailed job Aids ntieds aseesbment was performed to establish prioriies

among the sub-elements of each ISD block for which Job Aidr were to be completed.

These priorities were needed because time restraints imposed upon the research would

not permit equal developmental empaasis on all sub-element a-tivx•ies.

The needs ass,-srment for the P.ý. 'I, 1I and III Job tids was performed by means

of a detidled structured interview administered to 11 course developers and 10 trnining

analysts at the USAES. A copy of tbi lieeds Assessment Inventory used for tlh, literviews is contained in Appendix A. The interview dealt with topics such as the f.olowing:

* ISD products produced by the individual.

* The ISD input received by the individ"'l and problems associated with

use of this input.

I6
TASKS SELECTED FOR TRAINING,

INCLUDING CONDITIONS, CUES, INPUTS STANDARDS, AND ELEMENTS.

COMPLETE JOB PERFORMANCE

MEASURE DOCUMENTATION

__JF__ _____

PREPARE TERMINAL ANALYSIS FOR

LEARNING OBJECTIVE THE MENTAL

FOR EACH TASKSKILLS

FOR E T2.1 CATEGORY 2.3.1

DETERMINE APPROPRIATE PERFORM LEARNING

LEARNING ANALYSIS FOR THE

CATEGORY FOR INFORMATION

EACH TERMINAL CATEGORY

LEARNING OBJECTIVE 2.2 2.3.2

PERFORM LEARNING PERFORM LEARNING

ANALYSIS FOR EACH ANALYSIS FOR THE

TERMINAL LEARNING PHYSICAL SKILLS

OBJECTIVE 2.3 CATEGORY 2.3.3

PERFORM LEARNING

ANALYSIS FOR THE

ATTITUDES

CATEGORY 2.3.4

BLOCK

Figure 2. FLOWCHART OF BLOCK 11.1: DEVELOP OBJECTIVES

"lo7
* The ISD output produced by the individual and problems associated with

its production.

0 ISD training available to the individual and additional training required.

0 Availability of existing job aids and when available, the usefulness of the

job aid.

* Ideas for needed additional Job Aids.

The results of the interviews indicated that there was a considerable need for Job

Aids in all sub-elements of the blocks within the first three ISD phases. Specific needs

were identified for each of the ISD blocks except for ISD Block 11.3 (Describe Entry

Bahavior) for which interviewees could not identify any specific sub-elements in need of

assistance.

The major needs for Job Aids identified in the structured interviews were incorporated into a Needs Assessment Review Form which was then administered to 75 instructional development personnel at the USAES and 33 development personnel at the

USASC&FG. For each need listed on the Review Form the respondents were asked to

rate whether they "Strongly Agree," "Agree," "Disagree," "Strongly Disagree," or,

"Cannot Judge." A copy of the Needs Assessment Review Form is contained in

Appendix B.

Major findings from the Needs Assessment Review Form which shaped the content

of the Job Aids are as follows:

(1) Specific guidance is needed for completing all ISD activities.

(2) Examples of the products are needed.

(3) Sources of information necessary for implementing the ISD Model (e.g.,

where to obtain "frequency of task performance" data) are needed.

The Needs Assessment conducted for the Phase IV and V Job Aids was conducted

only at the USASC&FG with 27 instructional development personnel who were currently

working in the ISD Blocks for Phase IV and V. The specific questions covered for each

ISD Block are shown in Appendix C. The major results of this needs assessment are

as follows:

- ISD IV.1 Implement Instructional Management Plan. Aids are needed which

provide mathematical formulas for determining resources needed for implementation of the management plan. Guidance is needed for reviewing and

revising the Instructional Management Plan.

- ISD IV.2 Conduct Instruction. The instruction that instructors receive currently for this ISD block is adequate mad as a consequence, there is no nied

for a Job Aid for the actual conduct of instruction. It was agreed that a

Job Aid for administration and analysis of student opinions concerning

instruction would b4 helpful.

- ISD V.A Conduct Internal Evaluation. The USASC&FG currently uses a

checklist for conducting internal evaluation. Instructional guidance for

answering the questions contained on the checklist was considered to be

needed.

- ISD V.2 Conduct Extemal Evaluation. Respondents exprese the need for

general guidance in performing an external evaluation.

- ISD V.3 Revise System. No respondent was able to identify a clear neci for

a specific Job Aid for this ISD Block.
Step 2. Analyze Existing Job Aids

In Step 2 ISD literature was searched to identify existing job aids that would be

useful for the research effort. Two major sources of job aid information was TRADOC

Pamphlet 350-30, Interservice Procedures for Instructional Systems Development (1975)

and the McDonnell Douglas Corporation Bibliography of Aids for Course Material Development (Logan, 1976). Each job aid identified was carefully analyzed to determine:

* Its relevancy.

* How it fits into the sub-element blocks.

* How it feeds information into other ISD blocks.

* The degree of similarity with other job aids.

* How it might be combined with other job aids to form a more comprehensive Job Aid.

* The amount of the total research effort that would be required to complete

the job aid if it were selected for inclusion in the final listing of the Job

Aids to be completed.

In general, no existing job aid was identified that could easily be translated into the

interactive format proposed for the Job Aids to be developed in the research effort. However, the literature search did produce information that was cataloged for later use in Job

Aid development. For example, much of the ISD materials produced at the USASC&FG

later proved to be extremely valuable in the Job Aid development effort.

Step 3. Identify Candidate Job Aids

Based on the Job Aid needs assessment conducted in Step 1 and the analysis of

existing aids performed in Step 2, candidate job aids were identified in Step 3.

Step 4. Refine, Modify, or Complete Development of Aids

It had been anticipated that existing aids would be identified that would be useful in

the Job Aid research effort. It was thought that those aids would in most cases require

only miýnor modification for presentation in an interactive format. This modification

was to be accomplished in this Step. However, as already noted in the discussion of

Step 2, no job aids were identified that could be easily modified for presentation in an

interactive format. As a consequence, this stop of the research was not performed.

Rather, the information contained within some existing job aids was incorporated into

the Job Aids developed in this project.

TASK 2: DEVELOPMENT OF JOS AIDS

Previous research (Schulz, 1976a, 1975b, Schulz, Hibbits, Wagner & Seidel 1979)

indicated the utility of preparing Job Aids that make use of a flowchart format. While the

Job Aids developed in this project were intended to be used initially off-line in a printed

version, they were to be developed so as to facilitate possible later translation into on-line

computer versions. Therefore, each Job Aid developed in this project is composed of two

documents-Desctiptive Authoring Flowcharts and Job Aid manuals. In addition, there is

a document which provides an introduction to the use of the Job Aids and general information regarding useful resources for ISD implementation.

9
Job Aid Format

Introduction to the Use of Job Aids and Job Aids Resource Manual. The introduction describes the Job Aids, how they were developed, and provides general information

on how to use the Job Aids. The Job Aids Resource Manual provides general information

concerning CODAP (Comprehensive Occupational Data Analysis Programs), conducting

field surveys, and the use of various panels of personnel in the ISD process. Figure 3

shows the major topics included in the Introduction to the Use of Job Aids and Job Aids

Resource Manual.

Descriptive Authoring Flowcharts. The Descriptive Authoring Flowcharts are the

primary documents used in the Job Aids. They direct the user to specific guidance,

examples and references provided in the Job Aid Manuals. Figure 4 shows a representative page of descriptive authoring flowcharts. The flowcharts are easy to use in that they

include only five flowchart symbols. The symbols are standard flowcharts symbols except

for the one that looks like an open book or manual. This symbol is used to cue the user

that he/she is to go to the Job Aid Manual for additional information or guidance. The

directions contained within the flowchart symbols are frequently supplemented by narrative information provided in the margin of the Descriptive Authoring Flowcharts. This

narrative information is sufficient to provide an overview of the activities that are to be

performed in the Job Aid. The Descriptive Authoring Flowcharts are also sufficient to

act as an executive summary for supervisors or commanders.

Job Aid Manuals. As stated above, the Job Aid Manuals provide the specific guidance, examples, and references necessary to help the user produce the products for the

ISD activities covered by the Job Aid. In addition, each Job Aid Manual has associated

with it one or more worksheets to use in the development of each product. When the

user is referred to the Job Aid Manual an attempt has been made to provide four types

of information: (1) the purpose for performing the activity called for in the flowchart

block, (2) sources of information for performing the activity, (3) specific step-by-step

guidance for performing the activity, and (4) if the activity involves filling in a worksheet,

an example of what the worksheet looks like after the activity has been performed. In

addition, the flowchart block for which guidance is gj',.cn is reproduced at the top of the

page of the Job Aid Manual. Figure 5 shows a representative page from a Job Aid Manual.

{Note that flowchart block 9 is also represented in Figure 4.)

As previously stated, the Job Aids were intended to alleviate the needs expretsed

during the needs assessment survey. In some cases, guidance is readily available from

other sources for meeting the needs; or in a few cases, time constraints placed on the

research project would not permit the development of some needed guidance. As a consequence, the Job Aids do not always provide complete guidance or information for all

activities in the ISD Block covered by the Job Aid. Therefore, a brief summary of the

Job Aids for each ISD Block is provided below.

The Job Aids for the first three Plhses of the ISD Model are available as Army

Research Institute documents (Schulz & Farrell, 1980a, 1980b, 1980c, 1980d, 1980e,

1980f). Job Aids for the remaining phases are in press at this time (Schulz, Wagner &

Hillelsohn). The Job Aids are grouped by ISD Phase. That is, there is a separate Descriptive Authoring Flowchart mmaual for the analysis, design, development, implementation,

and control phase of the ISD model. Similarly, there is a Job A'd Manual for each of the

to
INTRODUCTION TO THE USE OF JOB AIDS

What are Job Aids?

What is the Classification System Used in TRADOC Pamphlet 350-30 and

Job Aids?

What Sources of Information Were Used in the Development of

Job Aids?

What Job Aids are Presently Available and What Do They Consist of?

What are the Main Parts of the Flowchart Manual?

What Flowchart Symbols are Used in the Flowchart Manual?

What Instruction is Provided Within the Flowchart Symbols?

What is the Purpose of the Supplemental Information Provided in the

Flowchart Manual?

What is the Flowchart Block and Page Numbering System?

How Do I Use the Job Aid Manual and What are the Main Parts of it?

What is the Purpose of the Partial Flowcharts at the Top of Some Pages

of the Job Aid Manual?

What is the Purpose of the Questions Written in Script That Appear on

Some Pages of the Job Aid Manual?

What is the Purpose of the Completed or Partially Completed Examples

of Worksheets?

What Does This All Mean and What Do I Do Now?

JOB AIDS RESOURCE MANUAL

How Can the Resource Manual Help Me?

What Are Some Sources of Job Significant Data?

Which Sources of Data are Included in the Resource Manual?

Figure 3. MAJOR TOPICS INCLUDED IN THE INTRODUCTION TO THE

USE OF JOB AIDS AND JOB AIDS RESOURCE MANUAL

I'
Taskkvetr ncue lltsspelre

Inventry Noin this skill level.

Guidance for Obtaining * Task inventory should be available as output

Task Inventory List frorn ISO 1. 1 Analyze Job.

* If procedures for listing tasks by skill level

w~era not complete. sie persons resoolnSibae

for compiling task list.

*When list has been obtained return 'a Manual.

ISO 1.2 Salect Tasks for Training, Bliock 6,

page Al.1.

Obtan Ctiuc3Fl ak *Critical task identilication and final selections are ricoroedo on the Ciaicl:3

:1Identification Stoees Task Identification Sheet (MT). The MT Sheet, theref ore. provides

I' CTI) documented rtcord for these deitsions wid the tcaut (fo each.

*For quidtance in obtaining CTI Sh~eets, see M;nuai. papgi A'.2

Obtain Ikforration *I nlOrrnallonf fteedt lot "Aettrg UP" the CT Sheet includits INh faowinq

Netetgry fot Setting .- MOS code

Upo CTI 5Wt( - skill level - Task 10 hurnitit

- Talk tinventory

F Or soejicAc 01 M1twWrmA toi M&ee0 ?J.page A- 11

i ~.r nVIShiem. 0 Nu'beir tafh CTI 911et litj U~ many as .vecwv wo '1110~n th'e wAsk~

Oa a ~4r"Ixt i Attach a copy ot the task Iftivntory to the Itvt C'1 Sheet to hillp oVC

M rOS Code. i~ttvte'lst dc h uta ee

Skill Lirvit. Task frtVte1k tWhc titntof

*Euilhze.t See Miazi P41At4.

Attach Taul Inetor

Figure 4. REPRESENTATIVE PAGE FROM DESCRIPTIVE

AUTHORING FLOWCHARTS

12
from block 8

04W. inItcfwtnec * Itwqt* wd for "u" up" Ow ClI Stat mduft ft taMl~wM.

U9 CTI - Tulw Skiff UIMmt omtvJ I" , 

to block 10

Whatu do Ic 1need ~whedo I ge it?

* The following information is needed before you can set up the CTI Sheet. Sources

for locating each item of information are identified and in most cases throughout

this manual the source will be rated as "excellent," "good," or "fair." Understandably not all sources of information will be readily available to you.

Infornmtion Required Source(s) Remuks

MOS and Skill Level O Supervisor 0 Excellent source

Sdesgn*tion * AR 611.1 (for a new MOS) 9 Excellent source

0 Output of ISO 1.1 Analyze Job 0 Excellent source

Task ID. Numben 0 Numbering system prescribed in

TRAOOC Circular 351.28. if already assigned * Excellent source

* Temporary numbering system of your

own design. e.g. I through 5W. if

auth•rized nunrthm have not been

assined 0 Excellent source

Task inventory * Outiut of ISO 1.1 or job nalyisit e Excellent source

0 List currently in use O Good &ource

Figure 6. REPRESENTATIVE PAGE FROM JOB AID MANUAL

'3

-Ik
PHASE I - ANALYZE

ISD 1.2 Select Tasks/Functions

The Job Aid for this block assists the user in making the decision of which tasks are

to be selected for training and which are to be rejected. The output from using the Job

Aid is:

e A final list of tasks for training.

* A list of tasks rejected.

* A summary of the data upon which the selection decision was based.

ISD 1.3 Construct Job Performance Measures

This Job Aid provides guidance for performing a Ask analysis of each of the tasks

selected for training. The output from use of the Job Aid is:

* Information concerning the conditions under which each task is performed.

* Cues indicating when it is performed.

* Standards indicating how well it should be performed.

* References and training tips.

* Identification of task elements or performance steps.

This information is recorded in a format which can be placed in the Soldier's Manual.

understood by the soldier lteaning the task, and used as an evaluation tool by someone

supervising the task.

ISO 1.4 Analyze Existing Courses

The purpose of this Aid is to eliminate duplication of effort in several ISD proctsses.,

"Thus, it contributes to one of the overall goals of ISb--to increase the cooperative development and use of training throughout the military servics, Specific guidance is provided

for analyzing existing courms. T"he output from this Job Aid can be:

SThe generation of an initial task list for a particular MOS.

SIThe identification of suitable courses for training. *• The identificltion ofcoumOes, or larwt of coums. whidii may be suitable

after revision by coupe dewelopment persounel.

ISO 1L6 Select Instructional Setting

The purpose of the Job Aid for this ISD Mlock is to trovide guidance in choosing

irttuctionsl settUeB (training locations) for tasks selected for training. Use of the Job

Aid will result in a listing of all critical taWks in which each task is assijned for training

to one of the following instructional settingp:

* Institution (resident school training),

* Supervih-d on-the-job training (SOJT).

* Self-study.

14
PHASE II - DESIGN

ISD 11.1 Develop Objectives

The purpose of the Job Aid for this ISD Block is twofold. First, it guides the

user in the actual writing of correctly stated learning objectives. Second, it explains the

mechanics of deriving learning objectives from Task Summary Sheets (developed in ISD

1.3 Construct Job Performance Measures) by means of learning analysis. The outputs

from use of this Aid are:

0 Terminal learning objectives for each task to be trained.

* Learning objectives for task elements.

ISD 11.2 Develop Tests

In this Job Aid the user is provided guidance in deciding whether tasks and task

elements should be tested by means of written or performance type test items. Information and guidance is also provided for the preparation of written test items. The Job

Aid does not cover the development of performance tests, nor does it cover preparation

of administrative materials or validation of test items. The output consists of written

test items which will be used to test the ability of soldiers to correctly perform critical

tasks or task elements.

ISO 11.3 Describe Entry Behavior

The goal of the Job Aid for this ISD Block is to provide the guidance necessary to

test the entry behavior of trainees and to utilize this information to adjust the beginning

point of instructional units; and to verify or revise the assumptions made in the learning

analysis performed in ISD 11.1. Develop Objectives, about entry skills and knowledges to

be trained. The output from using this Job Aid is a validated learning anulysis for each

task.

ISD 11.4 Determnne Sequence and Structure

This Job Aid provides guidance for sequencing and grouping learning objectives to

help assure that when instructional nmaterias are developed, each learning objective is

placd in the best relationship with other lemaring objectivt% to:

* Produce the most leamring n tile shortest period of time.

o Hetp the soldier make the trnsition from one skill or body of knowledge

to another,

* Assure that supporiJng (enabling) knowledges and skills aremquired before

suhsAqutnt subi-t matter is introducmd.

LUs of the Job Aid will result in the proper sequenting of all tasks rad task elements.

Is

., !
PHASE III - DEVELOP

ISD I11.1 Specify Learning Events/Activities

The purpose of the Job Aid for this ISD Block is to help the user to select appropriate learning events and activities for training sessions in which critical tasks will be

taught. Learning research has identified learning activities and conditions which are

essential for teaching different kinds of learning objectives. In this Job Aid, the user

learns what those activities and conditions are. Use of this Job Aid will result in a listing

of learning activities that could be used for teaching the learning objectives for each critical task.

ISD 111.2 Specify Instructional Management Plan and Delivery System

The Job Aid for this ISD Block enables the user to apply a systematic approach to

the selection of delivery systems (media) for presenting instruction to trainees. This

systematic approach to media selection requires consideration of the nature of the objectives: their learning categories/subcategories, learning activities, and instructional settings.

Such factors as equipment and facility constraints, cost of procurement, cost of replacement, and development costs are also considered in the Job Aid. However, the Job Aid

does not provide guidance for conducting a cost-effectiveness analysis of media, nor for

the actual development of the Instructional Management Plan, The output from this Job

Aid is a complete identification of the characteristics of a delivery system to be employed

for teaching each task and task element.

ISD 111.3 Review/Select Existing Materials

In the Job Aid developed for this ISD lock guidance is provided to help instruc.

tional developers evaluate existing materials in order to determine their usefulness in tle

training program they are preparing. Thie Job Aid results in a list of all existing materials

vdiich have been considered for use in the cxurse developmnem activity. Each item of

SMAtrifal is evaluated as to whether it should be accepted or rejectd for us, in the course.

If the nmaterial is to be accepted. suggestions for revsions (if nxeded) are documented.

Each item of material is therefore classified as:

* Accept for use without rtmvision.

0 ,Rvise for use.

ISO 111.4 Develop Instruction

11he Job Aid for this ISD Block has two goals. T1he main goal of ,11e Job Aid is to

•--,,ist in producing training materials for the following media: audio-only, audio-vtsual,

and wtitten text. A secondary goal is to familiarite the user with the ways in which the

various production persotmeý can help in the instructional development effort. The

product resulting from use of the Job Aid is a first draft of instructimml nmatcials.

16
ISD 111.5 Validate Instruction

Guidance for conducting twr; types ot instrucuiunal validation is provided in the Job

Aid for this block of the 1SD mr lel. They are: individual validation trials and group

validation trials. Validation as used in this block of the ISD is the process by which

material (course, module, lesson, etc.) is uested and revised until it teaches 

what it was intended to teach. The Job Aid defines each of these types of trials and

provides guidance in when and how to use them. The outcome from use of this Job

Aid is a set of validated instructional units.

PHASE IV - IMPLEMENT

ISD IV.1 Implement Instructional Management Plarn

The purpose of the Job Aid developed for this ISD Block is to provide guidance in

the techniques to use for reviewing and revising (when necessary) the Instructional Management Plan developed in ISD 111.2. The output from the use of this Job Aid is an accurate

Instructional Management Plan suitable for implementation in ISD IV.2 Conduct

Instruction.

iSD IV.2 - C :'.duct Instruction

Tf, i Job Aid emphasizes the collection of student performance and survey data

during vne conduct of instruction. It does not cover the actual conduct of instruction,

inasmuch bs this guidance is available from a number of other sources. Student Survey

Forms and Instructor Survey Forms are provided which permit students/instructors to

provide input as to needed changes in the instructional program. The Job Aid provides

guidance as to when th.' survey forms should be administered and how to summarize

data received from them. The output from this Job Aid will be student/instructor data

useful for determining needed changes in the instructional program. Student/instructor

data are addressed in the internal evaluation report prepared in ISD Block V.1.

PHASE V - CONTROL

ISD V.1 Conduct Internal Evaluation

The purpose of the Job Aid for this ISD Block is to provide the uSer with guidaiee

in conducting an internal evaluation of courses which were developed using the Job Aids

described above. The Job Aid assists the user to answer the following questions:

* "Is the instruction providing the studentz with the necessary knowledge

and skills to meet the objectives in a satisfactory manner?"

* "To what extent did the development effort conform to the ISD proc.-ss?"

* "flow good are the products resulting from the various ISD activities?"

A detailed checklist and guidance for its use is provided for gathering the information

needed to answer the above questions. The output from this Job Aid is an Internal

"Evaluation Report which is used to modify (where necessary) the products from previous

ISD Blocks.

17
ISD V.2 Conduct External Evaluation

The purpose of the Job Aid for this ISD Block is to help the user to conduct

external evaluations of trainin.; programs. Emphasis is given in the Job Aid to determine

whether a schooi-taught course adequately trains its graduates. Graduate and Supervisor

Survey Forms are provided in the Job Aid and instructions are given as to how these are

to be administered and how the data derived from the survey are to be summarized. The

output from use of this Job Aid will be an External Evaluation Report which will be used

in the next ISD Block, V.3 Revise System, to determine revisions needed in the instructional program.

ISD V.3 Revise System

Guidance is provided in this Job Aid for making a careful analysis of data relating

to doctrine changes, soldiers' performance on Skill Qualification Tests (SQT), and instructional program deficiencies reported in the Internal and External Evaluation Reports.

Aigcrithms are provided for determining required course revisions based on the above

information. The outcome from this Job Aid will be a report detailing the revisions

needed in the instructional program.

The -lob Aids described abuve were formatively evaluated at the USASC&FG. The

formative evaluation was conducted in Task 3, Evaluation of Job Aids. A discussion of

Task 3 follows,

TASK I EVALUATION OF JOB AIDS

Evaluation of Job Aids - Level 1 (Informal Review of Known Aids)

The first level evaluation in this project was carried out as part of the Job Aid development effort, and is described more fully in '

t

js t 1. Project staff reviewed published

guidance And known ISD aids, attended workshops on the uae of aids such as the Instruc.

tional Quality Inventory (IQI)(Ellis, et al., 1979), and visited a number of TRADOC

installations to learn about any useful forms. instruments, or guidance that should be

incorporated into the Job Aids.

Evaluation of Job Aids - Level 2 (Formative Evaluation of New Aids)

Previous information indicated that all ISD Phase 1-1i1 activities were being cotitinu.

ously performed at the USAES at Ft. Belvoir, Va. However, in early visits to USAES, it

was discovered that it would not be possible to collect evaluation datar for ISD Phase I

JoL Aids inasmuch as there was no one who would be working in this area during the

proposed evaluation period. Toe USASC&FG at Ft. Gordon, Ga., was chosen as the

alteriative location, and evaluation plmas were modified to accommodato the new site.

In Task 1, those ISD activities in most need of procedural guidance mad assLstance

were identified. It was against these needs that the Job Aids were evaluated. The focus

of this evaluation was whether or not thle problems noted during the needs assessment

wore solved or ameliorated by the Job Aids.

A detailed formative evaluation plan was prepared which .%t no procedures for

gathenng evaluation data. The evaluation plan was described to USASC&FG personne,

and approval obtained prior 'w its implementation.

'B

*
One of the features of the evaluation plan was the collection of baseline data. Its

purpose was to determine how well school personnel were performing ISD Phase I-I11

activities without the assistance of the Job Aids to be provided in this project. A copy

of the instrument used for the collection of baseline data is shown in Appendix D.

Following the collection of baseline data the Job Aids were provided to Ft. Gordon

personnel to use in their ISD activities. USASC&FG personnel were given a very short

training session in the purpose and use,ýof the Job Aids. It was emphasized in the training session that the focus of the evaluation was the Job Aid itself, not the person who

was working with it. Job Aid deficiencies were to be pointed out, but so too were those

aspects of the Job Aid that were deemed to be helpful (e.g., more efficient procedures

or specific guidance, etc.).

After each individual completed using the Job Aids he/she was interviewed concerning the practicality and usefulness of the Job Aids. The structured interview form used

for these interviews is shown in Appendix E. In addition to interviewing Job Aid users,

their immediate supervisors were also interviewed. In this latter interview the major

emphasis was the supervisor's perceptions of any reduction or impiovement in the

efficiency of their subordinate's work from the use of the Job Aids. Finally, all personnel (Job Aid users and their immediate supervisors) were asked to identify specific

changes, deletions, or additions that would increase the accuracy, timeliness, and usefulness of the Job Aids. Any serious problems that were identified were attended to as

soon as possible and revisions made to the Job Aids.

In the evaluation of Phase IV and V Job Aids, "expert" review was the main

approach used in assessing the Job Aids developed for these ISD Phases. Opinions

regarding user acceptance were also obtsined, as was information on problems that these

personne, expected to experience while using the Job Aids.

Evaluation of Job Aids - Level 3 (Impact on Instruction)

The Job Aids were designed to facilitate many steps in the ISD process--and as such,

ultimately should have an effect on instructional quality and relevancy. Those criteria

are measured by a change in the behavior of students who undergo training in the course

resulting from the ISD process. Although accumulation of student data could have

indirectly served to assess the adequacy of the Job Aids, it was decided that this activity

not be performed as part of the formative evaluation of the Job Aids. Rather, more

attention was paid to the determination of Job Aids' usefulness and the need for

improvements.

Findings

The formative evaluation plan proposed in this project produeme much useful

information for the revision and improvement of the Phase 1-ill Job Aids. The plan

was followed for the Phase 1-Ill Job Aids to the extent that:

- IHseline data were obtained from USASC&FG personnel regarding their

ISD activities before Job Aids were introduced.

- Acceptance and use data were obtained following intervention with the Job

Aids from some of the personnel from whom baseline data were collected.

19
The data obtained by the procedures outlined above were examined to determine

the acceptance and utility of the Job Aids, and to see if any improvements needed to be

made that would increase their efficiency and/or effectiveness. The effects of the Job Aids

were most readily observed when the same USASC&FG personnel who participated in

Task 1 (needs assessment) were the ones from whom the data were obtained regarding

their use of the Job Aids. Data obtained from others highlighted the generality of the

Job Aids' usefulness.

Background data were collected from personnel at U.S. Army Signal Center and Fort

Gordon, Fort Gordon, Ga., who were engaged in activities covered by ISD Phases I-III. Background Information Forms and ISD Activity Checklists (see Appendix D) were administered to over 200 individuals. Approximately 50 of these individuals were visited one

week later and interviewed by project staff concerning their ISD activities during the

preceding two-week period. The specific individuals interviewed were selected on the

basis of the ISD Activity Checklists completed the week before. In this way, a comprehensive set of information was gathered regarding the job activities of personnel working

in the ISD process. The problems they encountered, the time to perform certain

activities, and their opinions regarding the effectiveness and efficiency of the ISD

process (prior to the intervention of Job Aids), were among the topics of most interest

that were accumulated in the baseline data. Similar sets of data were obtained follo'ing

their use of the Job Aids.

The results of this evaluation were unexpected. First, the intensity and extent of

needs expressed by the USASC&FG and USAES personnel was much greater than had

been anticipated. As was described in Task 1, Job Aids were perceived as needed in all

blocks of ISD Phases I-III, and covered practically all activities within each block. These

needs were supported by the baseline data collected in the ISD activity sheets. These

data showed that the ISD activities took considerable time, due in part to the inaccurate

and untimely information generated by inexperienced individuals. With this situation

as the baseline condition, the reaction to the Job Aids was judged as positive.

All (100%) of the individuals given the Job Aids thought they were Good or

Excellent. The only negative comments were editorial in nature. Problems with the

flowchart format, as seen by its developers, were not confirmed by the users. Follow-up

structured interviews (see Appendix E) with 57 of the users, elicited so much support

for the draft Job Aids, that a change in project plans seemed essential. For example,

there was the refusal by some of the users to part with the draft Job Aids on which their

editorial comments were written, unless they could be replaced with other copies. Also,

eopies of the Job Aids were requested by other personnel in the school who were not

participants in this study, but wanted to use them in their own ISD work. In addition,

the Phase I and II Job Aids are being incorporated into a USASC&FG course that

teaches the analysis and desigp of instructional development. This is only some of the

evidence of acceptance that influenced the change in direction taken in the project.

Fvaluation plans were curtailed and resources shifted to Job Aid development which

permitted a broader set of activities t0 be covered by each Job Aid than had been

proposed initially.

In the formative evaluation of Phase IV mad V Job Aids the data emphasized

editorial changes to the documents. While it is true that some of the same personnel who

responded in the needs assessment, evaluated the Phase 1-I1l Aids, this was not the case

for the Phase IV and V Job Aids. Due to time and resource constraints, the ultimate users

were not able to participate in the evaltzation. Rather, only "expert" critiques of the

draft Aids were obtained and the Job Aids modified accordingly. However, these Phase

IV and V Job Aids will be sent to USASC&FG for review prior to their release by ARI

for general use.
RECOMMENDATIONS

A major shortcoming of the evaluation was the fact that the continuous nature of

the ISD process was not accommodated. That is, each Job Aid was prepared for use as

a "stand-alone" set of documents that would assist those engaged in the activities covered

by a given ISD block. It was assumed that the information feeding into that blockthe information necessary for using a particular Job Aid-would be correct and available

in a timely manner. This meant that the output or products of one Job Aid would have

to be available for use by those who needed that product as input to their own Job

Aid. Thus, when the evaluation of Phases I-III Job Aids was planned to occur simultaneously, an adequate evaluation of the impact or utility of the Job Aid was made

extremely difficult and impractical within the constraints of this project's time and resources.

As one result, the focus of the evaluation shifted from a comparison of pre- and post-use

measures of the Job Aids, to a detailed critique and debugging of the Job Aids by actual

and potential users. There is still the need to evaluate the utility and impact of the Job

Aids when applied to the ISD process as a whole.

First, the Job Aids need to be consolidated. There are many examples of redundancy

and duplication of forms and effort that should be eliminated.

Second, the Job Aids should be used throughout the entire ISD process for a given

course development effort. In this way, a realistic, longitudinal evaluation of the Job

Aids can be accomplished. The contribution of each Job Aid to the ISD process, and the

eventual impact upon instructional quality and relevancy, could be assessed in this type

of evaluation.

21

' . . ¥ - -..

"- -- . . *" *"..... - ' ';• '•.• I " '•:•, ,, *, *"**.*,o*,.,,, :-• **--, :.. , . ,•,,•, ":. .. . .-
REFERENCES

Aagard, J. A., & Braby, R. Learning guidelines and algorithms for types of learning

objectives (TAEG Rep. No. 23). Orlando, Fla.: Training Analysis and Evaluation

Group, 1976.

Briggs, L. J. Handbook of procedures for the design of instruction. Pittsburgh, Pa.:

American Institute of Research, 1970.

Conoley, J. C., & O'Neil, Jr., H. F. A primer for development test items. In H. F.

O'Neil, Jr. (Ed.), Procedures for instructional systems development. New York:

Academic Press, 1979.

Crawford, M. P. Concepts of training. In R. M. Gagne (Ed.), Psychological principles in

system development. New York: Holt, Rinehart, & Winston, 1962.

Dyer, R. F., Matthews, J. J., Wright, C. E., & Yudowitch, K. L. Questionnaire construction manual (ARI Rep. No. P-77-1). Ft. Hood, Tex.: U.S. Army Research Institute

for the Behavioral and Social Sciences, ARI Field Unit, July 1976.

Ellis, J. A., Wulfeck, W. H., & Fredericks, P. S. The instructional quality inventory.

I1. User's manual (NPRDC Special Rep. 79-24). San Diego: Navy Personnel

Research and Development Center, August 1979.

Finan, J, L. The systems concept as a principle of methodological decision. In R. M.

Gagne (Ed.) Psychological principles in system development. New York: Holt,

Rinehart, & Winston. 1962.

Logan, R. S& An exploratory computer.based model for comparing ins~ructional alternatives and a bibliography of aids for course material development: A two part study

(Vol. I1). St. Louis, Mo.: McDonnell Douglas Astronautics Corapany.East,

September 1976.

O'Neal. A. F., & O'Neal, H. L. Author management systems. In H. F. O'Neil, Jr. (Ed.),

Issues in instructional systems detvelopment. New York: AXadernic Pre.ss, 1979.

Schulz. It. E. Lesson Al ONIFORAIS: An authoring aid for the PLATO IV CAI system.

(HumRRO Research Product ED.75.6). Alexandria, Va.: Hutman Resources Research

Organization, April 1975a.

Schulz, R. E. MONIFORMS as authoring aids for the PLATO IV CAI system. (HutnRRO

Tech. Rep. 75-6). Alexandria, Va.: Hunum Resources Itese.arch Organization,

May 1975b.

Schulz, R. E. Job aids: Descriptive authoring flowcharts for Phase IV-IMPLEMENT

of the Instructional Systenms Development model. Alexandria, Va.: Army Research

Institute, (in press).

22
Schulz, R. E. Job aia manuals for Phase IV-IMPLEMENT of the Instructional Systems

Development model. Alexandria, Va.: Army Research institute, (in press)

Schulz, R. E., & Farrell, J. R. Job aids: Descriptive authoring flowcharts for Phase IANALYZE of the Instructional Systems Development model (ARI Research Product

80-13). Alexandria, Va.: Army Resex.rch Institute, May 1980a.

Schulz, R. E., & Farrell, J. R. Job aid manuals for Phase I-ANALYZE of the Instructional Systems Development model (ARI Research Product 80-14). Alexan'-.ria, Va.:

Army Research Institute, May 1980b.

Schulz, R. E., & Farrell, J. R. Job aids: Descriptive authoring flowcharts for Phase IIDESIGN of the Instructional Systems Development model (ARI Research Product

80-15). Alexandria, Va.: Army Research Institute, May 1980c.

Schulz, R. E., & Farrell, J. R. Job aid manuals for Phase I--DESIGN of the Instructional Systems Development model (ARI Research Product 80-16). Alexandria, Va.:

Army Research Institute, May 1980d.

Schulz, R. E., & Farrell, J. R. Job aids: Descriptive authoring flowcharts for Phase IIIDEVELOP of the Instructional Systems Development model (ARI Research Product

80-17). Alexandria, Va.: Army Research Institute, May 1980e.

Schulz, R. E., & Farrell, J. R. Job aid manuals for Phase III--DEVELOP of the Instructional Systems Developmrent model (ARI Research Product 80-18). Alexandria,

Va.: Army Research Institute, May 1980f.

Schulz, R. E., Hibbits, N., Wagner, H., & Seidel, R. J. On-line authoring aids for instructional design (ARI Technical Report 387). Alexandria, Va.: U.S. Army Research

Institute, July 1979.

Schulz, R. E., Underhill, W. G., Hargan, C. S., & Wagner, H. Computer interactive author aiding:

Development of programming design guides (HuniRRO Final Rep. FR-ETSD-80-9).

Alexandria, Va.: Human Resources Research Organization, June 1980.

Schulz, R. E., Wagner, H., & Hillelsohn, M. J. Job aids: Descriptive authoring flowcharts for

Phase V -- CONTROL of the Instructional Systems Development model. Alexandria, Va:

Army Research Institute, (in print).

Schulz, 11. E., Wagner, |i., & Hillelsohn, M. J. Job aid manuals for Phase V - CONTROL of

the Instructioiwl Systems Development model. Alexandria, Va.: Army Research

Institute, (in print).

U.S. Air Force. Instructional Systems Devetopment (AF Manual 50-2). 1970.

U.S. Army. Systems enginee-ing of training (CONARC Reg 350-100-1). Ft. Monroe, Va.:

U.S. Army Continental Command, 1968.

U.S. Army. Interservice procedures for instructional systems development (TRADOC

Pam 350-30). U.S. Army Training and Doctrine Command, 1975.

Vallanc, T. R., & Crawford, M. P. Identifying training needs and translating them into

research requirements. In R. Glaser (Ed.), Trining research and education. Pittsburgh, Pa.:

University of Pittsburgh Press, January 1962.

23
APPENDIX A

NEEDS ASSESSMENT INVENTORY FOR ISD PHASES I, II, AND III

JOB AID DEVELOPMENT

I. IDENTIFICATION

1. Name:

2. Title:

3. Organization:

4. Are you a subject-matter expert in any military area(s)? If so, what area(s)?

5. Have you ever been an instructor at USAES? If so, what courses, positions,

when, how long, etc.?

6. How long have you worked in ISD?

II. JOB

1. In which USAES courses do you work or have responsibility for someone

els's work?

24

S' A
2. What are your responsibilities?

3. In what ISD activities are you presently working or supervising others?

How long for each?

4. What additional ISD areas have you worked on (or supervised in) during the

past year? How long for each?

What did you produce?
III. INTERFACE

1. From whom do you (or the individuals you supervise) receive input?

a. What problems, if any, do you (or your staff) have in using the input?

b. How much time per day/week/month do you (or your staff) spend on

the related tasks because of the problem?

c. What control do you have over the quality of the input?

d. Do you have any sugestions as to how this input could be improved?

I;I
2. What are the Output products resulting from your (or the individuals you

supervise) ISD activities?

a. To whom (or ISD block/activity) are the products delivered?

b. What feedback do you receive concerning the quality of the products?

c. (Supervisors only) How frequeitly do you send work back to your

staff for revision? Explain.

d. (Non-supervisors) How frequently does your supervisor send your work

back for revision? Explain.

27

.t:
IV. TRAINING AND JOB AIDS

1. What training have you had (military or civilian) on subjects related to

instructional systems development, educational technology, and the like?

2. What addtional training, if any, would you like to receive?

3. What job aids are available to you or your staff for performing ISD activities?

a. Which of these do you actually use?

b. How useful ar. the job aids?

c. How oould tUhey be improved?

:•4
4. What locally produced forms do you (or your staff) complete relevant to

ISD activities?

5. Which, if any, of these forms do you/your staff use? (Show interviewee

appropriate forms shown in ISD manual.)

6. What happens to the completed form? (How is it used, by whom, etc.?)

V. OTHER PROBLEMS

I What other problems do you encounter in your work that have not been

mentioned eawier?

,.-•- ' .
2. Do you have ideas about possible improvements in the ISD process that would

help solve this problem(s)?

VI. IDEAS FOR ADDITIONAL AIDS

1. If you had the choice of selecting five ISD aids for development, what ISL1

areas would they cover?

2. Do you know someone who can provide additional ideas on the subjects we

have discussed?

30
APPENDIX B

NEEDS ASSESSMENT REVIEW FORM FOR PHASES 1, II, AND III

JOB AID DEVELOPMEN1

NAME:

TITLE: Telephone Numoier:

ORGANIZATION (Division, Lrbnch, etc.):

How long have you worked in ISE?

Job Aids are being pr •pared for individuals who are performing the activities caled

for in TRADOC Pamphlet 350-30 (Interservice Procedures for Instiuctior.al Systems Development), Phases 1. I1, and Il1. These aids wili be given to USAES personnel to tly out and

evaluate in the months ahead.

In January and February 1978 some USAES individuals from ITADD and CDD were

surveyed as to the kinds of hell. they need in their work. The statements presented on

the following pages remresent the outcome of thie survey

We would like :! -r you to read each statement and check whether you Agree or Disagree with it. The scale presented has four points so that you can indicate the level of

your agreement or disagreement. If you cannot judge a particular statement because you

have no. worked in that phase of ISD, check the box labeled, "Cainot Judge." Please

check the ' Cannot -fudge" box only when absolutely necessary. A page has also been

provided for you to add other assistance you ne3d in your work.

Thank you for your cooperation. Youre response to the statements, and the added

comments, will provide greater assurance that the aids that we develop will indeed provide

the needed assistance to you in your job.

31

--- -
GENERAL Strongly Strongly Cannot

Agree Agree Disagree Disagree Judge

1. We need complete approved examples

of the products we are to produce. El El [] El El

2. We need help in finding sources of /

information. El El El El El

3. We need a general orientation to the

course development process. El l El El E

ISD BLOCK 1.2: SELECT TASKS/FUNCTIONS

4. We need help in identifying and

defining standards for tasks. 0 El El El El

5. More information is needed on con.

dition,* of task performance. El El El El C

6. We need assistance in selecting critica!

tasks at the appropriate skill levels. El El 0] El El

1. Examples of good task statements are

needed. El El E 0

8. Clearly spuiled out procedures aie needed

for identifying and describing critical

tasks and associated performance

standards. El0 0

ISO BLOCK 1.3! CONSTRUCT JOB

PERFORMANCE MEASURES

9. More accurate information is neetted

iegarding the stquance of steps In task

performance. 0 " 0 0

(la Realistic information is needed regarding

ac•uat job pefortmance, available equip.

ment, and rllerenceaL D 0 0 0 L

11. More complete and detailed Information -s

needed rearding tosk performance require.

metts to include anabling stlisiknowledieL. 0 0 LI 0 0

32
Strongly Strongiy Cannot

Agree Agree Disagree Disagree Judge

ISD BLOCK 1.4: ANALYZE

EXISTING COURSES

12. Documentation is needed on previous

training analyses and decisions. El El El El El

ISD BLOCK 1.5: SELECT

INSTRUCTIONAL SETTING

13. More guidance is needed on how to select

from all possible training environments. El 0 El El El

14. We need more guidance on the purpose,

content and format of Soldier's Manuals. E- [] El El El

ISD BLOCK 11.1: DEVELOP OBJECTIVES

15. More guidance is needed on how to prepare

learning objectives. El El El [) El

16. We need guidance on how to identify and

analyze a task's skill and knowledge

components. E E El 0l [0

17. Examples of appropriately identified skills

and knowledges are needed. 0 El El E- El

ISD BLOCK 11.2: DEVELOP TESTS

18. Wa need a standard forrat for preparing

SorV s. 0 D 0 0

19. We need a redesigned "Brown Book"

containing examples of SQTs. 0 0 0 0 0

"NO. There Is a need fo( aids in preparing

course criterion tetts. El 0 0 0 E

21, We need guidance on how to develop stlf

evalustion checklists end tcoresheem C 0 0 0 El

22. Guidance on how to write good test questions is neded. 0 0 El 0 0

231 Guidgnte is tiedod on how to determine

the difficulty of test qu6stans, 0 l 0 0 0

11• 33
ISD BLOCK 11.3: DESCRIBE Strongly Strongly Cannot

ENTRY BEHAVIOR Agree Agree Disagree Disagree Judge

No Aids waire reccmmended

ISD BLOCK 11.4: DETERMINE

SEQUENCE AND STRUCTURE

24. Examples of course maps are needed. C) r] 0 0 El

25. Examples of prescriptive outlines are needed. 0 11 0 .0 0

ISD BLOCK 111.1: SPECIFY LEARNING

EVENTS/ACTIVITIES

26. We need help Wo identify the critical

instructional materials. 0 0 0 0 0

ISD BLOCK 111.2: SPECIFY INSTRUCTIONAL

MANAGEMENT PLAN AND DELIVERY

SYSTEM

27. We need guidance on how to select media

appropriate to spitific objectives. 0 0 0 0 0

28. Guidance is needed on how to develop

an instructional management plan. 0 0 0 0 0

ISD BLOCK 111.3: REVIEW/SELECT

EXISTING MATERIALS

29. We need guidance on how to locate, select,

and review existing materials. 0 0 0 0- 0

ISD BLOCK 111.4: DEVELOP INSTRUCTION

30. Guidance Is needed on the selection, us,, and

evaluation of visuals. D 0 0 0 03

31. Aids ere needed for ttoryboarding soundtdido emns. 0 0 0 0

32. Gutiaince is needed on how to write effectively. Q 0 0 C 0

33. Aids are naded for obtaining and using

studtft data to Identify weakrm tn

trentin0 m(ie3a. 0 0 0 0 0

ISO BLOCK 111.5: VALIDATE INSTRUCTION

No Aids w*re reco mmended.

A, k

4r.
Please indicate in the space below other assistance you need in your work. Also, please identify, if possible,

the ISO Block to which this aid will be appropriate.

ASSISTANCE NEEDED ISO BLOCK

.36

...

k•• ••,.',,o, .,.• ; L • • • • ` • ` :`• •``` . ` `` ., • • • . x .. : • , •
APPENDIX C

NEEDS ASSESSMENT INVENTORY FOR ISD PHASES IV AND V

JOB AID DEVELOPMENT

IV1 IMPLEMENT INSTRUCTIONAL MANAGEMENT PLAN

Sample: Chief (or his Representative) of Each Academic Department

QUESTIONS

1. Name

2. Academic department

3. Courses taught in department-student load and frequency of classes

4. For which courses did DTD provide:

a. front-end analysis (Phase I)

b. design products (Phase I1)

c. course development products (Phase III)

d. instructional management plan

e. instructor's manual

f. student's manual

g. POI

h. instructional content

5. Were modifications needed in any of the products in 4 above? If yes, who made

these changes? (That is, did you send them back, or do it yourself?)

6. If not provided by DTD. are the products in 4 above available? Who prepared

then)?

"7. If DTD did not provide any of the products listed in 4 above:

Why didn't they? Organizational problems? Time, etc.?

8. What job aids are available for the implementation of the instructional management

plan?

9. What additional job aids would be useful?

3S
IV.2 CONDUCT INSTRUCTION

Sample: From each academic department, as appropriaik, one or two instructors

selected on the basis of:

1. Type of instruction presented, e.g., self-pi.tked, CBI, conventional, etc.

2. Instructional experience

3. Position within instructional team

4. Student load

QUESTIONS

1. Name

2. Course taught

3. Instructional responsibility

4. Student load

a. per class

b. per year

c. How much time is there between classes for lesson changes?

5. Instructiona experience-type, length of time, etc.

6. Subject-matter experien e

7. Who developed your iistrction?

8. How frequently are changes made in:

a. PO0

b. Instructor's Mnuai

c. Student's M'nual

d. instruction, cocntent

e, rvcords maintair.ed

f. Atudent load

g. lesson plans

9. Who makes the changes in the materials above?

V'. Are changes fed back to DTD?

10. How frequontly does management plan change due to:

a. unexie-ted increase/decrease in student loan

b. equipment shortages

c. envirenmint

d. supples and materials

e. availability of instructors

37

.L W I 1.M

;.':',•'" "L :•','•:-..-. a - . ,; '9, .,4'".. .. . ---•"•' •'"•°'; •'• • • ' -• '4'' " • *• _" • ..•• <*;'•.''• . .' '
11. What forms are you required to submit--to whom?-COPY available.

12. What job aids are available to you? (FM's)

13. What aids would be useful to you in presenting instruction?

I

iA

3 8.

l ?Q
V.1 CONDUCT INTERNAL EVALUATION

Sample: Representative From Internal Evaluation Group.

QUESTIONS

1. Name

2. Department

3. Is there an internrl evaluation conducted for each course for which the signal school

has propanency? If so:

a. Who conducts it?

b. When is it conducted-how frequently?

c. What are the main steps of the evaluation?

d. What guidance is available (TRADOC, local SOP, other (get copies).

e. What standard forms are used? (Get copies)

f. What other data is collected?

g. How is the data summarized?

h. How are the results used? Who are they sent to?

i. How do the results feed into other ISD Phases?

(1) If not used what are the reasons?

(a) timeliness

(b) they have their own source

(c) format

(d) other

4. What problems exist, especially between divisions?

5. How does ISD V.1 differ from 111.5?

0. What aids would be useful?

39
V.2 CONDUCT EXTERNAL EVALUATION

V.3 REVISE SYSTEM

Sample: One or more individuals from Directorate of Evaluation

QUESTIONS

1. Name

2. Who determines the evaluation procedures used? TRADOC, local SOP, other.

3. If mailed questionnaires are used in the evaluation:

a. What is the sample size?

b, Who is sampled? e.g., graduate, supervisor, etc.

c. How soon after graduation is individual surveyed?

d. Are questionnaires MOS specific? If yes;

(1) Are copies available?

(2) What is the level of task description contained on questionnaire?

If no, obtain copy of general questionnaire.

e. How is the questionnaire administered (e.g., mailed to individual, included

in 201 file, mailed to unit. etc.)?

f. What control procedures are used to insure questionnaire return?

g. What is the rate of return?

h. Are there any follow-up procedures?

i Prior to graduation, what information is provided student concerning field

feedback (i.e., answering questionnaire)?

4. For on-site evaluation vLaita. What MOS's sampled

b. Frequency of evaluation

c. Sampling procedure and size

d. NMake-up of evaluation team

e. What data is cotlected

5. How is evaluation data sumniarized? (miad, on-site)

6. flow useful 1s the data from the mailed questionnaires?

7. Who are the results sent to? (mail and on-site)

8. How do the results feed into other ISD Phases (i.e., do they use it)?

If not used, what reasons are given:

a. Timeliness

b. Data available from other sources

c. Format

d. Other

40
9. What role, if iny, do SQTs play in the external evaluation? How are results used?

10. Of the guidance, procedures, and forms, (get copies) used by your people, which would be especially useful to others? What problems that you had in past were

solved?

11. What additional guidance, or job aids, would you like to have for new people

entering your directorate? Why? What problems would they solve?

41
APPENDIX D

:: zBACKGROUND IN FORMATION

ISD ACTIVITY CHECKLIST

Please answer each question as it pertains to your current work and/or previous experience. All responses will

be CONFIDENTIAL. This information will not be reported on an individual basis to assure the privacy and

anonyrniiy of your answers.

1. Name __________ ____________________ 2. Gradr, _________

3. Date _______ _ _ _ _ ~ -_________

4. Division ______________________5. Branch ___ ______

6. Bldg. # _____ ___ ______ 7. Telephone#~______ ___

8. Immediate supervisor _ _ ~ _______________

9. Current job _______-..-_____________

10. How long have you served in this job? _____________________________

t ~11. Please check (V) in the chart below all the ISO activities in which you ha~ve had some experience.

O Preparing critical task listing11 Makiny miedia methods selections o1 Conducting field valida Ition of ctittcil task list CiLocating selectrg. and reviewing ex~sing mautei alt o Preparing job task perfoimance -nea-,ire .1 evelopiiy correspondence suLUourse o Preparing task summary sheeti/iob task ilita caids. ltc DI evelopinig TEC matcr~als o Preparing list of reference ioatei a's to, CM-SN4 O!pivtlop-ng fesdent leston ;)!.in

0l Selecting instructional setting lewhter Iti-nirg is ito be conducted) l . Deveiop~n. P F sheets (k-,koikojlssl oWriting training objectives iTL.0s, L.0%. CL-0s. and LSOI Developing sell-Pared perftormanc' fle-ssonl guide ElPreparing test administration "lnuaril ITAMI [J Developing Aý, script%. T V scripts. etc

ElPreparing SOT imterials LI eve'loping ptoqtanmmed text, CMi. etc ElPreparing other tests (criterion, etc.) Oevi tCv piiig other instruciloi (trelidentifon-iesidetl) oConduf:ýing validation of tkests by subleci mratter exileits I]Coriductini (JoeviopmnintoI !esting khne-on-one)

fJConducting field volidatio:' of tiests Conductuiii ninsll group validation trials o Making revision following validation testss KiMking reyvs~ois flcslowing simi11 glouv va~ledotkon triali

El Examining student entrv behav~ini V lntitctng 'at9i' group) val~ilition i ttoiCSeque.acing of instructional rnltet.,ýl Icoitrse nwtpl 11 ther i~cdoI _______ ______ oclasifyving learning objectives lenwlomesti by typs's of ite"lnrig

12. Please check (~)in the chart below, all the ISO) activities that you have per formied o.. will petforn- an a

singe MOS.

oPlr~patirig ciltical task listing I-k~l Niakng rn~ ilethocls telecti,.

0l Conducting field validatioin of crtilcal wika list . Locatring serect-Si. ritd rcvie-wtr.,. linatet-al-s r.s o3 Pre1PAring fob task peifornanci, mehasures IC)vilirycorreospondencv iu

0 Preparting talk llummrylt sheets/lob task daita cardji. oft: LJ Developing TEC nsate-rials o Preparing list of refetrence trjtetials lfor Chl'sl I. 1Detrilop-nji twddrnt lesvon plan o Selecting insituctional setting (whlee Itrali-n ii. to be i.Oi'ducttl~l Develo~tniq P E %beets lwotkbooksi o Writing training objectives (TLOs. 1.0%. CLO%. arnd LSO ~ 171 Deloping sell paced &Vttorman't Ilirstissn guide OPfteParirg test administration mranual (TANA) IiDevelop-ing AV sctints. T V tcripts. etc o Preparing SOT mnalteials 11Developing prganh eikr. CAl. rtc

0l Preparing ot0e tWitl Icriterion, ect I (.1 Oevel tiqovss ther intrtucition rsCtn!Oncint o Caflducttng validation of testts by subject tvattet eflrts I I. Conducting cleveloiprsewit testing lone-onsorsel o Conducting field validation of tests11 ~dutn stall gioup val~da~ion tlioal o Making revrzmon following validation tests ( Making levisiolns folloliing inisatl gti3up validation trials

) 0 Examining stuident entry behavior 1Conducting isi" gisup vallidaotion trials

0Socquencing of instructional material tcrsutre map) IOither (U"10~~y ______ o Clauwfvrng teanrnig objectives fceitmentil by tlrpes of It-atrnii

42

-4% &i
ISD ACTIVITY CHECKLIST

Please answer each question as it pertains to the Instructionatl Systems Development (lSD activity you have worked

on during the past two weeks. rhis information will not be reported on an individual basis to assure th,ý privacy

and anonymity of your answers. If you have any additional comments, please enter them at the bottom of the form.

1. Name _________________________________ 2. Date_________

3. Please check (J) in the chart below the ISO activity that yvou have worked on during the past jweeks.

Ol Preparing critical task listing El Making media methods selections

C1 Conducting field v~ilidation of critica-l task list El Locating selecting, and reviewing existing materials

0 Preparing job task perf ormance measures El Developing correspondence subcourse

O Preparing task summary sheets/job task data cards, etc. El Developing TEC mnaterials

El Preparing list of reference materials for CM/SM El Developing resident lesson plan

El Selecteng instructional setting (whsere training is to be c'inductedl El Developing P.E. Sheets (workbooks)

0l Writing trilining objectives ITLOs. LOs, CLOs, and LSs'i 0 Developing self-paced performavnce (lesson) guide

Cl Preparing test administration mansial (TAM) 0] Developing AV scripts, T.V. scripts, etc.

El Preparing SOT materials El Developing programmed. text, CAl, etc.

El Preparing other tests (criterion. etc.) El Developing other instruction (resident tnon-resident)

0l Conducting validation of tests by subject matte- ,xperts El Conductinq developmental testing (one-on-one)

Ul Conducting fieldi validation of tests El Conducting small group validdtion trials

El Mrking revision following validation tests El Making revisions following smell gro;op validation trials

El0 Examining student entry behavior El Conducting large group validation trials

El Sequencing of instructional material (course map) El Other (specify)

El Classifying learning objectives (elemeents) by types of learning

4. Did you complete (at least 'n draft form), t"~ activity checked above? Ye~s._____ No.

5. Estimate the total number of hours that you spent warkirso n'n the ISO activity checked off above. Hours

6. What percentaW of the time that you worked on ýzis ISO activity was spent on:

(a) Initial Preparation%

(b) AeiwiovvIevision %____

Wc Other (expiain)%

7. What problems (it any) did yi.u experience in perfewnrminy your vvork i.-s khas activity? (Pteau, check all that ame

aplvicable.)

(,I) Guidane was not available (h) Input to mny ISO work was unsatisfactory -

(b) Guidance wim Inaccuirate ti) Input to my ISD work was late

Ic) Could :ýot get needed Informatin - (1) Not enough tiime to gaiher ilnfnsmation~__

(d) Informration was Inaiccurate - (k) Not enough time to develop material

(e) References; were ;-ta of d"l, 01) Not enough tirne to review/re.-.ise material

if) DA n~sov rmtceia tnough su,ýrvisloin (mi) I d,'d not receive enough trairc..otJ to do thlis

(g) Not enouqoh help ffoni other per-rinnel -ork satisfactorily

in) Had no problems

Other (pleass describe)

43

I __________________
8. If the input to your ISO work was not satisfactory, what did you do?

(a) Reported it to my supervisor

(b) Returned it for revision

(c) Did it over myself

(d) Not applicable

(e) Other (please describe)

9. Was the output (product) from your work returned to yuu for revision? Yes. No.

If "Yes," what had to be changed?

(a) Errors had to be corrected

(b) Format was wrona

(c) References were wro-ig

(d) Incorrect wording

(e) Documentation was inadequate

(f) Other (please describe)

COMMENTS:

PIea pOace this form in the sttached enyAwope wWd Oimme tt in the HutnKRO contaiher Iowted in

W. JohM 0. n0,movlch's office (WB14 19704) or in Mr. John A. Myers' offbio (BO. 25701).

Thank you for your coolpration.

F4

..............-....-. :"•., ,... ' -. -, ... ,",.'•. ,'•:,•'}'•••:"'-.~",:-,. .,.:...- ...;;. •.-
APPENDIX E

JOB AID FOLLOW-UP INTERVIEW FORM

inte rviewer

Date

BACKGROUND INFORMATION

Name Grade

Division/CMF

MOS (working on)

Current Job _ How long in job?

How long in ISD block for which Job Aid designed?

Any formal training in this ISD block? If yes, what and how much?

Did you complete a Background Information Sheet a few months ago? If no, have

them complete a Basd-line Data Background Information Sheet.

Who is your supervisor?

FLOWCHART

1. To dhkt extent did you actually use the Flowcharts?

2. Were the symbols used understandable?

3. Was written material on side understandable and useful?

4 45
FLOWCHART (Continued)

4. Were there times when you were referred to the manual unnecessarily? Times when you

should have been referred and were not? Which blocks?

5. Did you ever go to the Manual to see an example when not required to do so? If yes,

how frequently?

6. Was there a need for expansion/collapse of some flowchart blocks? If yes, where?

7. Were there too few/too many supervisor checks of your work called for in the aid?

Where should they have been added or deleted?

8. What is your overall evaluation of the flowcharts? How csn they be improved?

,NR.CTTON TG JOB AIDS

9. How goo4 a job did it do in eolaeining what the Job Aids were? How can it be

ILmproved?

N46

U 9 .... -``•`*.`.`•••`••` •.••• • `••••• • • • • .',••4-' i-•: ' <%-..,,: ... •,.,•*" .
RESOURCE MANUAL

10, Did you use the Resource Manual? How well did it cover: CODAP, School Conducted

Field Surveys, and School convened panels?

MANUAL

11. Were the partial flowcharts at the top of the page useful? To what extent did you

use them?

12. Was the overview to the job aid useful to yoQ? How can it be improved?

13. Were the questions used to introduce each area useful?

14. Is the text in the mmiual too difficult to read? Explain.

15. Were ther inuaccuracies io t"e i-fo~mation presented? Explain.

'is 4g
MANUAL (Continued)

16. Were important sources of information omitted? If yes, what are these sources?

17. Was inappropriate or obsolete information presented? Explain.

18. How useful were the references provided? Were they correct? What additional

references or sources of information do you usu.lly use?

t

19. How useful are the examples provided?

20. What is your overall evaluation of the Manual? How can it be improved?

21. 11cw uveful were the Wocksheets provided? Ho" c~an they be improveO?

-"48

I., A-1
WORKSHEETS (Continued)

22. Did the Worksheets require too little/too much documentation?

GENERAL

23. Was the reading level of the Job Aid (flowcharts, introduction, etc.) too high/

too low?

24. What did you do before you had this Job Aid that was better or more efficient

for performing the work in this area?

Z5. Do you think that by using this Job Aid you have turned out a better quality

product than without the aid?

26. In the long ran, do you think that use of the Job Aid will turn out products

faster or slower than without the aid?

27. Do you think the Job Aid Is too long/too short or about right in length?

49
GENERAL (Continued)

28. Will you continue to use this aid in the future? If no, find out why.

29. What type of person do you think would find this aid useful? (i.e., inexperienced/

experienced)

30. In order to use the Job Aid you needed materials from individuals who had

performed ISD activities preceding yours. Did the materials used hinder

or help you in your work?

1 60

II

'I.
DISTRIBUTION

I WS Ar(MY WESTERN CO-lNANU ATTN: AtPL

I I)EiAHTMEN1 OF T'-j NAVY 1RAININO ANALYbIS AND EVALUATION UPOUP

I HUUA ATTN: L'AA,-Ei-:

* I US PACIFIC FLELF HUMAN HLbOUkCt. mANA~tMFNT DETACHML.NT

I HOq ILAIA ATTIN: ATCAT-UP-W

?~ HU0A RESEARCH AN.) STUULýk. OF(,

I MILliARY UCCUPATI)NAL UEVtLQOPMEN1 DIV UAVC-MbIW-U* RN 952C HOFFMAN 8LU6 I

4 UASD (NRA AND L)

I Ho TLATA TECHNICAL LIBRARY

I HQ0A OUCSPkH

I HUoA ATTN: DAM[-ISI

1 USA AVIArION SYS1wNS CUML) ATTN: URSAV-1I(i.H

I USA ~uORADCOM ATTN1: AMSLL-PA-I*4

I HEA~wUARTLRS US MARINt COMPS ATTN: CJUE. TMTT

I HEADwUARTERS9 UiS 4ARINE LUHPS ATTN: C'U~t MP1-20

? US ApMY EUROPE AN;) SLVLNTH AHMY

I IST iNFANTPY DIVISION ANt) PT. RILEY AITTitI AfZN-UPT-T

I CHIErs ATTITUUL + OPINION SURVEY DIVISION ATTN: ATZI-NCH-NA# HOFFMAN ULUG 11

I USa iNTELLIGENCE AND SECUNITY COMMAND ATTN: IAOPS-TNG-T

P HO TsiADOO TECHNICAL LIHHAkY

I NAVAL TRAININ'i IUJIPMENT LLN ATTN; TECHNICAL LIdPARY

I MILliARY (JCCUPAIIDNAL UL)VLLUPmi.INI U1NELIORATL AITNI ATZI-NCR-MS-M, WA 3N33 HUFFMAN BLUG II I DATA ANALYSIS UIVISIUN AIIN: Alil-NC'4-4U, HUFFMAN ULOG 11

I USA ,mILPLHCEN ATTN: DAPLCPOU-T

I RT.1 iNFANTRY DIVISION

I HUUA ARMY FORCE 'IDDtRNIZIAIUN CUUNDI'JATION OFFICE

I HW9A ATTN: DASv-PTIK

I 123U USARCOM HLSEkVL LENItH

I US AWNY SULDIEN SJiPPOHI CtNTLk

I USA i-ORCLS COM14AND AFL6. UtlPuFY CHIEP oP STAFF FOR LOGIhTICS

I (IS AWMY AIR UE0`ýNSL SCHODW. AlTNI ATSA-UTU

I OlREI.TUHAIE OF INAININ~i AITN. AlZ(J-1

I OI1tELTORAIF LIF CU*4NAI ULVLLOPMk.NrS AT INt AT~ka-O

1HQ0AHCDM mARINI. CORPýS LIAISON VfC

I IIEAwTMENT OF IH4L AL114Y US ARmy IN1tLLI(jFNCL SLCURITY COmMANU

I USA mI1SSILE CUMiA'4b ATTIN; Ukhml-NIN

I UJS A.mNY CECO14 ATTN: UNSU..-ATUP

I U'SA tUI4CLS CUM1$.04)

I PM TmAUE/ I US o4iLITAWY OI!,)NIC(T Of~ WJSHIN6;1UN UFL; (I EVUAL OPPORJTUNITY

I NAVAL CIVIL.IAN Pt.-SONNLL LUMO ',WUltiEN FLO QIV

?#? ARI kIAISON OFIýICE

I ?Tit ikRMY 1PAIN1NtU LOMMANU

I A4.4Y TRAINING~ SiPDURT ICANILH INUIVIUJAL TRAININL4 EVA(IIATION

I '400A. OCSUPS ['JUIVIUUAL ImAININ6

I NtIIJA* UCAUPS TRAINING tOIHLCh)WAIL

I HUUA. OCSLD~i MA1~4TENANC1. mANAt.LlP4ENT

I HUi)A, OILS STUViY 3)FfiCt

I Us.i. NAVY Tk~NAIN ANALYbIS tvALUATIJN 6kiOljP

IUSACuEC ATTN: atE- -LX- tUMAN PACAOkS

*I AttIN SM-ALC/OVHC

I %,SA~ui&OS/TAC 51u1014 ARMY AUVISIJM

I INrEm-UNIV SEMI'dA4 ON AWMLLP 0Ul.L5 . bUC

* I OAbA (RUA) liEPtily fUk Nt.,LNCt A14U TECHNnLDtY

I OFC uF NAVAL 14 A.4Ctt

I AFHA~i/LHI

I AFHkL/LJ4LG

I AIR4 tORCE HUM4A~N WEOUtC0~ LAU ATTN! AfHR~L/TSW

I NAVY PERSONNELt W ANt) U CLrdItI4 UiktCTJM (I PNUGIRAmS

I NAVY PERSONNtL 0 AND) 0 CtNIN ti4

51
2 OFC UF NAVAL RLSLEARCH PHbNONNEL AND TRAININ6 RESEARCH PROGRAMS

1 NAVAL PERSONNEL R + 0 CENTER/

I OFC uF NAVAL RLSLARCtI PHUJECI UHýICIA, f-4V1RONMLNTAL PHYSIOLOiY

1 MCFAi4N-GRAY + ASbDCIATLS, INC*

1 NAVAL AEROSPACE MEDICAL R.'LH LAb A1RiDRtNý RANOER RESFARCH

I DEPTI. OF NATIONAL UEýLNCE ULýFrJC(L ANJ CIVIL INS11TUrF. OF ENVIR ME,j

I NAVAL AEROSPACt- MEDICAL RS(l-i LAO AERDSPACE PSYCHOLObY DEPARTMENqT

1 USA IRAL)OC SYSIF.MS ANAL'YSiý> A(..IIVIIY ATTN: ATAA-TCA

I HEAO~wUARTERSt COAST GjUAR0 CHIti-9 PSYCHULOGICAL RSCti RP

I JSiA WESEARCH AN,) TECHNULU O.Y L~t / -Q ( C O L 1 USA MUHILITY EUi'1PMENT R AND U LOMU AITN: UNUMETO(COL

I NIGHI VISION LAtI ATTN: UkNSEL-0v-SDU

I LiSA IRAINING F3UAHD ATTN: ATB.,-IAl-TA

1 USA mATE~lEL SYSTEMS ANALYS:IS ACTIVITY ATTN: URASY-C

I NAFEL HUMAN ENI\&EENINl., bkAN(ýri

I HAlTt~LLE-CoLUMbIIS LABHOATUkILS TACTICAL TECHNICAL OFc

I USA mRCTIC TEST Cý-N ATTN: AMSTL-PL-TS

IUSA k.OLD REGIONS TEST CLN1 ATTN: STECR-OP

I D-FEiqSE LANGUAGE- INSTIIUIL FONLIGN LANGUAGE CEN

I HO WNAIR DIV UF 'JEUROPSYLHIATRY

I US'A LLECTHQNIC tiARFARE LAb Ctilt', INTLLLIGE11CE HATER CDEVEL SUPP O~F

I USA HSCH UEVEL + STANUARUiLA ku~t UoK.

I USiA ,.&SEARCH AND )LVELUPMLENT LAbS CHILF, BEHAV SCIENCES DIV* FOOD SCI LAtI

1 NAVAL AIR SYSTLAS COMMAND ATIN: AJR-ZJ13

1 IISACuEC TEC~HNICAL INFORmAl ION CLNlER

I USkAKL LIHRARY

I HU"Tiý RESOURCEb KSCH O~K6 (riUMWRU)/

I SEVILLE RESEARCHI CORPORATION

I USA iRAJOC SYSILIAS ANALYSIS> ACTIVITY ATTN: ATAA-SL (TFCH LIbRARY)

I UNIFuRMED SERVICLS UNIT Ut THE 1t.ALTH bcl OLPARIMENT OF PSYCHIATRY

I HUMAii RESOURCES RSCH ORGý (HUMkRU)i

I HUMRmO L1RRARY

I HATTtLLE REPORTS Llt8RANV

I RANU CORPORATlON ATTN: LItIRARY U

iGRON)NkER LIHRRAY ATTN: ATLF-W,'-L t8LOUI 1314

I CENTtH FOR NAVAL ANALYSIS.

I NAVAL HEALTH R$CH CEN LIURARY

I NAVAI, PERSONNEL k ANU U Lt-N LIMNAWY Alt N: CUDE P1 06

I AIR FORCE HUMAN krESOURLES LAb ATTN$. AFMPL/OTS

I HQ , i-To MUACHuCek ATTN: ILt.H W0 UIV

I USA uCAI)EMY OF HtALTH- S)CILNCLS iJIMSON LIMNART (W)CUMFNrS)

I SCrIOuL OF SYSTtmb AND LObIbTICS/

I ERIC PROCE:SSINU A~4U REFERNlCE. FAL AC.JUISITIONS LIO4NARIAN

I OEPAN1MENT Of 1*i NAVY l"AININGU ANALYSIS ANU EVALOATION GiP

I NATIuNAL CENTER 1KW HE.ALT?1 STAIIýOICSI

I U514A DEPT OF bitHAVI0RAL SLA AWi LLAUERSHIP

I fi) ijUMINION UNIVERSITY tjtRýDNMANCL AbSESSMLNT LA,4OkATORY

I IISA LOMMANT) AND U;ENERAL STAFF CULLWLt ATTN: LIBRARY

IUSA IRANSPORTATIUN SCHOOL USA INANSP TECH INFO AND) RSCH CEN

INMRt)L PROGRAM MANIAGLR tJri HUMAN I'LRJt4,4ANCL

1 USA allMINCEN ILL.*JICAL RLSEAHCrI UkANCH L113HARV

i USA s-IELU ARTY 14U )

I NAr L.LEAH1NGt1OUSt FOR ME.NIAL rtt#1lti PIk) PAHKLAWN OLOG

I U OF TEXAS CEN 1Jk) COMMUNICATION kSCiI

I INSTATUTE FOR ukIENSL At'4ALVSESo

I USA iRAINING SDP$DNT CENTER ATTIN: ATIC-'isT-PA

I AFiikt. TECI4NOLOuY J1f C (H)

I PRUMvE. UNIV DoE..' OF PbYClIULOGICAL SCIENCES

I USA I4OUILITY EUUI'lME.NT R AMU U LO.MMANJ, ATTN: DUHU&4-IG

I (IA Uz, ARMY RETHA4IN~iu. uL, Rk.SfMNCK t VALUAlION UIVISION I ~ ES'.CPOAI4HAZEi.TINL ~~I CORPtJRATION 5

45

A-*
I USA ALROML)IACAL HNESEARCMH LAji SCIE.NTIFIC INFON?4ATION CENkLR

I RAY~T-sON SERVILI. CUMPANY SDPLNCLK LAtiJHATQNY

IHUAA4 RESOURC'. PIAA\J(iMEN1 LEN, SiAN U1~ti

I LJS1VA DEPT OF -4- SCI + LL-AULNtibIV

I UIS MiIiTARY ACALut'iY OEI*hH OF- dlitfURY9 HLOG bOl

I USA~ iNfELL16EN Ct.14 AND )(.H ATIN: SCHOIOL L163RAkY

1 USA~ iNTELLTGLNCF-- CLN ANt) 'LH A ITIN: A T!:I -LP

I MAi I iE. CORPS I NS I I I Uf

I NAVAL SAFETY CLTR/ A N -UAINLSC FI

I US CuASr blUARD ING3 CLN AI:VUAINLSC FIL

I GALLAUOET CULLL"LE LtAHNqi'JL, CtiNlt

I kl~iSAvNC AN~i VT. HJCr\LR ATTN: AIL(U-LS

1 US A1mMY AVN TNU. LibRARY ATTN: LHIt.F ~.ItAtARIAN

I USA AIR UtFENSL. SCHOOL AIIN: ATSA-UT

I kJSAAvNC ATTN: AIw-L)

I US MiLITARY ACAUtM~Y DIRLCIOR OV INSTITUTIONAL RS)CH

I IJSAAuS-Lh3RARY-IIUCUMENIS

I USA zjERGLANTS rIAAJ)H ACAULMY AT IN: LEAHNIN6 kLSOURCLS CENTER

I USA INFANTRY HUAN.) AT1Nq A~lti-it-AE

I USI INTELLIGENCE -,LN ANJ) SCH A11N: ATý:I-UTStFL

I UISA uHONANCE CLiq AND SCH ATIN:- ATS>L-TL)-IAC

I LIS-A ARMOR SCHOOL ATTN: AILrx-1 1

I USht ARMOR CENTELi DIRELTuKATE %.)i CUMfIAT DtEVELLPMLNTS

I NAVAL POST;IRALUAIE 'SCH AliN: iuUULLY K(NOX LIbRARY (CUIDE 1424)

I USA ikANSPoRIAIION SCHUUL L)Lti,0Y ASST. CU)MMANUANT EDUCA. ILCHNULUGY

i USA .aIGNAL SCH-UJL AND HT. 60HUQN ATTN: ATLH-LT

I1I ISi , iUARTLPMASkHR SCM ArIN: AlbN-lN6i-lm-V.T

I WSLA %tlLlTARY PULiCt. SChOOL ATTN: L16RAHY

I US.% ARHOR CENTLW + FT, KNUX 0týiCL OF AwmON FORCE MGT + STANL)ARUIZAIION

I CHIf,ý OF NAVAL ýIJC(ATIUN ANt) 10t'

ItiStA 31iNAL S~ojWot * VT. jUNUHUN LUUCATiONAL TL.CHNOLOGY DIVISION

1 HO AIC/)kP1U li'4ININU SYSILMS oLVtL0P4LNT

U tSi INSIITUTtK FiH MILIlARV ASS1S1lANCE ATTN: ATSU-TU-TA

I US AX16Y ARmOR CLt'1tR AT~iq: ATIY'TU-P4U

I USA AIN IJEFFNhL t.ýNTtks H.* bl-jSS ATI4; MTLC-0Im

1 UiSA wUAHTLWMAS~rW SCIOVL uINL(:lfJHAIl UIý IWAININU ULVELOPMENTb

I US CuAST titAkU ACAL)t.4Y

I l!SA &fIANSPORA1AIU' SCIIUOL Ujwt.LIORAlt . IW AINING * OOCikNf.I

I S4 INFANTRY Si-HO3L LIbRANY/

I USA INFANTRY S-t4%Q)L ATTN* AIS.4-I-V

IiiS A,.jNY INFANTky _%LMU(.L u1!N: A1SH-Cj,

1I US A INFANIRY SCWJL AiTNt AfrSýi-LUO-L-tu

IUSA 4 NFAN1RY SCLIV)L ATIN:* AThti-t.V

IUS& 4 P * CHEMtM S3ti,HThit, Lt.N * 1,. mCLLL.LLAN AI:ATZN-PTS

I ITSt e'P * LHEI. !SM/INUj CtN + FT. NCLLLt.LAN Ulk. COMSAT OLVELOPMENT

I USua AP * LHýi bSrl/1Nu Cl-N * IT, M(.LLL,.LAN UIR. TkAINING Ut.VtLUPMENT

I IJSm m.P * CHEM b:ti/TI4%j Ct.N * 1-1, N4.CLLt.LAW ATTNI ATIN-MP-AC&.

I W)A INSTIlti'1 Uý ALInINISIHATIU14 AITNI W**SIULNr TRAINING; AANA(3tMkNl

I uSA tIELU AHTILLL-ly S)CHOOL MuriNIS Sw~.IT LIIIRA44Y

I USA olbI'ITLTE Ui, AorAINISIHATIOtv AL.AOUt4IC LItHA1RY

I tI]Sc *AR~ CO~LLijt ATTN: LlowAH'v

I U5 tNbINtEk SCLAuX! LIlIjAAY AtA) LEAW4N(N RLSU'URCES LE~NTEN

I USA mkM~Ol- SCMOUL tUSARmtO Al I t4: L~dWNY

I 11S CuAST tIUANO 1,C40LMY LLIOAMY

I USA ikANSPORTAT1ON SCHUUL TWAt.'ONIUHArIUN SCHOOL LIIINAPY

I UN(;A~vILA1IONAL kýLCTIVtNLt.S CIN * SCti aTTN: L18RNAIIAN

I US AmMY INI'LhL~ot4Lt ctNTti$ * sCtHUOL ATTN4: ATSI-TO

1 (IS ANMY INTELLIvLhL'CI. Cb-41tk * SLHOOL ATTN: ArSI-RM-M

I tIS AmMY LNTELL1IvitNILL CLNILH 0 Si..tIOOL Af7T4- ArIfj-E

I U;S MmkttNL C014Pb EWICAILUN CLNIt'k

I is.' rILLU A1,11ILLE4Y SCHO~t. UlwLLIUR41t iJF CUO*4S5 LJEv * rRAININ(i

i iLp~Aq4mLNI Of TI-L AIN hjt AIN UtNdVLNS1TY L16RAWY (ATC)

53
I USA LHAPLAIN Ct.NT~k SCriQUL ATiN: ATSC-ID-OU

I USA tHAPLAIN Ct'jT;EH + ý)(HUUL A f IN: ATSC~I-TD-)

1USA LHAPLAIN CLIjlN SCiv AnN ATSC.-lDI LISA i-HAPLAIN C;L,\IEH + bCHOOL ATIN:- A I C-uoSj-LLC

I HO TKADOC THAl,\ii'J, UEVELUPMtNT INSTITUrf.

2? RRITiSH tkM4ASSY 4NITISti ULFLNLL bTAIF

? CAAAuIAN JOINT IAIFF

I CULS (w) LWNRAHY

I FRNLNH ARMY ATlAC-iL

- I AU'bTg-AIV EMB3ASSY UFFLN5Lw MIL11ARY AWU AIR ATTACHE

3 CA4qAuIAN DFFENCr. L.IALSUIN )IAF!- ATTN: QOUr'jSELLON, ULFFNcE R ANU D)

I ROY~AL NETHERLAN\JS EWtSASSY MILIIAkY AIIACHE

I CANAUjIAN FORCFý bASE CURNWALLIS ATTN: PFNSUNNLL SeLECTION

P CANALJIAN FORCES) PEkSUNNEL APPL RSCh uNliI

I A~iAY PERSUNNtL IiLSt.AtICHh tSlAhLlhtlMtNT

6 L-IHRm.RY Ul- CONUINL$S EXCHJANGE. 4NL) (,FT tIV

I OEFFtqSE Tt.CtINLI.AL INIOHMAI ION C04N ATTr4: )TIC-UUA-2?

140 LDi3RARY OF CUNutFLSS UNII UOLUmjLN1S EX.PEIoTIN6~ PROJE.CT

IUS (3uVERNMENT llkilI'TING OHk. LlhHARY, PUBsLIC OUCUMFENTS DEPAIRTMLNT

I LIUS G3uVLNNMENT PiiI\J1ING Oh.. LIt3HAHY ANUL SIATUTORY, LIH. 01Y (SLL)

I TH?. ARMY L16RAKY ATTN; ARMY !SrUUILS ý)C

N~lIMMEk kF ADUkESSWSC 20Ub

TW)AL NuMdffN Of* CUPIES ,ib

54

- - ~ 6