Techulcal Report 426

AN ANNOTATED BIBLIOGRAPHY FOR

INSTRUCTIONAL SYSTEMS DEVELOPMENT

Melissa Berkowitz and Harold F. O'Neil, Jr.]

EDUCATIONAL TECHNOLOGY AND SIMULATION TECHNICAL AREA

C-,

U. S. Army

Research Institute for the Behavioral and Social Sciences

August 1979

Approved for public release; distribution unlimited.

00.
_.wood

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WILLIAM L. HAUSER

JOSEPH ZEIDNER Colonel, U S Army Technical Director Commander

I NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Please address correspondence concerning distribution of reports to: U. S. Army Research Institute for the Behavioral and Social Sciences,

ATTN PERI-P, 5001 Eisenhower Avenue, Alexandria. Virginia 22333.

FINAL DISPOSITION ':This report may be destroyed when it is no longetr needed. Please do not return it t0

the U. S. Army Research Institute f or the Behavioral and Social Sciences.

NOTEi The findings in this report ore not to be construed as an official Department of the Army position. unless so designated by other authorized documents.

I--qq
SECURITY CLASSIFICATION OF THIS PAGE (Men Date Entered)

REPORT DOCUMENTATION PAGE READ ISTRUCTONS R SyBEFORE COMPLETING FORM

|. REPORT NUMBER 2. GOVT ACCESSION NO, 3. RECIPIENT'S CATALOG NUMBER

Technical Report 426 /

4. TITLE (and Subhllte - 5. TYPE OF REPORT & PERIOD COVERED

SYS'EMS DEVEILPMENT. - Final

7. AUTHOR(a) S. CONTRACT OR GRANT NUMBER(q)

S MelissaBerkowitz e--Harold F.jO'Neil, JrJ

a. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASK AREA A WORK UNIT NUMBERS

U.S. Army Research Institute for the Behavioral 673 -'_- and Social Sciences PERI-0K . i /. ,2.....22A764)

5001 Eisenhower Avenue, Alexandria, VA 22334 ". ..

It. CONTROLLING OFFICE NAME AND ADDRESS ,

Office of the Deputy Chief of Staff for / Augt--r979

Personnel -U BER OF PAGES

Washington, D.C. 20301 52 74. MONITORING AGENCY NAME & ADDRESS(If dflferen t.rm Cont*oln. Office) IS. SECURITY CLASS. (of this report)

--.-.. . ... .. unclassified

LL4 "- L/ , I Sa. DECL ASSIFICATION/DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abstract entered In Block 20. it dilfferent from Report)

1S. SUPPLEMENTARY NOTES

19. KEY WORDS (Cotinue on reverse ide Ifnec..ry and identify by block number)

Instructional Systems Development Model

ISD

Authoring aids

" TRACT Miartbas avre atta Nu neamrsW said Idemilty by block numnber)

The purpose of the study was to determine the availability of how to do

manuals for instructional systems development. The Interservice Procedures

for instructional Systems Development Model (ISD, TRADOC Pamphlet 350-30) indicates the steps to be taken for the development and conduct of training.

Authoring aids, procedures, and techniques were identified as annotated

references for each block of the ISD model. Directions for a future research

plan were identified based on the lack of authoring aids available.

D 13 rwDiON INov s IS owsoLETE Unclassified

j SECURITY CLASSIFICATION OF THIS PAGE flWlm De E,,eeed)

L

iI. --. ,!
Technical Report 426

AN ANNOTATED BIBLIOGRAPHY FOR

INSTRUCTIONAL SYSTEMS DEVELOPMENT

Melissa Berkowitz and Harold F. O'Neil, Jr.

Submitted by:

James D Baker, Chief

EDUCATIONAL TECHNOLOGY AND SIMULATION TECHNICAL AREA

Approved by:

Milton S. Katz, Acling Director

ORGANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCE'-

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

August 1979

Army Project Number Computer Based Educational Technology

20762722A764

Approved for public release; distribution unlimitel.

4, ill
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready for implementation at the time of publication are presented in the last part of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military agencies by briefing or Disposition Form.

lV
FOREWORD

The Computer-Based Educational Technology team of the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) performs

research and development in areas of educational technology with applicability to military education and training. Of interest are instructional

delivery systems with a special focus on the development of accompanying

courseware in the most efficient and cost effective manner possible.

The development and implementation of such systems address the problem

of training individuals who must produce good courseware in a reasonable

time, at an acceptable cost.

This Technical Report provides an annotated bibliography of materials available to support instructional development. It also identifies

those areas in which further research and development is required.

Appreciation is extended to Dr. Richard Braby, Department of the

Navy, Training Analysis and Evaluation Group; Dr. John Ellis, Navy Personnel Research and Development Center; and Dr. J. Dexter Fletcher, Defense Advanced Research Projects Agency; for reviewing an earlier draft

of this report.

The research effort is responsive to the requirements of RDT&E

Project 2Q762722A764, Educational Technology and Simulation Technical

Area of the FY 78 ARI Work Program.

0 H ZEIDNER

cncal Director

V
AN ANNOTATED BIBLIOGRAPHY FOR INSTRUCTIONAL SYSTEM DEVElOPMeNT

BRIEF

Requirement:

To determine the state-of-the-art in instructional systems development and to specify critical research gaps.

Procedure:

A review of the instructional development literature was conducted.

An annotated bibliography was produced which organized references according to the steps of the Interservice Procedures for Instructional Systan'

Development Model (ISD, TRADOC Pamphlet 350-30). Each reference was

classified as (1) an authoring aid, i.e., a "how-to-do-it" manual which

directly guides the author through an activity identified by the ISD

model, (2) a procedure, i.e., a listing of the activities involved in

ISD or (3) a technique, i.e., a methodology for accomplishing an activity but lacking sufficient detail to be a procedure.

Findings:

(1) In Phase I-ANALYZE, authoring aids are available for Select

Tasks/Functions; Construct Job Performance Measures, Analyze Existing

Courses; and Select Instructional Setting. Authoring aide are unavailable for Analyze Job.

(2) In Phase II-DESIGN, authoring aids are available for Develop

Objectives and Develop Tests. Authoring aids are unavailable for De- scribe Entry Behavior and Determine Sequence and Structure.

(3) In Phase III-DEVELOP, authoring aids are available for all

activities involving print as a medium. However, authoring aide are

unavailable for computer-based instructional materials.

(4) In Phase IV-IMPLEMENT, authoring aids are unavailable.

(5) In Phase V-CONTROL, authoring aids are unavailable.

vii

Il II ... , , , • , . _ . . . . . . . .. . .
Utilization of Findingst

The identification of aids, procedures, or techniques will be of

immediate use to those involved in instructional systems development.

In addition, these findings will form one basis for a future research

and development plan.

viii

I
AN ANNOTATED BIBLIOGRAPHY FOR INSTRUTIONAL SYSTEMS DEVELOPMENT

CONTENTS

Page

INTRODUCTION . ................... . . . . .. I

PURPOSE ......... . . . . ....... . ....... 3

METHOD ..................................... 3

REFERENCES ........... ............................ 7

RESULTS .............. ........................ .... 9

Block I.1 Analyze Job ..................... 9

Block 1.2 Select Tasks/Functions . . . . . .......... 13

Block 1.3 Construct Job Performance Measures ........ 15

Block 1.4 Analyze Existing Courses .... .............. 16

Block 1.5 Select Instructional Settings ... ............ . 17

Block II.1 Develop Objectives ..... ................. . 18

Block 11.2 Develop Tests ................... 20

Block 11.3 Describe Entry Behavior .... . ............. 22

Block 11.4 Determine Sequence and Structure ..... .......... 23

Block III.1 Specify Learning Events/Activities . ........ . 26

Block 111.2 Specify Instructional Management Plan

and Delivery System . . . . . ............... 27

Block 111.3 Review/Select Existing Materials ... ........ . 31

Block 111.4 Develop Instruction ...... ................. 32

Block 111.5 Validate Instruction .... ............... . 36

Block IV.1 Implement Instructional Management Plan . . . ... 38

Block IV.2 Conduct Instruction ............. . 39

Block V.1 Conduct Internal Evaluation .... ............. . 40

Block V.2 Conduct External Evaluation . . . . ......... 42

Block V.3 Revise System . . . . . . . . . . . ......... 43

CONCLUSIONS ........... ........... . .... . .. 44

Authoring Systems ............................. . 44

Summary of Availability of Authoring Materials

by ISD Phase . . . .. . . . . . . . . .......... 45

Summary and Plans for Research and Development . . . . . ... 48

DISTRIBUTION ........... .......................... 51

ix
LIST OF FIGURES

Figure 1. Interservice Procedures for Instructional

Systems Develoment Model - . .. . . . . . . . . . 2

2. Availability of authoring materials for

the ISD model. . . . . . . . 49

x
AN ANNOTATED BIBLIOGRAPHY FOR INSTRUCTIONAL

SYSTEMS DEVELOPMENT

INTRODUCTION

The need for the Instructional Systems Development (ISD) was identified in field tests of individualized instructional training systems.

It was difficult to bring individuals who used these systems to the

level of competence needed to produce good curriculum materials in a

reasonable time at an acceptable cost. Similar difficulties have been

noted with other systems employing computer-based instruction. This

problem is especially acute in the Army because of the high turnover

of personnel. Further, since future Army training and education systems will be individualized, the need for this technology is likely

to intensify.

Currently the instructional systems development process is time consuming. It is not unusual to find that it takes one hundred or more

hours to develop and evaluate one instructional hour of self-paced material. For example, Orlansky (1979) provides evidence for variability and

length of time among authors to produce instructional material. His data,

which summarize all available DOD studies in computer-based instruction,

indicates that authors vary by a factor of 10 among themselves. In addition, several sources (e.g., Avner, 1979) have indicated that the training

process takes at least six months before authors can be expected to produce useful materials and a year before full production.

The Army's first attempt at instructional systems development was

called systems engineering (United States Continental Army Command Regulation 350-100-1, February 1968). The most recent effort of the Army

to provide standardized procedures for thii development and conduct of

training is the Interservice Procedures for Instructional Systems Development Model (ISD, 1975). The model provides for the assessment of training needs; the design, development and implementation of instruction and

the assessment of instructional quality through a five phase process (see

Figure I). The first phase is ANALYZE, which provides guidance on task

analysis, the selection of tasks to be trained, the development of measures

of job performance and the selection of the appropriate environment for

training. DESIGN is the second phase which establishes the objectives,

test items, and sequence of the course. The entry behavior or skills

the trainee arrives with are also noted during this phase. The third

phase is DEVELOPMENT during which the instructional materials are created.

Existing materials are examined so that new course materials are devised

only when appropriate others do not exist. A plan specifying all activities of the learner is made as well as a plan for pilot testing the newly

e1
1.1 1.2 1.3 1.4 1.5

Analyze Select Construct Analyze Select

Job Tasks/ Job Per- Existing InstrucANALYZE Functions formance Courses • tional

ANLZ Measures Setting

Develop Develop Describe Determine

PHASE II Objectives Tests Entry Sequence &

DESIGN " Behavior 1 Structure

111.1 111.2 111.3 111.4 111.5

Specify Specify Review/ Develop Validate

PHASE III Learning Instruction Select Instruction Instruction

DEVELOP -IJ Events/ o Management o Existing Activities Plan & Deli- Materials

very System

1V.1 IV.2

Implement Conduct

PHASE IV Instruc- Instruction

IMPLEMENT Management Plan,

V.1 V.2 V.3

Conduct Conduct Revise

PHASE V Internal External System

CONTROL - Evaluation D Evaluation •-

FIGURE 1. Interservice Procedures for Instructional

Systems Development Model

£# 2
developed materials. During IMPLEMENTATION, Phase IV, the plan of Phase

III is activated with particular attention to the personnel needed to

accomplish the plan. The quality of the instruction is assessed during

Phase V CONTROL in terms of new skills acquired by the trainee and the

fulfilled needs of the Command. As is shown in Figure 1, the five phases

(identified by Roman numerals) have been elaborated upon in 19 blocks,

(Arabic numbers), each with its own inputs, outputs, and feedback loop

following the requirements of a systems engineering approach. A brief

example on notation is Block II.1 which refers to the second phase-DESIGN

and first block--Develop Objectives.

The remainder of this report will briefly discuss a problem observed

with the implementation of the ISD model, provide an annotated bibliography addressing the problem and pose directions for further research

and development.

PURPOSE

The activities of military instructional developers have been specified by a TRADOC Regulation (Pamphlet 350-30) which is formally known as

the Interservice Procedures for Instructional Systems Development Model

(ISD). This document provides "procedures" to be followed when devising

a course of instruction. These procedures reflect the state-of-the-art

when the model was devised. Unfortunately little guidance beyond "what

needs to be done" is provided by the ISD model.

The present report was designed to review instructional development

resources in an effort to provide more proceduralized guidance to individuals tasked with implementing the ISD model. These individuals are

referred to as course authors. By definition authoring materials may

take the form of research studies, textbooks, handbooks, computer programs,

audiovisual sources, or technical manuals as long as the intention of

the resource is to guide the analysis, design, development, implementation, or control of instruction. Several topics are outside the boundary

conditions of this report. These include research and development related

to cost-effectiveness (e.g., Seidel and Wagner, 1979) and artificial intelligence applications in instruction (e.g., Gentner, 1979).

METHOD

The present research was intended to update and supplement a previous literature review and annotated bibliography of authoring materials which was prepared and updated by Logan (1977, 1979). Many of

Logan's conventions were adopted so that this report could be used in

conjunction with his. However, this document includes the most cogent

of Logan's references so that it is complete in itself as well. The

3
bibliography was formed over the course of four activities: search,

classify, summarize, and assess.

A literature search, beginning with documents published in 1960,

was conducted manually through the library of the U.S. Army Research

Institute for the Behavioral and Social Sciences, Psychological Abstracts and abstracts from meetings of relevant professional organizations. Computer-based data searches were made with the Defense Documentation Center (DDC), the National Technical Information Center (NTIS), and

the Educational Resources Information Center (ERIC). Some of the descriptors used in the search activities were:

Audiovisual Communications Media

Behavioral Objectives

Computer-Assisted Instruction

Computer-Managed Instruction

Education

Educational Program Evaluation

Individualized Instruction

Instructional Materials

Job Analysis

Systems Analysis

Task Analysis

Test Construction

Testing

Training

Documents were also made available through personal contacts with researchers and practitioners knowledgeable in instructional systems development. Copies of relevant documents were made on paper or microfiche

and manually stored.

All documents were then classified according to the 19 block scheme

indicated by the ISD model (see Figure 1). In some cases a document applied to more than one block. In those cases it was annotated for each

block.

Next, a short summary was written after a thorough reading of each

document. If a document was summarized, both authors felt it was of

value. The purpose of the summary statement was to identify the document as an authoring aid, procedure, or technique. An authoring aid

was defined as a "how to do it" manual which directly guides the author

through an activity identified by the ISD model. A procedure was defined as a listing of the activities involved in a block of the ISD

model. A technique was defined as a methodology for accomplishing an

activity identified by the ISD model but lacking sufficient detail to

be d procedure. Finally, the status of authoring aids and procedures

a.Giiaole wao made on a block by block basis.

4
In the RESULTS section which follows, the annotated bibliography is

presented with a summary of the purpose of each block. The references are

organized by block and then alphabetically within block. A. letter preceding each reference indicates whether it is an authoring aid (A), procedure (P), or technique (T). A reference numbered A 2.4.8 identifies

an authoring aid for Phase II-DESIGN, Block 4-Determine Sequence and

Structure which is the eighth source. The notation in Arabic numbers

identifying the source is consistent with Logan (1979).

A Status section for each block indicates the availability of authoring aids sufficient to guide an individual through all activities specified by the block as well as the availability of relevant procedures and

techniques which could be developed into authoring aids. A figure indicating the availability of authoring aids, procedures, and techniques

by phase and block follows, using the A, T, and P notation established

earlier.

In some cases the status of a block may have indicated procedures but

no techniques. It can be assumed that if a procedure exists a technique

was available prior to its development. Educational Research Information

Center or National Technical Information Service numbers were noted when

available.

It should be noted once again that the main intention of the annotated bibliography was to identify authoring aids. The inclusion of procedures and techniques was intended to direct developmental efforts

toward authoring aids.

5

Of
REFERENCES

Avner, R. A. Production of Computer-based Instructional Materials. In

H. F. O'Neil, Jr. (Ed.), Issues in instructional systems development. New York: Academic Press, 1979.

Gentner, D. R. Toward an Intelligent Computer Tutor. In H. F.

O'Neil, Jr. (Ed.), Procedures for instructional systems development. New York: Academic Press, 1979.

Logan, R. S. A State of the Art Assessment of Instructional Systems

Development. In H. F. O'Neil, Jr. (Ed.), Issues in instructional

and systems development. New York: Academic Press, 1979.

Logan, R. S. A survey analysis of military computer based systems: A

two part study. Volume I: A survey and annotated bibliography of

aids for instructional systems development (Report No. MDC E1570).

Arlington, VA: Defense Advanced Research Projects Agency, 1977.

(AD 406 902)

Orlansky, J., & String, J. Cost effectiveness of computer based instruction in military training. Alexandria, VA: Institute for Defense

Analysis, 1979.

Schulz, R. E., Hibbits, N., Wagner, H., & Seidel, R. J. On-line authoring aids for instructional design (Technical Paper). Alexandria,

VA: U.S. Army Research Institute for the Behavioral and Social

Sciences, in press.

Seidel, R. J., & Wagner, H. A cost-effectiveness specification. In

H. F. O'Neil, Jr. (Ed.), Procedures for instructional systems

development. New York: Academic Press, 1979.

Training and Doctrine Command. Interservice procedures for instructional systems development (5 vols.) (TRADOC Pamphlet 350-30).

Fort Benning, GA: Combat Arms rraining Board, August 1975.

United States Continental Army Command. Systems engineering of training

(CONARC Reg 350-100-1). Fort Monroe, VA: Headquarters United

States Continental Army Command, February 1968.

MI4 7
RESULTS

AN ANNOTATED BIBLIOGRAPHY

Purpose of Block 1.1 Analyze Job

To determine which activities compose a job, what level of performance is adequate, under what conditions the job is done and in what

order.

References

P 1.1.1 Ammerman, H. L. Stating the tasks of the job (Vol. 2).

Performance content for job training. Columbus: The

Center for Vocational Education, The Ohio State University, 1977.

Procedures are given for conducting a task inventory.

P 1.1.2 Christal, R. E. The United States Air Force Occupational

Research Project (AFHRL-TR-73-75). Lackland Air Force

Base, TX: Air Force Human Resources Laboratory, Occupational Research Division, January 1974. (AD 774 574)

Christal reviews the development and progress of the Air Force Occupational Research Project which began in 1958. The project has produced the Comprehensive Occupational Data Analysis Programs package

(CODAP). CODAP is a computer-based system which inputs, reduces, and

analyzes raw data from job inventories. It also generates reports

identifying individual and group job characteristics and between-job

differences. It is currently used by all of the Armed Forces.

T 1.1.3 Control Data Corporation. Instructional task analysis

(Pub. No. 76360712). Minneapolis, M: Control Data

Data Corporation, 1978.

This document reviews three techniques for task analysis--() job task

analysis, (2) information-processing task analysis, and (3) hierarchical task analysis.

P 1.1.4 Folley, J. D. Jr. Development of an improved method of

task analysis (Tech. Report). Villencia, PA: Applied Science Associates, June 1964. (AD 445 869)

Folley provides a detailed procedure for task analysis indicating what

information is needed and how it should be obtained and recorded.

T 1.1.5 McCormick, E. J., De Nisi, A. S. & Shaw, J. B. The use

of the position analysis questionnaire (PAQ) for establishin2 the job component validity of tests (Report

No. 5). West Lafayette, IN: Purdue University, 1977.

(AD A042 270)

9
A research program is discussed which used the PAQ, a structured job

analysis technique to establish "job component validity" of tests. The

PAQ is a complicated questionnaire which enables the respondent to

analyze a job in relation to 187 job elements describing job related

behaviors.

P 1.1.6 Merrill, P. F. Task analysis--An information processing approach. National Society for Performance and

Instruction Journal, 1976, 15(2) 7-11.

This article reviews Gagne's hierarchical task analysis procedure and

presents an alternative approach--information processing analysis.

Merrill suggests that two skills have an hierarchical relationship if

learning one skill is prerequisite to learning the other. An information processing relationship may be observed if "the outputs of one

operation are required as the inputs for another operation."

T 1.1.7 Montemerlo, M. D. & Eddowes, E. The judgmental nature

of task analysis. Proceedings of the Human Factors

Society Annual Meeting. Detroit, 1978.

This paper examines the state of the art of task analysis. Montemerlo

and Enddowes suggest that "task analysis" defies task analysis. They

use the term "proceduralizable" to characterize those tasks which can be

described. Task analysis which is largely judgmental does not lend

itself to proceduralization. Their argument is supported by seven factors which indicate that task analysis is an artistic, creative, synergistic, multi-purpose, problem solving, global, interpersonal, political,

and cognitive task. They conclude that task analysis cannot be reduced

to one specific procedure.

P 1.1.8 Powers, T. E. Selecting presentation modes according to

personnel characteristics of users and the nature of

job tasks, Part I job tasks. Baltimore, MD: University of Maryland, Baltimore County, July-Dec. 1977.

(AD A038 511)

In this volume, Powers examines the usefulness of ten job task categories in an effort to provide a uniform procedure for classifying any

job task. Seven categories were found to be highly applicable to many

technical job tasks. Other volumes examine a task analysis technique

which distinguishes the way technical information is presented according to personnel aptitude and category of job task.

T 1.1.9 Task Inventory Exchange. Brochure. Columbus, OH: The

Center for Vocational Education, The Ohio State

University, 1975.

10 I'
The Task Invf ntory Exchange (TIE) is a national clearinghouse for occupational task inventories which are lists of job tasks. The goal of

the TIE is to increase the sharing of task inventories in order to

reduce duplication.

P 1.1.10 Training and Doctrine Command. Job and task analysis

regulation (TRADOC Reg. 351-4). Fort Monroe, VA:

Headquarters, United States Army Training and Doctrine Command, February 1979.

This document defines Army policy, objectives, procedures, and requirements for the conduct of job and task analysis. TRADOC Pamphlet 350-30

is the basis for the regulation.

P 1.1.11 Training and Doctrine Command. Job and task analysis

handbook (TRADOC Pamphlet 351-4). Fort Monroe, VA:

Headquarters, United States Army Training and Doctrine Command, February 1979.

The handbook details procedures for job analysis, task analysis, and

the selection of tasks for training. The handbook was designed as a

reference for analysts and not as a complete authoring aid to prepare

trainees to perform job and task analysis procedures.

P 1.1.12 Washington State Department of Personnel. Task analysis

handbook. Olympia, WA: Washington State Department

of Personnel, July 1973. (PB 226 682)

This source presents a detailed task analysis procedure for personnel

analysts which includes examples of the forms task analysts use. Unfortunately it falls short of an authoring aid because it does not step

a task analysis trainee through the procedures.

T 1.1.13 Yaney, J. P. A critical review of the instructional

technology mechanism of task analysis. Improving

Human Performance: A Research Quarterly, 1973, 3(2),

64-70.

Yaney offers a critical review of the instructional technology mechanism of task analysis. He discusses assets and liabilities of three task

analysis techniques: (1) cognitive analysis, (2) master performance and

(3) field analysis. Yaney presents an operational cost model for task

analysis. He concludes that the costs typically outweigh the benefits.

11
Status

A variety of techniques and procedures exist for task and job analysis

(Control Data Corporation, 1977; Amerman, 19771 Folley, 1964; and

Powers, 1977). The notion of devising an inventory of job tasks is

particularly well developed. A clearinghouse for the dissemination of

task inventories has been established (TIE) as well as a computer-based

system (CODAP) to analyze data gathered by job inventories. The TRDOC

Pamphlet (351-4) indicates the job and task analysis data which must be

collected but does not provide the analyst with the rationale for making

the decisions involved in gathering this data. An authoring aid for this

block is unavailable.

1.1\ ",x2 \I i3 1,,.4 ,,,, ,i \"

Analyze \Select \Construct\, \Analyze \ Select 

'

PHASE PASLE I I Job \Tasks/ \Job Per- \ \Existing \Instruc- Jo Functions formance N o\Courses tional

\Measures \Setting

PT

iI

Q, 12
Purpose of Block 1.2 Select Tasks/Functions

To determine the tasks to be trained based upon certain selection

criteria and resource availability.

References

P 1.2.1 Ammerman, H. L. Identifying relevant job performance

(Vol. 3). Performance Content for Job Training.

Columbus: The Center for Vocational Education, The

Ohio State University, 1977. (ED 146 371)

Ammerman, H. L. & Essex, P. W. Deriving performance requirements for training (Vol. 4). Performance content

for job training. Columbus: The Center for Vocational

Education, The Ohio State University, 1977. (ED 146 372)

Volume 3 provides a procedure for conducting surveys to determine

relevant job tasks. The details offered on the types of survey questions to be devised are useful. Volume 4 presents a task selection

procedure employing prediction equations which determine the frequency

of task performance.

T 1.2.2 Page, E. B., Jarjoura, D. & Konopka, C. D. Curriculum

design through operations research. American-Educational Research Journal, 13(l), 1976, 31-49.

The authors present a technique which calculates values for the components involved in instructional decision making. This would make a most

useful authoring aid considering the required activities of Block I.I.

According to Page et al. operations research has rarely been used to

determine what to teach.

A 1.2.3 Schulz, R. E. Job aid for select tasks for training

(HUMRRO Draft Manual). Alexandria, VA: Human

Resources Research Organization, September 1978.

Schulz provides an authoring aid for Block 1.2 following the ISD text.

A system of worksheets for the collection of data, which supports selection decisions, appears to be a good technique for organizing a great

deal of task analysis input.

Status

Page et al. (1976) offer a technique for selecting tasks for training.

Further development could make this a useful authoring aid. Amnerman

and Essex provide procedures for task selection. An authoring aid following the ISD text is offered by Schulz (1978)

13
.1.2 "1 3 \I. 4

\Analyze, Select \Construct,\ Analyze \Select

PhSE I \Job Tasks/ \Job Per- \ "Existing \Instruc-

\ \ 0 Functions ,"formance o 'Courses , \ ional ,measures N' , \etig 

APT

14
Purpose of Block 1.3 Construct Job Performance Measures

To devise job performance measures (JPMs) for each task selected for

training. JPMs are tests which indicate how well an individual can

perform a job.

References

T 1.3.1 Ammerman, H. L. & Essex, D. W. Deriving performance requirements for training (Vol. 4). Performance content

for job training. Columbus: The Center for Vocational

Education, The Ohio State University, 1977. (E 146 372)

Volume 4 provides a brief introduction to an algorithmic method for

calculating level of task performance.

P 1.3.2 Gael, S. Employment test validation studies. JSAS Catalog of Selected Documents in Psychology, 1974, 4, 95.

(Ms. No. 711)

Gael discusses a procedure for the design and selection of instruments

used to test clerical performance of telephone company employees.

A 1.3.3 Schulz, R. E. Job aid for conducting task analysis (HumRRO

Draft Manual). Alexandria, VA: Human Resources Research Organization, September 1978.

Although the title differs from that of Block 1.3 the activities described by this authoring aid are those of 1.3. The aid provides a handbook for collecting and recording information which operationalizes each

job task.

Status

A procedure exists for the development of job performance measures for

clerical staff (Gael, 1974). It is reasonable to expect that this procedure could be applicable in other vocational settings. An authoring

aid exists following the procedures outlined in the ISD model (Schulz,

1978).

\I.1 \.2 1.3 I4 \1.

\Analyze \Select Construct \Analyze \Select

\Job \Tasks/ Job Per- \Existing \Instruc-\\\

PHASE I A _ .T

ANALYZE Functions\ oformance I"\Courses , 0\tiona1

Measures "Settinq

15
Purpose of Block 1.4 Analyze Existing Courses

To determine the degree to which a course includes the same tasks which

the command needs to teach.

References

A 1.4.1 Schulz, R. E. Job aid for analyzing existing courses

(HumRRO Draft Manual). Alexandria, VA: Human

Resources Research organization, September 1978.

Schulz provides an authoring aid for Block 1.4 of the ISD model.

Status

An authoring aid exists for this block which was designed following the

guidelines of the ISD model (Schulz, 1978). Procedures and techniques

other than those specified in the ISD text itself were unavailable. A

validation of the Schulz aid would be particularly valuable for determining

further developmental efforts for this block.

1\2 1. \1.3 1.4 \1.5

\Analyze \Select \Construct Analyze \Select

PHASE I \Job \Tasks/ \Job Per- Existing \Instruc-\

&\\Functions poformance o Courses I\tional

ANALYZE \unctions \Measures Core\Setting

A 1P T \\ 

' 16

o.IF
Purpose of Block 1.5 Select Instructional Settings

To assign tasks to one of the following instructional settings: (1)

Job Performance Aids, (2) Self Teaching Exportable Packages, (3) Formalon-the Job Training, (4) Installation Support Schools or (5) Resident

Schools.

A 1.5.1 Schulz, R. E. Job aid for selecting instructional settings

(HumRRO Draft Manual). Alexandria, VA: Human Resources Research Organization, September 1978.

Schulz provides an authoring aid which guides the trainee through Block

1.5 of the ISD model.

Status

The purpose of this block is unique to military instructional settings

which limits the availability of references. Schulz (1978) has developed an authoring aid covering the five settings stated in the purpose of this

block. Techniques indicating the distinctions and characteristics of the

five settings identified by the model and procedures for assigning tasks

to all of these settings would be useful. Authoring aids following other

instructional models would be of interest.

\l.2 \N-, 1.3yz \I.4 N 1.5

\Analyze 'Select \Construct\ \Analyze Select

PHASE I Job \Tasks/ \ Job Per- \Existing Instruc- ALZ \Functions '\formance b\Courses So tional PASAYE 10ob\ ~ \Measures Setting

P\ APT\ A PT A

I 17
Purpose of Block 11.1 Develop Objectives

To develop learning objectives for the learning categories of information mental skills, physical skills, and attitudes.

References

P 2.1.1 Gibbons, A. S. & Hughes, J. A. A method of deriving hierarchies of instructional objectives. Proceedings of

the Human Factors Society Annual Meeting. Detroit,

1978.

Gibbons and Hughes offer a procedure for transforming a task list into

hierarchies of instructional objectives. This document is a detailed

authoring procedure which utilizes terminology of the ISD model. It

would be a good candidate for an authoring aid effort

P 2.1.2 Hillelsohn, M. J. Instructional objectives. Teaching

with computers: A guide to creating instruction for

computer presentation. Unpublished manuscript, 1979.

(Available from Human Resources Research Organization,

Alexandria, VA.)

Hillelsohn provides a procedure for devising objectives for computerbased learning. With the inclusion of opportunities for practice this

document could be readily modified into an authoring aid.

T 2.1.3 Roudabush, G. E. An empirical structure for reading objectives. A paper presented at the annual meeting

of the American Educational Research Association.

Chicago, April 1974. (ED 095 194)

Roudabush's research examines hierarchical organizations of reading objectives. Roudabush briefly discusses instructional objective literature and provides considerable detail about reading behaviors.

A 2.1.4 Schulz, R. E. Job aid for developing objectives (HumRRO

Draft Manual). Alexandria, VA: Human Resources

Research Organization, December 1978.

This document is an authoring aid which follows the procedures outlined

in Block II.1-Develop Objectives.

I °" 18
Status

The Gibbons and Hughes (1978) effort and the Hillelsohn (1979) effort

are useful procedures for developing objectives. The latter is particularly of interest because it is a procedure for devising objectives from

a task list which is the focus of this block. Both could be modified to

form authoring aids. The Schulz (1978) aid follows the ISD model.

NJ~ 11I.2 \11I.3 \\11I.4 N

Develop \Develop \Describe \Determine\

PHASE DE II Objectives \Tests \Entry \Sequence & II Behavior PStructure

19
Purpose of Block 11.2 Develog Tests

To write test items for each learning-objective.

References

P 2.2.1 Conoley, J., & O'Neil, H. F., Jr. A primer for developing test items. In H. F. O'Neil, Jr. (Ed.), Procedures

for instructional systems development. New York:

Academic Press, 1979.

Conoley and O'Neil provide detailed procedures for the construction of

multiple-choice test items. Guidelines for revisions are embedded in

test item examples.

P 2.2.2 Gronlund, N. Preparing criterion-referenced tests for

classroom instruction. New York: Macmillan, 1973.

Procedures for the development of criterion-referenced tests are provided. Gronlund discusses the decisions which must be made by the test

developers--i.e., the level of mastery desired, scope of the subject matter covered, and type of test item. Also indicated are practical rules

for writing test items.

P 2.2.3 Plumlee, L. B. A short guide to the development of performance tests (Professional Series Pamphlet 75-1).

Washington, DC: U.S. Civil Service Commission,

Test Services Section of the Personnel Research and

Development Center, January 1975.

Plumlee provides an overview of a procedure for development of psychomotor tests. Task analysis, test reliability, and validity are also discussed. This source directs the reader to more detailed references.

A 2.2.4 Roid, G. & Haladyna, T. A handbook on item writing for

criterion-referenced tests. San Diego, CA: Navy

Personnel Research and Development Center, 1979.

Roid and Haladyna provide an authoring aid for the development of criterionreferenced tests. The authoring aid has been designed to be compatible

with the ISD model.

A 2.2.5 Schulz, R. E., Hibbits, N., Wagner, H. & Seidel, R. J.

On-line authoring aids for instructional design

(Technical paper). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social

Sciences, in press.

Schulz et al. provide an authoring aid for the development of test items

following Block 11.2 guidelines.

20
A 2.2.6 Swezey, R. W. & Pearlstein, R. B. Guidebook for developing criterion-referenced tests (Report No. 287-ARI8(2)-

IR-0974). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences, August

1975. (AD A014 987)

The manual devised by Swezey and Pearlstein is a well developed authoring aid for test construction in an Army environment.

Status

Test item development is replete with procedures and authoring aids from

civilian and military environments. The Roid and Haladyna (1978) handbook and the Schulz authoring aid have been designed with attention to

the ISD model.

\\ 11.2 \Develop N Develop ,1.3 1 4II.4 \Describe \Determine

\ObjectivesN Tests \Entry X \SequenceD&1

\0 t b oMBehavior2 \Structure '

21
I

Purposes of Block 11.3 Describe Entry Behavior

1. To develop and administer a test of entry skills and revise instruction according to the results.

2. To devise a pretest which measures the trainee's current knowledge

so that instructional units mastered may be bypassed.

References

No references could be identified for Block 11.3.

Status

The test development aspects of this block are most likely covered by

procedures and authoring aids identified in Block 11.2. Procedures need

to be defined to identify the trainee's entry skills. Authoring aids can

then be developed.

\II\II.2 H I.3 \11.4

\Develop \Develop Describe \Determine

\Objectives \ Tests Entry "Sequence &X

Ap Behavior m0,Structure

DESIGN N

II

22

" 22
Purpose of Block II.4 Determine Sequence and Structure

To devise a learning hierarchy which specifies the sequence in which

instructional objectives must be learned.

References

P 2.4.1 Airasian, P. W. A method for validating sequential instructional hierarchies. Educational Technology.

December 1971, 54-56.

Airasian provides an explanation of the notion of an instructional hierarchy. He presents a model for testing the validity of an established

hierarchy of instructional tasks. This procedure is needed; however,

a procedure for determining the interrelationships between tasks is a

higher priority which is not addressed.

T 2.4.2 Airasian, P. W. & Bart, W. M. The use of ordering theory

in the identification and specification of instructional sequences. A paper presented at the annual

meeting of the American Educational Research Association. Chicago, April 1974.

Airasian and Bart review a technique, which is an outgrowth of Guttman,

for ordering instructional tasks. As it stands this document is too

complex. A simplified version of ordering theory would be useful for

an authoring aid.

T 2.4.3 Boozer, R. F. & Lindvall, C. M. An investigation of

selected procedures for the development and evaluation of hierarchical curriculum structures. A paper presented at the annual meeting of the American Educational Research Association. February 1971.

A study assessing the usefulness of Guttman's scalogram analysis and simplex analysis for a mathematics program is presented. These techniques

provide procedures for investigating the sequence of instruction in terms

of student performance. Boozer and Lindvall affirm the value of scalogram

and simplex analysis, however, this report lacks sufficient detail for

the novice instructional developer.

P 2.4.4 Dansereau, D. R., Evans, S. H., Wright, A. D., Long, G.

& Actkinson, T. Factors related to developing instructional information sequences: Phase I (Report

No. AFHRL-TR-73-51(I)). Lowry Air Force Base,

CO: Air Force Human Resources Laboratory, Technical

Training Division, March 1974. (AD 777 832)

Dansereau, D. R., Evans, S. H., Actkinson, T. A., and

Long, G. L. Factors relating to the develoeaent of

optimal instructional sequences (Report No. AFHRLTR-73-51(II)). Lowry Air Force Base, CO: Air

Force Human Resources Laboratory, Technical Training Division, June 1974. (AD 783 843)

23
Phase I provides a review of instructional sequencing literature. The

results of an experiment conducted in Phase II indicate that a multidimensional scaling procedure for generating instructional sequences was

useful and reliable.

T 2.4.5 Dansereau, D. F., Long, G. L., Evans, S. H., & Actkinson,

T. R. Objective ordering of instructional material

using multidimensional scaling. Journal of Structural

Learning, in press.

Dansereau et al. discuss multidimensional scaling as a technique for devising instructional sequences and the interactions of sequencing with

type of material and individual differences. A good review of sequencing literature is provided but it is insufficient for a procedure or an

authoring aid.

P 2.4.6 Durell, A. B. A computer simulation study of measures

for validating learning hierarchies. A paper presented at the annual meeting of the American Educational Research Association, Chicago, April 1974.

Durell examines procedures for validating a hierarchy of tasks.

P 2.4.7 Hillelsohn, M. J. Organizing instruction. Teaching with

computers: A guide to creating instruction for computer presentation. Unpublished manuscript, 1979.

(Available from Human Resources Research Organization, Alexandria, VA.)

A procedure for structuring and sequencing computer-based learning materials is provided.

T 2.4.8 McDade, C. E. Subsumption versus educational set: Implications for sequencing of instructional materials.

Journal of Educational Psychology, 1978, 70(2), 137-

141. (ED 187 983)

McDade tested two sequences of educational psychology instruction. A good

review of the literature on instructional sequencing is provided.

P 2.4.9 Phillips, E. R. Development of optimal instructional sequences. A paper presented at the annual meeting of

the American Educational Research Association, Chicago,

April 1974. (ED 097 374)

Phillips reviews research supporting the notion that there can be an optimal sequence of instruction. Procedures for developing and validing instructional sequences are also presented. This is a readable literature

review which could supplement the ISD text.

24

eI
T 2.4.10 Reigeluth, C. M. In search of a better way to organize

instruction: The elaboration theory. Journal of

Instructional Development, 1979, 2(3), 8-15.

Reigeluth provides a brief introduction to the elaboration model of

sequencing instruction. The model suggests that instruction should be

organized in a "general-to-detailed" pattern rather than a hierarchical

sequence. A user's manual is in press.

T 2.4.11 Seidl, N. W. & McKeen, R. L. More on the use of student

generated learning hierarchies. Improving Human Performance: A Research Quarterly, 1974 3(2), 71-80.

Seidl and McKeen examine learning hierarchies for algebra instruction

devised by lou and high ability students. Although significant results

were not obtained, the notion of students devising their own learning

hierarchies deserves more attention in light of current efforts to individualize instruction.

T 2.4.12 Wellens, B., Lenke, J. M. & Oswald, J. H. An investigation of a proposed hierarchy of mathematics concepts

and skills using ordering theory. A paper presented

at the annual meeting of the American Educational

Research Association. New York City, March 1977.

The ordering theory approach to the construction of instructional hierarchies is reviewed. Wellens et al. investigated the effects of a

hierarchy of mathematical skills and concepts generated by the ordering theory technique.

Status

Procedures for devising learning hierarchies exist (Dansereau et al.,

1974, in press; Durell, 1974; Hillelsohn, 1979). Research studies have

examined optimal sequencing of instruction (Phillips, 1974) and the development of learning hierarchies by students (Stidl and McKeen, 1974). A

variety of theories exist for sequencing instruction; however, no reference

approaches an authoring aid.

I\.2 \1I3 11.4

\Develop \D \XN \Decribe Determine

\Objectives \Tests \Entry Sequence & PHASE 11 1Ojctvs\) Behavior 1 tutr

DESIGN \ tutr

\ \ PT

25

I
Purpose of Block 11I.1 Specify Learning Events/Activities

To classify learning objectives and identify "learning guidelines

necessary for optimum learning to take place."

References

T 3.1.1 Aagard, J. A. & Braby, R. Learning guidelines and

algorithms for types of training objectives (TAEG

Report No. 23). Orlando, FL: Training and Analysis Group, march 1976. (AD A023 066)

Aagard and Braby have identified 11 classes of instructional objectives

(such as rule learning) and have devised a training strategy for each

class. The strategies specify learning events which are compatible

with the instructional objectives.

A 3.1.2 Courseware, Inc. Author training course. San Diego,

CA: Courseware, Inc., January 1978. Also in H. F.

O'Neil, Jr. (Ed.), Procedures for instructional

systems development. New York: Academic Press,

1979.

The Author Training Course is a series of authoring aids designed to

assist course authors in converting instructional objectives into instructional materials. The course follows the guidelines of the ISD

model. Particular attention is paid to developing instruction which

maximizes learning.

Status

The Aagard and Braby (1976) technique would make a useful authoring aid.

The second reference available for this block is a course designed for

course authors who are tasked with Block III.1 of the ISD model.

III.1 \111.2 \1 III 3 \III.4 \ 'III.5

Specify Specify \Review/ \Develop \Validate

Learning \Instruction \Select \Instruction\ \Instruction\

DEELOP Events/ Activities b \Management \Plan & Deli- bo\Exi \Materials sting

"very System .K \>Z\

26
Purpose of Block 111.2 Specify Instructional Management Plan and

Delivery System

1. To specify the delivery system through media selection.

2. To prepare organizational plan for the instruction which indicates

instructional activities of students and instructors.

References

P 3.2.1 Braby, R., Henry, J. M., Parrish, W. F., & Swope, W. M.

A technique for choosing cost-effective instructional

delivery systems (TAEG Report No. 16). Orlando, FL:

Training and Analysis Group, April 1975, Revised

October 1978.

The Training Effectiveness, Cost Effectiveness prediction Technique

(TECEP) is a procedure designed to assist experienced training specialists in the selection of instructional delivery systems. The selection

is based upon the media requirements of the training objectives and the

cost-effectiveness of the various delivery systems which meet these

requirements (see T 3.1.1).

P 3.2.2 Bretz, R. The selection of appropriate communication

media for instruction: A guide for designers of Air

Force technical training programs ( eport No. R-601-PR).

Santa Monica, CA: RAND Corporation, February 1971.

(AD 721 443) !

Bretz discusses a procedure for media selection which is based on learning

objectives, course content, student characteristics, and teaching method.

T 3.2.3 Department of Defense Audiovisual Information System. A

user guide for the audiovisual products data base.

Arlington, VA: Office of Information for the Armed

Forces, DoD Directorate for Audiovisual Activities,

1975.

This document is a user guide for a computer-based network of audio-visual

products. The guide is largely concerned with the manipulation of the

Hazeltine terminal which is the delivery mechanism for the system.

T 3.2.4 Keeler, F. L. & Rizzo, W. A. An evaluation of microfiche

reader types for use with programued instruction (TAEG

Report No. 35). Orlando, FL: Training Analysis and

Evaluation Group, August 1976. (AD A029 714)

27
One study compared several types of microfiche readers and indexing

techniques, in an attempt to explore ease of reader usage with Navy recruits. A second study examined differences in efficiency with the use

of microfiche versus traditional paper copy. No significant differences

in performance were observed. The studies answered important questions

concerning microfiche media but did not approach an authoring procedure

or aid.

P 3.2.5 Lindahl, W. H. & Lin, B. W. An heuristic approach for

the scheduling of Navy Training Courses (TAEG Report

No. 52). Orlando, FL: Training Analysis and Evaluation Group, December 1977. (AD A048 183)

A procedure indicating steps and sources for scheduling Navy training

sources is presented. The advantages of automated scheduling are

examined.

T 3.2.6 Lonigro, J. F. & Eschenbrenner, A. J., Jr. A model for

selecting the media in technical/vocational education.

In H. Hitchens (Ed.), Selecting media for learning:

Readings from audiovisual instruction. Washington,

D.C.: Association for Educational Communications and

Technology, 1974.

Lonigro and Eschenbrenner propose a model for media selection which considers types of learning required by task, desired level of mastery,

matches between types of learning and media, and media production costs.

T 3.2.7 Auefli, T. W., Ruefli, J. C., O'Neil, H., Jr., & Smith,

C. M. A scheduling strategy for computer-managed instructional systems. Socio-Economic Planning Science, 1978,

12, 145-152.

Ruefli et al. describe a technique for and a test of a scheduling system for computer-managed Air Force training. The scheduling system

assigns the student to the next lesson based on previous performance

and computer terminal availability.

A 3.2.8 Schulz, R. E. Job aid for specifying instructional management plan and delivery system (HumRRO Draft Manual).

Alexandria, VA: Human Resources Research Organization,

January 1979.

Schulz has devised an authoring aid which guides an author through the

procedures specified by Block 111.2 of the ISD model.

P 3.2.9 Smith, E. H. Quality assurance of media devices and

courseware (AFHRL-TR-75-37). Lowry Air Force Base,

CO: Technical Training Division, Air Force Human

Resources Laboratory, October 1975. (AD A018 867)

28

. . . . .
Smith presents procedures which assure quality in the development of

visuals and recordings. Procedures to select cassettes and films are

briefly discussed. Smith focuses on specifics of good cassette and

film production.

T 3.2.10 Spangenberg, R. W. Overview of mediated courseware for

learning centers (AFHRL-TR-76-37). Lowry Air Force

Base, CO: Technical Training Division, Air Force

Human Resources Laboratory, June 1976. (AD A033 304)

Spangenberg presents guidelines for media selection based on the type

of information which is to be learned (i.e., intellectual skill, cognitive strategy, verbal information, motor skill, attitude). An effort

to rewrite this document following ISD terminology would make it a

useful procedure or authoring aid.

A 3.2.11 Stolovitch, H. D. Systematically selecting media: A

mathematical procedure. NSPI Journal, 1977, 16(7),

11-15.

Stolovitch provides an authoring aid for devising "your own" media/media

attribute matrix for media selection. A mathematical analysis of a completed matrix provides input for decision-making which incorporates (1)

the range of media available and (2) the media characteristics required

for attaining training goals.

Status

Procedures and techniques exist for the selection of media based on learner

characteristics (Bretz, 1971) and type of learning required by the task

(Lonigro & Eschenbrenner, 1974; Spanyenberg, 1976; Braby et al., 1978).

Several documents examine the use of different types of audiovisual media

in depth including cassettes, films, and microfiche. An authoring aid

exists for media selection which considers type of media and desired

learning outcome (Stolovitch, 1977). Another intention of Block 111.2

is to devise a schedule for the progress of instruction. A procedure

and a technique exist which provide guidance on scheduling traditional

and computer-based courses (Lindahl & Lin, 1971; Ruefli et al., 1978).

The Schulz (1979) authoring aid covers media selection and the preparation of an organizational plan.

29
111."11.3 11\I1.4 \111.5

\Specify Specify \Review/ \Develop \Validate

\Learning Instruction \Select \j "Instruction\ \\Instruction\ PHAS III \Events/ Management •)Existing

DEVELOP \Activities Plan & Deli- "Materials

very System Q \ APT

30
Purpose of Block 111.3 Review/Select Existing Materials

To review existing instructional materials and examine the correspondence with instructional objectives of the current training need.

References

P 3.3.1 Merrill, M. D.V Reigeluth, C. M. & Faust, G. W. The

instructional quality profile. In H. F. O'Neil, Jr.

(Ed.) Procedures for instructional system development. New York: Academic Press, 1979.

Merrill et al. discuss a procedure for diagnosing and correcting weaknesses in existing instruction through the examination of instructional

objectives. An authoring aid for this procedure is in production

(P 3.4.11, P 5.3.1).

A 3.3.2 Schulz, R. E. Job aid for review and selection of

existing material (HumRRO Draft Manual). Alexandria,

VA: Human Resources Research Organization, October

1978.

Schulz has prepared an authoring aid for Block 111.3 of the ISD model

which follows the procedures outlined in the ISD text.

Status

The authoring aid for this block, which is now in production (P 5.3.1)

will provide guidance in the analysis of existing instruction. The

authoring Aid provided by Schulz (1978) follows the procedures of the

ISD model.

\111.1 \N ,,III.2 111I.3 ',11I.4 " 1"1I .5

\Specify \Specify Review/ \Develop \Validate

PHAS I \ \Eeat/ ~ M namnsut~ Eitn \Learning \ Instruction , Select \Instruction\ \Instruction\ PHIASE III \Events/ N., 1\anagement NI Existing

DEVELOP \Activities \Plan & Deli-, Materials \\\\\\\ \very System

A T\ A P T A P

31
Purpose of Block 111.4 Develop Instruction

To prepare new instructional materials for learning objectives which

are not covered by existing materials.

References

A 3.4.1 Bell, N. T. & Abedor, A. J. Developing audio-visual instructional modules for vocational and technical

training. Englewood Cliffs, NJ: Educational Technology Publications, 1977.

Bell and Abedor provide an authoring aid for the development of selfinstructional audio-visual materials based on specified instructional

objectives.

A 3.4.2 Braby, R., Parrish, W. F., Guitard, C. R. & Aagard, J. A.

Computer-aided authoring of programmed instruction

for teaching (TAEG Report No. 58). Orlando, FL:

Training Analysis and Evaluation Group, June 1978.

This report describes a computer program (AUTHOR) that generates programmed training manuals which teach symbol recognition. The present

document details computerized authoring system which produces a learnpackage containing a pre-test, drills, and criterion tests based on

author input of a set of symbols. The next two sources are companion

documents to A 3.4.2 and therefore do not have reference numbers.

Guitard, C. R. Computer-aided authoring sXstem (AUTHOR)

users guide, volume I (TAEG Report No. 59). Orlando,

FL: Training Analysis and Evaluation Group. June

1978.

This document is an operator's manual which accompanies the previous

reference. It also lists the computer program for AUTHOR which assists

in the preparation of training materials.

Braby, R. Computer-aided authoring of instructional

materials (Tech. Note 2-79). Orlando, FL: Training

Analysis and Evaluation Group, June 1978.

Braby revi ws the computer-aided authoring tec-hnique presented in TAEG

Reports No. 58 and 59. He also reports that materials produced by computer-aided authoring routines were found to produce performance superior

to those using traditionally developed materials.

A 3.4.3 Braby, R., Kincaid, J. P., & Aagard, J. A. Use of

mnemonics in training materials: A guide for

technical writers (TAEG Report No. 60). Orlando,

FL: Training Analysis and Evaluation Group,

July 1978.

32
Braby et al. provide an authoring aid for writing mnemonics (memory aids

to be included in training materials for classroom instruction. Nine

mnemonic techniques are reviewed and details on when each technique

should be used are given. The following document is a companion document to A 3.4.3.

Ainsworth, J. S. Symbol learning in Navy technical

training (TAEG Report No. 66). Orlando, FL:

Training Analysis and Evaluation Group, January

1979.

This report is an evaluation of the technique developed by TAEG for

writing mnemonics (Report No. 60) for symbol learning and a feasibility

demonstration of the computer-based authoring system (A 3.4.2). Handbooks produced by the computer-aided authoring routines were found to

be desirable as well as the complete algorithm for symbol learning

during morse code training.

A 3.4.4 Courseware, Inc. Author training course (Data Item No.

0002AD). San Diego, CA: Courseware, Inc., January

1978. Also in H. F. O'Neil, Jr. (Ed.), Procedures

for instructional systems development. New York:

Academic Press, 1979.

The Author Training Course is a series of authoring aids designed to

guide course authors through Block 111.4 of the ISD model (A 3.1.2).

The goal of the course is to have authors develop instruction for

specified instructional objectives.

A 3.4.5 Deterline, W. A. & Lenn, P. D. Coordinated instructional

systems. Palo Alto, CA: Sound Education, 1972.

This is a lesson book which forms a self study course on instructional

design. The book is a thorough authoring aid which guides the author

through the development of two lessons. Deterline and Lenn indicate

the time needed to reach mastery in their course of instructional

design is an average of 35 hours.

P 3.4.6 Hillelsohn, M. J. Production of CBLM. Teaching with

computers: A guide to creating instruction for computer presentation. Unpublished manuscript, 1979.

(Available from Human Resources Research Organization,

Alexandria, VA.)

Hillelsohn presents a procedure for putting instructional materials

into a computer system. The steps involved in programming, inputting,

and debugging the system are detailed.

P 3.4.7 Joyce, R. P., Cheznoff, A. P., Mulligan, J. F. & Mallory,

W. J. Fully proceduralized job performance aids:

Handbook for JPA developers (Tech. Report). Vallencia,

PA: Applied Science Associates, December 1973.

33

4'
Joyce, R. P., Cheznoff, A. P., Mulligan, J. F. & Mallory,

W. J. Fully proceduralized job performance aids:

Handbook for JPA managers and training specialists

(Tech. Report). Vallencia, PA: Applied Science

Associates, December 1973. (AD 775 706)

Joyce et al. provide detailed procedures for the development of Job Guides

and Troubleshooting Aids for Air Force equipment maintenance. The development of this information into an authoring aid would give input for

those tasked with Job Performance Aid development.

A 3.4.8 Kern, R. P., Sticht, T, G., Welty, D. & Hauke, R. N.

Guidebook for the development of Army training literature. Alexandria, VA: Human Resources Research

organization, November 1976. (AD A033 935)

This document is an authoring aid for the development of training

manuals. The guide presents examples of training manual problems in a

"before and after revision" format.

A 3.4.9 Schulz, R. E., Hibbits, N., Wagner, H. & Seidel, R. J.

On-line authoring aids for instructional design (Technical paper). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences, in

press.

An authoring aid is provided for Block 111.4 of the ISD manual.

A 3.4.10 Shriver, E. L. Fully proceduralized job performance aids:

guidance for performing behavioral analyses of tasks

(AFHRL-TR-75-38). Lackland Air Force Base, TX: Air

Force Human Resources Laboratory, Occupational Research

Division, June 1975. (AD A015 059)

Shriver amends the Joyce et al. (see P 3.4.7) documents with finer details

for job performance aid development.

P 3.4.11 Wulfeck, W. H., Ellis, J. A., Richards, R. E., Wood,

N. E. & Merrill, M. D. The instructional quality

inventory: I. Introduction and overview (Special

Report No. 79-3). San Diego, CA: Navy Personnel

Research and Development Center, November 1978.

The Instructional Quality Index (IQI) is a procedure for the development

of instruction based on the learning behaviors and outcomes specified by

the instructional objectives. Authoring aids for the IQI are in preparation. The IQI was originally called the Instructional Strategy Diagnostic

Profile (ISDP) (P 3.3.1, P 5.3.1).

34
Status

This block is the best developed in terms of the availability of authoring aids for print and audio-visual using the ISD model (Courseware, 1978; Schulz et al., 19771 Wulfeck et al., 1978; Braby et al. 1978) perhaps because it is the most basic to the intent of the model. Authoring aids

are available for the programming (Hillelsohn, in press) and are unavailable for the development of computer-based instructional materials.

\I ;; III1 . 1.2 \III.3 \ I . \III.5 ' \Specify \Specify \Review/ Develop \Validate

\Learning HAE Instruction ~Actiitie & eli- \Pln "Matrial \Select Instruction )Instruction\ DEVELO DEVEOP I\Events/ Activities.X p\\Management \Plan & Deli-N Io\Existing\I \Materials\\

T, , "' APATP A P

35

.. ... m . . . . . n
Purpose of Block 111.5 Validate Instruction

To debug instructional materials by trial with students.

References

P 3.5.1 Dick, W. Applications of formative evaluation to the

instructional design process. A paper presented at

the annual meeting of the American Educational Research Association. New York City, April 1977.

(ED 132 135)

Dick reviews procedures for formative evaluation which must be part of

an instructional development plan so that evaluative information is

gathered during each phase of the developmental process.

P 3.5.2 Kandaswamy, S., Stolovitch, H. D. & Thiagarajan, So

Learner verification and revision: An experimental

comparison of two methods. AV Communication Review,

1976, 24(3), 316-328.

Kandaswamy et al. examined the cost effectiveness of two procedures for

testing instructional materials on students and making revisions based

on their feedback. Instructional materials can be piloted with one student or with a group of students. Their research findings indicated no

significant difference. These results indicate that the more cost effective method would be trials with one student at a time.

T 3.5.3 Rayner, G. T. An empirical study of a methodology for

the revision of systematically designed educational

materials (CAI Center Tech Report No. 24). Tallahassee, FL: Florida State University, March 1972.

(ED 067 877)

Rayner developed and tested a model which identified revisions for instructional materials based on content and procedural deficiencies.

The model was found to be empirically useful for revising a health education course. (See Block V.3.)

A 3.5.4 Schulz, R. E. Job aid for validating instruction (HumRRO

Draft Manual). Alexandria, VA: Human Resources Research Organization, October 1978.

Schulz provides a detailed authoring aid which steps the trainee through

validation procedures identified by the ISD model.

P 3.5.5 Wulfeck, W. H., Ellis, J. A., Richards, R. N., Wood, N. E.,

& Merrill, M. D. The instructional quality inventory:

I. Introduction and overview (Special Report No. 79-3).

San Diego, CA: Navy Personnel Research and Development

Center, November 1978.

36
The IQI is a quality control procedure designed to evaluate instructional

objectives, tests and materials designed by the ISD method (P 3.3.1,

P 5.3.1).

Status

Procedures, techniques, and authoring aids following the ISD model (Schulz,

1978; Wulfeck et al., 1978) exist for pilot testing newly developed instructional materials.

<1III.l "111I.2 N \III.3 INN ,"III.4 111I.5

\Specify N \Specify N \Review/ l Develop Validate

\Learning \Instruction \Select K\ Instruction\ Instruction

PHASE III \bents/ p \Managemeenite System

DEVELOP \Activities " "Plan ,~very & System Deli-, " \Materials ~

A N\ A P T P A P T

37
Purpose of Block IV.l Implement Instructional Management Plan

To initiate plan which specifies trainees, learning activities, and resources (this is achieved by making and completing checklists).

References

The authors could identify no references for this block.

Status

Some references included in Block 111.2 discuss implementation. An

authoring aid would be useful for activities such as the development of

checklist used during implementation.

IV.1 \IV.2

Implement \Conduct

PHASE IV Instruc- \Instruction\

IMPLEMENT tional

Management

Plan

3

38J
Purpose of Block IV.2 Conduct Instruction

For the instructor to initiate the instruction using the setting, methods

and media specified by the management plan.

References

P 4.2.1 Hillelsohn, M. J. Implementation of CBLM. Teaching

with computers: A guide to creating instruction for

computer presentation. Unpublished manuscript, 1979.

(Available from Human Resources Research Organization,

Alexandria, VA.)

Hillelsohn provides a procedure for initiating computer-based instruction by detailing the tasks of the course administrator.

Status

The Hillelsohn (1979) procedure would make a useful authoring aid for

computer-based instruction. Procedures and authoring aids are needed

which focus on the implementation of traditional classroom instruction

utilizing a variety of media.

',IV.1 IV.2

\Implement Conduct

PHASE IV \Instruc- N Instruction \tional ]

IMPLEMENT \tP.nag

\LManagement

\Pl an

3

I
Purpose of Block V.1 Conduct Internal Evaluation

1. To determine student progress.

2. To improve quality of instruction.

These goals are achieved through the collection of data from students,

instructors, and supervisors and the evaluation of this data. Data

must indicate the relationship between student entry behavior and

achievement on the terminal learning objectives. The ISD effort is evaluated as well. The development of the instruction is monitored to see

that it is consistent with prescribed ISD procedures.

References

P 5.1.1 Borich, G. D. A systems approach to the evaluation of

training. In H. F. O'Neil, Jr. (Ed.), Procedures

for instructional systems development. New York:

Academic Press, 1979.

Structured decomposition is a program evaluation procedure in which data

is collected in each component of a program. Discrepancies observed

between planned and actual outcomes are examined.

P 5.1.2 Champagne, A. B. & Klopfer, L. E. Formative evaluation

in science curriculum. Journal of Research in Science

Teaching, 1974, 11(3), 185-203. (ED 088 718)

A procedure for the formative evaluation of a new science curriculum is

presented. Sources of evaluation data considered are the curriculum development plan, student instructional materials, student behaviors with

respect to science content, student behaviors in the classroom, the teacher

and course marketability.

P 5.1.3 Elsbree, A. R. & Howe, C. An evaluation of training in

three acts. Training and Development Journal, 1977,

(July), 1-19.

Elsbree and Howe discuss procedures for the evaluation of training which

focus on (1) the establishment of the general scope and objectives of

the evaluation, (2) the production of a blueprint for conducting the

project and (3) obtaining data, interpreting data, and reporting to the

client. Flow charts of each of these phases are provided as well as a

narrative providing an example specific to Civil Service training.

T 5.1.4 Hall, E. R., Lam, K., & Bellomy, S. Training effectiveness assessment: Volume I, Current military training

evaluation programs (TAEG Report No. 39). Orlando,

FL: Training Analysis and Evaluation Group,

December 1976. (AD A036 517)

40
Hall, Z. R., Rankin, W. C. & Aagard, J. A. Training effectiveness assessment: Volume I1 Problems, concepts,

and evaluation alternatives (TAEG Report No. 39).

Orlando, Pt: Training Analysis and Evaluation Group,

December 1976. (AD A036 518)

Volume I examines interservice training evaluation efforts. Volume II

discusses problems which have been observed in Navy training evaluation

efforts. A variety of evaluation techniques are reviewed with guidelines indicating the most appropriate use of each.

P 5.1.5 U.S. Civil Service Commission. Training evaluation: A

guide to its planning, developnent, and use in agency

training courses (Pamphlet T-13). Washington, DC:

Bureau of Training, U.S. Civil Service Commission,

May 1971.

This pamphlet describes a procedure for assessing effectiveness of

internally developed and administered training courses.

P 5.1.6 U.S. Civil Service Commission. A process for the evaluation of training. Washington, DC: Training Leadership Division, U.S. Civil Service Commission, April

1978.

This document provides a procedure for formulating and implementing an

evaluation plan.

Status

Procedures and techniques exist for student assessment (Champagne &

Klopfer, 1974; Hall et al., 1976). Procedures for planning internal

evaluation are also available (Borich, 1979; Elsbree & Howe, 1977; U.S.

Civil Service Commission, 1971, 1974). Unfortunately authoring aids

and procedures which follow the ISD model are unavailable.

V.1 V.2

Conduct \Conduct \Revise

FPHASE V Internal \External \- stem

CONTROL Evaluation bo\Evaluation 1

'I PT

41
Purpose of Block V.2 Conduct External Evaluation

To determine whether trainees meet the needs of the command.

The external evaluation examines survey data of trainees in the field,

job performance measures, and evaluation of supervisors.

References

P 5.2.1 Department of the Air Force. Training and evaluation

course reviews (ATC Regulation 52-1). Randolph Air

Force Base, TX: Headquarters Air Training Command,

Department of the Air Force, September 1971.

This document is a regulation which details procedures for internal and

external evaluation. The intention of the data collected is to provide

feedback for ISD course improvement.

P 5.2.2 Pennell, R., Harris, D., Schwille, J. Appraisal of Air

Force training course field evaluation system (AFHRLTR-76-63). Lowry Air Force Base, COt Technical

Training Division, Air Force Human Resources Laboratory, October 1976. (AD A035 641)

Pennell et al. examined the Air Force field evaluation procedure with

the intention of improving data analysis and reporting. Recommendations

were made which focused on the design of the evaluation instrument and

frequency of evaluations.

P 5.2.3 Walker, R. A. Post-hoc evaluation of S-3A instructional

systems develoxnent. San Diego: Courseware, Inc.,

1978.

Walker reports on procedures used to evaluate a training course for an

anti-submarine warfare aircraft (S-3A) developed by an ISD approach.

Status

Procedures exist for conducting an external evaluation for military

training courses. Authoring aids are needed which follow the ISD model.

v1 V.2 \V.3

\Conduct Conduct \Rev i se

\Internal >1 External \System CONTRoLPHA Evaluation oN Evaluation Jn

42
4I

Purpose of Block V.3 Revise System

To revise the system based on internal and external evaluation reports

and changes in instructional needs due to new doctrine, procedures,

or weapons systems.

References

P 5.3.1 Ellis, J. A., Wulfeck, W. H., Merrill, M. D., Richards,

R. E., Schmidt, R. V. & Wood, N. D. Interim training

manual for the instructional quality inventory (NPRDC

TN 78-5). San Diego, CA: Navy Personnel Research

and Development Center, February 1978.

The Instructional Quality Inventory is an analysis procedure for evaluating

and revising instructional materials. This document, which is currently

undergoing revision, is a manual designed to train instructional developers

in the use of the IQI (P 3.3.1, P 3.4.11).

T 5.3.2 Gropper, G. L. Diagnosis and revision in the development

of instructional materials. Englewood Cliffs, NJ:

Educational Technology Publications, 1975.

Gropper reviews types of student failure and program failure which indicate needs for program revision.

T 5.3.3 Rayner, G. T. An empirical study of a methodology for the

revision of systematically designed educational materials

(CAI Center Tech Report No. 24). Tallahassee, FL:

Florida State University, Marcy 1972. (ED 067 877)

Rayner has designed a model for revising the content and procedures of

instruction which incorporates instructional objectives, diagnostic tests,

prescriptions, and posttests. The model has been successfully tested on

health education courses.

Status

A procedure and technique exist for revising a course based on internal

and external evaluation data. An authoring aid (Ellis et al., 1978;

P 3.4.11) following the ISD model is in preparation. More procedures and

authoring aids would be valuable.

\V IV.2 V.3

\Conduct \ \Conduct Revise

PHASE V \Internal N \External System

CONTROL \Evaluation, ibpEvaluation W

N \XX

T PT

43
CONCLUSIONS

The following section presents references for authoring systems.

The intent of these systems is to provide authoring aids or procedures

for all steps of the instructional systems design process. A summary

is then provided indicating the availability of authoring aids, procedures, and techniques by phase. Finally suggestions for research

and development are made in light of the availability of such materials

as noted in the summary.

Authoring Systems

Several systems have been developed or are in development which

guide an author through complete phases of an instructional development

effort. The systems use a variety of media for delivery and generally

incorporate the procedures identified by the ISD model.

Control Data Corporation. Computer resource for effective

author training in education (CREATE). Minneapolis,

MN: Control Data Corporation, 1978.

CREATE is a curriculum for the development of individualized instruction. A variety of media is used to deliver the course including

printed text, videotapes, and CAI. The student author is expected to

take the course at a learning center so that all media are readily

accessible. The curriculum is modularized so that a program of

instruction can be designed for each student.

Department of the Air Force. Training handbook for designers

of instructional systems (AFP 50-58). Washington, DC:

Department of the Air Force, July 1978.

The Air Force training handbook is a set of authoring aids or procedures detailing an instructional systems development model (AFM 50-2).

The major steps of the model are: (1) Analyze Systems Requirements;

(2) Define Education or Training Requirements; (3) Develop Objectives

and Tests; (4) Plan, Develop, and Validate Instruction; and (5) Conduct

and Evaluate Instruction. Instructional developers receive guidance

in filling out a series of forms for each step of the ISD process.

Hughes, J., O'Neal, A. F., Kearsley, G., O#Neal, H.,

Fee, M., Roli, L., & Hermanns, J. Functional specifications to computer aided training systems development

and management (CATSDM) support functions (Tech. Report

NAVTRAEQUIPCEN 71-C-0018-1), Orlando, FL: Naval

Training and Equipment Center, July 1978.

44
This reference is a specification for a system of computer programs

which manages the total ISD process.

Harris, W. P. An authoring system for on-the-job environments. In H. F. O'Neil, Jr. (Ed.), Issues in instructional systems development. New York: Academic Press,

1979.

The Lincoln Laboratory has devised and field tested a system which enables subject matter experts (SMEs) to create computer-deliverable onthe-job training materials. A Lincoln Terminal System was used to

deliver the authoring system. The authoring training lessons were

created to guide SMEs through the development of CAI materials without

providing instructional development theories.

Status

A system incorporating all blocks of the ISD model is currently unavailable. The Department of the Air Force Training Handbook (1978) is not

directly applicable for Army training developers because of its reliance

upon Air Force terminology. The Hughes et al. (1978) authoring support

system will be most appropriate to this need once it has been developed.

Throughout the prior reference section a series of authoring aids prepared by Schulz (Human Resources Research Organization) have been noted.

The aids have been developed under contract to the U.S. Army Research

Institute for Behavioral and Social Sciences. By October of 1979 thirteen aids covering Phases I, II, and III (excluding Block 1.1) will have

been developed and evaluated.

Summary of Availability of Authoring Materials by ISD Phase

1.1 1.2 1.3 1.4 1.5

Analyze Select Construct Analyze Select

PHASE I Job Tasks/ Job Per- Existing InstrucANALYZE • Functions • formance D Courses N tional

Measures Setting

PT APT APT A A

45"

q45
Job informat! -n is collected and analyzed during Phase I. Tasks

are identified for training and tests are developed to show job mastery.

An analysis of existing courses is made to determine the availability

of courses designed to teach the tasks selected for training. The best

match between selected tasks and instructional setting is then determined.

Schulz (A 1.2.3, A 1.3.3, A 1.4.1, A 1.5.1) has devised authoring

aids for all of the above activities except job analysis. The Schulz

aids provide forms to be filled out covering each procedure identifieA

by the ISD model. Although an authoring aid does not exist for Block

I.1-Analyze Job, there are procedures available which could be converted

to an authoring aid format (Training and Doctrine Command, P 1.1.11).

11.1 11.2 11.3 11.4

Develop Develop Describe Determine

PHASE II Objectives Tests Entry Sequence &

DESIGN •• Behavior •Structure

APT AP PT

The Design phase utilizes the job analys:s data produced in Phase I.

Learning objectives are devised for each task and test items are produced for each learning objective. Pretests are developed to examine

the trainees' knowledge prior to instruction so that the level of instruction is appropriate to the needs of its users. An analysis of

the learning objectives is made to determine the most appropriate order

for the instructional activities.

Authoring aids exist for the development of objectives (Schulz,

A 2.1.4), and test items (Roid & Haladyna, A 2.2.4; Schulz, A 2.2.5; Swezey &

Pearlstein, A 2.2.6). Procedures are available for sequencing instruction

by creating learning hierarchies (Dansereau et al., P 2.4.4, T 2.4.5;

Durell, P 2.4.6) but an authoring aid is not available. Neither techniques,

procedures, nor authoring aids are available for Block II.3-Describe Entry

Behavior.

4

46

I1
III.1 111.2 111.3 111.4 111.5

Specify Specify Review/ Develop Validate

Learning Instruction Select Instruction Instruction

PHASE III Events/ • Management J Existing • • DEVELOP Activities Plan & Deli- Materials

very System

AT APT AP AP APT

During Phase III the learning objectives are classified so that the

optimal instructional events to produce learning can be identified. A

plan is made which specifies the allocation of training personnel, instructional materials and plant requirements for the conduct of instruction. At this time a determination of the most appropriate instructional

delivery system (considering a variety of media) is made. A review of

instructional materials is made to determine whether existing instruction meets the requirements of the learning objectives. New instruction

is developed for objectives not taught by existing materials. A pilot

test is then conducted to try out the instruction.

Phase III is the best developed in terms of the availability of

authoring aids for print and audio-visual materials. At least one aid

is available for each block. Authoring aids are needed however for the

development of computer-based instructional materials. A considerable

number of procedures exist for media selection and the development of

instruction.

IV.l IV.2

Implement Conduct

PHASE IV Instruc- Instruction

IMPLEMENT tional

Management

Plan

P

The plan which specifies the trainees, learning activities, and

physical resources is initiated during Phase IV. The instructor receives an orientation to the course, conducts the instruction, and

records student data.

Authoring aids are unavailable for both blocks of Phase IV. Procedures were also unavailable for Block IV.. The single procedure

identified for Block IV.2 could be converted to an authoring aid;

however, its scope is limited to computer-based instruction.

47
V.I V.2 V.3

Conduct Conduct Revise

Internal External System

PHASE Evaluation • Evaluation • CONTROL

PT P PT

During Phase V the data collected about student performance are

evaluated to reveal deficiencies in the instruction. On-the-job performance of trainees is also of interest to determine whether the instruction meets the needs of the Command. Finally the internal and

external evaluation data are summarized to provide input for revisions

to be made on the completed instructional system.

Authoring aids are unavailable for the evaluation phase. A variety

of procedures exist for conducting internal and external evaluations but

none use the activities and terminology specified in the ISD model. The

procedure for revising instruction devised by Ellis et al. (P 5.3.1) will

evolve into an authoring aid directly applicable to Block V.3.

Summary and Plans for Research and Development

In Figure 2 the availability of authoring aids for the complete

ISD model is summarized. Briefly, an authoring aid is needed for

Block 1.1-Analyze Job. Procedures exist for job analysis so an authoring aid could be readily developed. Block 11.3 would require a greater

effort because procedures would need to be developed as well. Phase IIIDEVELOP has a sufficient number of authoring aids and procedures to

recommend no further effort for print materials. However, authors aids

are required for computer-based instructional materials. Phases IVIMPLEMENT and V-CONTROL need authoring aid development. Phase V would

be a higher priority than Phase IV because quality control is essential

to an instructional design effort.

Three concluding comments regarding this state-of-the-art assessment

of Instructional Systems Development are salient. First, there exists a

considerable lack of information on the best ways of managing the instructional systems development process. Second, the interrelationship between

the ISO process and instructional strategies is an important one, yet the

knowledge and technology base for understanding this relationship is lacking. An ongoing state-of-the-art assessment in instructional strategies

will address this issue. Third, although most persons in the field believe

that the instructional system developeent process is an expensive one, there

is a lack of accurate detailed ISD cost data.

48

It
I l 1.2 1.3 1 4 1.5

Analyze Select Construct Analyze Select

P EJob Tasks/ Job Per- Existing InstrucANALYVZE Functions formance Courses • tional

AAYEMeasures Setting

PT APT APT A A

11.1 11.2 11.3 11.4

Develop Develop Describe Determine

PHASE 11 Objectives Tests Entry Sequence &

DESIGN "0 10 Behavior 1 Structure

APT AP PT

111.1 111.2 111.3 111.4 111.5

Specify Specify Review/ Develop Validate

Learning Instruction Select Instruction Instruction DEVELOP -§0 Events/ Management N Existing • -

Activities Plan & Deli- Materials

very System

AT APT AP AP APT 

IM IV.2

Implement Conduct

PHASE IV Instruc- Instruction

IPASEN V tional

IMPLEMENT Management

Plan

I, P

V'.1 V.2 V.3

Conduct Conduct Revise

Internal External System

CONTROL Evaluation • Evaluation Le

Legend

P T P T I A Authoring Aid

P Procedure

T Technique

Figure 2. Availability of authoring materials for

the ISD model.

49

,, ,,I . .
The final product of this research and development effort is

viewed as an integrated instructional system composed of a series

of field tested self-paced modules. The modules will be ISD authoring

aids, an author management system, and a computer-based instruction

course to teach how to use both the job aids and management system.

The curriculum will be designed so that extensive resources, instructor

personnel or facility support will not be required. The modules will be

available in the form of audio-visual materials, books, and documented

computer programs. Evaluation data regarding its cost-effectiveness

will also be available.

50
DISTRIBUTION

ARt Oi~tribuiooj List

4 OASD (M&RA) 2 HOUSACDEC. Ft Ord, ATTN: Library

2 HODA tDAMI-CSZ) 1 HOUSACDEC. Ft Ord. ATTN: ATEC-EX--E Hum Flactors

1HODA IDAPE POP) 2 USAEEC, Ft Benjamin Harrison, ATTN: Lilvaiy

1 HODA IDAMA AR) 1 USAPACDC. Ft Beiilameii Harrison, ATT'N: ATCP --HA

1 HODA JDAPE HRE-PO) 1 USA Comm-Elect Sch, Ft Monmnouth, ATTN: ATSN -EA

1 HODA ISGAD-ID) 1 USAEC. Ft Monmouth, ATTN: AMSEL - CT HOP

I HODA IDAMI-DOT-C) 1 USAEC, Ft Monmouth, ATTN: AMSEL -PA P

1 HODA IDAPC-PMZ-A) 1 USAEC, Ft Monmouth, ATTN: AMSEL- SI-CA

I HODA JDACH-PPZ-A) 1 USAEC, Ft Monmouth, ATTN: C, Fact 0ev Br

I HODA (DAPE-HRE) I USA Materials Sys Anal Agcy. Aberudeen, ATTN: AMXSY -P

1 HODA (DAPEMPO-CI 1 Etigewoocl Arsenal, Aberdeen. ATTN: SAREA SL H

iHODA IDAPE DW) 1 USA Oid Ctr & Sch, Aberdeen, ATTN: ATSL-TEM--C

1 HODA (DAPE-HRL) 2 USA Hum Engr Lab, Aberdeen, ATTN: Library/Dir

1 HODA (DAPE-CPSI 1 USA Combat Arms Trig Sd, Ft Benning., ATTN: Ad Supervisor

1 HODA (DAFO-MFAI 1 USA Infantry Hum Rsch Unit, Ft Banning, ATTN: Chief

1 HODA (DARO*ARS-PI 1 USA Infantry Od, Ft Benning. ATTN: STEBC- TE- T

I HODA (DAPC-PAS-A) 1 USASMA. Ft Bliss. ATTN: ATSS- -LAC

I HODA (DUSA OR) 1 USA Air Def Sch, Ft Blist, ATTN: ATSA CTD ME

1 iIfDA (DAMO-RORI 1 USA Air Oct Sch. Ft Bliss, ATTN: Tech Lit,

I1 IIOIA (DASGI I USA Air Del Ad, Ft Miss, ATTN: FILES

I HODA IDA I 0P0I 1 USA Air Del Ad. Ft Bliss. ATTN: STEAD -P0

I Clii-l, Constilt Div (OA-OTSGI. AdeI1,hi, MD I USA Cmii & General 5Sf College. Ft Leavenworth. ATTN: Lib

1 Mit As%t. Hum Res, Ot)DR&E. QAD (E&LS) I USA Cmcf & General SOt College, Ft Leavenworth. ATTN: ATSW-SE- L

1 110 LISAHAL. APO Seattle,. ATTN: ARAGP-R 1 USA Cmd & General Sti College, Ft Leavenvwrh. ATTN: Ed Advisor

1 HO First Army. ATTN: AFKA-01 TI I UISA Ciirnlitned ArIM Crntit Dr'v Art, Ft Leavwnworth. ATTN: Depdi

2 HO Fifth Army, Ft Sam Houston 1 USA Combinad Arms Cmbt Dev Act. Ft Leavenworth. ATTN: CCS

1 )it. Army S51 Studies Of c. ATTN: OAVCSA IoSP) 1 USA Comnbinad Arms Crrtbt Dee Act, Ft Leavenworth, ATN: ATCASA

I Ott- Chief Of1 St,. StRhiei!' Ofc 1 USA Combined Arms Cmnbt Des Act. Ft Leavenworth. ATTN: ATCACO -E

I tDtS'E R, ATTN: CPSfOCI' 1 USA Corimlsd Arms Cmlii Des Act, rt Leavi-rwoeth, ATTN: ATCACC-4I

1 rho Army Lili, Pesitarloii, ATTN: RSB Chief 1 USAFCOM. Night Vision Lai), Ft Belvroir, ATTN: AMSEL-NV-SD

I The Army Libi, Pentagfon, ATTN: ANRAL 3 USA Computers SCnset, F% t lIVOIT, ATTN: Tech Library

1 Olc. Asat Sect of the Army (R&D) 1 USAMEROC. Ft Belvoir, ATTN: SrSFA- -00

I Tech Support Ofc. OJCS 1 USA Eng Scb. Ft Aelvoir. ATTN: Library

I USASA, Arlilton. ATTN: IARD-T 1 USA Topographic Lab. Ft Belvoir, ATTN: ETI TO-S

1 USA Rsch Ofci: Durham, ATTN: Life Sciences Dir I USA Topographic Lab, Ft Belvemir, ATTN: STINFO Centter

2 SJSARIEM, Natick, ATTN: SGRD-UE CA 1 USA Triliographic Lab, Ft Belvoir, ATTN: EnL GSL

I LISAIrc, iI :.iro9i, AllN- !;t ll(;MO A I lIAMicllo.lr di tHivliia T:C I S

I USAIMA, FtBrdrg,ArrN: AlSU-CTOOM 1 USA Intelligence Ctt &Sch, Ft Huirhucti, ArTN: ATS-CTD-MS

I USAIMA. Ft Bragg. ATTN: Martuat Lib 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-TE

1 US WAC Cir & Sch, Ft McClellan. ATTN: Liti 1 USA Intelligence Cir & Sch, Ft Huachuca. ATTN- ATSI- TEX -GS

I US WAC Ctr & Sch, Ft McClellIan, ATTN: Togl Dir I USA Intelligence Cti & Sch. Ft Huachuca. ATTN: ATSI-CTS-OR

1 USA Quartermaster Sch. FtI Lee. ATTN: ATSM-TE 1 USA Intelligence Ctr &Sch, Ft Huachuca. ATiN: ATSI-CT- -OT

1 Intelligfence Material Dev Ole. EWL, Ft Nolabird 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI- CTt)-CS

I USA SE Signal Sch, Ft Gordon, ATTN: ATSO EA 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: DAS/SRD

I USA Chaplain Ctr & Sum, F1 Hamiltonm, ATTN: ATSC-TE-RD I USA Intelligence Ctr & Sch. Ft Huachim.i. ATTN: ATSI- TEM

I USAISCH. F t Eustis, ATTN: Ediic Advisor I USA hItelligence Cl, & Sch. Ft Huacfsuca, ATTN: Library

1 USA War Collegle., Carlisle Barracks, ATTN: Lib 1 CDR, HO Ft Huachuca, ATTN: Tech Ret Div

2 WRAIR, Nemiirpyrti.aiti Div 2 CDR, USA Electronic Pryg Grn, ATTN: STE FP MT-S

1 DLI. SOA. Mintriey 1 HO. TCATA, ATTN: Tech Library

1 USA Concepit Anal Aricy. Bethiesda, ATTN: MOCA-MR 1 HO, TCATA, ATTN: AT CAT-OP-O. Ft Hoodl

I USA Concept Anal Agcy. Bethesda, ATTN: MOCAJF 1 USA Recruiting Cmd, Ft Shetidan. ATTN: USARCPM-P

1 USA Arctic Test Ctt. APO Seattle. ATTN: STEAC-PL-Mi I Senior Army Adv.. USAFAGOD/TAC. Elgin AF Aux FkIt No. 9

1 USA Arctic Test Cir, APO Seattle, ATTN: AMSTE-PL-TS 1 94O. USARPAC. OCSPER. APO SF 96558, ATTN: GPPE -SE

I USA Armament Cmld, Redstone Arsenal, ATTN: ATSK-TEM 1 Stimiori Lib, Acadamy at Healtht Sciences. Ft Sain Houston

1 USA Armament Cmd. Rock Island, ATTN: AMSAR-TDC 1 Marine CutAp hnst.. AT IN: Dean-MCI

I FAA-NAFEC, Atlantic City, ATTN: Library I HO, USMC. Commandant. ATI N: Code MTMT

1 FAA NAFIEC, Atlantic City, ATTN: Human Engr Br 1 HO. USMC. Commandant. ATTN: Code MPI-20 28

1 F AA Aeronautical Ctr, Oklahoma City. ATTN: AAC-44D 2 usCu. Academy. New Lonclon, ATTN: Admnission

2 USA Fid Arty Sch, Ft Sill. ATTN: Library 2 URLG Academy. Now London. ATTN: Library

I USA Armor Sch. Ft Knox, ATTN: Library I USCG Training Ctr, NY. ATTN: CO

1 UISA Arnmor Sr, Ft Knox, ATTN: ATSA-DI.F 1 USCG Training Ctr. NY, ATTN: Educ Svcr.Ofe

I USA Armor Sch, FtI Kniix, ATTN: ATSA OT TP 1 US5CG, Pvychol Res Bi, DC, ATTN - P 1/62

1 USA Armor Sch. Ft Kiiox, ATTN: ATSA-CD-AD I HO Mid-Range Ar. MC ot. Ouaentico. ATTN: P&S Div

51
1 US Marine Corps Liaison Ofc. AM4C. Alexiandria. ATTh- AMCGS -r I Def A Civil frat of Envio Medicine. Canada

I USATRADOC, Ft Monroe, ATTN: ATtlO-ED I AIR CRESS. Kensilton, ATTN: info Sye St

6 USATRADOC. Ft Monroe, ATTN: ATPR AD I Militerveiogaak Tjaneaie Copenhage

I USATRADOC, Ft Monroe, ATTN: ATTS- EA 1 Military Attadue French Emiaesiy. ATTN: Docc Sec

I USA Forces Cmd, Ft McPherson, ATTN: Library 1 Medecin Chat, C.E.R.P A.Afrwlel, ToIon/Pdavall fiance

2 USA Aviation Test Sd. Ft Riurer, ATTN: STEBG- PO 1 Prin Scientific Off, April Hum Engr Asch Div. Ministtiy

I USA Agcy for Aviation Safety, Ft ARucker, ATTN: Library of Defense. New Delhi

I USA Agcy for Aviation Safety. Ft Rucker, ATTN: Educ Advisor 1 Pairs Rich Ofc Library, AKA. Israel Defanse Forces

1 USA Aviation Sch. Ft Rucker. ATTN! PO Drawer 0 1 Ministeris .an Defeneia. OOOflKL Aid Soc..al

1 HOUSA Aviation Sys Cmrl, St Louis, ATTN: AMSAV--ThR Psychologische Zakarr. The Katiaa, Neithearv

2 USA Avialirni is Test Act., Fodwaids AFA. ATTN: SAVTE- -T

I USA Air DOi Sich, Ft Him,. ATTN: ATSA TFM

I USA Aui MWilila Asci & Dev Lai), Molftt Fitt, ATTN: SAVOL AS

I USA Aviatioin Scti, Res Tog Mgt, Ft Rucker, ATTN: ATST-T-RTM

1 USA Aviation Scti, CO, Ft Rucker, ATTN: ATST-D-A

1 HO, DARCOM, Alexandria, ATTN: AMXCO -TI

1 HO. DARCOM, Alexandria. ATTN: CDR

1 US Military Academy. West Point, ATTN: Serialt Unit

I US Military Academy. West Point, ATTN: Ole of Milt Ldnitip

1 US Militaiy Acaidemy. West Point. ATTN: MAOR

IUSA Standaidiiation Gn, UK. FPO NY, ATTN: MASE -C

IOft: of Naval Fisch, A loigton, ATTN: Code 452

3 Ofc of Naval Atch. Arlington, ATTN: Code 458

I Ofc of Naval Rsch, Arlington, ATTN: Code 450

1 Olc of Navail Fisch, Arlington, ATTN: Code 441

1 Naival Aeinspic Med Ret Lab. Pensacola, ATTN: Acost Sch Div

INavif Aernisnc Med AMs Lab, Peitocola. ATTN: Code 1561

INavail Acrostic Med Aes Lab, Pensacola, ATTN: Code LS

I Chiof of Navirs, ATTN: Pets-OR

INay Oceaniograpihic, DC, ATTN: Code 6251, Charts &Tech

1 Center of Naval Anal, ATTN: Doc Ctr

I NavAirSysCom, ATTN: AIR- .5313C

I Nay SuMed. ATTN: 113

1 NavHeIlcopterSubSqua 2. FPO SF 9661

I AFIERIL (PT) Williams AFB

1 AFIIRL (TT) Lowry AFB

1 AFHAL (AS) WPAFB. OH

2 APHAL (DOJZ) Brooks AFB

I AFHRL fDOJN) Lackland AFB

I HOUSAF (INYSO)

I HOUSAF (DPXXA)

1 AFVTG (RD)1 Randolph AFS

3 AMRL (fIEI WPAFB. OH

2 AF Inst of Tchf, WPAFB, Off, ATTN: ENEISI

1 ATC (XPTDI Randoll ih AFC!

I UISAF AertiMrr Lih. Broiiks AFB (SUL 4), ATTN: DOC SEC

I AFOSR (NI. Arlington

I AF Log Cmrl, McCfelIion AFB, ATTN: ALC/DPCR8

1 Air Force Academy, GO, ATTN: Dept of Bel Scn

5t Navers & Dev Ctr, San Diego

2 Navy Marl NeuroPsychiatric Rsch Unit, Son Diego

I Niv Electronic Lai), San Dierto, ATTN: Ret Lab

I Nay TrirqCen, San Diego. ATTN: Code 9000--Lili

1 NwvPostGraScli. Monti -y ATTN: Code SAa

1 NivyPiistGraSch. Monterey. ATTN: Codle 2124

INivTrngEquipCtr, Orlando. ATTN: Tech Lib

1 US Dept of Labor, DC. ATTN: Manpower Admin

1 US Dept of Justice, DC, ATTN: Drug Enforce Admin

1 Nat Orir of Standards, DC. ATTN: Computer Info Section

I Nat Clearing House for MH- Info, Rockville

IDenver Federal Ctr, Lakewood, ATTNV: OLM

12 Defense Documentation Center

4 Dir Psych. Army Hit, Russell Ofcs. Canberra

I Scientific Adtir. Mil Sd, Army 14g. Russell Ofcs. Canberra

1 Mil and Air Attache. Austrian Embassy

I Irnilie do Recherche Des Facteurt. Humaine de la Defense

Nainale, Briiisrfr

2 Catiadiian Joint Staff Washington

1 C/Air Staff. Royal Canadfian AF, ATTN: Pars Std Anal Br ~.3 Cthie. Canadian Def Rich Staff, ATTN: C/CRDS(W)

4 it tih Dni Staff, Ilnitish Embiassy. Washington

52