-

~~~.~~~~~~~~~ --

Tschn~

cal Report 387 ~

ON-LINE AUTHORING AIDS FOR

INSTRUCTIONAL DESIGN

• Russel E. Schulz, Nancy Hibbits, Harold Wagner and Robert J. Seidel

Human Resources Research Organization

EDUCATIONAL TECHNOLOGY AND SIMULATION TECHNICAL AREA Ill ,

.0

13J U. S. Army

Research Institute for the Behavioral and Social Sciences

July 197

~~ 

~ 

- 0 2 ‘

- ~~

Approved for public release; distribution unlimited.

~~~~~~~~~~ ~~~~~~~~~~~ ~

•1

~~_~~~~• — 

— _

~~~~ •_•

~~~~~~~~~~~~~~~~~~~~~~~~~~ : ~~ ~ ,

~~~~~~~~ ~~~~~~ ~~~~~~~~ ________
—I

U. S. ARMY RESEAR CH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating A gency under the Juris diction of the

Deputy Chief of Staff for Personnel

WILLIAM L. HAUSER

JOSEPH ZEIDNER Colone l . U S Army

Technical Director Commander

Research accomplished

under contract to the Department of the Army

Human Resources Research Organization

NOTICES

DISTRISUTI OIII: Primary distribution of th is report hN been made by ARI. Please Iddrass cOrrespondsn~e concerning distribution of reports to: U. S. Army Research Institute for the liheviOrsI ~~

d Social Sc’,nces. ATTN. PERI.P, 5001 Eissnho~~r Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroysd tvhsn it is no longer needed. Please do not return it to

tn U. S. Army Research Institute for the B.havio,al sd Sociol Scienaps.

~~IL Trig finding, in this report are not to be construed as an official Department of the Army position. Li unlass 50 designated by ether authoriad documents,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1

~

1 /

UNCLASSIFIED 

~~ / SICUNITY CLAISIrI CATION oc THIS PAO~ (Wha, Iii . 

~~~~~~~ ~~~~ ~ — _L ~~~~~~~ r A ~~~I#%kI ~ READ INSTRUCTIONS

i~~rvs~ u uu~

..uM~ N I A I IUI~ r

~~ u~~ REVOKE COMPI

~

EflN O FORN

r. NIPONT NUMIZN 5. GOVT ACCSUION $0 I NtCIPitwT ~~ c~~~ At oo i.u~~asN

Technical Report 387 ~

5~~~~!b

~~(aid$ubHiS.) ,

~ ~~~VPt Or NAPONT IPINIOO COVCR(O

____ ‘1 1 ~~ ,~ J-I~

INE AUTHORING AIDS FOR INSTRUCTIONAL~ ~ 

J ~~~na~~Rej~~ 1ESIGN . • 

• ___ 4.7.E~-..ll~,a.as. Y Nu MSAN

J ~~

LiJIAkt~ D- 1.

~~AUTHON(.1 j_ .,_—_ — - -•,~~

— ~~~

. • $L TR.,... , VR OR N? NUMSt~~ I)

V~ ,/“ “ f R u~~e1 E.fSchulz, Nancy. Hibbits, Harol agnerj •.~~~ .

——- —

~~~ IL~~~~~~~~rtJ ~~~~~del T ~~1 DAH~ l9-76-C-~~4 

~

)

5. PIN~~ONNlNO ORGANIZATION HAM S AND ADORS$$ 10. PN

~~GI~

AM j

~~~~~&T.PROJ CT . TASK

Human Resources Research Organization

300 North Washington Street ..,,

~2Q~627l7A76

Alexandria , VA 22314 “ #P’~

AR~~ öre 2887

It CONTROLLING O.PIC( NAM( AND ADONSIS ,~~ ~~~~ . 

~—~=

U.S. A rmy Research Institute for the Behaviorqlj u~~~~~ 19 7

and Social Sciences L NUMSIROF PAGIS,

’

5001 Eisenhower Avenue. Alexandria , VA 22333 198 •

• 15 50 7ON~N5 ASE1SC, sANS S AbDR*$~~

II 

~~DIsiaI Nail CaifrsJlIiig ~~~~~ IS. SICUNITY CL ASS. (. •

Unclassified

Is.. O(C~ As1IPICAtiON/D~’aiIO~~~~ N G

ICHIOUI I

ii bs?mSu?0e eTAVPst ~~~~~~~

Approved (or pU) .. re&~~

,se; distribution unlimited

17. OISTNISUT1ON $TATININT (el ~~

. aAefr•ci ail ,.d ii 5t.rA 20. U ~~

U.rs,i I frau

tS. $Ue tIMtNTAWr NOTES

This project was jointly funded by the U.S. Army Research Institute and the

Defense Advanced Research Projects Agency . Dr. Leon Nawrocki (ARI) and

Dr. Harold F. O’Neil, Jr. (DARPA ) served as Contracting Officer ’s Technical

Representative and agency point of contact respectively.

IS. KEY WORDS (CWffiivS en ,.au.. idd •~ ia a. v aid D4 itH~

. S~ stock naub.t)

Computer-Administered Instruction system engineering

Computer-Assisted Instruction IPISD

CAl test development

author aids lesson development

authoring aids

AUS1KACT (

~~~ I~~~~ en eocs,a sI

~~ N .. aay d fduNH~ S~’ block mNOev)

“The purpose of the research was to conduct a development and feasibility

demonstration of on-line , query-based authoring aids for selected blocks of

the Interservice Procedures for Instructiona l Systems Development (IPISD )

model. The model indicates techniques and procedures for the development and

conduct of interservice training . The blocks selected from the 1PISD model

for this demonstration were Block 11.2, Develop Tests, and Block 111.4, Develop Instruction. The research was designed to provide authoring aids (“How

to do it” guidance) for developmental activities identified in the blocks.

~~ ~~

:51

~ W3 tDlTlON OF I NOV SIll OSSOI.tTt UWCLASSIFIED

SICURSTY cL.AsaI,IcAr,os, OP ThIS PASS (

~~SiI 0*. tiuiorsd)

i

,

1 J( (

~ ~~~~

‘ / (

~ ~~ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Tschnlc

~ Riport 387

ON- LINE AUTHORING AIDS FOR

INSTRUCTIONAL DESIGN

Russet E. Schulz, Nancy Hibbits, Harold Wagner and Robert J. Seidel

Human Resources. Research Organization

L. H. Nawrocki, Team Chief

Submitted by: Michael Kaplan, Acting Chief

EDUCATIONAL TECHNOLOGY AND SIMULATION TECHNICAL AREA

Approvid by:

Frank J. Harr is, Acting Dlrsctor

ORGANIZATIONS ANO SYSTEMS

RESEARCH LASORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

July 1979

Army Proj.ct Numb r Educational and Training Tschnology

2Q702717A754 Contract 1S76.C.0041

• Approved fo~ public ralease ; distribution unlimited.

_____ - .• 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ft. .c- # ‘ J ~~ ~~~~~ ____
. -- -- .

~

ARI Research Reports and Technical Reports are intended for sponsors of R&D tasks and for other research and military agencies. Any findings ready for implementation at the time of publication are presented in the last part of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military agencies by briefing or Disposition Form.

I

Ju. 

~

.ii’icatiOfl_._—___~

__~I

k~

-:

~~~~~i. ‘. Ll/ Ol

L)ist i

~j’C~~ .Ld
FOREWORD

The Computer-Based Educational Technology Team of the U.S. Army

Research Institute for the Behavioral and Social Sciences (ARI) performs

research ançl development in eares of educational technology that apply to

military training . Of interest are computer—based instructional delivery

systems that focus on developing the accompanying instructional courseware in the most efficient and cost-effective manner possible. Development and implementation of such systems will help solve the problem of

training individuals to produce good courseware in a reasonable time , at

an acceptable cost.

This Technical Report describes a development and feasibility demonstration of two author aids designed to assist individuals in developing

tests and instruction. The project was funded jointly by ARI and the

Defense Advanced Research Projects Agency (DARPA) . To accomplish this

research, ARI ’s resources were augmented by contract DARC l9-76-C—0041 with

the Human Resources Research Organization, an organization selected as having unique capabilities for research and development in this area.

Personnel at the U.S. Army Engineer School (USAES) , Fort Belvoir , Va.,

provided guidance and assistance throughout the project: Dr. Everett

Rompf , Mr. Jack Ainsworth, LTC Ernest Larson, MAJ John Harvey , MAJ Ramile R.

Rebello, 1LT D. Bunn, SFC Alton J. Blanchard , and SFC Leon M. Loomis. In

add ition , Dr. James Kraatz, PLATO Services Organization , Computer—Based

Educational Research Laboratory , University of Illinois at Urbana , and

Ms. Beverly Hunter and Mr. Richard Rosenblatt of HumRRO also contributed

to the research effort.

The entire research work unit area is responsive to the requirements

of TDT&E Project 2Q7627 17A764 , “Educational and Training Technology ,” the

1977 ARI Work Program .

hnical Director

V

• ~~~~~~ . - -

~~~~~~~~~~~W)J~~~~~~~~~ . _____ ____ 

~~~~~~~ — -—~~~~~~--~~~~--
~~~~ •.— 

ON-LINE AUTHORING AIDS FOR INSTRUCTIONAL DESIGN

BRIEF _______________________________________

Requirement:

The purpose was to examine the feasibility of providing “how to do

it” guidance (authoring aids) for the instructional design and development

tasks identified by the Interservice Procedures for Instructiona l Systems

Development (IPISD) model. The usefulness of the IPISD model depends on

authoring aids which enable training personnel to translate IPISD procedures into instructional products. The authoring aids developed by this

research should be useful for computer-based and off-line instruction and

be generalizable to differing subject matter areas.

Procedure:

Au thoring aids were constructed , implemented , and tested . The authoring aids were developed to be used on the PLATO IV Computer-Assisted Instruction (CAl) system. The first step produced flowcharts which detailed

the steps of the IPISD Blocks 11.2 (Develop Tests) and 111.4 (Develop Instruc tion). On—line author aids as well as off-line versions were produped

to assist the author in preparing materials for CAl and non-CM delivery

of instruction.

Three levels of evaluation were conducted . An informal evaluation

on existing IPISD materials was performed , and a formative evaluation on

the newly developed authoring aids. Finally, the in~;tructiona l materials

were evaluated by military authors and administered to U.S. Army Engineer

School trainees.

Findings:

The feasibility of on-line aids for implementing IPISD Blocks 11.2

• (Develop Tests) and 111.4 (Develop Instruction) was demonstrated through

the evaluations. User acceptance of the aids was high , and the time required for development of test and lesson material has been significantly

reduced .

Utilization of Findings:

Based on these findings , the development of authorinq aids for additional blocks of the IPISD model was initiated.

vi

~~~~~~~~~~~~~~

—
-

~~~~~~~~~~~ -- ~~~~ - -~~—~~~~~ 

ON-LINE AUTHORING AIDS FOR INSTRUCTIONAL DESIGN

CONTENTS

• - Page

CHAPTER I: BACKGROUND 1

Systems Approach to Training 1

Problems of Implementing Instructional System

Development Models 3

The Need for Author Aids 6

Evaluation 11

CHAPTER II : PURPOSE 17

CHAPTER III : APPROACH 18

CHAPTER IV : PROJECT ACTIVITIE S AND ACCOMPLISHMENT S 21

Task 1. Analysis and Determination of Required Author

Aid Elements 21

Task 2. Conversion of Flowcharts to Interactive Program - . . . 27

Task 3. Evaluation 36

Task 4. Revisions 42

CHAPTER V : RECOMMENDATIONS 44

Modification of Current Author Aids 44

Addi tional Author Aids for IPISD Blocks 11.2 and 111.4 45 • Conversion of INQUIRY Authors Aids to a CAl System Other

than PLATO IV 45

Development o’ ‘

~

u thor Aids for IPISD Blocks Other than

II.2 and II~ 46

F Author Chara~ ~stics 46

REFERENC ES 47

List of Figures

Figure 1. Detailed Breakdown of Activities to be Performd e in

Each Phase 4

2. Flowchart of Block 11.2: Develop Tests 22

3. Flowchart of Block 111.4: Develop Instruction 23

t.i

- 

~

— —~~~~~~~—.--.

--•—- 

—

~~ l,JE ~~~~~~~~~~~~~~~~~~~ -••----. -.-“.

-- - •-- • - •-- 

~~•

.-‘~ .-.- ‘•—~~~• .

•.•.

•• --S.- A
IT ‘te 1

4. Illustrative Flowchart Exp ansion of IPISD Block 2.6 • 24

5. Illustrative Flowchart Expansion of IPISD Block 2.6.1 25

Lis t of Tables

1. Characteristics of Practice Question Author Aid

(Multiple Choice) 31

2. Characteristics of Practice Question Author Aid

(True—False) 32

3. Characteristics of Practice Question Author Aid

(Constructed Response) 33

4. ENCOA Lessons Authored on PLATO Using INQUIRY Aids 38

APPENDICES

A. Readability Level Formul a

B. Flowcharts of IPISD Block 11.2 (Develop Tests) and Block 111.4

(Develop Instruction)

C. Detailed Flowcharts of Author Aids for IPISD Block 11.2 and

Block 111.4

viii

— ~~~~~~~~~~ — 
Cha pter I

BACKGROUND

The purpose of this chapter is to provide an overview of the stateof—the—art related to instructional systems design and evaluation . Major

subjects addressed in this chapter are :

• Systems Approach to Training

• Problems of Implementing Instructional System Development Models

• The Need for Author Aids

• Approaches and Techniques for Evaluating Instruction

SYSTEMS APPROACH TO TRAINING

A revolution in the technology of training within the militar y and

industry began when the systems concept was applied to the development and

conduc t of training . Even now , aft~. 25 years , the full potential of

applying the systems approach to improve the effectiveness of instruction ,

improve on—the—job performance and lower the cost of training has no t been

realized . Even so, results from applications of the systems approach in

terms of improved instruction , increased relevance in what is taught , and

lowered costs have been so dramatic that , at present , the systems approach

to training has permeated civilian training (and education as well)--

• training in business and industry, training in the military services , and

training in other agencies of the federal government .

Many different names, terms , and variations are or have been used for

the systems approach to training . Some of the names are : “systems

engineering of training,” “curriculum engineering ,” “systems approach to

training ,” “instructional systems development ,” “training situation analysis .”

“modern instruct ional technology .” Even among those using a particular

name , there are many divergences in definitions , part iculars of techn ique

and procedures , and effectivenes’. with which the systems approach is

applied . Despite such variations , the comon thrust and orientation of

these applications is pre—eminent , .~

specia1lv as they contrast with tradit ional approaches to train ing.

The essence oi the systems approach to training rests in ident i f yi n g

explicit end states tha t are to be achieved through training and in defining

sets of orderly , objective , and explicit procedures to do that which i

~ necessary to achieve these end states in the most comprehensive , reliable ,

effective and efficient manner.

The systems approach defines a process which ~~~~~~~~~~~~~ :

~~~~“: ~~~~ ‘ ~~~~

that is ultimately to be performed and ~~~~~ t h ~ ~~~~~~~~~~~ who is to learn
— ~~~~~~~~~~~~~~ T~

to perform that job . Traditional approaches , by contrast , focus upon ~‘

. ‘:~‘

~‘:r f ~”:.~ ~~~~~~ •.

~~~• 

catalog ” ~~~~ ~~~~~~~~~~~ that tend to be more of the “school

the student 

variety and are , generally , onl y appro ximately pertinen t to what will be doing later .

mation 

In addition , most traditional approaches place the burden for infor- transfer upon the students rather than on the instructional materials. Whether the instruction is rapid or slow , complex or dull , .~nden 9~,t 5 ~~~ feasible 

t By contrast , in the systems approach , it is

the instru

to 

ctio

eng

fla 

ineer flexibility into the instruction and , so, to adapt ~ Special 

stem to individual differences among the students. consideration is given to:

• Evaluation of the needs of each individual student . • The nature of instructional conten t to be imparted . • The lnstructjo~

ia1 decision rules that mediate between student needs and instructional content .

The systems approach is just what the name implies: a systematic

will 

process 

be 

for specifying the desired products of training and selecting what

be, and 

taught , how it will be taught , what the presentation mechanism will

on 

evaluating the effects of each phase of the process. It focuses student performance as a determinant of content. Its proper application

been 

can hardly 

given 

fail to improve instruction where only incidental attention has

unconventional 

to these functions. Thus, in the systems approach setting ,

(usually 

clusters of instructional material may be used for a uniform

the same 

small) group of students , each of whom is being prepared to perform

‘

job. Major efficiency is achieved by directing instruction :

thereby assuring 

~ to the student and to what the student will use on the job ,

adapting 

relevance and efficiency, precluding oversights , and instruction to the individual .

During the past 20 years, many attempts have been made to cod~ definitive fv a

in this 

technology for the systems approach to training. Early efforts area included those by HumRRO on behalf of the U .S. Army early l9SOs in the

mid—l950s. The 

and 

Hu

the 

mRRc) 

development of the tJSAF personnel subsystem approach In the

starts with the development 

model, for example [1,2,3 ,4], is a seven—step process which

that , a job 

of a man—machine system analysis model. From

of 

model is developed which then leads to both the specification

the 

knowledge and skills required for adequate performance of the job, and

ability 

proficiency test development. The proficiency test measures the

job 

of the student to perform actual job tasks, thereby assessing the

kn owled

proficiency of the student . From the specification of skills and

those 

ge, one may determine the instructional objectives , which Is to say , specific requirements for an instructional program . 0nc~ the instructional objectives have been determined , then a training program can he constructed. The seventh step is the evaluation of the training program.

— 

One of the m ore notable of recent systems approach efforts is the I

[5

nterservice 

,6,7

Procedures for Instructional Systems Development (IPISD) Model ,8,91. This model was prepared by the Center for Educat ional

•~~ — —~~~~~~~~~•. - ,

- - 

~~~~~~~ .- -~~~~ ~~ - : 

-- •

- - •- - - - ~~

.- ~~. . - . • ~• ~ 

~~~~~ —

•• —~~..
Technology at Florida State University under contract with the Inter—

service Coninittee for Instructional Systems and Development , involving

the Army , Navy , Air Force and the Marine Corps. The IPISD contains

standardized rationale , terminology , and basic concepts of instructional

systems . These have evolved by developing and recording the results of

efforts in theoretical and guidance materials required for actually performing instructional systems development. Prior to this effort , the Air

Force had undertaken a large activity to develop, define , and record a

definitive technology for instructional systems development [10], and the

Army had embarked on an ambitious five—year program to systems engineer

all of its training courses [11]. Some of the development of the systems

approach to training has gone on outside the Services , particularly in

industry [12 ,13,14]. In addition , Mager [15] and others such as Glaser

[16], Ainmerman [17], Krathwohl [18], Bloom [19], Melching [20], Gagne [21],

Esbensen [22], Bond [23], and Butts [24], to mention only a few, have

made significan t contributions to systems approach models through their

research in the development of behavioral objectives and sequencing of

instruction . In the Navy , much of the work dealing with the systems

approach has been carried out by USNTEC with reference to simulation

(e.g., [25,26]). In addition , the Navy has initiated several major

efforts related to training systems design of a more general nature [27,

28 ,29].

The IPISD model shows promise as a useful tool in instructional

system development activities and is presently undergoing preliminary field

evaluation . The model consists of five major phases which can be conceived

under the ADDIC rubric :

A analyze

D design

D develop

I implement

C control

Figure 1 is a breakdown of the five phases into more detailed activities

(blocks) comprising each phase.

PROBLEMS OF IMPLEMENTING INSTRUCTIONAL SYSTEM DEVELOPMENT MODELS

Early applicat ions of the systems approach to training were accomplished

by expert training developers . In the l960s, the possibility of having

laymen use these models to achieve the success of the experts , by imitating

their actions , was explored . The use of an ISD manual by existing military

personnel with little or no experience in training program design may cost

a fraction of the cost of hiring or contracting experts to do the development .

Even so, the cost/effectiveness of the model will still depend on the

effectiveness of the model , or tools , in enabling laymen to produce

effective instruction .

3

____________ - 

~~~~~~~~~~~ ~ :.. - 

~~~ -. 

.—— -.-——.—
ThE BLOCKS IN EACH PHASE ARE:

— 

~~~~ :LYZE 

~~~F~~~CTI ONS MEASURES 

~~~ANALYZE ~~~SELECT 

1

k ~ 1 12 

~~~DESCRIBE kDET ERM IN E — ~~~DEVE LOP ~~~I)EVELOP 

~~~ENTRY V SEOUENCE & OBJECTIVES TE STS REHAVIOR STRUCTURE

till

1 SPECIFY

— ~~~~~~~~~~~~ ~~~~~~ ~~~~X~~~~~~

ELECT 

~~~~INSrHUCTION ~~~INSTRUCTION

ACTIVITIES DELIVERY

SYSTEM

i~~ 1

~~ IMPLEMENT

— INST RUCTIONAL CONI)UC I

MANAGEMENT INS1RUCTION

PLAN

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Detailed Breakdown of Activities to be Performed in Each Phase (TRADO C PAN 350—30, Executive Summary and Model, August 1975)

Figure 1

4

~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
In the past few years , ..problema with attempts at implementinR ISD

models by laymen have surfaced . Montmerlo [30] conducted a comprehensive

review of ISD state—of—art and problems of implementation . His conclusion

was th at “available ISD—type methodologies will not allow the layman to be

as successful as the expert. ” (The particular IPISD manuals with which we

are concerned , however , were not a part of this review.)

Montmerlo cites the paper by Ricketson , Schulz and Wright [311 as the

— 

“most significant article concerning the problems of ISO ,” because it

represents “the only empirical evaluation of an ISP—typ e methodology .”

Ricketson , Schulz , and Wright studied the CONARC R~G 350—100—1 and its

implementation by Army instructional developers. Although IPISD is intended

to be a considerable improvement over 350—100—1 , many of the same problems

do apply . For example , Ricketson , et al., found that “High rates of

F personnel turnover within some curriculum development groups have resulted

in a general reduc t ion of systems engineering program productivity .”

Assuming this to be a continu ing reality in military instructional development , the need becomes evident t o p rovide autho ring aids whi ch can be

quickly learned by new developers . The study found , among other things .

that developers tended to develop training programs that employed the

same techn iques with which they had been taught , since they did not have

the ability to assess other training techniques and equipment.

IPISO and other ISO manuals are intended to have general applicabilit y .

However , it has been clearly recognized by many experts that the same

methodologies cannot be applied to the universe of training problems . The

literature on task analysis, for examp le , contains a number of articles on

the impossibility of using the same method for all tasks (32 ,33,34).

The IPISD Executive Summary 6, Model [91 also emphasizes the need for

different methodologies in the statement, “The extent that one used the

interview method , the observation method , or the occupational survey method

depends on the nature of the job being analyzed , the job data already

available , and the availability of analyses resources.” [9j

Wh ile the “what to do” may remain relatively constant across training

problems , the “how to do it ” may vary enormously. This again is why the

instructional s stems designer needs a wealth of aids to refer to in

dealing with a specific training problem in a specific subject area.

While the IPISD manuals do provide far more references to the literature

than previous manuals did , they do not provide specific “how to do it”

guidance for specific design and development tasks . [9 , p. 124]

Another major problem area of ISO implementation in general is the

management of the lustructional development process. For example , when a

change is made f 11 the conditions of a part icular test item , this has many

ramifications backward and forward in the ISP process . The management of

these changes, including the communicat ion among various members of the

IPI SO team , is complex and usually requires some management aids.

Discussions with training staff at Ft. Belvoir provided us with practical

evidence that It is in the area of management of the ISO process that

major problems cont inue to he found .

a 5

~~ --
Other problems of using IPISI) relate to the background skills of the

team leader and of the members of the development team . This is pointed

out in the IPISD Executive Summary [9 1, in the Montmerlo study [30), and

in many othe r sources [35 ,36,37].

The IPISD model is advanced over other systems approach models in

providing guidance to the training manager. However , the IPISD manuals

are not presently intended to provide specific procedures for every instructional situation that can be encountered . Some situations are now covered

only by the general principles underlying the Model. If IPISD is to have

a “fair chance ” of be ing accepted by training managers, it is essential

that tools and author aids be developed that will permit training personnel

to readily and effectively translate reconuuended IPISD procedures into

meani ngful instructional products. This rationale forms the basis for

the initiation of the present project.

THE NEED FOR AUTHOR AIDS

Author aids are any products used in accomp lishing one or more steps

of the IP ISD p rocedures. Under this definition , thousa nds of guidebooks ,

research studies , texts , p rofessional articles, and tech nical reports

could be considered as aids. If an instruc t iona l system desi gner we r &s

fa mil Ear with the full range of aids available , be wou ld in fact he an

expert in the field of IPISD and therefore not be a subject of our concern .

The problem for the author (any member of the development team) , is to

know what aids to use when , to know they exist , to have access to t hem in

a t imely way , and to have some facility and judgment in their application .

In this effort two of the blocks of the IPISD model vere selected

for eval uation. Block 11.2, Develop Tests, and Block 111.4, Develop

Instruction . The first block , 11.2, was required in the RFQ . Our

choice of Block 111.4 is based on a number of mutually supportive

genetal and specific reasons. Many leading instructiona.l technologists

and designers have concluded from their experience that the Development

Phase of instructional preparation is significant component of the

systematic approach to producing qual itv instruction . It is expensive , t ime consuming, critical , and requ ires spec ialized capabilities. Van Pelt

and Rich [38), for example , speak from experience in the Army training

environmcnt “There is no question that much t ime is wasted by writers

casting about for a reasonable set of guidelines to follow that will result

in 1es~ons requiring a minimum of editing and revision .” For the Navy , Aaga r d and Braby [39) ha ve emphasized the need for an algorithmic approach

to translate basic learning events into instructton a llv meaningful task

categories “ . . . in a manner that emphasizes the flow of events and the

combining and sequencing of learning gu ide l ines in the design of a training

program. . .“ (p. 7). In the civilian sector , Lipson [40] has stressed the

need for “

. . . increased investment in development of instruct iona l

materials. ” The “homemade ” variety doesn ’t have “

. . . the qual it ies of

craftsmanship, artistry , nor the proper incorporation of what is known

about effective instructional design to he widely used .” Industrial

developers øf CAL (Simonsen and Renshaw [4 11) have stressed that “

.

6
the cost of lesson preparation actually more than doubles the projected

cost of an hour of CAl and cannot be ignored.” Recent analyses (formal

and informal) by the Training and Evaluation Group (TAEG)1 support this

assertion for the general case of individualized instruction that involves

a systematic approach to the development of materials.

On a specific and practical basis we have learned from the Curriculum

Development Personnel at the USAES that they have experienced the most

difficulty in using IPISD with Block 111.4, Develop Instruction. A

comparison of test construction to general development of instruction

suggests that in develop ing tests (IPISD Block 11.2), authors need help

in performing such activities as the following:

• Developing test items that actually test the Terminal Learning

Objectives (TLO), Learning Objectives (LO), and Lesson Steps (LS).

• Constructing hands—on scorable units of Skill Qualification

Tests (SQTs).

• Developing scoring procedures .

• Writing test items that will help identify bad instruction .

• Devising test items that will support remediation strategies.

• Generating test items and alternative forms of items.

• Managing the test development process, e.g., have all TLOs been

tested?

• Obtaining reliability measures on test items.

• Determining the validity of test items.

In Block 111.4, Develop Instruction , a variety of aids are needed .

Some of the activities and decisions which require support include :

• Ensuring reading level is appropriate to the audience.

• Deciding what kind of drill and practice is needed , and how much ,

for a given task.

• Deciding how the student will be able to obtain additional help.

• Determining the nature , frequency, and type of feedback to provide

to the student during the instruc t ion.

‘Personnel Commun ication , Dr. Richard Braby of the Training and Evaluat ion

Group (TAEG).

7

&&

A
-

~ 

- - - -

~~~

As an overriding concern , aids are needed to help reduce the t ime it takes

to develop quality instruction and tests, and to make the development

process as efficient as possible.

A wide variety of aids exist. Recently Logan L421 completed a survey

of existing tools/procedures which could be used by instructional developers

in conjunction with the IPISD. The results of this survey indicated that

aids exist for a number of IPISD components. Unfortunately, a considerable

technical background and level of expertise is required for their use .

Thus , even when aids are available , there remains the problem of using

them without imposing an undue burden on the author.

The existence of these aids testifies to the recognition that they are

needed. However , in the case of author aids it is not the variety and

quantity in the universe that counts. What matters is that the appropriate

aid be available , easy to use and accessible at the ri ght t ime and place.

The majority of existing materials are more of the “what to do” nature

than the “how to do it” variety. There are a number of general resource

guidelines already available to aid systematic development of instruction .

The following discussion of these resources is illustrative and not intended

to be exhaustive of the field . Because they are general , many of these

handbooks , manuals, etc., are difficult to categorize in terms of specific

single phases of IPISD. However , many examples can be categorized as giving

guidance p~~marily in Analysis , Design, Development , Implementation , or

Control . For instance , Harless [43] emphasizes the importance of “front—

end analysis” to solve human performance problems . For his target aud ience , primarily management personnel , he recommends defining the problem , investi gat ing its characteristics and studying alternative solutions and their

costs before making any decisions. McKnight ’s work [44] on tailoring

military training by systems and job analysis provides another useful

resource for the Analysis Phase. The recently produced Marine Corps

training guide for task analysis [45] is another such aid .

Hager ’s techniques [46] are classic as aids to stiuplating precise

instruc t ional objectives and thus fit into the Design phase of IPISD.

The TAEC approach to categorizing instructional tasks according to

particular learning algorithms [39] can in some instances be useful for

design purposes.

The brief book by Pipe [47] for beginners falls between Design and

Development. It describes the systematic procedures necessary to begin

writing programs. He neglects the details such as frame writing or

program format , concentrating on the “practical” issues of steps that

precede writing, testing programs, and avoiding pitfalls. Drumbeller ’s

handbook [48 1 ta even broader. In his guide of curriculum design for

individualized instruction he has highlighted the need for materials to

“have built—in comprehensiveness.” His systems approach provides guidelines in the form of a detailed model for curriculum design . It includes

defining objectives , analyzing sub—objectives and integrating them into

the learning experiences. Wong and Raulerson ’s [49] gu ide spans all the

steps of IPISD in brief but they spend more t ime and give more detailed

8

- ~~- _______ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :

~~~~~:~

i

~~~- 

~= .,. — -- 

-
aid for Design and Development. In the latter, they provide guidance for

media selection based on variations in stimulus requirements of the learning -

tasks. Their selection guide is useful, and while not apparently theoretically derived from learning principles as is Briggs ’ approach [50], It

may be more helpful to the layman . As with most such selection guides ,

the user still must choose from two or three potentially equivalent alternatives . Wong and Raulerson ’s model [49) is based in general on a strict

sequential view of learning .

A number of published texts are available as aids for various parts

of the Development Phase of IPISD. Markle [513 has discussed the construction, format, and sequencing of the frames in her texts, both of which are

programined.Sperry [52] with its Instructional Program Develqpment Workbook,

has developed and used comprehensive plans for a workshop (along with a

workbook) on instructional program development. Its structure follows the

philosophy of mathetics that includes demonstration , prompt , and release

exercises. Espech and Williams [53) In their Handbook for authors of

programmed instruction describe the process of constructing programs with

major emphasis on editing, testing , and analysis. The finished product is

then assured of being a “packaged change of behavior.” Hawkridge , Campeau ,

and Trickett [543 provide a rather unique resource to help the evaluator

prepare his reports. While it is written towards a school system audience ,

its clear , concise approach should make it usable in a military context .

More recently , and still under development , llillelsohn is employing a

programmed instruction approach to creating and managing computer—based

learning materials [55). ThIs effort is expected to provide an additional

means for implementing several components of the IPISD model.

The most relevant example for the Army as an aid to 1ti~

plementat ion of

instruction is the military training manual, FM 21—6 [561. Its format is

readable and comprehensive, replete with examples for the Instructors.

Not a great deal of useful materials is available for the Control

phase per se. However , Cogan ’s case study approach [571 Is illustrative

of such aids which could be a useful resource to training managers.

Having noted some of the available resources , certain ~~~~~~~~~~~~~ are in

order. Rather than providing actual help in performing the authoring work ,

or even detailed “how to do it” guidance , most of the existing author

aids may serve simply to reinforce or broaden the guidance provided in

IPISD manuals. Aagard and Braby [39 ] very carefully note the practical

limitations of the use for their algorithms and guidelines to general

approaches. “The task categories and related guidelines are not at a

level that will accommodate any training setting .” Briggs’ handbook [50 ]

may also serve such a broad guidance function . Care should be exercised

in the selection of aids to be integrated with IPISD , so that

IPISD authors are not confronted with a confusion of different yet

similar models, sets of jargon , procedures or forms. Existing general

manuals differ from one another in that they:

9
• Include different steps or different name s for steps.

• Include different methods of accomplishing each step .

• ProvIde dIfferent levels of specificity in the detail included

under each step .

• Provide different formats for reporting the work accomplished

under each step.

Thus, to provide the most efficient aids to IPISD developers, the guidance

found in some of these manuals and guidebooks needs to be translated and

integrated into the IPISD framework , rather than referred to in its source

form.

One example of a useful “how to do it” guide is the Guidebook for

Developing Criterion-Referenced Tests [58]. To make the guidance in this

book readily accessible and useful to the IPISD author , the Ideas need to

be integrated into the IPISD framework. Another Important aid in the

testing area is the recent Manual for Developin~ Skill Qualification [59 ]

and the Procedures for Validating Skill Qualification Tests [60].

In some cases, the author needs actual assistance in performing an

activity, rather t han simply how to do it information. For example ,

automated readability indices can take some of the workload from the author.

Some aids are specific to a particular method or theory of instruction .

The layman author needs some basis for usIng that particular approach or

method , and needs to know that It is one of several alternatives. Thus,

for example, Markle’s [513 texts might be relevant and useful to an author

who has decided to follow her particular approach to programmed instruction .

Alternatively, Sperry ’s comprehensive plans f or a workshop and workbook

[52 ] on instructional program development as noted earlier follows the

philosophy of mathetics.

In the more specialized areas of computer—based instructional development , an array of automated aids have been produced . The TICCIT project

[61], for example , uses highly structured forms for text preparation , and

hIg hly proceduralized production techniques for authoring teams. Similarly,

Project IMPACT [62—68 ] developed standard formatting aids for authors,

sophisticated techniques for logic and text separation , instructional

management , etc. Again , these aids are highly specific to a part icular

instructional strategy or method and (particularly in the case of TICCIT)

frequently constrained by systesi hardware and software constraints.

The array of automated aids for authors of computer—based materials

includes: special programming languages, test item generators, scoring

algor ithms , recordkeeping facilities, objectives data banks, text editors ,

graphics aids , student response analysis algorithms , data analysis routines ,

statistical subroutine packages .

10

-

~

— 

--~~~~~

.-

~~~

— j
Among the major problems for users of computer—administered instruction

(CAl ) Is the high cost of developing quality instructional material . The

problem is exacerbated by the fact that new CAl programmers frequently

require several months of training before they are able to produce quality

instruction within an acceptable time frame . These individuals view their

role as that of subject—matter experts and educators rather than programmers.

To satisfy this audience , authoring aids are needed that will permit

educators to rapidly develop quality instruc tion without extensive CAt

language training.

As a first step in meeting this need , and a forerunner of the present

research , HumRRO completed the development of sets of author aids called

MONIFORMS [6:1,68]. These aids assist an author in generating question—and—

answer type practice items. The author interacts with the computer which

leads the author step—by—step in the creation of items , answers, feedback ,

remediation , etc. The resulting practice items can then become part of

either an on—line or off—line course of instruction . The computer dialog

can also be used off—line in the form of a checklist for the author.

MONIFORMS were developed specif ically f or the PLATO IV system TUTOR

language. However, the concept of programming templates which permit —

authors with limited programming experience to create test and lesson

materials has wider application . While MONIF0R1~ are a valuable first step ,

there still existed a need for more advanced author aids. Preliminary study

at }IumRRO indicated the feasibility of developing author aids which through — interrogation of the course author would automatically convert lesson content

and structure into executable program code . Therefore, the author would

require no previous programming experience and thus make the aids much -

easier to use. This concept formed the basis of the approach for the current

effort.

In summary , with respect to authoring aids:

1. There exists a very rich array of a wide variety of materials ,

handbooks, guides, and automated aids which could serve to help in the

IPISD process.

2. There exists a very real need for these aids.

3. The selection of specific assistance to be integrated into the

framework of IPISD is a task that is yet to be completed .

EVALUATION

In the previous sections , we have discussed the need for specific

aids or tools which can assist authors to apply the IPISD process in the

preparation of instructIon . In developing and tailoring aids for authors ,

we are in effect developing instruction——for authors. Once any instructional product is under development , a continuous process of evaluation and

11

4 .

~~~~~~~~~~~~~~~~~~~~~_tI~~~~~~~ ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ‘-.-

~~~~~
r ________________

revision needs to take place . This evaluative process should also be

applied to the development of author aids——an approach we have taken in

the present research effort. The following discussion of evaluation provides

background and rationale for our approach to evaluation.

Evaluation Defined

Evaluation is the process of delineating, obtaining , and providing

useful information in judging decision alternatives [

~ 9]. It is an action—

related process which has as its major characteristic the determination of

value, worth, or merit. The evaluation process is conceived as continuing

rather than as having a discrete beginning or ending. Evaluation should

facilitate the continuous improvement of a program. It should stimulate,

nor stifle, Instructional development .

Evaluation procedures may be categorized as formative or sumniative

[

~~J . Formative evaluation is that process which validates instruction

during on—going initial program development . The results of this evaluation are acted upon immediately in program modification. In other words,

the practice of conducting tryouts of draft materials during program

development , followed by measures which provide an assessment of the

materials which lead to their revision Is referred to as formative eval—

uation . Formative evaluation is performed for the purpose of diagnosing

and correcting the weaknesses of a program.

Anyone involved in the revision of instruction may be engaged in

formative evaluation (in the loosest sense of the term). What is presumably

being done is being done because the developer or someone else has judged

the existing course as unsatisfactory . As new materials are developed , they

are constantly being “evaluated” as better or worse than that which already

exists. However , it is a formal program of formative evaluation employing

various assessment techniques which Is the keystone of the IPISD process and

which provides the link between course content and course improvement. By

explicitly stating objectives and criteria, one can properly determine if

the program is achieving its goals, or if goals are to be modified .

Summative evaluation is performed for the purpose of assessing a

fully implemented training program with respect to its ability to produce

graduates who can perform to minimum standards of performance. Also, the

evaluation can determine whether or not efficient and effective use was

made of educational resources. Suinniative evaluation should occur after

instructional development , improvement , and stabilization of operational

and administrative activities. This may vary from one training program

to another. In some cases, training objectives may not be measurable

at the desired time of evaluation because they are either too costly to

measure or are long—term objectives.

Results of a sutmuative evaluation , while of interest to the developer , are of primary concern to those who will decide whether or not a program

is to be continued or adopted . Summative evaluation , therefore , prov ides

the basis for policy decisions that do not necessarily concern revision

of the program or product (71 ,72 1.

12
The differences between formative and summative evaluations are mainly

in their purposes and the timing of their application. Formative evaluation

is continuous in nature and serves to refine a given program through an

iterative feedback process; summative evaluation produces final judgments —

concerning the degree to which program objectives and goals have been

attained . The information obtained from a sununative evaluation allows the

user to judge whether a program meets his needs, whether it shou ld be wid ely

disseminated , and if alternatives exist , which are to be preferred .

The discussion which follows will concentrate primarily upon tormative

evaluation as the purpose of the effort reported herein was to evaluate —

author aids in their initial development stages.

Eva luation Models

The formal distinction between formative and sununative evaluation is

attributed to Scriven [70]. However , the purposes for which such evaluation

data are used have been discussed for many years in the training and

education lIterature. Cronbach [73] stated that “the greater service

evaluation can perform is to identify aspects of the course where revision

is desirable. ” Early models of the systems approach to training development

contain quality control components which emphasize the need for feedback

for program improvement. Smith [14 1 described the purpose of a quality

control system , “. . . a means for continuous monitoring of the quality

of the graduates and for improving the training when it is deficient .”

Quality control procedures are needed both at the school and in the

field . Information from both locations must be “fed back” so that the

instructional program can be appropriately adjusted . Schools require two

types of feedback information . The first type assesses the ability of a

course graduate to perform acceptably those tasks which the instructional

program claims to teach. This type of information assesses the ability of

the instructional program to teach well whatever it is that it claims to

teach . In most instances , this assessment can be made at the school.

A second type of feedback information deals with the discrepancies

between the course graduate performances and field requirements. “Relevancy

control” information assesses whether or not the instructional program

teac h es the appropriate subjects or tasks, and whether or not the student

can transfer these capabilities to the field. Also , this feedback should

provide information dealing with changing field requirements and with more

precise descriptions of job activities.

Baker and Alkin (75] point out that the evaluative process was an

integral part of programmed Instruct ion development which antedated the

surge of interest in forma t ive evaluation during the past decade.

Recent models of the formative evaluative process include those of

Stake ( 63, Scriven (77 ], Stuff lebeam , et al. ~693, Sanders and Cunningham

(78], and Rippey [79]. Scriven [77] feels that it is best if formative

evaluation is performed by someone other than the developer. Scriven

calls his approach “goal free evaluation” which calls for the evaluator

13

I; ____

-
-- --—- - - ~~~-

~~~ - - ~~~~~~~~~- - --~~~~~~~~ -~~~~~ -~~~~-- - — - - - -

to assess the actual effects of the program . The evaluator operates without knowledge of the purposes , goals, or objectives of the program

developers . Another model is described by Stake [76] as “responsive

ev aluation ” which calls for the evaluator to be external to an instructional

development activity, and therefore to have a certain independence and

objectivity that is presumed not to be present in an internal evaluator.

The Stake model provides a process evaluation strategy which contains a

two—stage procedure : the first determines congruence between what is

intended and what is actually observed (that is, discrepancies from

program specifications), and the second with making sure the program has

the type and quality of components implied by its objectives.

Less dependence is p laced on t he external evaluato r by Stu f f lebeam

[691 . As this model emphasizes the need for evaluation data to serve

decision—making purposes in a timely manner , it permits the evaluator to be

part of the development team . The “process” evaluation component calls

for provision of feedback continuously during program implementation . In

a similar framework , Sanders and Cunningham [78] identify four stages of

tho formative process. The first is called the predevelopmental stage ,

which seeks to identify needs. The second stage is called evaluation of

objectives in which one develops , revises , and clarifies objectives. The

third stage is called interim evaluation , and seeks to evaluate each piece

of the instruction as it is developed . The final stage is called product

evaluation , in which the program as a whole is evaluated , after which it

may be recycled for further development.

Churchman , et al. [801 discuss the question of whether to use internal

or external formative evaluators. They make the point (with which we agree)

that in practice the formative evaluator will become so involved in the

program that the objectivity expected from an external evaluator will be of

little significance during the formative process.

Transactional Evaluation (Rippey [79]; Seidel [81]) differs from other

evaluation models in that it focuses on the effects of perceptions of project

team members and the user population. Its usefulness in fortr~

’tive evaluat ion comes from its emphasis on making explicit the relationships , roles,

problems and possible solutions as perceived by developers and potential

users of the instruction . The formal involvement of these people in clarifying the goals and objectives of a given program contribute to improvement

during its early formative stages.

Formative Evaluation Techniques

The same measurement techniques and procedures may be employed in

formative and summative evaluation . It is the purpose to which the evaluative effort is put and the time when it occurs that distinguishes between

the two types of evaluations .

The application of experimental design to evaluation problems conflicts

with the principle that evaluation should fac ilitate the continual improvement

of a program. Experimental design prevents rather than promotes changes in

the treatment because treatments cannot be altered in process if the data

14 

—.-- - -----—.- - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —‘--~---. -~~~~~ --‘ -~~~~~~~
about differences between treatments are to be unequivocal [821. The

experimental design type of evaluation is useful for making summative

decisions but almost useless as a device for making decisions during the

planning and implementation of a project [23] .

In formative evaluation , the developer is looking for what the

researcher often takes great pains to avoid. Instruction changes as a

function of his activity, both as it is being developed and as it is

implemented in pilot or field tests. The sun*native evaluator , on the

other hand , as does the researcher , goes to great lengths to hold the

program constant.

The choice of design for a formative evaluat ion is a complicated

dec ision depending upon a number of considerations : cost , utility ,

practicalit y , ~

.

~

neralizability, etc. Campbell and Stanley [83] have —

discussed the major considerations in the choice of a design . The evaluator

needs to be concerned with replicability in tha t if the effect of Instruc—

t ion cannot be reliably established , then , of course , dec isions about how

to make it better are meaningless.

The most frequently used desi gn in instructional evaluations Is the

single group pre—test/post—test design [84]. In this quasi—experimental

design , a single group of students is first tested to determine how much

of the criteria behavior they possess , then are administered the instruc—

tion , then tested again. If learning gains are demonstrated , the produc t

developer concludes he has a successful product. The p robl em with suc h a

design is th’it it allows many other plausible rival explanations for the

observed results. In addition , a very serious limitation is the unreli—

a b i l i ty of change scores [85].

Pre— and post—testing is usually considered inadequate for formative

purposes. Continuous monitoring permits correcting problems as they occur ,

tends to increase the aspects of the program that are included in the

evaluation , and consequently improves the usefulness of the evaluation

itself.

One problem with monitoring is in collecting data representative of

the performance of the program such that It is typ ical of the full range

of the intended usage of the system. This collection of performance data

needs to be done without disturb ing the performance of the sYstem being

monitored , which is difficult. Another problem is assimilating and interpreting the results. It Is easy to collect massive amounts of confusing

~~taunless one establishes monitoring experiments with clear hypotheses in

mind [ 86] .

Th e IPISD guidance [87] recommend s such tryouts , as follo~~~: “ If

the student who tries out the instruction experiences difficulties , it may

be profitable to again test out the instruc t ion , after revisions , on

another student. Beyond prac t ical considerations of t ime there is really

no limit to the amount of pre—test tryouts that can be conducted until the

instruc t ion is successful. ”

15

~~~~~~
If the task of the forma t ive evaluator is to monitor programs in order

to provide evaluation data leading to improved instruction , then it is not

surprisi that the focus of most researc h on forma ti ve evalua t ion has been

at the d - —acquisition/evaluation—utilization juncture [75 ].

One interesting researc h questio n relates to the selection of subj ects

as a data source for various formative evaluation efforts. There are those

(Including IPIS D ) ~h ich recommend tha t formative evaluation data are obtained

from single learners in linear fashion with repeated tryouts. Essentially

this tech ni que consists of p lacin g the author with a student as he/she uses

the materials. Ideall y, th~ student will help the author locate ambi guities ,

errors of sequence , and the lik e, and allow the author to test his assumpt ions concerning the thinking processes which will be emp loy ed b y students

using the materials [78].

*

An unpublished stud y b y Robeck (as reported by Baker an Alkin [751)

tested the feasibility of using a single student as the data source for

forma tive evalua t ion lead ing to the revision of an instructional program.

The study demonstrated that observation of a single student is an economical

method for significantl y improving Instruction . Aside from this stud y,

very little research on this techn ique has been performed. The r esent state

of knowledge consists of a number of conflicting “tips” on how to implement

the procedures. Some recommend that high ability students he used , others

recommended low ability. Some sources argue that students can only clean

up semantic and sy ntac t ic errors , while othe rs insist that the student can

make more substantive suggestions concern ing sequence , intended prerequisites , etc. At present , even a simple experiment comparing the quality of

instructional products which have and have not used ind ividual student

t ryouts as part of the development has not vet been done .

As was stated earlier , a variety of techniques can be used for formative evaluation. The pu rpose for which the Information is gathered determines

whethe r it is formative or summative . The ultimate criterion of an instruc—

tional program , however , is a change in the behavior of students. Deterinin—

ation of whethe r or not tha t purpose was met requires a demonstration of

such changes. The IPISD guidance states that one needs to examine in detail

the responses of the learners on criterion tests [87]. A combination of

tests , observations , in terviews , and affective measures is required to amass

the data necessary for the formative evaluation and improvement of instruction .

The specific techniques and approach used in our project to evaluate the

author aid s will he discussed in Chapter IV of this rer ~

rt.

16

-
Chapter II

PURPOSE

The purpose of the research effort described herein was to conduct a

development and feasibility demonstration of on—line , query—based author

aids. The research was designed to include author aids for Blocks 11.2

(Develop Tests) and 111.4 (Develop Instruction) of the Interservice

Procedures for Instructional Systems Development (IPISD) .

Specifically, the activities of the project were to result in author

aids which :

• Are suitable for creation of both on—line and off—line instruction .

• Are generalizable for differing subject matter areas.

• Are documented in a flowchart form to permit timely conversion as

appropriate to other CAl systems.

The utility of the author aids developed was to be evaluated and

revised as necessary with military authors/instructors preparing

operationally relevant instructional material .

4

17

—V——— ~~~~~~~ ~~~~~~~~~~~ -

~~~~~——~~~~

-. 

~~~

.—- r-r -

~~~~~~~~~~~ :

~~~ ~~~~~~~~~~~~~~~~
--

~

Chapte r I II

APPROACH

The goal of the project was to construct , implement and provide a

feasibility test of on—line authoring aids which can be integrated ~‘ith

the IPISD model. En order to attain the objectives of this p roj ect , the

approach taken was:

• User—oriented

• Guided by the IPISD model

c Multi—level in its parallel development/evaluation activities.

A cooperative working relationship was established with instructional

and curriculum development personnel of the ‘U.S. Army Engineer School

(USAES) , Ft. Belvoir , Virginia. Input from USAES personnel was an important

influence in the selection of author aids which would help the USAES

instructional development team to implement the IPISD.

Author aids to be developed in this project were presented on the

PLATO IV computer—assisted instruction (CAl) system . During the course

of the p roject , fou r PLATO IV terminals were located in the NumRRO

laboratory , Alexandria , Virginia. In addition , 8 terminals located at

Ft. Belvoir , Virginia , were also available during the project.

The Engineer Non—Commissioned Office Advanced (ENCOA) course was

selected for this project in consultation with USAES curriculum development

and training personnel and with the agreement of ARI . Arrangements were

made to permit participation of four instructors (2 NCOs and 2 Officers)

who teach this course.1 The ENCOA course covers a wide range of technical

(“hard”) and soft skills. It was thought that if authoring aids were

developed which would be useful for handling instruction and testing of hard

skills (e.g., straightforward mechanical work) as well as soft skills such

as problem—solving, the set of authoring aids would be more applicable to

other courses and other schools than if just the hard skills were chosen

for the targeted materials. Therefore , the subject matter selected for

this project was a nine—hour block of instruction from the ENCOA course ,

covering such items as field fortifications emplacement construction ,

U.S./foreIgn mine warfare doctrine , and protective mining. The work

involved in this section of the course includes computational problem—

solving, as well as procedural tasks. School personnel had defined training

objectives as a result of previously applied systems engineering principles.

‘One Officer was transferred from the USAES during the course of the project .

Theref ore, only three instructors participated in the research effort.

- 

18

~~~ ~~~~~
_______________________ 

-

The I P1St) model was a compa t t hi e and useful guide in di’s I going he

technical approach fot’ the proposed p to ~i’c t . I~

ach of he rq htc’i’

m a y or procedura I Phases——Anal v~ e Desig n , and Devo top——were pert itwut to

the act tvit tt’s undertaken In this project . The Lir,

~~’t~

e d ’ students ” )n

this case were the authors and th~ Ins t rtic I en.i I (oct15 was (lii’ ant her at d

The mul t I le vel nat nrc of tb pro j e’e t shou t ~

1 he eons I derod here .

liumRRO personnel dove I oped and i’va I nat ~

‘d ant her aids . These ant hot at ts

were then used ~~ PSAES Inst m eters to dovi’ 1op and v ail d:,t e Inst m e t ton .

Thu s I te m at lye , par allel :le I Lvi ties occurred at di (t i’t’en t I i’ve Is I n t Ito

project . Cuiding ;%l I these ci forts was t he l I ~l SI) i*~de I It set t - in pail I cut am

the f i rst three procedural l’hases . For i’xamp I i ’, the approach to ant her

aid development and vat tdat ton drew Its guidance spec I ( ical lv from ll’ I SI)

fiIt~ck~ 111 .4 and 111 . ’.

In It tall ~ , a sot o I dot ailed I lowctt ~ rt s were ~~~~~~~~~ t t.ue t i’d to dose ii lie — info rmat ten element $ and f,’.i t ures r equ I rod liv inst rue t tona l dev~’ I ~

‘pers t o

performing the steps of II’ 151) tlloeks II . .‘ (Dove iop Tests) and 111 .4 (Develop

Inst ruct t on) . The fl.~~

chart we ti’ desIgned to hi’ stil ft e I en t lv di’ tall i’d

and annotated fo,’ ready adaptat Ion to any svst em 1 .e. ret at l vi’ l v

hardware or software I ndepotidc ’n t ‘I

Inasmuch as the PLATO I V svs i’m was c onsI d ered a research veb I cli ’

on 1 v • c are was t aken to max I ml c hardware .‘i’ so I wa i’ i’ I ode pondi’nci.’ of t he

aids. On— I Inc au thor aids as well as of I -- line v ’rs Ions we re suppi led to

assIst the author In prepa ii ng Inst rue t tonal and Ii’S t mate ml a is fo ci her

CA I or non—C ,~ I ~le live rv of In st rue t ten .

The mu It I —1 i’V0 I nat u r

~’ of the i’s’ sea reh act I v i t I i ’s is c I i’a r I ~ demeit —

st r at ed liv the thi’i’,

’ 1 i’vi ’l s of ova l nat Ion undertaken In I he project . The

first leve l was an In lemm a I ova l nat ion e I ox _ t ,s_ t ln,~ 1)’ iSP gut dancr’ , p rocottttmi’s

and au thor aids. Ut~ni~~ l stat I as users et ’ it Is guidance , were the primary

source of cvaluat Ion data at thi s level .

l evel .

~ was dl roe I cii toward a form at lvi ’ i’

~~’:i I nat ton 01 ni ’w ant her a I its

and procedures tk ’v~ I oped spec I f I call v for on—t i no app it cat (en to I1’1 SD

Blocks 11 . .

~ and I II .4 . I i’ve 1 I oval nat ion ;IsSi ’ssCil the adequacy of t he

instruc t tonal materials e teat cii hr the mlii tarv authors. Those mater Ia Is

were then admi n 1st o red t o 1’ .5 . Amm ~ Eng I noel’ School I r* f n~~’s ~~~ pr~~ I ded

an addi t tonal data source

Revision act iv it tos occurred cent Inuousl v t broug hout the perIod of

project per formane e . The purpose of t hes~ rev I s Ions was to assure max i mum

Ut III tv of the (I owcharts and author aids In Imp I emi’nt lng the 1 )‘lSI) precess

The test Items and lesson materIal were not ri’yt sod as a has is 01 1 rattee

data, h~ causi’ of I I nw lImit at lens , hut tht’~~i’ dat a we me I tteovpera t i’d as part

of the research conclusion .

The pro _joe I activities we me dIvided Into t oiti’ major Tasks, liti ’s,’ we i

~ ’

Task I . Anal vs Is snil Do term I tut ti on of Requ I rod Ant her A l ii Element s

‘~

_____ -- 

,

~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ 

~~~~~~~~~~ ~~ -~~ -

~~~~~~~
- ~—~~~~

-- —--— “---

~~

- - -- 

~~~~~~~~~—‘- ---- - ‘-~

—~

Task 2. Conversion o

~ ’ Flowcharts to Interactive Program

Task 3. Evaluat ion of the Programmed Materials

Task 4. Revisions

The activities and accomplishments in each of the Tasks are described in

the following chapter.

F

20

.1.

— —~ ~~ ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ 

•-

~~~~~~~~~ — ~~~~~~

-

~~~~

—

~~

-

~~~~~~~

- ~~
Chapter IV

PROJECT ACTIVITIES AND ACCOMPL ISHMENTS

TASK 1. ANALYSIS AND DETERMINATION OF REQUIRED AUTHOR AID ELEMENTS

In Task 1 a detailed set of flowcharts was constructed which provide

instructional system designers with the means of performing the procedures

called for by IPISD Blocks 11.2 (Develop Tests) and 111.4 (Develop

Instruction) . Activities during Task 1 were conducted in two Phases:

(1) expansion of IPISD flowcharts so as to provide greater detail of the

specific activities required for each block (What To Do), and (2) selecting,

identifying , designing and flowcharting of author aids for completing the

activities (How To Do).

Develop What —To—Do Flowcharts

The IPISD flowcharts for Blocks 11.2 and 111.4 shown below in Figures

2 and 3 were used as the basic framework for the RumRRO—developed f lowcharts. The IPISD flowcharts provide a broad description and sequencing

of necessary activities. However , because of their global nature , t hey

provide only minimal assistance to the instructional systems designer.

Each element of the IPISD flowcharts was expanded into detailed

step—by—step sub—elements that must be performed (or considered) in

completing the specific flowchart block. With respect to the IPISD activity,

Develop Tests (IPISD Block 11.2), the procedural steps described in the

“Guidebook for Developing Criterion Referenced Tests” L58 1 were used

heavily in the identification of the sub—elements. Figure 4 is an example

of how one such IPISD element , 2.6 (Determine Scoring Procedure ) from

Block 11.2 was expanded into sub—elements.

It was found that the activity descriptions shown in sub—element

blocks were not always sufficiently descriptive of the activities required

by the block. Consequently, it was necessary to further flowchart several

of these sub—element blocks. An example of further flowchartlng of sub—

element block 2.6.1 (Determine Qualitative Scoring Procedures) is shown in

• Figure 5. The blocks in italics refer to blocks already flowcharted in

IPISD. All other blocks are HumRRO flowcharts. Blocks outlined in boldface are blocks for which author aids were developed in the proj ect. A

check mark above a block indicates that existing author aids have been

identified for that block. The narrative on the right of the flowchart

further clarifies the block and lists any references to existing author

aids.

The product which resulted from the Phase 1 activities is itself a

valuable author aid for instructiona l system designers . It provides a

~~ step—by—step enunc iation of activities that must he performed . The revised

‘

~ 

21 -

I’ ‘

~~~~ ‘ -“.

•

_,

~~~ I.’ i

~~ ’~~~ TT~ ‘

~~~~~ •~~~ .j .1

~

. - 

~~~~ __________ •I’ .

~~ ~ ~~~~~~~~~~~~~~~~ ~~~~~~~~~ 

‘— .—-

~~~~~~~~~ ‘~~‘
- ‘

~~~ _w~’- • - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -~ -~~~~--—-~~~~~ • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TASKS SI LECTE’D FOR

INPUT S TRAINING , JPMs ,

I LOs. LOS . AND LSs

DE T IRMIN E HOW 

~~~~~~~~UT -OFF DETAILED Til t —~

i TEST SHOULD BE ~~~~

~~~~LI

TRANSLATE ILOs , DETERMINE

LOs, AND LSs IN1 O SCOR ING

TEST ITEMS PROCEDURIT

________ 

2.? ________ 

2.6

SE T IRAINING COLLECT ‘3ASE LINE

STAN DARDS DATA

_Li

RANK OIIDER ( 

~ 

oc

S1Ut1EN1S — ~ ( II 3 (IF HIOUIRIT D) 

2,4

Flowchai t of Block 11.2: DEVELOP TESTS

FIgure 2

I’,-

22

li

~~~~~~~
I

lEARNING OUi( :LIV I S

m OM BLOC K .1~ lEst I T E M S I 1

~OM I1IOCKiI 7

l EARN ING GUIDE LINES ANt)

AC1 IV ITIL s i HUM u 0cK 1U1. 1 IN UI OIL IVITI1Y svs ; ~ M ANt)

MANAGI MINI Pl AN

F HOM (tIOCK 1ff?

MtO tR IALS St ([CIII) IN

DLOCKiIL.3

IDEN1 IF V INS1 RIICTIO NA L 1

NEEDS AND CONS I H AI N 1 S ..

——--U-—

1

~ 

IDLNl IFYAVA L~~~~~

’ 1

[ R I S OU R C F S ? 7 J

DEVELOP

IN St HU C1 ION 2 3

PRhlISI F IR S t 1 DRAM M A 1LItIA I S 2 4 1

I’A HI USI H

INSIIWCTION

F$owch~rt of Dlock 111.4: DEVELOP INSTflUCTION

Figure 3

23

-

~~ 

- ,

~~~ 

.-- ‘I-~ ~~~ ~~~~~~~~~~
Determin, qualitative scuu Ing

procedures.

2.6. 1

Determine quantitiative scoring

procedures

2.6.2

Determine when scoring will Will interference scoring be used?

occur, after or during test . See Swezey & Peeristern , Guidebook for Developing CriterionReferenced Tests.

Arlington, VA: U.S. Army Research Institute for the Behavioral and

2.6.3 Social Sciences, Auguat 1975.

Determine whether scoring will Develop form f or recording correct and incorrect answers of class

be done by hand or machine. and individuals. Develop method for determination of number of

correct answers (or a scoring key). p. 6-6.

2.6.4

Write scoring directions.

2.6.5 -

Perform scoring procedures Use on-line or off-line readability aid.

tryout. See Swezey & Peerlste in , p. 7.3

Perform read.bility check.

2.6.6

Revise procedures as indicated

in tryOUt.

2.6.7

Illustrativs Flowchart Expansion of IPISD Block 2.6

Figure 4

- 

24

—~~~~ -

~__— _________________ ____________
Determine Qualitative

Scoring Procedures.

2.6. 1

Deter min, whither partially

correct responses will be

evaluated.

2.6.1.1

I

Determine schem. for Especially applicable in construct ed response format.

assessing pirtlally correct

answers.

2.6.1.2

I

Determine whether presentat ion For oral and constructed response (essay ) tests .

of response , 

~ ammar , spelling

or punctuation will affect

score.

2.6.1.3

Outline scoring guidelines for

behavioral checklists or

scales.

2.6.1.4

Ill ustrat ive Flowchart Expansion of Block 2.6. 7

Figure 5

I

!

~

i 23

, iI _ ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ _____
and expanded flowcharts were produced as off—line materials. As such , they can be converted to checklists and used as procedural guides by

designers of instruction.

It was not feasible within the limits of this project to produce

the fully expanded flowcharts on the PLATO terminal . However, the

information contained in many of the blocks was incorporated into the online author aids and as such provide on—line guidance in accomplishing the

objectives of each block.

Identify and Reference How-To-Do-It Author Aids

In Phase I the detailed activities (sub—elements) needed for developing

tests and instruction were defined and arranged into sequential order. In

Phase 2 each sub—element was examined to determine specific authoring aids

desirable to accomplish the sub—element . In other words , Phase 1 describes

what must be done, and Phase 2 defines author aids for doing it. Time

constraints did not permi t the development or selection of author aids for

every sub—element. Therefore , aids were provided for those sub—elements

which were identified as of highest priority for potential users. The

selection was based on such factors as:

1. Available HumRRO expertise gained from previous experience in

autho r aid development ;

2. A review of the literature to identify aids already available

for use; and

3. Opinions of instructional systems designers at USAES concerning

aids they considered would be helpful to them.

After identifying the author aids needed for the IPISD sub—element

blocks , the next step accomp lished in Phase 2 was that of including

references to the author aids in the flowcharts developed in Phase 1. The

purpose of this step was to identify for users of the flowcharts those sub—

elements for which author aids were available and to refer them to a

reference (hardcopy or on—line) which more fully detailed the specifications

of the aid .

Throughout Phases 1 and 2 , care was taken to assure that the flowcharts

were sufficiently detailed and annotated to be of practical use to thstruc—

t ional system designers and would readily permit adaptation to any system,

i.e., be hardware or software independent.

Task 1 activities resulted in an Interim Report and Guide to the Use

of Flowcharts [88J . This report contains flowcharts provid ing detailed

guidance on the procedural steps necessary f or implementing IPISD Blocks

11.2 and 111.4 , and identifies sub—elements of these blocks for which author

aids were developed . In addition , the Inter im Report includes a guide to

the use of flowcharts which we felt is necessary inasmuch as many authors

• may not be familiar with a flowchart format.

26

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-
The flowcharts prepared in Task 1 are of value from three stand points:

(1) They are useful as tools for instructional systems designers in the

implementation of the IPISD process; (2) they may be used as a model for

detailing the processes covered in other IPISD Blocks in terms of level of

detail , style , and format; and (3) they may be used in the preparation of

on—line author aids on any CAL system.

TASK 2. CONVERSION OF FLOWCHARTS TO INTERACTIVE PROGRAM

In Task 2, author aids identified in Task 1 were developed for presentation on the PLATO IV computer—assisted instruction (CAl ) system. Inasmuch

as the PLATO IV system was considered a research vehicle only, care was

taken to insure that the author aids developed could be readily modified

to be hardware or software independent. Where possible the author 8ids

were also created so as to have application for “off—line ” use .

In the context of the present project , an instructional system developer

(author) may be working on—line in an interactive mode with a computer. in

this case , he is termed an “on—line author. ” If an author is not working

di rectly with a computer , he is referred to as an “off—li ne author. ” Even

authors who are developing CAl materials work in both off—line and on—line

modes. For example , some authors use preprinted CRT layout sheets to write

their text , and then have clerks key the text into the computer. Currently

authors of most military instruction typically work off—line , although they may

have access to computer support for such things as test scoring, statistical

item analysis , or other aids.

Author aids were developed to assist the on—line author in preparing

instructional and test materials for both CA ! and non—CAl delivers- of

instruction . CAl was the principal mode used in this research because the

CAL mod e provides opportunity for ease of gathering and analyzing data

regarding both student and author activities. Off—line versions of these

aids will assist off—line authors in preparing both CAL and non—CM materials.

PLATO lessons “inquiryl” and “inquiry2” can be thought of as master

author aids. These lessons incorporate the individual author aids identified

in Task 1. Lesson “irtquiryl” deals with lesson development (IPISI) Block

III.4~

1 , and “inquiry2” with test develqpment (IPISD Block Il.2’

~.1 Individual

author aids for inquiryl and 2 can be roughly categorized in four different

classes which are discussed below. The four categories for each master

author aid are :

~

‘These lessons were available on the University of Illinois PLATO IV CAl - -~ system at the time this effort was completed .

27

- 

-

e’. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ “

~~~~~~~~
Lesson “inquiryl” (Lesson Development)

1. Tutorial author aids for Lesson Development

2. Author aids for management of the lesson development process

3. Author aids for lesson content development

4. Author aids controlling within lesson branching

Lesson “inguiry2” (Test Development)

1. Tutorial author aids for Test Development

2. Author aids for management of the test development process

3. Author aids for test development (e.g., test instructions and

test items)

4. Author aids for post—test reporting of results , review and

remediation actions .

Lesson Development (Lesson “inquiryU’ )

1. Tutorial Author Aids . This series of author aids provide instruc—

tional system designers (authors) with guidance in the preparation of lesson

materials. On—line the guidance is automatically presented at appropriate

points during the lesson development process. In addition to the forced

presentation , authors may review any specific guideline as desired . The

guidelines are available in both on—line and off—line versions. Specific

guidelines included in this series of author aids are:

• Instructional Sequencing Rules

• Guidelines for the Preparation of Terminal Learning Objectives

• Guidelines for Reducing Reading Difficulty Level

. Guidelines for the Preparation of Text Material

• Guidelines for the Use of Practice Question Formats (General)

• Guidelines for the Preparation of Multiple—Choice Practice

Questions

• Guidelines for the Preparation of True—False Practice Questions

• Guidelines for the Preparation of Constructed Response

Practice Questions

The “off—line” version of these author aids will be of use for all instruc—

tional modes .

2. Au thor Aid s for the Management of the Lesson Develo pment Pro cess

• ~~~~~~~~~~~~~~~~ ~~~~ ~~~~

Instructional content is, of course , based on Terminal Learning

Objectives (TLOs), Learning Objectives (LOs), and Learning Steps (LSs) .

However , how these are sequenced in the instruction may very well determine

whether an instructional module is effective or ineffective . (A module

as here defined begins with an LO or LS and is usually followed by 5-10

frames of text and practice questions which teach the LO or LS.) To

assist authors in the creation of modules and the sequencing of instruction ,

worksheets have been prepared for off—line creation of LOs, LSs, text

28

- ~ ,- •~~~- -.- - - • ~-‘- 

— -‘. 

. 

-~

•.— - • —~~..~~~~ —~ - ———~

-. £

I .I

~ _______ .

~~~~~~ __.___~~ _&__— r ~~~~~~~~ ~~ 4
frames and various types of practice questions. These worksheets can be

used for most instructional modes but are particularly suited for CAl and

progransaed instruction.

• Learning Obje~ t--f ven (LO) and Learning ~~~~~~ (LS)

Manageriien t

Learning Objectives (LOS) and Learning Steps (LSs) are the backbone of

the IPISD process. They dictate the content of both instructional material

and test items. The instructional system designer must attend to them

carefully to insure they are represented in the instruction and test

situation. As described above (Sequencing of Instruction) each module of

instruction begins with an LO or LS. The author is required to input the

LO or LS prior to inputting instruction for a given module. During the

preparation of the instruction for a module , the associated LO or LS is

available to the author as a continual reminder of the instruction to be

addressed. A by—product in the CAL version that is available to students

studying a particular module of instruction is the option to access the

LO or LS statement underlying the instructional module (see Student

Controlled Branching Author Aid below.) The “off—line” version of this

author aid will be of use for all instructional modes.

a Reading [ f f ~~~~? tj ~ Index

When preparing any instructional material it is essential for the

author to consider the intended audience for the material [89—97]. Therefore, an author aid was prepared for use on the PLATO system that auto—

matically computes the reading difficulty of text material , question stems,

and feedbacks provided the student as the material is inputted into the

system .1 In using the aid , authors specify the reading ability level of

the intended audience and if this level is exceeded the computer so informs

the author who can then revise the material to a lower reading level. An

off—line version provides the formula and identifies the components required

for computing the reading difficulty index. Obviously, this author aid

is far stronger in its on-line version since the author is not required to

compute the index. However, it can be used manually in off—line

instructional modes.

3. Author Aids for Lesson Content Development

• Text Crea t

~cn and Ed:~ t~~

This author aid will be most powerful for development of CAl

materials. In CAL form it permits authors to create CAL executable textual

material without a knowledge of the programming- language required by the

system. The text may be placed at the author ’s option any place on the

screen and permits revision after initial creation. It incorporates other

author aids such as the reading difficulty index (described earlier).

1

Available in Appendix A.

29

. ,..

. - ~~~~~~~— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• ~~~~~~~~~~~~~~~~~ I ~~~~ ‘r

~

This is ~ctuallv a series of several author aids which allows

creation of practic e questions. The author aids do not require a knowledge

by authors of a computer programming language. The aids are of primary

value in a CAt or PT mode. Detailed characteristics of each of these

author aids are shown in Tables 1, 2 and 3.

4. Author Aids Controlling Within-Lesson Branching

a -

~~~. ~~~~~~~~~~~~~~~~ :‘:‘~~~~~~~~~:-:

Today , in many instruc t ional modes such as CAl , P1 and other forms

of self—paced instruction , students frequently must demonstrate a mastery

of current instruction before being allowed to go on to new instruction .

Based on their performance , some students may be required to review certain

port ions of the instructional material while others will go through the

instruction without forced review. That is , students are branched depend—

ing upon their particular needs. This process requires that student

performance be continuously monitored . Author aids have been provided to

assist authors in these efforts. Data collection aids for practice questions

provide continuous student monitoring. Other aids provide guidance to the

author on how to use-monitoring information (e.g., the number of attempts

a student is permitted at a practice question , the conditions under which

the student is required to review instruction, etc.). These aids have

primary application in self—paced modes of instruction .

a ~~~:

~ i.

-’; ~~— _ ~~~~~~~~ ~~~~ ~~~~~~~~~ :,,.

Students themselves , frequently know when they need additional

assistance and should have the opportunity of accessing this assistance

whenever they desire. However , they must be able to identify what assistance

is available and the means for accessing it. This ser ies of aut hor aid s

provides students the options of accessing auxiliary information , returning

to previously studied material ’- and , if permitted by the author , of branching

to the end—of—lesson test from anyplace in the lesson . These aids make use

of aids al ready developed for other purposes. For example , the management

and sequencing aids provide specific statements of TLOs , LOs and LSs asso—

~iated with each instructional module . The student—directed branching

aids permit the student to temporarily branch to these statements whenever

desired . The author aids provided for student—directed branching will have

wide application independent of the instructional mode used .

Test Develop~ent (Lesson “inquiry2”)

1. Tutorial Author Aids. This series of author aids is similar to

those discussed in the tutorial author aids for lesson development . These

aids , however, provide both on—line and off—line guidance in the var ious

‘That is, hack—page . return to beginning of lesson or beginning of module .

30

- 

- - - ~~ ‘ ..4 

~~~~~~~~~~~~ -,.

.-

~.- —

~~~~ ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~ ~~~~~~~~ --

~ — ---~~-— —-~~ — —
- ---— 

Ta ble 1 . CHARA CTERISTICS OF PRACTICE QUE STION AUTHO R AID

(MULTIPLE CHOICE)

• Question stem has maximum length of six 5O~

-character lines.

• Three—six anèwer alternatives (including correct answer) permitted.

• Each answer alternative has maximum length of two 40—character

lines.

• Correct answer position randomly selected . (Author may select

other position if desired.)

• Author “cued” if answer alternatives differ in length by more than

± 20 characters. (Author has option of revising.)

• Author specifies one—three attempts student permitted on question .

• Author can create correct answer congratulatory message . Maximum

of five 40—character lines.

• Incorrect answer feedback messages may be specific to response

given , or general feedbacks which may be different for different

attempts. Incorrect answer feedbacks are limited to five 40—

character lines.

• Correct answer given student if number of permitted attempts

reached without student correctly answering the question .

• Reading difficulty of question stem and feedback messages

automatically computed . If desired reading level exceeded ,

author has the option to revise material.

- 

31

:

~. - 

~

--- 

~~~~~~~~~~~~

—

--

~

-

~~~~~~~~~~ 

-

~~~~~~
______________ - - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I

Tabl e 2. CHARACTERISTICS OF PRACTICE QUESTION AUTHOR AID

(TRUE-FALSE)

• Question may be a maximum of six 50—character lines.

• Author specifies one or two attempts student permitted on question.

• Author can create correct answer congratulatory message. Maximum

of five 40—character lines.

• Incorrect answer feedback message provided if two attempts -

permitted. Messages may be a maximum of five 40—character lines.

• Correct answer given student if number of permitted attempts

reached without student correctly answering question.

• Reading difficulty of question and feedback messages automatically

computed . If desired reading level exceeded , author has the option

to revise material.

32

~~~~~~~

. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

~~~
~~~~-- -~~~~~~~~~~~~~~~~~~~~~~ 

---- -

Table 3. CHARACTERISTICS OF PRACTICE QUESTION AUTHOR AID

(CONSTRUCTED REPONSE)

— a Question may be a maximum of six 50—character lines.

• Author specifies one—three attempts student permitted on question .

• Student response analyzed for one-four correct or partially

correct answers.

• Student response analyzed for one—four anticipated incorrec t

answers.

• One—four congratulatory messages permitted depending upon number

of correct or partially correct answers specified by author.

Messages may be a maximum of five 40-character lines.

• One—four wrong answer messages permitted depending upon number

of incorrect answers specified . Messages may be a maximum of

five 40—character lines.

• Author has option of permitting misspelling of answer; words in

answer to be out of order; extra words in answer; and disregarding

the capitalization of answer.

• Correct answer given student if number of permitted attempts

reached without student correctly answering the question .

• Reading difficulty of question and feedback messages automatically

computed . If desired reading level exceeded , author has option

to revise material.

33

- --~~~~~~~~~ -— -—

~ 

- - 

~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~ 

- -.- - - - - - 

-
facets of test development . Like the lesson development tutorial author

aids, they are automatically presented to authors at appropriate points

during the test development process and are available for review at any

time . Specific guidelines included in this series of author aids are.

• Guidelines for using “inquiry2” author aids

• Guidelines for the Preparation of Terminal Learning

Objectives

• Guidelines for Writing Test Instructions

• Guidelines for Reducing Reading Difficulty Level

• Guidelines for the Preparation of Multiple—Choice Test

Items

• Guidelines for the Preparation of True—False Test Items

• Guidelines for the Preparation of Constructed Response

Test Items

• Guidelines for Assigning Scores to Test Items

• Guidelines for Post—Test student review of test items

and Remediation Strategies

2. Au thor A id s for Management of the Test Devel opment Process

• Sequencing of Tcot Items

Worksheets are provided to authors for off—line creation of

TLOs, LOs, LS test instructions, and test items. These aids permit authors

to organize and sequence their test items prior to input into the computer.

The worksheets are useful for all modes of instructional delivery .

• Terminal Learning Objectives (TLO), Learning Objectives (Lu),

~

zn (1 team ing f-cr (LS) Mni~~~’~c;i t

This author aid is somewhat different from the corresponding aid

used for creating TLOs, etc., in the lesson development process. Instructional system designers (authors) input all TLOs, etc., into the computer

in the sequence in which they wish to cover them in the test. (See sequencing

of Test Items above.) The author aid then maintains records of which TLO, etc., has been addressed in the test and in the on—line version . The

computer “cues” the author as to the TLO they should next address. The

off—line version of this author aid will be of use for all instructional

modes.

• Reading Difficult! ? f nJ€ -x

This author aid is identical to the one discussed earlier in the

Author Aids for Management of the Lesson Development Process.

3. Author Aids for Test Development

• ‘rca tion of Teat Instru ct-ions ~

md Ej i t ing

This author aid is similar to the text creation author aid

previously described . The aid will be most useful for development of

34

- - ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~ ~~~~ 

- - - -- 

~~~~~~~~~~~~ ~~~~~~~~~~~~ ~
_____________________________________ — - ‘I

student instructions for computer—administered tests. It can also be used

for on—line development of instructions for other instructional modes such

as programmed texts, etc. The use of the author aid does not require a

knowledge of the programming language required by the system. The aid also

permits revisions to be made to the instructions after initial creation.

• Te8t Item Creation

A series of author aids was developed to be used for creation

of representative types of multiple—choice , constructed response and true—

false test items. They are similar to the practice question author aids

(see Tabl es 1, 2 and 3) except that they do not provide correct and

incorrect response feedbacks, nor a variable number of permitted attempts.

• Timing of the Test

This author aid permits authors to establish , if desired , a time

limit for individual items in the test or a time limit for the entire test.

In the CAl version of this author aid , the computer maintains a record of

elapsed time and takes appropriate action based on the elapsed time. This

author aid is most useful in a CAl mode.

a Test Ttc-r: SL’orinq

Te st item scoring author aids are provided to assist test developers

in establishing test scoring procedures. These aids include such considerations as: setting cut—off scores, differential weighting of various

answers to a test item (i.e., correct , partially correct , and incorrect

answers), and/or differential weighting of different test items. Off—line

versions of these aids consist of guides , checklists, etc. On—line versions

are similar but are prepared in a “query” format . The aids are useful for

test scoring for most instructional modes.

4. Author Aids for Reporting of Results , Review and Remediation

Actions

• !‘

~~vt:~

ng of Results

Subsequent to test item scoring (discussed above) authors can

establish the minimum passing score required. This author aid then scores

the test and automatically reports to students their obtained score and

the minimum score required for passing-. The aid is most powerful in a

CAt format but may be also used in other self—paced modes of instruction.

• Pc~~t-Tc~ t Review of Teat It ems Missed

These author aids permit the test developer different student

review options for test items missed . For example , if the student passes

the test with less than a perfect score, the author may elect to show the

student the correct answer to items missed . Or , in the case of students

who fa il the tes t , the author may elect to: (1) show the students test

items missed without providing the correct answers, (2) show test item~

35

- ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
missed and include the correct answers, or (3) not permit a review of

items missed . This author aid is primarily of use in a CA! or programmed

text mode.

• Rented-l.a tion Actions

This is a series of author aids which permit the author to select

the type of action that will be taken if a student fails the test. The

actions possible in these aids are as follows :

(a) Re—administration of instructional lesson followed by . re—

administration of test items previously missed . -

(b) Re—administration of instructional lesson followed by re—

administration of entire test.

(c) Immediate re—administration of test items previously missed .

(No re—administration of instructional lesson.)

(d) Immediate re—administration of entire test. (No re—

administration of instructional lesson.)

(e) Re—administration of test items missed. Give student

option of reviewing instructional lesson first.

(f) Re—administrat ion of entire test. Give student option of

reviewing instructional lesson first.

(g) No re—administration of instructional lesson or test——

student is finished with lesson or goes to new lesson.

In the on—line version of these author aids , failing students are auto—

matically branched as directed by the author . Therefore , these aids are

most powerful in a CAl format. However , the principles underlying the

aids can be employed in any instructional mode .

TASK 3. EVALUATION

Th ree levels of evaluation were undertaken in this project. The first

level was an informa l evaluation of existing IPISD guidance , procedures and — author aids. Six HumRRO personnel with technical expertise in systems engineering procedures judged the ease and effectiveness with which selected

IPISD procedures and guidelines could be used to develop instruction. Where appropriate , these author aids were referenced in the flowcharts developed in Task 1.

The second level of evaluation was a comprehensive formative evaluation

of the new author aids and procedures developed for application to IPISD

Blocks 11.2 and 111.4. Three instructors from the ENCOA course [one off icer

and two NCOs (ES) ) serve d as study participants and as a data source for

evaluation . These instructors used the aids to create test items and lesson

material . Evaluation data were gathered as the authors developed their

36
instructional material . Formative evaluation of the author aids was

ac complished by examining author performance and acc eptance of the

aids. Critical weaknesses in the aids (i.e., those which impeded the

progress of the authors) were remedied immediately upon diagnosis of the

problems .

In the third evaluation level , the adequacy of the instruction created

by the military authors was assessed . This instruction was administered to

U.S. Army Engineer School trainees who provided the data for evaluation .

The ultimate criterion of instruction is evidence of desired changes in - -

trainee behavior (i.e., Does it “teach?”). In order for the author ’s developmental activities to be adequately assessed , trainee performance and

attitude data were collected .

In the intial stages of the second level formative evaluation , HumRRO

staff functioned as “test item developers” and/or “preparers of lesson

material .” Their role in this study was to find errors and faults in the

directions , requirements, procedures , etc., of the author aids. We then

used these data to make needed revisions of the author aids.

Once the author aids were considered ready for application to actual

course content , the three TJSAES authors were given training in using the

aids. A brief 15—minute familiarization/train ing period in using PLATO

preceded each individual’s involvement in the project. They received

instruction in signing on and off to the system (which included signing

into the appropriate }hnnRRO lesson) . A brief (approximately 5 minutes)

orientation to the PLATO keyboard was then presented to each author. This

included : use of the edit keys; editing techniques; and use of the help

sequence keys (e.g., HELP , BACK , NEXT , etc.). Descriptions of system

crashes, transmission errors , and other system abnormalities were provided

along with instructions on how to proceed under these circumstances . The

authors were then permitted to practice with the keyboard before they

started inputting their lesson/text materials , and all of them chose to do

so.

Following familiarization training on PLATO, the three USAES instructors/

authors were given a brief explanation of their role as authors and then

training in the use of the author aids. Project staff members provided the

training in a one—on—one , tutorial mode .

Following training in the use of aids , instructors prepared and input

on—line in the PLATO system test items and lesson material. Each of the

instructors developed a lesson and the related test items in their content

specialty as part of a 2— 3 hour block of different , but related, subject

matter from the Engineer NCO Advanced (ENCOA) course . Table 4 lists the

subject—matter blocks selected for this project.

The ENCOA course had undergone systems engineering and USAES personnel

provided a set of well—defined terminal learning objectives. Test items

were prepared which reflected these objectives . An additional advantage to

the ENCOA course wa s that both NCO and Off icer instructors were available as

authors. Hence , the utility of the aids could be evaluated across a wide

range of background skills and experience .

37

__ 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ --- -

~~~~~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Table 4. ENCOA LESSONS AUTHORED ON PLATO USING INQU IRY AIDS

Average

Compl etion Completion Time

Time to Time in in CAl Versi on

Lesson Au thor Create Lesson Current Course (minutes )

Field Fortifications NCO—l 48.5 hrs . 4 hrs. 94.3 (N 9)

Emplacement

Construction

US/Foreign Mine Officer 41 hrs. 2 hrs. 60.2 (N = 11)

Warfare Doctrine

Protective Mining NCO—2 35 hrs 3 hrs. 38.6 (N = 9)

TOTAL 124 hrs . 9 hrs . 191.1 minutes

(or)

3.22 hrs.

—,- -- .--- - 

38 

___________
- - T ~~ - -- -.~~~ - - - -

Although each author was required to input his material into the

computer , a HumRRO staff member was present to assist him in the process

on a one-to-one basis. No instructor needed to know the TUTOR language or

have previous TUTOR experience because the INQUIRY author aids were designed

so tha t code was automatically generated .

Data collec tion, to a large extent, resulted from direct observations

of the authors creating and inputting their instruction and from structured

interviews with the authors. We gathered user acceptance data , and information on various areas of difficulty that the authors experienced while using

the aids.

The authors using the INQUIRY system were encouraged to comment at any

time on their progress. Monitors were present at every inputting session

to note any problems encountered or comments made by the authors. These

comments were used later to make changes in the system so that it was

easier to use. After the authors inputted all of their material , they

received a questionnaire asking their opinions of CAT and the INQUIRY system .

In addition to interview/questionnaire data, performance data were collected .

Such items as the time to create a given frame of text or test item on—line , the number of times a piece of text had to be re—input , errors in attempts

to apply a particular aid , calls for help from the monitor , etc., were

recorded .

In the third evaluation level, we assessed the instruction created by

the authors using the INQUIRY aids. To the extent feasible, student—

identified areas of difficulty in the instruction were associated with the

use of particular author aids. In this way , we tried to determine whether a

poorly designed aid led to unclear instruction or to problems with the tests.

Twenty—two students1 went through the lesson material for about 2— 3

hours each to assess the quality of the instruction created with the author

aids . All students received preliminary training on using PLATO. As all

the students could not go through all the instruction and testing within

the time allotted by USAES for this project , only two of the three lessons

were presented on a random basis to each student .

Presentation of lesson content occurred on—line, as did the administra—

tion of the post—tests based on the TLOs. In addit ion to collecting

cognitive data regarding student performance , exit questionnaires were

administered to obtain information regarding opinions of the clarity of

the instructional material , problems encountered in the practice and test

items , and attitudes toward the CAl instructional experience .

‘Prerequisites for selecting students were that they be NCOs who have entry

qualifications for the ENCOA course , but have not been exposed to the

material covered in these lessons.

39

‘

~~

f

~

___ ~~~~~—- - .

- - -- -.—-------- ~~- ,

-. 

~~~~~ 

-

~~~-- - ---

~‘---.‘~~~

- ~~~~~~~~~~~~~~~~~
~

—-- -- -~~~~ - -~~~~~

F 1nd ln~~

1. Lnstructlon and Test Development. The tutorial aids were presented

to each author prior to inputting. In no instance did the authors seek to

reread these aids which presented guidance on test and lesson development.

Thus, it cannot be concluded whether or not these guidelines were useful to

the authors. It appears that more emphasis on the applicability and value

of the aids is required in order for authors to pay attention to this

guidance. This may involve a considerable change in their presentation

format.

- As a result of the initial formative evaluation , the authors were

able to prepare test and lesson material with minimal difficulty. In

developing almost 360 frames of instruction and testing, a total of 65

problems were experienced by the authors as recorded by the monitors . Over

25X (18) of these problems were trivial errors caused by the author pushing

The wrong key . Fourteen instances were due to unclear INQUIRY instructions,

which were remedied as soon as possible after they were noted . Twenty—one

problems were due to “bugs” in the INQUIRY program which were eliminated as

soon as their diagnosis was confirmed . Six problems were noted as due to

PLATO system crashes and transmission errors . Six other problems arose

from miscellaneous reasons. Thus, most of the instructional and test devel—

opment activity undertaken by the authors occurred smoothly and without

undue difficulty .

2. Time. The blocks of instruction from the ENCOA course which were

put on—line are traditionally taught in 9 hours. The average completion

time was under 3 1/4 hours for this instruction including taking the

associated tests. (See Table 4)

The time to prepare the test items and lesson materials using the

INQUIRY system of author aids varied little from one author to another.

Times ranged from 35 hours for one NCO (15 hours on—line), 41 hours for

the Officer (14.5 hours on—line) , to 48.5 for the remaining NCO (18.5 hours

on—line).

3. ReadabilIty Index. This author aid provided information if the

reading grade level was surpassed for each text frame or test item . However , it was rarely used. That is, no matter what the index showed , the

authors chose to ignore it. About 220 text frames of instruction were

produced in this study. More then 50% (126) exceeded the pre—specified

reading levels. However, only 1 frame of instruction was revised by the

author as a result of this information. This was most likely due either

to a lack of confidence in the measure ’s validity , or to a lack of perception on the part of the authors regarding the criticality of reading leve l, or a combination of both . In any event , no changes have been made to this

aid yet. However, we believe that there are at least two possible changes

needed . First , authors should be given more instruction in the usefulness

of this aid together with more practice. Second , the options available in

the INQUIRY system to override this aid should be removed entirely or severely

constrained (i.e., within 1 grade level on either side of the pre—specified

one).

- 
~~~~-- - - - ~~~-—----~~ 

4. Multiple -Choice Items. As authors prepared multiple—choice

practice and test items , a pre—progranuned INQU IRY aid assigned the correct

answer alternative on a random basis. Authors were given the option to

change the designation of the correct answer alternative , and approximately

half the time exercised this option . Authors were , thus , indicating their

preference for retaining control over the manner in which they created

instruction .

Another author aid compared the lengths of answer alternatives and

indicated when they were unequal . This occurred in about half the items .

However , authors unanimously disregarded this information and left the

alternatives as they were. It appears that more restrictions on the author

aid are needed in order for these author aids to be used .

5. Constructed Response Items. In the constructed response format ,

authors used the following aids:

• The aid which permitted them to define the rigor with which

answers would be scored . Authors selected those options which permitted

misspellings , extra words , and optional capitalization . However, authors

did not permit the words in the answer to be Out of order.

• Authors made full use of the various aids available for

preparing response feedbacks and varied betw~

een providing trainees

specific as well as general feedbacks to both anticipated and unanticipated

answers . The most positive reaction by students was to the explanatory

feedbacks presented after each response to practice questions . The

INQUIRY author aids for presenting response feedbacks were used frequently

by the authors and , if possible , should be incorporated in off—line

instruction (e.g., using the guidance for preparing feedbacks in P1 texts).

• Authors were able to use the INQUIRY aids to specify anticipated

correct and incorrect answers . However , there appeared to be a problem

with anticipating all the answers which were given by the trainees.

The student attitude questionnaire data indicated a strong negative reaction —

to the constructed response questions provided by all the authors both

as practice and as test items . Student performance data supported this

result , as most difficulties were encountered when responding to constructed

response questions (both during learning and test taking). These results

appear to be due to those instances in which a “correct” answer as given by

the student is considered to be incorrect by the system .

The monitors had observed this problem as authors input their

material . The authors could not adequately anticipate all the synonymous

correct answers which could be given by the trainees. This problem is

particularly critical in CAl , as the evaluator “knows” whether the answer

is correct after seeing it. This finding suggests that the guidance for

preparing constructed response items be revised to clarify situations where • authors should or should not use constructed response questions . That is ,

• constructed response formats should be used only in cases where the number

of possible alternate correct answers is small.

41
I

6. Student Options. Of the student options , the ability to back up

to a previous frame (BACK) was considered helpful by almost all of the

trainees itt all three lessons. The other three options were : HELP——in

which the relevant learning objective was displayed ; LAB——in which the

student could return to the beginning of the lesson ; and DATA——in which

the student could go back to the beginning of the module. All three

options were rarely used , and so it was not surprising that students were

divided in their opinions about their necessity .

TASK 4. REVISIONS

Task 4 activities consisted of making revisions to: (1) flowcharts

developed in Task 1 and (2) author aids provided in Task 2. The purpose

of the revisions was to assure maximum utility of the flowcharts and author

aids in implementing the IPISD process. Revisions constituted a series

of activities which spanned almost the entire research period and paralleled

all development and evaluation actions in the other Tasks. Information

sources upon which revisions were based are as follows:

• As flowcharts were being developed , HumRRO personnel not directly

involved in the project provided input as to the clarity, utility and need

for revision.

• The expanded flowcharts developed in Task 1 were submitted to

instructional system designers at the USAES for review.

• Review of the Interim Report (which contains flowcharts) by the

COTR provided additional information for needed revisions.

• As on—line author aids were developed they were initially used by

HumRRO personnel to identify “bugs” in the aids which were corrected

before wider use was made of them.

• - The most important test of the utility of the flowcharts and author

aids occurred itt Task 3 when authors participating in the research effort

used the flowcharts and author aids f or developing instructional material.

Only minor “bugs” were identified at this stage since the flowcharts and

author aids had undergone extensive -re~

d~

ew and pre—testing . Any problems

encountered by the authors in using the flowcharts and author aids were

immediately corrected .

• The last information source for flowchart and author aid revision

• was to occur after the students had been administered the instructional

materials developed by the authors. A few such needed revisions were

identified as a result of difficulties students had with the instruction

that was directly connected with the author aid used for preparing the

instruction . Specifically , it was found that the author aids for preparing

constructed response practice and test items require additional developmental effort in guiding authors in the identification of what constitutes

a correct or incorrect answer. For example, the answer to a question might

be 820 meters. However , if the student answer ed 820 M (which should be an accep table answer), they were judged as having given an incorrect response .

42

_________ - ‘

.., 

~~~~~~~~~~~~~~~~~~ .~. ~~~~ ~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~ -—-. - -

~~
_ - _ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —.- 

a

As a result of the input received from USAES system designers and the

COTR, major formatting revisions were made to the flowcharts contained in

the Interim Report [88]. The revised flowcharts are shown in Appendices

B and C.

I

t

______
Chapter V

RECOMMEN DAT IONS

The evaluation of the author aids reported in the preceding chapter

has demonstrated the feasibility of on—line aids for implementing IPISD

Blocks 11.2 (Develop Tests) and 111.4 (Develop Instruction). User acceptance of the aids is high and the time required for creation of test and -

lesson material has been significantly reduced . Further developmental

effort of on—line author aids appears warranted . Continued development

effort should include five major areas which are discussed below . These

areas are :

1. Modification of selected current author aids developed in the

present project.

2. Development of additional author aids for IPISD Blocks 11.2

- and 111.4.

3. Development of author aids for other blocks of the IPISD model.

4. Conversion of aids presently programmed for PLATO IV CAl to

other systems.

5. Author Characteristics.

MODIFICATION OF CURRENT AUTHOR AIDS

There was insufficient time during the project to make all of the

modifications that were indicated during formative evaluation . These

modifications should be made if the lesson and test development author

aids are to be maximally effective . The specific author aids for which

we recommend modificaticn are:

(1) Reading Difficul ty Index. As was reported in the previous

chapter , experimental authors did not revise lesson or test material when

the material was written at a reading difficulty level in excess of that

intended. Hence, if reading level is critical , the author aid should be

modified to force authors to revise material when the reading level is

more than one grade above that desired.

(2) Author A id for Creating Constructed Response Questions. Authors

require additional guidance in determining how to use constructed response

questions appropriately . When constructed response questions are used ,

guidance is needed in the selection of the correct answers and alternate

forms of the correct answer (e.g., George Washington , Ceo. Washington ,

44
President George Washington , etc.). The need for this modification arises

from the level 3 evaluation . Students who were administered the experimental

lessons and tests had difficulty in answering constructed response questions.

Often , students would provide answers which were actually correct but with

an answer variation not anticipated by the author. Therefore , their answer

would be erroneously judged incorrect. When this happens , it is very

frustrating for the student and if it occurs frequently, it reduces the

perceived instructional value of the lesson to the student . To remedy this

situation , authors should be provided with detailed guidance on the use of

constructed response questions as well as guidance on the framing of correct

answer variations.

(3) Editing of Test and Lesson Material. With the present author aids

all editing must occur 2

~

J1 during the creation of text or questions. Once

material has been completed there is no provision for further editing.

This is a severe weakness of the present author aids. It is possible to

revise the author aids so as to permit text and question revision after

trial administration of the lesson . However , this is a major effort outside the scope of the current project.

ADDITIONAL AUTHOR AIDS FOR IPISD BLOCKS 11 .2 AND 111 .4

Although additional author aids could have been developed , this was

outt~ide the current scope of work . Additional aids which are desirable

include :

• Author Aid for creation of Matching Questions. (This aid is

presently in draft form.)

• Author Aid for creation of Arithmetic Manipulation Questions .

• Author Aid for creation of Multiple Choice Questions with more

than one correct answer.

CONVERSION OF INQUIRY AUTHORS AIDS TO A CA! SYSTEM OTHER THAN PLATO IV

As previously stated , the PLATO IV CAl system was considered to be a

research vehicle only. The goal was to develop and document on—line

• author aids that could be programmed on any CAl system . Therefore , a trial

conversion of at least selected author aids should be undertaken. This

undertaking would determine the extent to which author aids developed on

one CAl system could be converted to another CAl system and point out • difficulties to be expected in such a conversion . For example , rather than

using the TUTOR language , a system—independent language such as PLANIT

could be used as a test for the general usefulness of the on—line author

aids . Use of flowcharts which supported on—line aid development on PLATO

IV could be used for the basis for this effort .

- 

45
DEVELOPMENT OF AUTHOR AIDS FOR IPISD BLOCKS OTHER THAN 11.2 AND 111 .4

The present research has demonstrated , to a degree , t lit ’ utilit y of

on—lin e , query—based author aids in the implementation of IPISD . Howeve r,

before further work is initiated which is directed toward providing online’ author aids for all applicabl e Blocks of the IPISD mode , othe r research

is needed. Examp les of such research are discussed below . This study

does 81mw the benefit of flowchartlng to aid the Ii’ISI) process , and such

efforts should be undertaken for other tl ’E S D blocks.

AUTHOR CHARACTERISTICS

Authoring of CA! lessons requires a certain d iscip l i ne and l eve l of

competence which may not be present in all instructors assi gned to this

task. Aids are thus needed which constrain the author much more than was

done in INQUIRY , in order that useful guidance and techniques can be app lied

En creating effective Ins t ruction . The minimal prerequisites for authoring

both on— and off—i Inc materials need to be est ~b I (shed as well as the extent

to wit I cit aids can compensate for variable experienc e between personnel

if such a study ind icates that many individual proficiencies are lacking

and CSfliiOt be overcome by author aids , t hen a selection and class! ficat ton

prob~ in would have been uncovered and an assessment of “author” ob/dutv

posit Lou requirements is necessitated.

46

Li •

~~~~~~~~~~~~~ • ~~~~~~~~~~~~~~ 

__
__ -

RE FE RE NC ES

I. McFANN , R.H ., Bastes, J.A., and Taylor , J.F.. TRAINFIRE 1: A New

Course in Bas~ ç_Rifle Marksmanship,

~ Tec hnical Report 22. Alexandria ,

VA.: Human Resources Research Organ i zat i on, October 1955 .

2. HAIU€S, J.A. , K ell y, H.E., McFann , H.H., and Ward , J.S. TRAINF 1RE II:

A New Course in Basic Techn iques of Fire and Squad Tactics , Technical

Report 41. Alexandria , VA.: Human Resources Research Organi zation ,

July 1957.

3. BAKER, R.A. The Determination of Job Rcquirements for Tank Crew

Members , Technical Report 47. Alexandr ia , VA. : Human Resources

Research Organization , May 1958.

4. MacCASLIN , E.F. , Woodruf f , A .B., and Baker, R .A. An !mproved Advanced

Individual Training Prograni, Technical Report SQ . Alexandria , VA.:

Human Resources Research OrganizatIon , December 1959 .

5. Interservice Procedures for Instructional Systems Development. Phase 1:

Analyze. TRADOC Pamphlet 350—30, Augus t 1975.

6. Interservtce Procedures for Instruct ional System s Development. Phase ii:

Design . TRADOC Pamphlet 350—30, August 1975.

7. Interservice Procedures for instructional Systems Development . Phase’

III: Develop . TRADOC Pamphlet 350—30 , August 1975.

8. Interservice Procedures for Instructional Systems Development. Phase

IV and V: Implement and Control . TRADOC Pamphlet 350—30, August 1Q75 .

• 9. Interservice Procedures for Instructional Systems Development .

Executive Summary and Model. TRADOC Pamphlet 350—30, August 1Q75 .

10. U.S. Air Fo rce. Instruct ional S

~

y~~~ m Develqpmen!, AFM 50—2 . Washington ,

DC.: Department of the Air Force , i)ecemher 1970.

11. U.S. Army . ~ystems Rngin eertng of TraLn1i ~~. CONARC Regulations

350—100-1, February 1968.

12. SMITH, R.C ., Jr. The Eniineertng of F.ducat tonal and Tra 1nit~~ tests.

Lexington , MA.: D.C. Heath and Company , 1971

13. AmerIcan Airlines , Inc. Optimized Fligh t Crew Tra (ni1~

gL A Step Toward

Safer Operations. Fort Worth , TX.: Fligh t Training Academy , lQe’Q .

.47

~~~ ~~~~~~~~~~~~~~ .

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~. 

~~~~ — -

~ —
~ 

14. WYDRA , F.T. Learner Controlled Instruction: How Allied Supermarkets

Made It Work . Training, August 1975.

15. HAGER, R.F. Preparing Objectives for Programmed Instruction. San

Francisco : Fearon Publishers , March , 1962.

16. GLASER , R. Psychological Bases for Instructional Design. AV

Communication Review, 1966, 14, No. 4.

17. ANMERMAN , H.L., and Melching , W.H. The Derivation, Analysis, and

Classification of Instructional Objectives, Technical Report 66—4.

Alexandria , VA.: Human Resources Research Organization , May 1966.

18. KRATHWO -IL , D.R., Bloom , B.S., and Masia B.B. Taxonomy of Educational • Oblectives——The Classification of Education Goals——Handbook II:

Af fective Domain. New York : David McKay Company , Inc., March 1969.

19. BLOOM , B.S. (Ed.). Taxonomy of Educational Objectives——The Classification of Educational Goals——Handbook I: Cognitive Domain. New York :

David McKay Company , Inc., February 1968.

20. MELCHING, W.H. Behavioral Objectives and Individualization of

Instruction, Professional Paper 18—69. Alexandria , VA. : Human ResourceE

Research Organization, May 1969.

21. GAGNE , R.M. The Conditions of Learning . New York: Holt , Rinehart and

Winston, Inc., 1970.

22. ESBENSEN , T. Using Performance Objectives. Tallahassee : Florida

State University, Department of Education , April 1970.

23. BOND, N .A ., Jr., and Rigney, J.W. Specification of Training Objectives

for Computer—Aided Instruction. Los Angeles: University of Southern

California, Department of Psychology, June 1970.

24. BUTTS , D.P. Stating Instructional Objectives. First Experimental

Edition. Austin : The University of Texas, Research and Development

Center for Teacher Education, 1970.

25. JEANTHEAU , G.G. Handbook for Training System Evaluation. NAVTRADEVCEN

66—C—0l33—2. Orlando , FL.: U.S. Naval Training Device Center , 1972.

26. MICHELI, G.S. Analysis of the Transfer of Training, Substitution and

Fidelity of Simulation of Training Equipment. NAVTRAQUIPCEN TAEG

Report 2. Orlando , FL.: U.S. Naval EQuipment Center , 1972.

27. RUNDQUIST, E.A. Course Design and Redesign Manual for Job Training

Courses (First Edition). Research Report SRR 66—17 (revised).

San Diego: U.S. Naval Personnel Research Activity, January 1967.

28. BRABY , R., Michel i, C.S., Morris , C.L., Jr., and Okrask i , H.C. Staff

Study on Cos t and Tra ining Eff ect ivenes~ of Proposed Training Systems.

TAEG Report 1. Orlando , FL.: U.S. Naval Training Equipment Center , 1972

48

______ _______ 

-_— —
29. U.S. Navy . Design of Training Systems: Phase I Report, Volumes I

nd 2. TAEG Report No . 12—1. Orlando , FL.: U.S. Naval Training

Equipment Center , 1973.

30. 14DNTMERLO, M.D. Instructional Systems Development State-of—the—

Art and Directions for the Future. (Paper presented at 8th NTEC/

Industry Conference.) Orlando , FL.: U.S. Naval Training EQuipment Center , 1975.

31. RICKETSON, D., SChulz, E., & Wr ight , R.H. Review of the CONARC

Sy~

stems Engineering of Traini~~ Program and Its Implementation at

the United States Army Aviation School. Alexandria, VA. : Human

Resources Research Organization, April 1970.

32. GUSTAPSON , 1LW., Honsberger, W.D., and Michelson , S. Determination

of Task Analysis Content. Uses of Task Analysis in Deriving Training

*nd Training Equipment Requirements, WADD TR 60—593. Dayton, OH.:

Wr ight—Patterson AFB, Air Development Center , December 1960.

33. FOLLEY , J.D. Guidelines for Task Analysis. NAVTRADEVCEN 1218—2. Pt. Washington , NY.: US. Naval Training Device Center, June 1964.

34. RANKIN , W.C. Task Description and Analysis for Training System

Design. TAEG Technical Memorandum 74—2, July 1968.

35. HARTLEY , H.S. Twelve Hurdles to Clear Before you Take on Systems

Analysis. American School Board Journal, July 1968.

36. QUADE, E.S. ~ystems Analysis and Policy Planning: Applications in

Defense. NY.: American Elsevier Publishing Company, 1968.

37. KOBERG , D., and Bagnall, J. The Universal Traveler. Los Altos, CA.: William Kaufman, Inc., 1972.

38. VAN PELT, K.B., and Rich, J.J. Effective Writing for a Computerized

Training System. TRADOC Interim Report CTS—TR-75—l, January 1975.

39. AAGARD , J.A., and Braby R. Learning Guidelines and Algorithms for

Types of Learni

~~~Objectives. TAEG Report No. 23. Orlando , FL.:

March 1976.

40. LIPSON, J.I. Needed : A Collaborative Open University Network .

Productivity in Higher Education (Proceedings of Educational Technologies Symposium). Stony Brook, NY.: September 1973.

41. SIMONSEN, R.H., and Renshaw , K.S. CAl——B oon or Boondoggle. Datamation,

March 1974.

42. LOGAN, LS. An Exploratory Computer—Based Model for Comparing • Instructional Alternatives and a Bibliography of Aids for Course

Material Development: A Two Part Study (Volume II). St. Louis, MO .:

McDonnell Douglas , September.l976.

49 F ( __________ 

- — — 

.

~~ ~~~~~~~~~ ~~~~~~~~~ .

~~~~ ~~~ ~ - .

~~~ ~~~~~~~ ~~~~~~~~~~~~~~~ ~~~~~
__

__ -

~ - • • ---~.--- ------ ---.-- -•--~~~~~- -

~~~~~~ ~~~~~~~-••-~~~- - ---- - -~~ - - — - •

~~~~~~~~~

- 

I

43. HARLESS, W .G. et al. CASE : A Computer—Aided Simulation of the

Clinical Encounter. Journal of Medical Education, 1971, 46.

44. McKNIGHT, A.J. Establishing Performance Requirements. In An Experimental Program of Instruction on the Management of Traininj, Technical Report 70—9, by Donald Haggard, et al. Alexandria, VA.: RuRan Resources Research Organization, June 1970.

45. KURILOFF , A.H., Yoder , D., and Stone, C.H. Training Guide for Observation and Interviewing in Marine Corps Task Analysis. Training Manual III.

Los Angeles : California State University, 1975.

46. HAGER. R.F. Preparing Instructional Objectives. Palo Alto, CA.: Fearon Publishers , 1962.

47. PIPE, P. Practical Programming. NY.: Holt , Rinehart and Winston, 1966.

48. DRUMHELLER, S.J. Handbook of Curriculum Design for Individualized

Instruction. A Systems Approach. Englewood Cliffs, NJ.: Educational

Technology Publishers , 1971.

49. WONG , M.R., and Raulerson, J.D. A Guide to Systematic Instructional.

Design. Englewood Cliffs , NJ.: Educational Technology Publishers , 1974.

50. BRIGGS , L.J. Handbook of Procedures for the Design of Instruction.

Pittsburgh , PA.: American Institutes for Research, 1970.

51. MARKLE, S.M. A. Programmed Primer on Programming. NY.: Center for

Programmed Instruction, 1961.

52. Instructional Program Development Workshop Workbook. Great Neck, NY.:

Sperry Gyroscope (undated) .

53. ESPECH, J.E., and Williams , B. Developing Programmed Instructional

Materials. Palo Alto , CA.: Fearon Publishers , 1967.

54. HAWKRIDGE, D.G., Campeau , P.L., and Trickett , P.K. Preparing Evaluation

Reports: A Guide for Authors. Pittsburgh , PA.: American Institutes

for Research , 1970.

55. HILLELSOHN, M.J. Progress Report to Exxon Foundation——Project AUTHOR . Alexandria, VA .: Human Resources Research Organization, Ma rch 2 , 1976.

56. How to Prepare and Conduct Military Training . U.S. Army Field Manual

21— 6, November 1975.

57. COGAN , E.A. Case Study of Quality Control System. In An Experimental

Program of Instruction on the Management of Training, Technical Report 70—9, by D. Haggard, et al. Alexandria, VA.: Human Resources Research Organization, June 1970.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~ ~~~~~~~~~~~~~
58. SWEZEY, R.W., and Pearlstein , R.B. Guidebook for Developing Criterion—

Referenced Tests. Arlington, VA.: U.S. Army Research Institute for

Behavioral and Social Sciences, August 1975. —

59. VINEBERG , R., and Taylor , E.N. Manual for Developing Skill Qualification Tests. Alexandria , VA.: Human Resources Research Organization, January 1976.

60. RIRSHFELD, S.F., Young, D.L., and Maier , M.H. Procedures for Validating

Skill Qualification Tests. Arlington , VA.: U.S. Army Research Institute

for Behavioral and Social Sc iences, April 1976.

61. BUNDERSON, C. V. Team Production of Learner—Controlled Courseware: A

Progress Report. Provo , UT.: Brigham Young University, 1972.

62. The IMPACT Staff. Project IMPACT——Computer—Administered Instruction:

Preparing and Managing the Content of Instruction, IMPACT Text—Handling

Subsystem, Technical Report 71—21. Alexandria , VA.: Human Resources

Research Organization , September 1971.

63. WILLIS, L., Garneau , J., and Steizer, J. Project IMPACT Software

Documentation: V. File Activity Control System (FACS), Research

Product 51—72—4 . Alexandria , VA.: Human Resources Research Organiza—

tion, August 1972.

64. STELZER, J., and Garneau , J. Project IMPACT Software Documentation:

Overview of the Computer—Administered Instruction Subsystem. Technical.

Report 72—21. Alexandria, VA.: Human Resources Research Organization, August 1972.

65. WILLIS, L., and Stelzer , J. Project IMPACT Software Documentation: II.

The IMPACT Data Evaluation System--Version 2 (IDES—2). Research Product

D1—72—l . Alexnadria , VA.: Human Resources Research Organization, August 1972.

66. HILLELSOHN, M.J. Project IMPACT Courseware Subsystem: Volume I——

Innovative Procedures for Development and Administration. Technical

Report 74—1. Alexandria, VA.: Human Resources Research Organization, February 1974.

67. SCHULZ, R.E. MONIFOR~~ vs Authoring Aids for the PLATO IV CAl System.

• Technical Report 75—5 . Alexandria , VA.: Human Resources Research

Organization, May 1975.

68. SCHULZ, R.E. Lesson MONIFORM: An Authoring Aid for the PLATO IV CAl

System. Research Product ED—75—6 . Alexandria, VA.: Human Resources

Research Organization , April 1975.

69. STUFFLEBEAM. D.L., at al. Educational Evaluation and Decision Making .

Itaska , IL.: F.E. Peacock Publishers, Inc., 1971.

70. SCRIVEN, M. The Methodology of Evaluation . In Perspectives of Curriculum

Evaluation, by R. Tyler, R. Gagne, and H. Scriven. Chicago: Rand

McNally and Co., 1967.

51

~~~~ —~~~~~ - ~~~~~~~ ~~~~ ~~~~~~~~~ -~~~~
••- ~ • . —--- .

71. BORIcH, G.D. (Ed.) Evaluating Educational Programs and Product.

Englevood Cliff s, NJ.: Educational Technology Publications, 1974.

72. JOHNSON, C.H. The Purpose of Evaluation and the Role of the Evaluator.

Evaluation Research: Strategies and Methods. Pittsburgh , PA.:

American Institutes for Research, November 1970.

73. CRONBACH, L. Evaluation for Course Improvement. Teacher’s College

Record, - 1963,, 44(8) -

74. SMITH , R.G. Controlling the Quality of Training. Technical Report

65—6 . Alexandria, VA.: Human Resources Research Organization, June 1965.

75. BAKER, E.L., and Alkin, M.C. Formative Evaluation of Instructional

Development. AV Communication Review, Winter 1973.

76. STAKE, R.E. The Countenance of Educational Evaluation . Teacher’s

College Record, 1967, 65.

• 77. SCRIVEN, M. Pros and Cons About Goal-free Evaluation. Evaluation

Cousnent, 1972, 3(4).

• 78. SANDERS , J.R., and Cunningham, D.J. A Structure for Formative Evaluation

in Product Development. Review of Educational Research, 1973, 43,

79. RIPPE~ , R. (Ed.) Studies in Transactional Evaluation. Berkeley , CA. :

MeCutchan Publishing Corporation, 1973.

80. CHURCHMAN , D. et a].. The Theoretical Basis for Formative Evaluation.

(Paper presented at the annual meeting of AERA , Washington, DC , 1975.

81. SEIDEL, R.J. Transactional Evaluation: Assessing Human Interac tions

During Program Development (in preparation).

82. STUFFLEBEAM, D.L. Improving Educational Assessment: An Inventory of

Measures of Affective Behavior. Washington, DC.: National Education

Assoc iation, Association for Supervision and Curriculum Development, 1969.

83. CAMPBELL, D., and Stanley , J.C. Experimental and Quasi-Experimental

Designs for Research. Ghicago: Rand McNally, 1963.

84. MILLER, C.D. Comparative Effectiveness of Evaluation Designs.

(Revision of paper presented at American Psychological Association

Convention, ~~ntrea1, Canada , August 1973.

85. SJOGREN , D.P. Measurement Techniques in Evaluation . Review of

Educational Research, 1970 40.

86. JOHNSON, R.F. Needed : A Measure for Measure . Datamation, December 15,

1970.

52

imIIii~l
87. Interseriice Procedures for Instructional Systems Development.

TRADOC Pamphlet 350—30, August 1975.

88. HIBBITS, N., Wagner , H., and Schulz. R. Interim Report and Guide to the

Use of Flowcharts. Interim Report IR—ED—76—48. Alexandria, VA.:

Human Resources Research Organization, November 1976.

89. KINCAID, J.P. Making Technical Writing Readable (The Fog Count and an

Alternative). Human Factors Society Bulletin , 1972 , XV(5).

• 90. KLARE , G.R. The Measurement of Readability. Ames, 10.: Iowa State

University Press, 1963.

91. FLESCU, R.F. A New Readability Yardstick. Journal of Applied Psychology,

1948, 32(3).

92. TAYLOR, W. CLOZE Procedure: A New Tool for Measuring Readability .

Journalism Quarterly, 1953, 30.

93. MOWRY, H.S., Webb , W.B., and Garvin, E.A. Readability of Instructional

Materials and Methods for Analyzing. Research Report 816(02). St. Louis:

Washington University , Department of Psychology (undated). -

•

94. CAYLOR , J.A ., Sticht , T.G., Fox, L.C., and Ford , JP. Methodologies for

Determining Reading Requirements of Military Occupational Specialties.

Technical Report 73—5. Alexandria , VA.: Human Resources Research

Organization, March 1973.

95. STICHT, T.G. (Ed.) Readability of Job Materials. In Reading for Workiflg. Alexandria , VA.: Human Resources Research Organization, 1975.

96. KLARE , G.R., and Sina iko , H.W. The CLOZE Procedure: A Convenient

Readability Test for Training Materials and Translations. Arlington, VA.:

Institute for Defense Analysis , January 1971.

97. FATTU, N .A., and Standlee, L.S. Analysis of Reading Difficulty of

Selected Navy Materials. Indiana University: Institute of Educational

Research, March 1954.

53

— •-- ~--

~ ______ 

~~~~~~~~~~~~~~~ 

.• 

~~•
Appendix A

READABILITY LEVEL FORMULA

Figure the average length of a sentence in number of words. Figure the

average word length in number of letters.

1. Multiply the average sentence length by .5.

2. Multiply the average word length by 4.71.

3. Add the products of Steps 1 and 2 together.

4. Subtract 21.43 from the sum obtained in Step 3. This is the

readability level of the materials.

Here is the formula:

[ (.5 (verage sentence length)) + (4.71 (average word length)) 1 — 21.43

(from Kincaid, 1972)

A-l

-••------

~~~~~~~• ,••_—.

~~~ ,_. •. ~~~~~~~~~~~~~~~~ ~~~~ è-~—-~~ L~
Appendix B

FLOWCHARTS OF IPISD BLOCK 11.2 (DEVELOP TESTS)

AND BLOCK 111.4 (DEVELOP INSTRUCTION )

GUIDE TO THE USE OF FLOWCHARTS

The IPISD flowcharts for Blocks 11.2 and 111.4 (see Figures 2 and 3

in Chapter IV) were used as the basic framework for the HumRRO—developed

flowcharts. The flowcharts in this appendix expand each of the IPISD

flowchart blocks into detailed step—by—step components which must be performed (or considered) in completing the specific flowchart block. (In

the appendix IPISD flowchart blocks are noted by italics.)

Flowchart blocks which are shaded are blocks for which author aids

were developed in this project. Next to these shaded blocks are indicators specifying whether the aid is on—line and/or off—line . Blocks with

an asterisk (*) next to them indicate that existing author aids have been

identified . In these cases we provide a reference to the author aid that

is to be used at that specific point in the process of developing tests

and instruction. For some flowchart blocks supplementary information is

presented for clarification of a specific block’s activity statement.

For example , Figure B—l shows the first seven task elements required

to prepare multiple—choice test items . The total task elements can be

found on pages B-i thru B—9 of this Appendix.

The first task element in Figure B— i , Establish Testing Conditions

for Multiple—Choice Tests (2.2.1.2.2) is the task to be performed . The

task elements under this block must be performed , or at least considered

when preparing multiple—choice test items. For example , task element a,

“set readability level for test ,” is the first sub—task shown in the

figure . The asterisk (*) beside the block indicates the availability of

a non—HumRRO author aid . In this case , the readability level set is

contingent on the reading level of trainees. Since this block is shaded

it is identified as a block for which an author aid was developed in this

project. This aid is also designated as both an on—line and off—line author

aid . In the computer version of this author aid , the author is specifically

queried as to the reading difficulty desired for the entire test. Thereafter , all material input by the author is automatically checked to determine if the desired readability level has been exceeded .

Block b, “set minimum and maximum number of answer alternatives

including the correct answer ,” indicates that author aids were not developed . Block c , “determine if more than one answer is correct ,” has neither

an asterisk, nor is it shaded . This ind icates that no off—l ine author

aid has been identified and no aid was developed in this project for this

B—i
~

1

task element . The comment to the right of the block is intended to further

clarify the statement within the block.

Block d , “set time limits if any,” is shaded identifying it as requiring development of an author aid in this project. In the computer version

of this aid , authors have three options : (1) an untimed test , (2) time

limitation for individual test items, and (3) time limitation for entire

test . Again , the comment to the right of the block is for further clarifi—

cat ion.

Block e, “set conditions for test administration ,” has neither available

or developed author aids. The comment further clarifies the statement

within the block.

This completes the task elements identified as required for establishing

testing conditions for multiple—choice test items. The next major task to

be performed is the actual writing of the multiple—choice items (2.2.1.2.3).

The line coming out of the block indicates that in the actual flowcharts

t~ais task’s components are continued on subsequent pages.

The flowcharts in this appendix are on the pages listed below.

Block 11.2 (Develop Tests) —— Page B—4

Block 111.4 (Develop Instruction) —— B—35

B—2
- •--—---••• --••- •.•—.- —- —

Figure B—l. Flowchart for Establishing Test Conditions for Multiple-Choice Test Items.

Establish testing conditions for

multiple-choice tests.

2.2.1.2.2

* Set readability level for test. Contingent on reading level of trainees.

See: Peter Kincaid, Automated Readability Index, Human Factors

On line Society Bulletin, Vol. XV . No. 5, May 1972.

and

Off line a Author aid for reducing readability level

Set minimum and maximum

number of answer alternatives

including the correct answer.

b

Determine if more than one That is. is one (or morel answer alternatives correct , in addition

answer is correct, to correct answer?

C

Set tim. Hmib it any. Time limit for each item and entire test.

On line d

Set conditions for test Will all trainees take the same test? Will they alt be in the same room ?

administration. Will they be tested in shifts? (Some of these decisi ons are contingent

on available facilities and testing personnel.)

C

Writ. multlp(s.thoice items. Author aid for writing multiple.choice items.

On line • Author aid multiple’choice initial preparation worksheet.

and

Off line 2.2.1.2.3

* Non•HumRRO author aid. 

B—3

.. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~ - - ‘

~~ -~~~~~ --k ’ i

~~~J~~
11 .2 DEVELOP TESTS

Determine how detailed the test

thould be.

Develop test plan worksheet . What information t~

, u

~

e for this activity. (See Swezey and Pearlstein, p. 3-33).

2.1.0.1

Will Base decisions on resource teSt 

* Sample objectives at random, availability, time con- include all 

Keep selection process secret straints, and criticality! mi oh ectives Sample

to examinees. Importance of each obj ec- or a sample of 

tive. (Guidance on how to LO’s? 

2.1.0.2.1 sample objectives is in

2. 1.0.2 Swezey arid Pearlstein , p. 3-6)

All

Translate TLO’s and LO’s into TLO defines objective of test.LO’s test items, are transla ted into tests (or test items).

Read over action, condition, and

standard for each T1O. (Obtain

from Learning Objective Analysis

Worksheet.)

2.2.0.1

Specif y whether high or low item Base decisions on resource availability, time

fidelity is required, constraints, and criticality/importance of

each objective.

2.2.0.2

*~~ n HumRRO author aid. B—4

— --~-.. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -,
2.2

Identify

NO 

ln~~~~~ on 

NO 

Physical Skills 

NO

Mental Skills 7

2.2.0.3

Yes Yes Yes

2.2. 1 2.2.2 2.2.3

(Jeasuring) 

NO Consult examples below to _____

Attitudes choose learning category.

7

Yes

2.2.4

‘ LEARNING CATEGORIES

Mental Skills . Skills such as:

Problem solving Mapreeding

Concept format ion Comput er programming

DecIsion making

TestIng Inform ation—K now ing standa rd operating procedures , filling Out clerical forms

Physical Skills—Psirceptua l motor skills , such as typing, target shooting

Testing Attitudes—Cooperation, dedication, helpfuln ss, leadership

— —-——---

~~~~~——-— — -fl—-
-V - --- 

~

—

~~~~~

- - - I

Testing Mental Skills

2.2.1

Select type of test

Paper N Oral N Behavioral N and Pencil Test Performance

Test ? Test

2.2.1.1 ? 7

Y V V

2.2.1.3 2.2.1 4 Test Y

in CAl 2.2.1.2b

mode

Develop Paper and Pencil Test 7

N

2.2.1.2a

Select Item Format 

~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~

M t hi N Constructed V CompletIon! N E N a c ng R.~~nw List ay

On line On line

Off line Off line

V N V V

2.2.1.2.8 2.2.1.2.1 2.2.1.2.8 2.2.1.2.10 2.2.1,2.1

B—b

- - 

- ~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ 

, . — ---~~~~~
~~~

--

Establish testing condition s for

mul t iple~

choic e tests.

2.2.1.2.2

* Set readabili ty level for test. Contingent on reading level of trainees.

See: Peter Kincaid, Automated Readability Index , Human Factots

On line Society Bulletin , Vol. XV , No. 5, May 1972.

and

Of f line a Author aid for reducing readability level.

Set minimum and maximum

number of answer alternatives

including the correct answer.

b

Determine if more than one That is, is one (or more) answer alternatives correct , in

answe r is correct , addition to correct answer ?

C

Set time limits if any. ‘Time limit for each item and entire test.

On line d

I

Set conditions for test Will all trainees take the same test? Will they all be in the same room?

administration. Will they be tested in shifts? (Some of these decisions are contingent

on available facilities and testing personnel.)

C

Writs multiple-choice Items. Author eid for writing multiple-choice items.

Author aid multiple-choice initial preparation worksheet.

On line

and I

Off line 2.2.1.2.3

IL.~~
2.2.1.2.3

Make sure that item tests one

objective only.

a

Make alternatives similar in length

to correct ans~~r.

On line b

* Mike sure that readability level of Use on’line or off-line readability aid. Of f’Iine formula is at end

stem is not surpassed. of flowchart.

On line Author aid on reducing readability level.

and Peter Kincaid, Automated Readability Index , Human Factors

Off line c Society Bulletin. Vol. XV , No. 5, May 1972.

Check questions for grammatical Errors or phrasing of question might inadvertently indicate

errors, correct answer.

d

Avoid negatives in item stem. Negatives are confusing to test takers . State items positively.

Avoid using “none of the above”

as an alternative. —

e

Make sure that distractors Do not make distractors overly technical.

(options other than correct answer)

are plausible.

It only one alternative is correct,

make sure that it is unequivocally

correct .

9

I 8-8

Non’HumRRO author aid,

L __________ ___________________________ •
-- - -~~~~~~~~~~~~~~~~~~~~~~~~~~-

~~~~~~~~~ - -~~---—----- ~~~~—~~~ - -

~~~~-—-~~—-

2.2.1.2.3

Select position of correct ans~~r Chosen at random at author ’s option.

alternative. Aut hor aid question scoring form for recording correct answer

position and number of alternatives.

On line h

Consult subject matter experts They will check item fidelity and correctness.

or peers for review of items.

Consult test exp erts for final They will point out possible poor construction of items (and

review of test. weak or ambiguous items).

8-9

~~~~~~

—.. .-.- — ——— - . .. —-V .. :‘— ~ ~ ~~ ~~~ -

. “ 

~~ 

—“—

——~~~~~

——-‘ 

~~~~~~~~~~~~~~~~~~~~~~~~~

—V.- ~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ —— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~ ~~~~~~~~~~~~~~~~~~~~ _

~~~~~~~ .. ~~~~~ ..

~

. ,,,4
— — -.—— —

Establish testing conditions

for true/false test.

2.2.1.2.4

* Set readability level for test. Conting ent on reading level of trainees.

See: Peter Kincaid, Automated Readability Index, Human Factors

On tine Society Bulletin , Vol. XV , No. 5. May 1972.

and -

Off line - a Author aid for reducing readabili ty level .

Determine number or percent Will the items be 50% true and 50% false or - tr proportions?

of items to be true (and

false).

b

Set time limits, if any. Time limit for each item or entire test.

On line c

Set conditions for test Will all trainees take the same test? Will they all be in the same room?

administration. Will they be tested in shifts? (Some of these decisions are contingent

on available facilities and testing personnel.)

d

$ Write true/false Items. Author aid for writing true/false items.

Author aid true/false initial preparation worksheet.

On line

and

Off line 2.2.1.2.5

Make sure item tests one

ob~sctwe only.

a

Non~HumRRO author aid. B 10

t
~ -

2.2.1.2.5

* Mike sure readability levil Is not See: Peter Kincaid , Automated Readability Index , Human Factors

surpassed In item stem. Society Bulletin , Vol. XV , No. 5, May 1972.

On line

and Author aid for reducing readability level.

Off line b

Check item for grammatical

errors.

C

* Paraphrase material for test items; See Stevens and O’Neil (November 1974) for guidance

do not lift material straight from and examples.

text.

d

Avoid ambiguous and indefinite

terms (such as sometimes). Also

avoid use of negat ives and

negatively worded stems.

e

Keep true and false state ments

equal in length.

Be sure that item can be

categorized unequivocally true

or false.

9

I

Consult subject-matter experts They will check test for fidelity and correctness.

or peers for review of test.

h

Non-HumRRO author aid. B—il

~~I. - __ ___ 

_ _

__ _________________ 

- 

-. _

~,r~

_

~

’ ~~~~~~ ~~~~~~ —— 

~~~~~~~~~~ ~~~~~~~~~~~~~~
p. -C

2.2.1.2.5

-I

Consult test expert for final They will point out possible bad construction of items (and review of test. look for weak or ambiguous items).

Establ ish testing conditions for

matching tests.

2.2.1.2.5 —

* Set readability level for test. Contingent on reading level of trainees.

Author aid for reducing readability level.

See: Peter Kinc aid , Automated Readability Index, Human Factors

Off line a Society Bulletin, Vol. XV , No. 5, May 1972.

Set number of elements for

each column.

b

1

Non.

HumRRO author ~ aid.

6-12
2.2.1.2.6

Determine if more than one

element is to be paired correctly

with other column element(s).

C

Set tim. limits , if any.

d

Set conditions for test Will all trainees take test in same room? Will all trainees take same

administration, test? Some of these decisions are conti ngent on availab le facilities

and testing personnel.

0

Write matching test items.

- 

- 

2.2.1.2.7

Make sure item tests one

• objective only.

a

I

* Make sure readability level Is Aut hor aid for reducing readability level.

not surpassed.

See: Peter Kincaid, Automated Readability I ndex , Human Factors

Society Bulletin , Vol. XV , No. 5, May 1972.

Off line b

Make elements in each column Column items should be of similar category (e.g., nouns, verbs).

similar In length and type and

as short as possi~le.

J 

* 

Tm 0

11~ 

B~ l3
-S.---- _-

~~--_ _ - — .-•-. _SS• S — -_ ._ _

~-•_-.—_•,-__•__•____S._. ~~~~~~~~~~ ~~~~ •S~_-S-S~~ ~~~~~~~~~~~~ s. _._- ----S.--. . ~~~~~~~~~~~~~~~~~~~

2.2.1.2.7

Label each column.

d

Make columns with unequal This is so answers cannot be found by elimination. • numbers of elements.

0

Consult subject•matter experts They will check test for accuracy and fidelity.

or peers for review of tests.

• Consult test expert for final Ambi guous items or those of poor construction will be found

review of test. in thi; review,

8—14

—~ ~~~~ __________
_ - — --- ~~ -- -r- ’-— -

~~~~~~~~~~~~~~~~~ 

•. -- -

Establish testing conditions for

Constructed Response Tests using

completion list items.

2.2.1.2.8

* Sit readability level for test: Author aid for reducing readability level.

On line See: Peter Kincaid, Automated Readability Index, Human Factors

and Society Bulletin, Vol. XV . No. 5, May 1972.

Off line a

Determine whittier more then one Author aid for preparation of constructed response items.

answer Is correct end whether Author aid of examples of answer alternat ives.

On line answers can be partially correct.

and 

Determine it there are anticipated

Off line 

wrongansuers. b

Establish format for answering Author aid constructed response initial preparation worksheet.

questions.

On line

and

Off line C

Establish level of hints to be If listing, number of items required? What part of speech

given trainee, is missing?

d

Sat time limits, if any. Time limits for each item or entire test.

On line a

Set conditions for test Will all trainees take test in same room? At same time?

administration. Will they take same test? Some of these decisions are

contingent on facilities and available testing personnel.

Non~

HumRRO author aid. 8—15

,

. ‘

~ 5

~4~ ~~~~~~~~ - ~~~ “ • .

~

~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~ _

~~~~~~~~ ~~~~~~~~~~~~~~~~~~ — ~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —
—. --—-—~~

- -- --• - .—- — -—S. 

~~~~~~~~~~~~ ~~~~~~~~~

.-

2.2.1.2.8

Writs completion/lIst items for Author aid for preparation of constructed response items.

Constructed Response Tests.

On line

and

Off line 2.2.1.2.9

Make sure item tests one

objective only.

a

* Mike sure readability levl of Author aid for reducing readability level.

item is not surpassed.

On line See: Peter Kincaid, Automated Readability Index , Human Factors

and Society Bulletin. Vol. XV , No. 5, May 1972.

Off line b

Check items for grammatical If fiII’.n.blank format used, is question understandabl e

errors. with blank?

C

If a numerical response is required,

specify units to be used.

d

I

In completion, omit only

key words.

e

List possible correct answers. Author aid for prepar ation of constructed response items.

List enticipalad incorrect Author aid of examples of answer alternatives.

On line ssnwars.

and

Off line t

Non-HumBRO author aid.

~~ ~~~~~ . 

- i. ,. ~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~

‘

~~~

‘ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~
-

~~~~~~~~~

--- ~~~~~~

-

-~~ --.-

~~ ~~~~~~~~~~~~~ --—- -

~~~~~~~~~~~ - -

2.2.1.2.9

Consult subject’matter expert or They will look for fidelity and accuracy and may add other

peers for review of test. correct answer possibilities.

g

Consult text expert for final This review will pinpoint poorly constructed items or

review of test. ambiguous items.

H .

Establish Testing Conditions for

Constructed Response Tests

Using Essay Items.

2.2. 1.2. 10

* Set rs dsbllity level for test. Author aid for reducing readability level.

On line See: Peter Kincaid, Automated Readability Index, Human Factors

and Society Bulletin. Vol. XV . No. 5, May 1972.

Off line a

Determine it all items are to be Specify number of alternatives to be answered.

used or if a choice for answering

is provided.

b

Non~

HumRRO author aid. 8—17

I

I.. ~~~~~~~ - _______ --- — — --.———.~

—.p ~~~~~~~~~~~~~ ~~~~~~ .‘ ~~~~~~~ 

- .- - ~~~ S -

~~~~~

.— 

- ~~~~~~~~~~~~~~~~~~~~~~~ - 

~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~ 

- -
2. 2. 1. 2. 10

Determine if more than one answer

can be correct and if answers can

be partially correct.

I

C

Set limit for number of words

to be written for each item ,

if any.

d

— I

Determine if test answer will be

scored for grammar, spelling and

punctuation.

e

Set time limits , if any.

Set conditions for test Will all trainees take the same test? In the same room’ In

administration, shifts? Some of these decisions will depend on available

- 

facilities and testing personnel.

9

Write Essay Type Items for

Constructed Response Tests.

2.2.1.2.11

Make sure item tests one objective only.

a 

..
2.2.1,2.11

* Mike sure readability level of item Author aid for reducing readabili ty level.

is not surpassed.

On line See: Peter Kincaid, Automated Readability Index , Human Factors

and Society Bulletin, Vol. XV , No. 5, May 1972.

Off line b

Be sure item is phrased clearly.

Start item with action verb, such

as “Explain.”

C

Prepare sample correct answer and Observe time it takes to Construct answer.

acceptable alternatives, if any.

d

Consult subject-matter experts They ,theck test for fidelity and accuracy ,

or peers for review of test.

e

Consult test expert for final This check is for poorly constructed items and ambiguous items.

review of test.

I 1

Non’HumRRO author aid.

B-19

__ - - - - - - - S .. ~~,

~~ — S._____-i_ - - — — — —-— .

~~ ,—- S.— -

~~~~~~~
I

Develop Oral Test

2.2.1.3

Establish Testing Conditions for

Oral Tests.

a

Set conditions for test Will all trainees take identical tests? How will facilities

administration, be used?

al =

Determine if more than one Author aid of examples of answer alternatives.

answer is correct and if there

will be partially correct answers.

On line a2

Determine whether administrator’s questions are written or

oral.

a3

Determine whether correct Will verbal ability be part of answer evaluation?

answer is a phrase, word, or

exposition.

a4

Establish level of hints to be given Are number of components in answer to be delineated? (For

in items, example, explain the three parts of an experiment, etc.)

a5

8—20
- - --~~~~~~~~~~ -— -

~~~~--- S.S.- - —— -- --- -— — --— - ~~~~~~~~~~~~~~~~~~~~~~ -~~

2.2.1.3

Write Oral Test Items. Author aids for preparation of multiple’choice, true/false and

constructed response items.

On line

and

Off line b

Choose item format. Review 2.2.1.2.3, .5, .7, .9, and .11 to choose best format.

bi

Make sure item tests one

objective only.

b2

Make sure readability level is

not surpassed.

On line

and

Off line b3

Answer items to measure Check with time allotted for test.

time required to answer.

b4

List all correct answers. Author aid of examples of answer alternatives.

On line bS

Consult subject-matter experts They check for fidelity and accuracy and may add other correct

or peers for review of test. answer possibilities.

b6

‘Non’HumRRO author aid. 8—21

__

L - - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_i

~~~~i~

— —.--.--——-—-----—

~~~~~~~ —-.-—-,--~~~~~~~~

,, —‘--S. — 

~~~~~~~~~~~~~~~~~~~~~~~~~ 

—-S.

2.2.1.3

Consult test expert for fina l This will highlight ambiguous and poorly constructed items.

review of test.

I

Develop Behavioral Performance

Test.

2.2.1.4

IEstablish testing conditions for

behavioral performance tests.

a

Determine length of each

exercise.

4

al

P Identify facilities to be Find out if any restrictions are imposed on use of facilities

utilized, and personnel.

a2

8—22
- --—---—---——--.-—- - —-,- - -

2.2.1.4

r Identif y personnel to be Set up schedule for use of ~ ecif Ic areas and schedule for personnel

utilized, involved in testing.

a3

Determine sequence of exercise. “County Fair” type testing environment might be used .

a4

Determine if performance is If process, the behavior will have sub.components

process or product.

aS

Establish use and level of hints

to be provided during testing.

a6

Prepare Behavioral Performance

Test Exercises, Descriptions.

b

Describe the performance(s) Include time restrictions and facilities to be used in descriptions.

required for each objective. Describe locations and duties of testing personnel in description.

bl

Describe steps for correct See Vineberg and Taylor, 1975.

performance if process is to be

meesured -

b2

B—23

-- :- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ‘——~~~~ 

-— 

~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~
- ‘—~~ -~~~~~~ ——S.—- — ~~~~S._ -

2.2.1.4

Describe attributes of acceptable

finished product if product is to

be measured.

b3

Perform your own exercises to

check out facilities, timing and

personnel needed.

b4

Describe acceptab le hints to be

given if hints are allowed.

b5

Develop checklist of behaviors This will be necessary for scoring later on.

to be performed.

b6

_________

Consult subject.matter experts They will check for fidelity and accuracy.

or peers for review of tests. — -

I j

Consult test expert for final This check will illuminate ambiguous or poorly constructed

review of test . exercises .

‘

I

B-24

- S. ~~~~~~ — —.-‘~~~~ .

~ , ~~~~~~~~ — •.—~-—-—.-,— —-

-,

—- ~~~~~~~~~~~~~~ .~~~~~~~-
--~~~~--

Write Test Instructions. Test

= in CIA V 

~ 

Author aid for preparing

2.2.1.8 

mode - test instructions.

Prepare instructions for

administrator(s) of test.

a

Prepare initial and within test Check readability level of instructions to guarantee that reading

Instructions for trainees level has not been surpassed.

On line taking test. Author aid for reducing readability level.

and Author aid for developing test instructions.

Off line b

B—25

~~~~~~ 

i

~;, - - - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —
- —--S. -

- S. -

Testing /nferm%1,

2.2.2

~~

termj item format 

Item fo rmats are the same as th

~~ in the Mental ~~iII$ section

I 

The types of mental 

~~ilIs tested in this cat~ ory are recall and recognition The best formats for testing informetjo~ are multi.

I 

pie choice and matchj~~ (for recognj~j 0~) ~

nd a completion type of constructed response (for recafi)

2212

See behav~

orai performance test develop~~~~ actIvities

- 

8-26 L 

~ - - 

- . 

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~

,—

-

~~~~ ~~~ -

- — -‘——~~~~— —- -,-- S. 
— -S.- — - - S. 

- — -S.-- -‘ -

Tasting (Measuring) Attitudes

2.2.4

Determine whether attitudes are Check section These measures

to be measured by observation ~~~~~~~~~ Solicitation 2.2.1 2 — which (rwnees either

or by solicitation from trainees. - ~

) 

.

/_

__ 

especially comple- “agree” or ”disagree”

a \

\ ~,

/ ‘ tion & true/false. “dislike), 

or

Determine scaling techniques

to be used .

al

* Select measuring technique

for attitudes to be observed. A good Source for choosing observation instruments is Mirrors

for Behavior, edited by Anita Simon and E. Gil Boyer .

b

Describe behaviors which

demonstrate given attitudes.

bi

Establish rating system for Determine if trainee will be rated by one or more raters. administrator(s) to use. Create rating sheets for raters to use.

b2

2.2.1.8

\.,..., ,..“ Non’HumRRO author aid.

8—27

~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ £ ~~~~~~~~~~~ -~~~~~
S. — ~~~~~~~~~~~~~~~~ 

— 

S. —

Set Training Standards

2.3

Determine it test is for end of

training or within training.

a

Determine level of proficiency Setting of criteria is sometimes determined by reference

warranted within training, to job requirements or consensus of “experts.”

________________

Decide whether performance Somet imes trainees are required to “overlearn” so that decay

at end of training should equal of learning on the job is not detrimental to performance.

or surpass the job performance

measure.

I

C

Establish criteria for trainee Decide on number of LO’s to be met by trainee.

performance of LO’s,

d

I —

False Positives and False Negatives.

2,3.1

Perform validity check. Validity shows discrimination between masters (those who are “go”)

and non-masters (those who are “no-go”). See formula at end of

flowchart.

a

8—28

S. ~~ -- - — - •,

~ %~ S.- S. - - i_nh

-

‘ - :- - , ~~,
-S.— -- w__S. ----S.---

-

2.3.1

Decide on pitfalls of either Decide which type of false situation is more critical and

false positives or false negatives, which can be tolerated.

b

Rank Order Students (if required).

2.4

List scores from highest to Develop form for listing purposes.

lowest.

a

Provide identical T anks to trainees

having identical scores.

b

Set Cut-Off Scores Only if required.

2.5

Review training standards as

established (in 2.3) to set

cut-off scores.

a

Set cut off scores, recognizing Swezey says, “If the cost of a false positive (passing an

probability of false positives and incompetent man) is very high, the cut-off point should be

L false negatives. 

b 

set very high.” (p. 6’13)

- ~,

; B-29

I, -~~~-- - 

- 

~—~~~P~ z ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

--
2.5

Lower cut-off scores if required

by manpower needs or criticality

of task.

C

Determine Scoring Procedures

Qualitative Quant.

or 2.6.2

quantitative

2.6

Qual.

Determine qualitative scoring Decide what effects the scoring can have on trainee comparisons.

proced ures.

a

Determine whether partially

correct responses will be

evaluated.

al

Determine scheme for assessing Especially applicable in constructed response (list ing) format.

partially correct answers.

a2

Determine whether presentation Oral tests and constructed response—essay.

of response, grammar, spelling or

punctuation will affect response.

a3

8—30

S. - - - , . - - ~~ ~~~~~~ ~~ - - - - --—— - - -

- 

___ .

~~
— ~~~~~ -‘

2.6

Outline scoring guidelines for Checklist scoring is applicable to “process” types of

behavioral checklists or scales. performance .

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Determine number of points Author aid for scoring test items.

for each response.

On line

and

Off line a

Determine if responses are to be Could be used for multiple-choice or constructed response, — rank-ordered, where some responses are considered “more correct” than

others. Weighted responses?

b

Establish penalties for Esp ecially applicable in lure/False, or Multiple-Choice where

incorrect responses, probability of guessing correctly is high. Could also be used

On line in matching format. Formula or correction for guessing.

and Author aid for Scoring test items.

Off line - C

B-31

—S. 

‘

~-~~~..~~~-‘----- —~~ - —~~~~~~~,
-S. - -- -- S.S.S. ~

2.6.2

Determine whether partially cor- Author aid for scoring test items.

rect responses will be evaluated. Author aid of examples of answer alternatives.

On line

and

Off line d

Determine scheme for assessing Especially applicable in constructed response (listing) format.

partially correct answers . Author aid for scoring test items.

On line Author aid of examples of answer alternatives.

and

Off line e

Outline scoring guidelines for

behavioral checklists and attitude scales (if used).

f

Determine when scoring will Will interference scoring be used? See Swezey, p. 6-6.

occur, after or during test. Go/No-Go type?

2.6.3

Determine whether scoring will Develop form for recording correct and incorrect answers of

be done by hand or machine, class and individuals. Develop method for determination of

number of correct answers (or a scoring key).

2.6.4

Write scoring directions.

2.6.5

* Perform scorIng procedures Author aid for reducing readability level.

tryout. Perform readability

check, See: Peter Kincaid, Automated Readability Index , Human Factors

, Society Bulletin, Vol. XV , No. 5, May 1972.

2.6.6

Non’HumRRO author aid.

B-32

- 

, -S.- 

~~~~~~~ 

•

~S.S. - - —u- ~~~~~~~~ ‘

~~~~ -‘-q .

~ . —~ -~~ — e. ~~~~~~ -, ~~- -. -.-_ - - - -

~~~~~ 

—S. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~ 

- -, ~~~~~~~~~~ — -- -- ,~ — ~~~~~~~ ~~~~.
2.6

Revise procedures as indicated

in tryout.

2,6.7

Collect Baseline Data.

— 2.7

Decide how data are to be Usually not test developer’s responsibility.

stored , hand or machine file.

a

Write instructions for storage

of data.

b

Write instructions for gathering

background data.

C

Collect background and train- Develop form for intermediate storage of data.

ing data for each trainee.

d

B—33

_____ -- ~

~.
REFERENCES

HA1RIS, James H., Campbell , Roy C., Osborn , William, and Boldovici, John. Development of a Model Job Performance Test for a Combat

Occupational Specialty. Volume I. Test Development; Volume II.

Instructions and Procedures for Conductjn~

g a Functionally Integrated

Performance Test, Final Report FR—CD(L)075--6. Alexandria, VA:

Human Resources Research Organ ization , November 1975 .

KINCAI D , J. Peter. Automated Readability Index, H uman Fac tors Society

Bulletin, Vol. XV, No. 5, May 1972.

OSBORN , William and Ford , J.P. Research on Methods of Synthetic

Performance Testing, Final Report FR—CD(L)—76—l. Alexandria, VA:

Human Resources Research Organization, April 1976.

SIMON , Anita and Boyer , E. Gil (Eds.). Mirrors for Behavior II: An

Anthology of Observation Instruments, Vol. B. Philadelphia:

Research f or BetterSchools , Inc., Spring 1970.

STEVENS , Jane Close and O’Neil, Harold P., Jr. Suggestions for

Generating Test Items at the Comprehension, App

~

lication,_and Analysis

Levels. Austin, TX: University of Texas, November 1974.

STEVENS , Jane Close and O’Neil, Harold F., Jr. Some Guidelines for

Development and Review of Multiple—Choice Questions. Austin, TX:

University of Texas , May 7, 1974.

SWEZEY , Robert W., and Pearlatein , Richard B. Guidebook for Developing

Criterion—Referenced Tests. Arlington, VA: U.S. Army Research

Ins titute f or the Behavioral and Social Sciences , August 1975.

TNORNDIKE , Robert L., and Hagen, Elizabeth. Measurement and Evaluation

in Psychology and Education. New York: John Wiley and Sons, 1955.

TRORNDIICE , Robert L. (Ed.). Educational Measurement (second edition). Washing ton, DC: American Council on Ed uca tion, 1971.

VINEBERG, Robert and Taylor, Elaine. Performance Teat Deve1op~

ment f or

Skill Qualification Testing, Final Report FR—WD—75—5. Alexandria, VA: Human Resources Research Organization , June 1975.

-. ~

. :

. ~~~~~~~~~~~~~~~~ ~~~~~~~~ ,

~~~ ~~~~~~~~~ — ~~~~~~—
1

111.4 DEVELOP INSTRUCTION

1.0 Introduction

2.0 Pmc.du,,s

k~

.ntify ingtru

~~,n.I needs

(

~~and 

~~nst~ in~~ 

_ -

flumber of trainees 1

Determin, entry characterjs~

jC~ E.g., reading comprehension level. L isv.: to lnstruct

ol

[Identify budg.t restraints for

instructlon l development.

C

Identify tim. constraints for

[

tructional T:~

ent.]

~

B—35

_ _-
—,

2.1

I

Identify scheduling constraints.

I

D.t.rmine fidelity required during See LOAW

instr uction .

B-36
- .—,.— --.

——. . 

~~~~~~~~~~~~~~~~~~~~~~~ ~ 

—— -———- . -—-- ..

.———.-— -

~

—-.-

~

.-—.—--—. -— .

-.,-- —. -,.—,——..-—,.——.—. . -..— ------ -, 

~~~~~~~ ~~~~~ ~~

‘ .

—- —. .

.-

~

————.. —‘—.,—

I 

______________

Id.nti’f

~ avaiioble reeourvea.

2.2

Identify personnel knowledgeable

In various mod.simarli. of

instruction.

a

Prepare roster of coworker s

(piers) available for instruction.

ii

Prepare roster of available,

qualified instructors,

.2

Describe and document List all lIbrary/reference rooms, study rooms, and equipment;

available facilities . classoom and large lecture halls; practic, rooms/laboratories,

and equipment.

- ____ 1. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :

~~~~ . ~~~~~~~~
—

~~~~~~~~~~~~~~~~~

Develop Iflstruct,O,,

2.3

Prepare outline of TLO’s and LO’s

to be covered by instruction.

a

Determine mod. of instruction Base decisions on criticality of task , fidelity required, equipment

f or each section of course, availability, personnel availability.

Will learning be self-paced (learner controlled), tutorial, smallgroup or large group discussion, or teacher oriented?

b

Determine InstructIonal sequence. Author aId for preparing learni ng obg ectives ,

Author aid for sequencing of materials.

On line

and

Oft line C

l

~ ntIty locations within coons Author aid for preparing practice frames.

wiwse there will be practice.

On line

and

Off line d

Select first mode of instruction

for development.

dl

CHOOSE FROM LIST: 2.3.7 ~~lf ,each~

nge~porrabl

2.3. 1 Audio script. p.ckag.e (STEPS). Go to block explaining mode 2.3.2 video materials. 2.3.8 Suppl montaey instruction.

selected. 2.3.3 Audiovisual materials. 2.3.9 Adjunct pcop ms.

2.3.4 Pnnt.d materials. 2.3. 10 Job perlormanc, aiab (.N’As) . 2.3.5 Progr.mm& instruction. 2,3.?? Formal OJT. - 2.3.6 Platform lectures. 2.3.12 Other forms of mediated d2 instruction.

2.3.13 CAl.

B-38

_________________ —.- -—~~ . 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
________ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~-~--~ - - - - - - -

Writing an audio script only.

2.3.1

F

Identify section s of course for use Review TLO’s arid LO’s.

of audio in conjunction with other

t media.

a

= * Identify comprehension level See Brown , J .l. , and Carlssn, G,R. Brown-Carlsen Listening

of trainees. Comprehension Test . New York: Harcourt , Brace and

World, 1955.

b

.

Using outline, prepare plan

for script.

C

Further outline TLO’s and LO’s

pertaining to this section.

ci

__________

Plan use of audio cues, music ,

- voices and combinat ions of these.

c2

I I

Set up schedule f or script

-- development.

‘Non HumA RO author aid. ~~39
___________________________________ - r

~~~~~~~~~~’— - -— ——-.------- - 1

2.3.1

Establish time frame for

development. ‘

dl

I

Meet w ith audio equipment

personnel to set up recording

schedule,

d2

Write script from outlines,

a

I

Use two-sided script for m for Left side should contain speciil instruct ions fo. cues to music , other

~~ iting content . sounds; right srde contau ~

s actual ectt pt and directions to narrator

.1

Indicate pauses and uses of

other media on script form.

.2

Have script reviewed,

I

Give script to peer for review .

ft

I

B—40

- ,

.- —~~~~~ -- rn— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ £

~~~~t5&,
2.3.1

Meet with prod ucer to review

script.

f2

Revise scri pt if necessary to

change puce, clarif y sections, etc.

g

Prepar, required number of

copies of script for production

purposes.

h

PMke recording of script .

a—-

I

B-41
_________________________ 

Preparing video’only neatetrials.

2.3.2

I

Identify sections of course for Review TLO’s and LO ’s.

use of video in conjunction with

other media.

a

Prepare outline of instructiona l

sections applicable to video

medium.

b

Sketch or describe visual s

to be presented on storyb oards.

bl

Specify details of each Include special effects , lettering, shading, amount of detail.

p.cturelilluttration.

b2

Ensure that detail is correct and

not too complex or cluttered.

b3

Establish sequence of visuals to

be presented,

C

B—42

L.. -

~~~~~~ ~~~~~~~~~ ~~~~~~~~~~~ - — — -
. —- - —-——-- - ,———w ~~~~——- - -.. -

- ~~~~~~~—. -

~~

- .

~

—

2.3.2

I I

Identify timing of sequences.

‘

ci

Identify locations where videoonly materials are to fit with

other instruction.

c2

Set up schedule for development

of materials.

d

Establish time frame.

I

dl

I

Meet with illustrator/photographer .

to explain storyboards and

sequence of visuals.

d2

Arrang. f or TV production Determine what TV facilities are required and/or available,

facility to produce materials. (i.e., studio equipment, per sonnel, materials, etc.)

d3

Have video materials reviewed,

C

8—4 3

k. - ~~~— - ‘

~~~‘
- -

2.3.2

Give materials to peer for rev iew .

it

Meet with producer to revi ew

materials.

.2

Revise mater ials if necessar y to

clarif y sections, change pace, etc.

Have required num ber of copies

produced.

8—44 L

L. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4
~~~~~~~~~~~~~~~~~~~~~~~~ - -

7

~* a o ~si

~ lproducgion.

2.3.3

The a~~io in acadio visual,

2.3.3.1

Identify sections of course for Reviiw TLO’s and LO’s.

audiovisua l media.

a

* Identify comprehension level See: Brown. il . , and Carisen, G.R . Srown-Carlsen Listening of trainees. Comprehension Test . New York: Harcourt , Brace and

World , 1955.

b

Set up schedule for script

development.

C

Establish time frame.

a

ci

Meet with audio equipmen t

Personnel to set up recording

schedule.

c2

‘Non -HumRRO author aid.

j~

.I 8—45

: 0 ________ ___

~~~~ 

-

:_. _

~~~~~~~~ ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ -~~~
-- ~ . - --—-—- ~~—-—- — ,-, ~~

‘---- -,-

~~~~~~ 

-..-.-.~~

.----.~ -

2.3.3.1

Prepare outline of instructional

sections applicable to audiovisual medium.

d

Write script from outlin es.

e

Use two-sided script form Left side should Contain production instructions (use of music ,

for writing content. special cues ); right side should contain directions to narrator .

el

Indicate pauses and uses of other

media on script form.

e2

Have script reviewed.

Give script to peer for review .

fi

Meet with producer to review

script.

f2

B—46

- _______________
~~,— — -

~~~~

-—.——- -,.—

2.3.3.1

______________

Revise script It necessary to change

puce , clarify sections, etc.

g

Prepare required number of

copies of scr ipt for production

purposes.

h

* Record audio materials. See Closed-Circuit Television Production Techniques, by L.G. Goodwin &

1. koehring. (Indianapolis: Howar d W . Ssms & Co., 1970).

a

Th. visual in audio vösu.l.

2.3.3.2

Prepare outline of instructional Review TLO’s and LO s.

sections applicable to visual

medium.

a

Sketch or describe visuals

to be presented on storyboards.

11

‘Non4iumRRO author aid.

L ~~

- - . 

~~~~~~~~~~ ___________________
—,- ~~~~~~~~~~~~~

2.3.3.2

Specify details of each Include special effects, lettering, shading, amount of detail

plctur./lllustratlon.

a?

Ensure that detail is correct

and not too complex or

cluttered.

a3

Establish sequence of Vis~~l5

to be presented.

b

Identify timing of sequences.

bi

With first version of visuals ,

play audio accompaniment while

going through sequence.

b2

Set up schedule for

development of materials.

b3

Establish time frame.

b3-i

- -
2.3.3.2

Meet with illustrato r/photog rapher

to explain storyboards and

sequence of visuals.

b3~

2

Arrange for photographic Determine what photographic facilities required and/or production of visuals , availab le (i.e., studio , equipment , personnel , materials , etc.)

b4

Have visuals reviewed.

bS

Give materials to peer for review.

b51

RevIse materials if necessary

to clarify sections, change pace, etc.

b6

* Have required number of See The Video Handbook, NY: Media Horizons, 1972.

copies produced.

Non-HumRRO author aid.

8—49

- ..

-——— -.

~~~~. - -. . .~~~ -— ____ —

~

_

•

~~

;.

_ . ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —— — -

—~~ —~~~--~-~~~ 

L.,~~~ ~~~~~~~~... 

~~
.,-

~~~~ .

.—-..—-.- .,, - . ~ --—.— _

~~ —.-_ -.- .—- .---. -,- --—--— —--

~~~~~~~~~~ -—-—..-.- .

--

~ —--.- -- - --—— - -

_ ---- —--. ~

-.---—--,- .-..-. ~~

---

~ - -- ...---------- _

Pheducing a slide/tap. program.

2.3.3.3

Integrate audio tape recording

with visuals.

a

Determine app ropriate timi ng Refer to audiovisual outlines.

for each storyboard in outline.

al

Record pauses, audio cues,

voices, and combinations of both.

.2

* Have llustrator/photographer See The Video Handbook, New York : Media Horizons , 1972.

produce slides.

a3

Record audio portion on tape

including cues.

a4

Review integrated slide/tape

program.

b

r.

4.. Non’HumRRO author aid. B—SO

— ~~~~~~~ ~~~~~. ~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~ ~

, ~
-, - -

~

- -

----

~~~~

, - - - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

—-- - - —

~~~~~~~~~

-- . ----.--- ,—-.— —-.--

.

~~

. 

—~~~~~ --

F 

____________________________________________________

2.3.3.3

Present program to peers for

review.

bl

Revise audio and/or visual

1. materials if necessary to clarify

S sections, changes, pace, etc.

c

I —

Have required numb ers of slides

and tapes produced .

-

H a

Film Production. Not recommend ed.

t

2.3.3,4 

- -•• 

~~
- -...

_ _

* Phoducinga television prog ram. Follow steps for 2.3.2 (Preparing video materials).

See Closed-CircuIt Television Production Techniqu es,

Indi anapolis : Howard W. S.ms and Co., 1970.

2.3.3.5

Developing prin ted ,n.rerial&

2.3.4

Prepare outline of instruc tional Review TLO’s and 10’s.

sections applicable to print medium.

a

Determine sequence of

instructi on ,

b

Set time limits for using

materials in class.

bt

‘Non’HumRRO author aid.

8—52

- 

~~~ - - ~~ .. -- ~~~ ‘

~~~~ 

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~ ;r~ _~

_,_

~

L ~~~~~~~~~~ -
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-“ .-

~~ -r - -

~~~~ 

- - . - -

2.3.4

Determine location and type of

practical exercises and illustrations.

b2

1

~

Identify readi ng compre hension Use reading test scores.

level of trainees.

___

C 

* ~~~~iJaIs r

~! Ity level for Author aid for re&cing readability level

mu~

vWs, On line ~ See Peter Kincaid Automated Readability Index Human Factors

and Society Bulletin Vol XV No 5, May 1972

Off line d -

~~

--

~ -

~

..-i

Schedule preparation of

materials.

e

Set time frame for preparation

and production.

ci

Identify available facilities.

e2

Identify qualified personnel to

develop instruction.

‘Non.HumRAO author ald S3
C—- -

-

2.3.4

Meet with editorial and production

staff to discuss plans for instructional development.

I . 1

* Write Instru ction . See: Robert Gunning . How to Take the Fog Out of Writin g. Chicago: The Dartnell Corp., 1964.

* Write introduction and See W. James Popham and Eva Baker , Planning an Instr uctional summary of materials for Sequence, Englewood Cliffs: Prentice Halt , Inc., 1970. —

trainees.

fi

Indicate outside references

and any other aids used.

f2

Prepare remedial version s of

instruction if needed.

‘3

* Write primary instructional Tyler G. Hicks, Successful Technical Writing, New York : content. McGraw~

Hill Book, Co., 1959.

f4

Write practice exercises

and self-tests.

Non-HumRRO author aid. B—54

- , 

~~~~~ “r- ~ “ ‘ .

~~~~ ~

; 

~~~~~~~~~~ ~~

- 

~~~~~~~~~~~ 

~~~~~~~~~~ ~~~~~~~~ ~

-- ---.

~-
2.3.4

Have printed materials reviewed.

g

~

- I

Give to peer for review.

gi

Revise if necessary to clarify

sections, etc .

h

Have sufficient copies prepared

for use by trainees,

hi ~~~~
F— ‘-

~

--

~~~~~~ 

- ‘

-

~~~

C*vedoping programmed

instruction.

2.3.5

* Prepare outline for instruction. Review TLO standards and conditions. Review management

plan. See: Thiagarajan The Programming Process : A

Practical Guide , Worthington , Ohio: Charles A. Jones

Publishing Co.. 1971.

a

Determine inst.uctionai sequence. Author aid for preparing learning obiectives,

Author aid for sequencing of materials.

On line

and

Off line .1

Organize frames into logical Frames on the main track (if there is individualization) should

order, present all the information that a student needs in order to

master the subject matter.

al .1

* Plan frame size contingent on Relate frame size to expected student behav ior. Determine how

type of trainees , large a step toward mastery the student can reasonabl y be

expected to take in each frame. See: Markle , Good Frames

and Bad, New York: John Wiley & Sons, 1964.

.2

Make a rota i deaft of frame Author aid for preparing practice questions , Author aid for

including illustvetlons. instructional frame development.

Off line .3

Draw flowchart of frame

sequences.

a4

- 

~~

‘ Non’HumRRO author aid. 8—56

L. ~~~~~~~~~~~~~~~~~~~~~~~~ 

“ 

“

~~~~~~ 

L. ~~~~~~~~~~~ ~~~~~~~~~~ - - --——a-~~ 

--—- - ~~~~~~~~~~~ —~~~.--~ - - .

- -
_______

2.3.5

Locate areas foi practice exercises. Determine the amount of practice necessary beyond the

minimal range of examples. See: Markle, 1964,

a4-1

* Choose format of exerciass. Multi ple choice , true /fal se, completion? See: Narkle ,

Good Frames and Bad , New York : Joh n Wiley & Sons,

On line 1964,

and

Off line .42 Author aids for preparing multiple choice, true /false

and constructed response practice questions.

Locate potentiall y difficult

frames .

a43

Loca te areas for review , rest , and Where the program is extensive enough , provide isolated review

self- test. List references and demon- and test items as feedback to the programmer, as well as the student

str ation materials to be used by on how well the teaching sequence has gone. Prepare quiz which

student. tests students understanding of the material.

.4.4

* Determine type of prompt to be Make use of the thematic prompts (in context ). Only use formal

used in each frame , prompts (multip le.choic e format excluded ) when absolutely

necessary . See Thiagarajan & Mark le.

a5

Determine if there will be It the student is branched to remedial instruction , the material

branching. should represent a restatement of information covered in

the main track .

a6

Set sequence for branching.

a6’ l

4 I

Non-HumRRO author aid. B—57

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4
V ~~V

2.3.5

Determine if there are supp le’ Provide instructions on their use.

mentary mater ials.

al

Determine if there are supple- To be used with supp lementa ry materials? Alone?

mentar y exerc ises.

a8

Identify the types of responses

to be elicited in each frame.

89

I

Determine feedback to be given Will feedback contain explanatory material? Will examp les V

to trainee after responses , be used?

alO

* Write frames in rough draft , See Nesbit arid O’neil , Thiagarajan & Mark le.

b

* Write frames clearly in good English. Avoid introducing more points than can he responded to in

any one frame. See Markie.

bi

Provide good examp les on the Provide examples covering the variety of conditions the student

instruction, will cope with .

1,2

Non’HumRRO author aid. B—S 8

- - 

- 

~~~~~

- --

~~

--- -,
V V

2.3.5

C~ieck fr ames for accu,acy .

b3

* Check readability level of frames. See; Peter Kincaid, Automated Readability Index , Human

Factors Society Bulletin, Vol. XV . No. 5, May 1972.

On line

and Author aid for reducing readability level ,

Off line

Eliminate irrelevant material

from frames.

b5

Write practice exercises in

rough draft.

Off line c

* Choose exercises that are relevant See Markle.

to instructional content.

ci

Choose the number of exercises Do not make exercises too long at one time.

for the frames.

c2

Write exercises clearly, in good

English.

c3

Non’HumRRO author aid.

— — . - -4. . .

- . 

- ~~~~~~~~ 

~~

, 

V

-

~~~~ ~~~— -~~
-‘

2.3.5

Have peer review roug h draft

of program.

d

Try out draft on other

authors.

dl

* Have editorial review of See Thiagarajan.

instructional materials,

d2

* Revise materials as necessary . See Nesbit and O’Neil, Thiagarajan and Mark le.

‘Non~

HumRRO author aid,

B—60

——- —---.—- - - V~~~~~~~ _ L.’

L ti ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~

‘ 

~~~~~~~ ? ,‘ ~~~~~~~~~~~~~~~~~~~ 

.

~~~~~~ s - ’. :

~~~~~~~

- - -~~
F V ~~~~~~~ ~~~~~~~~~~~ 

.- — -- -‘—

~

_ - -- -- -

~~~

- - - V V

CMs,Ioping platform lectures.

2.3.6

Develop outline of instructional Review TLO’s and 10’s.

sections to be covered by lecture,

a

Divide outline into_ lecture Refer to management plan for time constraints.

periods of _ minutes each.

al

Determine time and placement Refer to management plan.

of tests , quizzes , discussion

periods, practice, other mediated

instruction , etc.

a2

Identify sequence of 10’s to be

covered by lecture.

a3

Prepare outline and exerci ses for

t alternate lectures if they are to

be develop ed for different student

populations.

a4

Prepare lecture notes to aid Note different concepts to get acro ss. Also , note where

delivery. supplementary materials are useful.

aS

8-6 1 

L~~~~~~~~~~~~~ 4

- — .,. — ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~ ._ .... ~~~~~~~~~~ ~~
_ V ~~~~ ~~~~~~~~~~~~~~~

2.3.6

Determine degree of student Will students listen only or will interactions be encouraged?

particip ation in lectures.

b

IV

Schedule facilities and

resources,

C

Determine classroom /lecture hail

requirements and schedule their use,

ci

~

V VI

Identify supplementary materials ,

if any, to be used in conjunction

with lectures and order them.

c2

Identify instru ctor s/lecturers

who will present instruction and

prepare ro ster,

c3

Practice lect ure in “dry run ” to

determine how much time it takes.

d

Prepare additional notes and/or

modify outline , if necessary.

dl

-- - ‘

~~~~ ‘;~~:

‘

: 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

~~~~~~~~~~~~~~~~~~~~~~

— -~~ ~—
2.3.6

Try out lecture on other

authors.

e

Revise lecture outline, notes,

exercises , etc., as necessary .

D velop Self’teaching exportable

packa~~s (STEPs/.

2.3.7

Develop outline of instruction al Review TLO’ s and LO’ s.

sections to be cover ed by STEPs ,

a

Identify sequence of 10’s to be Author aid for preparing learning objectives , covered. Author aid for sequencing of materials. On line

and

j Off line b

B-’63
-

~~~~

-— --

~~~~~~~~-——--- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~

2.3.7

Select places for exercises and

self ~tests.

bi

Identify the supplementary

materials and references to be

used with instruct ion.

C

* Write draft of STEP, See: Deterline Associates , How to Design and Develop Self

and Supervised Instruction: A Guide for Developin g

Corresp ondence Instruction , February 1975.

d

* Write introduction to include See: Preparing Extension Training, TRA DOC Pamphlet 350-31,

list and sequence of LO’s to be (Draft) , February 1976.

attained ,

dl

Establish standards of performance

needed to complete instruction.

d2

List references supplied and

describe their use.

d3

Assemble remedial and supplementary instructional materials,

d4

Non-HumRRO author aid. B—64

- ~~~~~~~~~ ~~~~ 

— - -

~~~

—--

~ 

—

~

-—

~~~ ~~

—. , 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~ 

‘

~~~~ 

- V~~~~~~ - __________
__ -

~~~~~ -~-—--—

2.3.7

Include exercises and self -tests V with scoring instructions.

d5

Supply branching directives conS Provide discussions regarding all responses.

tingent upon responses to

exercises and self-tests.

d6

Have STEP tried out by students

representative of targ et population.

e

Revise STEP if necessary.

B’-65

~—~~~~~

-=-

~~~~-- ~~~~~~~~~~~~~~ ‘ ——- ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~ ~, ~

V 

~~~~~~~~ —
— - -

~~ - — —— — - —V V V V ~~~~~~~~~~~

O.velpping supplementary

instruction.

2.3.8

Survey available resources for Review LO’s for requirements, V validated instructIonal materials

relevant to LO’s to be covered.

a

Prepare outline of instruction Identif y special instructions for their use, if any. to identify places for use of

supplementary instruction.

b

Relate self ~

tests and exercises

to supplementary instruction.

C

Modify supplementary instruction

to be in a form similar to that in

existing program.

là

B—66
____________________ - 

---- , 

~~~~~~~~~~~~~~~~~~

Developing ac*unct instruction.

2.3.9

Develop outline of instruction Review TLO’s and 10’s.

to be covered by adjunct

instruction.

a

Identify sequ ence of LO’ s to

be covered.

b 

V

Identi fy locations in sequence

for self-tests and exer cises.

bi

Collect and review all materials

to be used.

C

* Mske sure that text does not See: Peter Kincaid, Automated Readability Index, Human

exceed ~~~~~ ~~~~~~~ Factors Society Bulletin, Vol. XV , No. 5, May 1972.

On line level of trainees.

and Author aid for reducing readabili ty level.

Off line ci

I

Ensure uniformity of length and

difficu lty of instruction al unit.

c2

‘Non HumRRO author aid 8—67

A _ _
- - -

~~ -~~~~~~~~~~~~~~~~~~~ - - 

-V —

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~--

~ - - -

2.3.6

Write proceói ral instructions

for trainee.

d

List TL O’s and 10’s to be attained

by trainee.

dl

Wr ite correct responses for

student to comp are his answers V

on quiz or test items.

d2

Prepare exp lanations for

possible responses.

d3

Prepare branching or remedial

instruction s based on responses.

d4 V

V 

Provide suggestions for remedial V

exercises.

dB

Review instruction.

/

B—68

I.- - 

V ~~~~~~~~~~~~~~~~~~ J ~~ VV_ ~~~J1--

~ r -—
‘I’ - 

2.3.6

Have peer review materials

for accuracy and appropriateness.

.1

Have st udents representative

of target population try out

materials.

e2

Revise instruction, if necessary.

..inb Performance Aids (JPA ‘a).

2.3,10

Prepare outline of instruction Review 110’s and LOs

which identifi es small ste ps

requiri ng one specific action.

a

Group small steps into functional Note referen ce and supplementar y material relevant to each unit.

units,

b

8—69

~~IL. — -

~~~~~~~~ - ~~ ~~~~~~~~~~ — 

~~~~~~~~~~~~~~~ — ~~~- — ..á—- ~~~~~~~~~~~~ -
r

2.3.10

1

~

Develop JPA’s for unit.

C

1

~

* Plan JPA content including See: Full y Proceduralized Job Performance Aids , Handbook for JPA

checklists. Developers by Reid P. Joyce, et. al., Air Force Human Resources

Lab, A FHRLVTR.73.4 3 (III) .

ci

-VI

Plan use of visuals/il lustrations

in JPA’ s.

c2

Meet with production personnel

to discu ss layout and content of

JPA.

c3

Schedule production of JPA.

c4

Write instructions for JPA user . Include references.

On line Author aid for preparing test instructions.

and

Off fine

Review JPA.

8-70
-V -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -V

2.3.10

Have peer review JPA by performing task s using JPA .

di

Have novice perform tasks, noting

any problems, ambiguities.

d2

Revise JPA, if necessary.

Developing formal WT (FOJT) .

2.3.11

I V

Develop outline of instructional Review TLO’s and LO’s.

sections to be taugh t by FOJI.

a

3—71.

______ ~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ _

~ V ~~~~~~~~~~~~ -V-V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

- -

~~~

2.3.11

-t

Identi fy which sections will be

presented as demonstrations,

lectures , “ hands~

on ” performance ,

or mediated instruction,

b

Establish sequence of

instruction.

C

Choose locations for pract ice

and tests.

ci

I

Select reference materi als

and aids.

c2

Prepare FOJT content.

d

Obtain assistance from subject

matter expe rt .

dl

Prepare introductory

material.

8-72 

~~~~~~~ V- V - ~~~~~~~~~~~~~ 

L~~~~~ V~~~~~
-V - - -V . -

~ -

2.3.11

Write procedures for demonstrations and performa nce

portions of instruc tion .

d3

Prepare practice exercises

and tests.

d4

Prepare lectures and audiovisual See blocks 2.3.3 and 2.3.6 for further reference .

portions of FOJT .

d5

Establish time limits for

V tasks and demonstrations.

e

Establish standards for performance throughout the traini ng

period.

Prepare performance checklist

for use of supervi sor to eva luate

student.

fi

Prepare instructions for

supervisor to score student

performance.

V f2

B—73 

- ____

____________________ ______________ — ~~~~ V I

~~~~~ V~~~~~ V ~~~~. -
-

~~ 

~ V ~~~~~~~~~ -- .-V-V --V —-V—-V—-V- -----—.------V - 

I

2.3.11

Writ. *qctjons for students, Author aid for preparing test instructions.

On line

and

Off line g

Review FOJT.

h

Have knowle dgeable personnel

review instru ctio n.

hi

Try out FOJT in field site using

trainees typica l of target

populati ons.

h2

Revise FOJT , if necess ary.

V 8—74

V V I - ,

,..

~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~ ~~~~~~~~~~~~~~~~~~~~~~ — -

~ .
— -V—-V --

[

~~~ v Forms of n .di.f,(j

instruction.

L 2.3.12

rDev,lop outline of section s to 

~

‘J Review TLO’s and 10’s. covered by other forms of mediated instruction (e.g., CAl).

[ a

Fldentifv available resources and

facilities to use in instructional

development.

b

[

~

dentify availa ble personnel who

are knowl edgeable in using

resources to prepare mediated

instr uction,

C

Prepare instructional sequence and

strategies depending on characteristics of mediated instruct ion.

d

Establish branching, sequencing,

{id 

remedial paths 

:0r 

trainees, 

J

Prepare text , audiovi sual Illustrat ions , exercise s , tests , etc . that are sui ta ble for the medium

to be used,

e

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~
~~~~~~~ V~~~~ -V V~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -V -~~~~~-V~~~~~—--V - ---V -~~~~~- - - .— -,

2.3.12

Check rsadmbility of textual See: Peter Kincald, Automated Readability Index, Human

mecarials. Factors Society Bulletin , Vol. XV , No. 5, May 1972.

On line

and Author aid for reducing readabili ty level ,

Off line .1

Review instruction.

Have peer review materials

for accuracy and app ropriateness.

fi

Have students representative

of targ et population try out

materials.

f2

Revise instruction , if necessary .

C

‘Non•HumRRO author aid. B—76

- .

; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ -~~~~~
-V -~~~~~~

I- 

________ 

* Pr,tnt first dr af t nsei riols~ See: __________

Handbook for Designers of Instructional Systems ,

Air Force pamphlet AFP-5O•SS, Volume IV , Jul y 1973.

2.4

Choose a complete sequence for

testing (about 1/2 hours).

a

I

Select sample population sim ilar

to target population. - 

- -

b

Choose naive subjects.

bi

Instruct Si to note problems Author aid on attitudinal questionnaires for CAl.

in understanding Instruction,

iftustredons,

b2

Time Si as they progress

through mater ials.

b3

* Administer test of mater Ials See: Handbook of Procedures for the Design of Insti ucti on , V to trai nees. Leslie J. Briggs, Pittsburgh: AIR , 1970.

C

Non HumRRO author aid. 8—77
-V-~~

- - - -- 

-

~ - - -~~~~~~ - ---. --V-V-—--

- - -

2.4

V 

Trainees Check to see if instruction

t get all Yes is too long, if there is too — items correct. much infor mation.

ci

No 

________________________

Trainees Check vocab ulary and Length of instr uction may

do Yes sequences of instruction , have to be shortened. Number

poorly. of summaries of self-tests may

7 have to be increased . Supple .

c2 c3 mentary materials may have to

____________________________ be added.

No I

Revise areas where indicated,

2.4.4

8-78 ( t ___- 

- _________
- -V

V - - — - -—-

~

,-.,-.—.-.,—-— -

$1

Plsiere user instructiona

2.5

Pr.pa,, hii*uctorI ~~~~~~~~ Guide for the use of author aids in preparing CAl materials.

Online a

Describe instruction,

al

1

Discuss rationale for

instruction .

al-i

Identily the need for

instruction.

al ~

2

V~ 

- 

I

Identify the targ et of

instruction.

al-3

I

V Identify job(s) for which

stu dent will be prepared.

I al.4

I

____________ V 

~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~ V
V -V V~~~~~~~~~~~~ , V —-— - -:‘ V _VS ~~~~~~~~~~~ -~ - ~•

~_V-V-V ~~~V V

2.5

Identify degree to which

instruction train s student for job.

al -5

Prepare overview of instruction.

a2

Prepare outline of each lesson .

a2~ 1

Briefly describe contents

of each lesson.

a2-2

List lessons in their

proposed sequence.

a2-3

* Write plan of instruction, See:l4andbook for Designers of Instru ctional Systems, AFP 50-58. July 1973.

a3

Indicate LOs for each sequence or block of instruction,

8-80
—- -V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-V -- 

2.5 

V

Determine duration of train ing List time constraints for each sequence, practice exercise, etc V

for each instructional section,

IV

.3-2

Describe instructor requir.

~ ments/duties In each instructional

section.

a3-3

I __________

Describe media resource List the available faciliti es, the time and duration of their use and

requwements related to training the supervisory personnel needed. List all available training aids

aids and facilities , for each block or unit. List all mediated and supplementary

instructional materials , their location , etc.

a3-4

Target population d,acription.

a4

Identify trainees’ academic or Ref er to entry tests.

educational level, reading level,

verbal ability, etc.

a4-1

Identify previous training or List courses taken and/or hours of traini ng in specific

related knowledge and experience prerequisite areas.

of trainees.

a4-2

Identify r quired physical and List required physical skills and characteristics (e.g., coordination, personal character istics of trainees , motor skills) and personal qualities (e.g. , leadership, motivation, etc.).

a4.3

8—81
--V -

~~~~~~~~~ -V --—-- V

2.5

Identify administrative restraints. List all sequence ranks and grades of students.

a4~4

Testing inform.riün,

a5

Provide tern prepared In block 11.2.

On line

and

Oft line aS-i

Furnish ans~~rs to test items.

a5-2

Provide directi ons for

administration.

a5-3

Provide scoring procedures to be

used.

a54

Administration dI,vctions.

a6 

~~~~~~~~- ~~~~~~~~~~~ ~~~~~~
_________________________

__ - V

2.5

Describe scheduling procedures. List schedule for using facilities , materials, and personnel

resources .

aS- i

Discuss monitoring process.

a6-2

Provide instructions for

maintaining attendance records.

a6-3

Describe monitoring requirements for

self ~tests and practical exercises.

a6-4

Provide recommendations for Discuss needs of exceptionally fast /slow trainees.

handling individ ual trainee

differences.

a6-5

Describe procedures for keeping For example , ways to elicit student activity.

the student productively involved in

the learning process.

a7

Indicate recommendations for For example, discuss value of displaying high motivation

providing an environm ent conducive of teechers and easy access to aids.

to learning.

a8

L _ 

A

I’- ,-.‘

~

, 

- I. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~— —~—~~~~ - ~~~~~~~~~~
--— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

V~~=- V-V~ ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

2.5

Provide teaching types, methods, For example , indicate ways to change pace in instruction .

and techniqu es.

a9

P~apare stud nts’ guide.

b

Li st prerequisites for course

in terms of TLO’s.

bi

Explain framework of course. Descri be time frame, prerequisite assumptions, pre-tests , if any ,

post-test requirements /criteria , and course materials.

b2

Describe structure of course and its Explain sequence of course lessons, use of aids, facilities , and

environment, personnel in control of course , remedial branching, if present.

b3

Specify personnel to conta ct

when instructions or course

materials are not understood.

8—84
3.0 Outputs

Products,

Furnish all instructional materials,

exercises, self ~

tests, aids, etc.

a

Furnish instructions for using

all materials developed.

b

Other documev,tation.

3.2

Prepare an outli ne statement of Describe management of course and student s . instructional development plan.

a

I

Write a summary statement

of any deviation s from plan and

reasons for deviations.

b

Prepare a repor t detailing

development time, costs, and

problems/solutions,

C

8-85

-VV V-V V

~~~~~~ 

- V •• 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- -~~~~~

REFERENCES

BROWN, J.I., and Carloen , G.R. Brown—Carisen Lis tening Comprehension

Test. New York : Harcourt , Brace & World, 1955.

BRIGGS , Leslie , J. Handbook of Procedures for the Design of Instruction.

pittsburgh : American Institutes for Research (AIR), 1970.

Deter]J,ne Associates . How to Design and Develop Self— and Supervised

Instruction: A Guide for Developing Correspondence , February 1975 .

GOODWIN , L.G ., and Koehring , T. Closed—Circ uit Television Production

Techniques. Indianapolis: Howard W. Sanies & Co., 1970.

GUNNING, Rober t. How to Take the Fog Out of Writing . Chicago: The

Dartnell Corp., 1964.

Handbook for Designers of Instructional Systems. Department of the

Air Force : AFP— 50— 58, 15 July 1973.

HICKS , Tyler G . Successful Technical Writing . New York: McGraw—

Hill Book Co., 1959.

JOYCE , Reid P., et al . Fully Proceduralized Job Performance Aids——A

Handbook for JPA Developers. Air Force Human Resources Laboratory

AFHRL—TR—73—43 II.

KINCAID, J. Peter. Automated Readability Index , Human Factors Society

Bulletin , Vol. XV , No. 5 , May 1972.

MAI1KLE, Susan . Good Frames and Bad: A Grammar of Frame Writing .

New York : John Wiley & Sons , 1964 (second edition) .

NESBIT , Marion and O ;Neil, Harold F. Jr. Guidelines for Editing

Programmed Instruction. Austin : The University of Texas , November

1974.

POPHAM, W , James and Baker , Eva L. Planning an Instructional Sequence.

Englewood Cliffs , NJ: Prentice—Hall , Inc., 1970.

THIAGARAJAN, Sivasailam . The Programming Process: A Practical Guide.

Wor thington, OH: Charles A. Jones Publishing Co., 1971.

TRADOC Pamphle t 350—31 (Draft). Preparing Extension Ts aining, February

1976.

t 

B-87 

~

q” ~~~~~~~~~~ ~~~~

- -

,

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ________
-V ~~~~~ V V ~~~~~~~V V V

A ppendic C -

DETAILED FLOWCHARTS OF AUT HOR AIDS FOR

IPISD BLOCK 11.2 (DEVELOP TESTS) AND BLOCK 111.4 (DEVELOP INSTRUCTION )

The two sections of flowcharts in this appendix serve three purposes .

First , they further clarif y IPISD Blocks 11.2 and 111.4. The number at the

beg inning of each flowchart refers to the appropriate p lace they are to be

inserted in the IPISD blocks flowcharted in Appendix B. Additional block

identification was not made because of the complexity of the flowcharts.

The second purpose of the flowcharts is to provide directions for the

use of the on—line authoring aids for test and lesson development . These

flowcharts are detailed enough to lead an author from the beginning of

development (the learning objective ) to the finished product (on—line

for student use) .

The third purpose of the flowcharts is to show the logical flow of the

author aids (including where they fit into the larger IPISD process), so

that conversion of the aid is possible on other computer—administered

instructional systems . On—line demonstrations of the flowcharts are possible

by accessing HumRRO lessons inquiry l (test development) and inquiry 2 (lesson

development on the PLATO system .

The flowcharts in this appendix are on the pages listed below .

V Preparing CAt Test Materials —— Page C— 2

Preparing CAl Lesson Materials —— Page C—30

-

~~~ -— 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

,

-

VV

~~~~~_ 

~~~~~~~~~~~~~~~~~~~ ~~~~~~~ ~~~~~~~~~~~~ ~~~~ ~~~~~~~~ £ ‘

~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~
_____ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

~~~~~~~~~~~ ------ - -

ll.2.2.1.1.b PREPARING CAl TEST MATERIALS

Have

references N Obtain from the been site director obtained

V

Has See site director to dataset N obtain datas et for been — tem porary storag e designated of test

Transfer block b

from inquIry 3 and

make the necessury

changes. Refer to

the manual 
~ 

-

~~ 

- - -- 

Have

8000 and

N 

for creation

V

Type in dataset

name and lesson name

~~1

Type in 4 letter

code of test

Type in

tim e limit

I -

~~~~~~~

C-3

V ~~ ~~ ft ~

VAI .U

~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~
•V ~—VType in whether

or not student is

given the option

of branching to

the quiz

Study general test

guidelines

obje~ ive guJeli~~ o~~~t

~ e~~epara 

-

_

tons —

_______ — - 

0 ~~~~~~ ~~~~

r

_LV~~~,~~~~~ — - A
— 

------ — —

~~~~~

-

/ \ Study objectives.

See objectives V

V 

~~~~~~~y t e ~>-~~~~~~~~~~

_ _

-

N

Enter learning Make revision s.

objective. Is Revise

C— s

- ~~~~~~ _____
-- -

~~~~~~~~~~---.~ - - - ~~~~ - - ~~~~~~~ .

_____________ A

Write

mult iple-choice V 1

item p.c-i

N

Write V 2 true/false ________

N

Wr ite

constructed y 3

response (

~~~ 2

V _

____ ______ - 

-- ~~~~~~~~~~~~~~ II

V
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ — V V - ~~~~~~~ -------—- —-~~~~

- V -

Multiple choice item format.

Have Study guidelin es .

multiple choice N

test item guidelines

been studied

?

V

Use multiple choice

Has item test prepar ation

been created worksheet. ________

off line

7

N

Type in stem. Use option. 

V

C-i

____________ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ - ~~~~~~~~~ -- .— ~~~~~~~~~~~~~~~~
Make revisions.

Type in number Revise number

~t:~~~~es 

~~~~~ 

N :

~erfl~ ives. _____

including corr ect

Chose Type in answer Specify the

position alternatives includ ing correct answer.

of correct answe r correct answer in

alter na ive position desired.

Type in answer

alternati ves including

correct answer

(position randomly -

selected)

V 

c-s 

:

~~ ~~~~~~ — - -

—

- - - V
I

— 

Make revisions

itte 1

Complete scoring

form for item.

Finished N wit h learning B

objective p.C.16

7

V

Review next

learning objective to

determine action

required.

~ 

- 

~~~~~~~~~~ 

___

— .-~~~

- -

~~~~~~~~~~

---- -
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2

Have Study guidelines.

7 true/false N

(test item guidelines

\

~ been studied

V

/Has\ Use true /false

question y test item

worksheet.

Type in question. 

— 

Use option.

C—b

r

V - ~~~~~~~~~~~~~~~~~~~~~~~~~~~
- -

Type in whethe r

answer is true

or false

Type in ti me —

V permitted.

Complete scoring

form for item.

Finished N

~~~~~

Ewmn

~~~~~~~~~~~~~~~~~~~

c.’6

Review next

learning objective to

determine action V - 

- required.

B

~p.C-1OJ

s

—, 

C—il

~~~~~~~~ .-V - 

~

•::- - ‘

. 

• -

~ ~~ 

V 

— . 

- --—— - U — —-

- 

.

.

.- .

.

.

~~~

V .

-..

. 

~‘e. -. ~~~~~~~ 

- . .
~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~

3 Constru cted Response Item format.

Have Study guidelines.

constructed N

response test item

idelines been

studied

V

V / \ Use constructed

Has item V response test item

worksheet. ____________

N

Type in question. Make revisions.

Is V Revise

reading level item

exceeded 7

1

N N

C—12

-- - . . 

— —. - — - ~~~~~~ ~~~~~~~~~~~~ -— - -— . 

A ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~

-

~~~-—-~~~~~ ~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
- 

Study gujhi.

examp les.

Enter as many as —

6 correct, partially

correct or incorrect _______________________________________________________

answers.

Determine scoring

options.

Ignore

capitalization

Type changes if

N ____________ 

necessary .

C—fl

- ~~~~~~~~~~~~~~~~ ;-

~~~~ 

,

~~~

, 

~~- -

~~~~ 

-—-— —— - - V

L M.~ 

—
V V — ~~~~~~~~~~~~~~~~~~~ —- ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~

Type change if 

V

necessary. N ____ ___

Permit

extra words

in answer

7

Type change if

N necessary .

C—14

A
- - - __

Permit

words in y

out of order —

~~ I ges if

Type in time Timed permitted. item _____________

7

N

Complete scoring

form for item.

Review next

learning objective

to determine action

required.

-

~ 

~~~~ 16 

C-is

- —-V - - - ~ -. .

~~—---.-,.—I. ~~~~~~~~~~~~~~~~~~~~~~~
V ~~~~~~

B

Create

test Item V A

7 p.C6

N

Create test V Initial test N During test N

instructions instructions instructions

7 7 7

N V ‘I’

4 5 PV C -20 pc 2 1

Review V C

guidelines ~ c 7

N

Stop D temporarily PCI

N

C—lb

-V -- ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

~~~~ -- — ~ - -
~ -

Stop.

test is “ E

finished p.C-22

I

N

B

pC-16

L..
____________________________ —V. ~~~~V~~~_ V .~~~V. V. ~~~~~~~~~~ ~~~~~ V

C Review guidelines.

Return to index.

Select No more

guideline ~~~~~ 16

Type in guideline

number.

I 

_ _

Review guideline.

Another N

I guideline

I

V

C—l8

— ~~~~~~~ - - V~•

~ _V~~~~~~ •~•V~ ~~~~~~ 

- .:~~~~~w~~~~~—----- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ V~~ - V~V .V. V V •
- _ V ~~~~ _

~~~~~~~~~~~~~~

0 Stop temporarily.

Lesson Type in lesson name

for formatting V for final storage of

test changed tUtU ______

I

N

Sign off Sign into

Review test N (Hit STOP 1) inqui ry 2

on line when reedy to

1 continue.

V

See manual for

conversion

instructions.

Review test.

See manual for

instructions.

C—b9

V .- V ~~~~~~~- -

_____________
- V .- - -- V V. ~~~~~~~~~~

4 Create initial test instructions

Have

test guide.

lines been N Study guidelines _____ studied

1

V

I Write initial test /

‘ reading \ V /

‘ 

Revise ‘

\ V —1 instructions ......... ( level ~~~~~~~~~~~~ instructions \.. .... Make revisions —

I__ _ _

N N

Another

page of N B

Instruct ions P.C 46

1

C—20

~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~
— V .

VV ~,

5 Create duri ng test instructions

Have

any test

items been

created p.C-20

V 

1

V

- 

Write dunng test 

__<

/

‘

~~~~~~~~~~~~~

V

l

~~~~~~~~~~~~

V 

~~~~~ns

Another

page of B

instructions p.C.16

1

C-21

~~~~ V . - - —~~~~~ - - V VV ____V

~~~~~~~~

~V
VV ~~~

£ Stop, test is finished

Test

instructions N 4

created p.C.2O

1

V

Read scoring

guidelines

Review y of test D

on line p.C.19

1

N

Lesson

for Type in new

formatting y lesson name for

test final storage —

V changed of test

7

N

C22

_____ 4’
Review scoring

worksheets

Type in scores

and assign score

to each answer

Is tota l . -

score N - -;_~

of test Vi

correct

7

V

Specify minimum

passing score

Will

student review N items missed

if test is __________________

I

Select student

V V review option

C—23

V - 

. - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

L ~~
_ _

Specify review

option desired

~~~ 1

-

~ I C—24

- V - . ~~~ ~~V V _~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~
- -- V~~~~~~ - -— - — - — - _________________

Read.

ministra t ion

of instructional N £3

lesson p.C.26

1

V

4

It

Then

readm,njstra.

tion of test V Type in items previously E5 option 1 

missed p.C—29

I

N

Then

reedminjstre. y Type in

tion of entire opti On 2

N

£3

.C-26

C—25

- - V It ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ V V. - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

______
— - - --~~~~ ---~~~~~~~~~~~ V.- - V

~ 

--V — —--V ~~~~~~~~~~~~~~~~~~~ - ———- -— - -- ~I

~~~

£3

Reedminis.

7 tra~~~ of \ N

test items pre. 

~~~~~~~~~~~ 

Es ~ \viously missedZ %P.C.27 J

V

Give

Z student- \ I

/ option of \ N I Type in reviewing iflstruc. Option 3 tion first E5

7 P.C-29

V

Type in

Option 5

E5

.C-29

C- 26
E4

Read.

mini strat ion

of entir e N E6

test p.C-28 I

V

Give

student

option of N Type in reviewing option 4 E5

Instru ction p.C-29

first

7

V

Type ji,

option6

V 

ES

p.C-a

C—27

—‘---V - 

V ~ 

:V : 

V~~~~~~~~~~~~~~ V V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —-& ~~~~~~~
E6

No tedmin istr a- E2 tion of

instructional p.C-25

lesson or

test

I

V

(E5~~

C—28

— -‘- V 

. . ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ j V ;p 

- V
~ 

-_-

See manual on

conversion

instructions

Review test

End CAl

test creation

See Appendix B, page 8-25

C—29

- -V.— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

PREPARING CAl LESSON MATERIALS

111.4.2.3.13

Have

references N Obtain from the site dire ctory.

been

obtained?

V

See site director to obtain data sot

for tempora ry storage of instru ction.

Has data N

set been

designated?

V —

See site director for PLATO lesson

to be used for final sto rage of

Has PLATO N instruction

6 part lesson been —

designated?

V

Transfer block b from

inquiry 3 and make the

necessary chang es.

Refer to the manual .

See site director tot creation

instructions.

and 4000 word N

commoos been _ _

V --~~~~~~~~~~‘ —~~~~~-

~~~AI. -
-

~~ ~~~~~~~~~~~ 

Type in data set name

and lesson name.

~~~~~~~~~~~~~~~~~~~~~ Determine reading level Subtract 1 fro m reading of trainees, level determined. Has reading N

difficulty level been —

decided

?

V

Practice, Familiar Practice

with PLATO N with editing V

keyboard keys —

Study guideline descri bing Study guidelin es for modules and lessons , preparing practice fra mes.

- 

E

~

:

~~

°

~ ~~~~~~~~~~~~~~~~~~~~~~~ 

U~~~ear~ ng ob~c~ve 

-

C—31

— - V ~

VV V_._ ~~~~~ J V - ‘

V V 4~~ ’ 

~~~~~ j ~~~~~

-

.. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -V. .

Type in learning Use options listed on

objective. screen.

Make revisions.

Reading

level exceeded V Revise V

on LO LO

I

V 

N N

H

p.C-33

-_~~~~~~~~~~~~~~~~~~~~ -V. ~~~ V V.~~~~~~_ ~~~~~~~~~~~ 

~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ - 

* — V — V V. V

~~~~~~~~~ ~~~~~~~~~~~~~ - 

- -V-V.- ~~~~~~~~~~~~~ ~~~~~~~~~~ 

i
~

-V--V 

V

H

Text N actice N Review N Start N Stop N creation frame guidelines a new St N temporarily )

O1 7 ~ creation 1 module 7 7 7

V V V V y V

P.C-34 

G K L M N F P.C’36 p.C-51 p.C.50 P.C-52 p.C.~~~ —

4

-

~~~~~ 

. -

~~~ !.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

: ‘ __ __

-V ~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -~--~-1-
V- V ~~~~~~~~~ V ~~~ - -

G Text creation “

Guidelines y

on text creation

studied p.C.35

7

N

Study text frame

preparation guidelines.

Create outline for

instruction.

to be 

~~~~~~~~~~~~~~~~~~~~~ 

[

vise outline.

Use text preparation

Has text V worksheets for off ’line

off tine 

_

creation.

Set margin. (Use the arrow keys )

L - !V~

’

I-V —‘a.’ ,_,~~ _1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~ _k~~~~~ V V ~V._ -‘.-

~~~~-. — -‘- ~~~~~~~~
---V.- .-- - - -- V- - — - -—- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

Has text

been created v Use text prepara.

off-line tion worksheets.

I

N

J

Use option . Are

options to be

used

1

N ___I —

Use arrow keys to

Is , ~. set margin. margin

correct

I

— 

Type in tex: 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Make revisions

Return to index

Create

another text N

C-35

_____ - “I ~~~~~~~~~~~~~~~~~ ~~~~~~~~~ ~~~~ _______________
V -

K Practice frame creation,

K 1 Multiple-choic e format.

Study guidelines Use Mul*ip$e’choice

onm

G

u~1~~~~~~ce N 

E~Fm4 ~~i arm

_

-1

- 

Type in stem. Any Use option.

options V

Used 

_______ 

~

I] V 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Make revisions.

C—36

~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~

~~
- -~~~

Type in number of 4e

~

ie Revise number of

i

~~~&gco rr.ct alternatives 

N answer alternatives. 

—

answer. >3~~6

Type in answer Specify the correct

Chose y alternat ives answer.

V position of correct including correct —

answer answ er in position

alternative desired.

Type in answer

alternatives

including correct

answer in position

randomly selected.

V ~~~~~~~m~~ves y 

Make revisions.

Type in number

attem pts permitted

(1.3).

C—37

~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ — -~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - r ~~~~~~~~~~~~~~~~~~~~~ -
~~~~~

V . • V ~~_ V- ~~~~~V.~~ V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ,— -- —----- -

~~~~~~~~~~~~itted

N 

Rev~~ number of 

1

ai~

s

1

er feedback. 

~~~~~~~~~~~~~ 

Make revisions.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

7 iry again \VV f~~?

çfeedback based on ) ~ ‘ tp.c’~~ ~~ spectfic answer /

\I/

N

K1.1

p.C-39

I

C-38

— 

~ 

- —- -- - . ~~~V V .~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - - - - -- --
-1 — ~

XL?

Type in feedb c~ for anempt n.

reading level 

V Revise 

Made revisi ons.

exceeded feedback

N N 1

Is number

of feedbacks V

‘- number of Another V K

attempts multi ple.cho ice pC

minus I practice item . -36

7 7

N 

V

N

H

p.C.33

~~~~ ns 

~~~~~~~~~~~~~~~~ 

Make revisio ns. 

1

feedbac r t

wrong answer 

} 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —<iI

~

E~

E:Ii>

~

-—-- 

-

Make revi sions.

}

More

C—39

V ~~ t’ -

~~~~ . 

-

~~~— — ~~~~~~~~ 
-V. V V V. V~’ ~~~~

!FL ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~ F

_ 1

Type in question. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~ Use option. 

V —

~~~~~~~~~~~~~ 

<(

“

~~~~~~~~~~ 

~~ Make revisions. 

—

Type in whether

answer is true

or false.

-: 

~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~ :k

V 

Make revisions. 

1

C—40

X V

-: ,, ~~~~~~~~~~~~~~~~~ V. ~~~~~

-V - ~~~ V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -
r

~~~~~~~~~~~~~~

V V VV - --—-— V— V.——- ~~~~~~~~~~~~~~~~~~~~~~~ V.-V~~~~V V.V. VV ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -VV.

1 or 2

attempts 1

perm tted

Type in try again

feedback.

Make revisions.

rea~~ng~~vel ~ 

1

Another

true /false ______ K2

practice item P.C-40

7

H

p.C-33

C—41

~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

.
____ ____ V.--.--.V--~~~~~~~

K3 Constructed response format.

~~~~~~~ deFines 

~ : ~~~~~~~~~~~S~~~~~ ! 

1

Type in stem. Any Use option.

options V

usd 

_________ 

__j

Make revisions.

reading level ‘~

‘ Revise y

exceeded in —

Study examples. Assistance

needed to enter

correct answers

__ 

1

Enter as many as

4 correct answers.

I

C—42

-, 

-V — 

~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .

~ .. -

~

- ~~- — ~~~~~~~~~~~~~~~~~~~~~~ “~~~~~~~ -
-

~ 

V.V~~~~~~~~

-V

~~ 

-

Determine scoring

options.

<

“

\~~ 

Type change if

C— 4 3

~~~~~~~~~~~~~~~~~~~~
Different

fe~~~~~~or 

~

Ty pe general correct answer

feedback.

c~~~±’~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make revisions.

K3.1

p.C45

C—44
~~~

—

~

‘—-

~~~~~~~~~~~

:--— --- 

~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —..—— - ~~~~~~~~~~~~~~~~~~~~~~ ~~~ I

I

1(3.1

Anticipated N 3 wrong answers

for item

7

‘9

Assistance Study examples.

ne.ded to ent,r V

anticipated Wrong answers

7

N

Enter as many as 

F

four anticipated

wrong answers.

C—45

____ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Type correct

answer feedback 

Make revisions.

for answer 1. Is y Revise ~

y

readmg level feedback

Ef~: ::r . ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Make revisions.

p,C 45

C—46

~~~~~~~ •~~~~~

..
Type in number f ~ °~~~~~P~

Are

(

‘ attempts 

~ —4~ c49 ) greater than 2

7

V

Different

try again V 

feedbacks for ~~~ p.C-48

each attempt

N

Type general Make revisions.

try again feedback . Is y Revise V

— reading level feedback exceeded 7 7

N N

Are N ~~~ 3 attempts p.C49 permitted

7

V

C—4 7

__ -. _________ — —

~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~
1(3.3

Type try again 

feedback for Make revis ions. _<

~ > <:E:i:>{

I ~

L

~~

i 

~~~ ~

zI

~

E~

::

~

- L 11

C-48

T —~~~~~--

-~-—~~~ ~~-

~~~~~~~ ~~~~~~~~~~~~ ---~.- ~~~~~~~~ 
_ -

_— — . -

~~

_. —_-— - -. -- __

- .— .—.—.-._a___ . — - -

~ - - - - - - . ------- -_r---__-.---_ - - —

1(3.4

Another N H .

constructed response p.C~

33 item

7

V

1(3

p.C-42

C—49 

~~~~~~~~~~~~~~ ~~ -- --
_____________________________________________________ ~

1

— . .—- -

M

Use learning

Learning V obI.ctive

obèectwe created preparation —

off line worksheet.

7

N

Enter learning

obiective.

______________________

~~~~~~~~~~~~~~~~~~~~~~ 

~~~ 

Make revisions 

—

C-50

I _

~~

_ 

- ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~ 

.—— ~~~~~~~~~~~~~~~~
I 

I 

- ..—---——— I

- Return to index.

No more

Type in guideline

1 number.

P Review guidelines

I

~~~~~~~~~~~~~r N

.

1.

C-51 I __________________ ______ 

~~~~~~~~~~
- r 

-

~ 

-

~~ 

- -

~

N Stop tempo raril y.

Lesson Type in new lesson

- 

for formatting \f name for final

instruction changed ~

tor.ge Of

7 instr uction . —

N ___ 

—

Sign of t.

Review lesson N (Hit STOP 

Sign into inqu iry 2

on line 

when teedy to

Continue.

V

See manual for

conversion

instructions.

__________

Review lesson.

See manual for

instructions .

C—52

~~~~~~~~~~~~~~~~~~~~~~ 

---.

~~~

—.

~~~~~~ -: ~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-

~~ I. 

- 

-~
-- —C- ~~~~~~~~~~~~~~~~~~~~~~~ - -

F Stop, lesson is finishe d.

Type in new

Lesson lesson name for

for formatting final storage of

instruction instructions.

shanged 

See manual for

1

conversion

instructions.

Review lesson.

End CAl lesson

creation.

See Appendix B

C—53

L~. ~

—: -

~

-

~~~

--- 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
--- 

:

Best 

Available 

Copy 
DISTRI B UTI ON

AHI I-il.. . I st

.1 OASD (M&RAI 2 IIQUSACEJI( It 0’) ATTN L iwar’.

- , H()OA uiiA’,

)( i S!’ ))1)USACDE C Ft 0.) . ATTN A T I C IX F il o, Fart,,, ’

.

IUN1A IOAP I PBP - , 

~

; t i[lc Ft ~~~~~~~~~ ‘)i, s,,, ATTN I iIs . ’

~s

HOt tA ii WA \ HI I LJSAI’ACDC . Ft 13e. i,uu.,. Ha u s,, Al IN A1(P HR

ii) P A It) API 14)11 II ii I ‘‘A Comm El, 5.-I Ft Mc.. ru ,utii - AT 1 N ATSN F A

14(J(IA )%RD ioi 1 USAI( ’. Ft M01’. - . 

~h Al TN AMSE t (‘I HOP

I IHIDA IL1AMI oor ci I USA FC, It Morn,,,,,, P. ATTN . AMSf I PA P

1 HODA IDAPi PM! A~ I USAIC . Ft Mi, ,’.’ ii ATTN AMSE L SI (‘6

I ‘40 04 IDACH PP.’ A~ 1 USAIC. Ft M~r,mout)u ATTN C Fact flu.. B.

I HI)r)A 1DAPI HREi I USA Materials Sys A,.aI A,j’... A(wu.Ice,, ATTN AMXSY P

I )4(M)A )()API MINi IH 1 Eulip.woo.i A,ser’aI Aheudeen ATTN: SARFA I H

I HODA iOAt’I OW’ I USA Ooi Cli & Sd. AIu,’.dee.i ATT N ATSI, TFM (

~

1-4011)4 WAPI 1-461 - 2 USA Hn,ri F.uqr I..). Aho,,iee,. AT TN L,iirar’, flu,

I 14 00A (DAPE t.,PSI I USA Couuutuet Aim’. t..q Rd Ft Beuu’u.uuu i. ATTN Ad Suiu,,cso.

I H)JDA OAF 0 MI Ai I USA Iiu)J”ti’. Hum RSC)u Unit . It Btnninq. ATTN C)..’-’

I HODA IDAHO ARS P) 1 USA I.uIauitr’. But , Ft B.’nnurig ATTN STFBC i I

I •400A IDAPc. P45 Al ‘ USASMA . Ft Bliss ATTN ATSS i PC

I40f)A ( OLISA OR) I lISA A,. Del Sc). Ft Ri’s, AI1N ATSA CT!) Mt

ill -U n II3AMIi 6061 1 LI SA Au. flit Sc),, ft Bliss , AT1N levi, Ii,

(I IA’ ,l - 1 USA Air 0,) Bc) F I RI.’..., ATTN FillS

I )uHIiA iflAtOPt i 1 USA A~

u flu) Br) Ft Hi.’..., ATTN 51(80 P0

C it 0... iD”i i)IS( ) . A,)eIi ),,, MO I USA Cund & General StI College . Ft Lr,uvenwOut’ u . AT) N Ii.

I ‘

i

~.i AssI uI,~u,, I.’-, OtM’.k&I OAt) (F &IS) I (‘Si C-n,) a. Gsuwral St) Co?i, -

~u’ It Leavenwo. ri , Al TN Al SW SF I

Ii) ‘\ Aii Al ‘(PD S.’.,til. . A TIN ARAGP It I LISA C’”.) A Geuuer~

uI 5,1 C.uii.’,p’, It ~ , .-,,s’. ,, ’ ‘(‘TIN Ii Als o’,

1 HO F irs) A ruuuy, A TTN. iF IA UI T i I I ISA ‘ “) - ed A ‘ slid’) P’s A, I I is ’’ ion ti ATTN fleluC’?’

HO F ft)r Aniriy F~ 5jn. I4,,urtOn 1 USA Combined Arms Cmbt Os.. Air . Ft leeses’.wo,th ATTN: ~ CS

I I) . ’ . Aims SI) Studies OIc , ATTN OAVCSA (DSP) 1 USA Combu nid Arms Cmlii 111ev Act , Ft Leavenwor th, ATTN: ATCASA

I (Ii. u ’l,wi f 5,) Sr .1, - - , it),- I lISA 1 ,,,,,) -‘‘ . ‘d Arms (‘miii (1,’, Art , Ft I ravunwortli 41111. A1’CACO -F

ii Si~i H Al TN CI’S cS) ’ 1 USA (‘our,) .,

~‘ ,1 A ,,.’. ( nit’, 1).”. Air , It t..ss.-uuwouth . ATTN ATCAC r i i

I thu A..,,’. L it’ I’eu,tw uou . All N RSI) I’).,ef t USAF I.’UM Night Vision Lulu Ft fl”hu’~” Al ‘N ANSF I—NV - SD

I P’r Army L i ’ , Pu’ntaqo.. ATTN ANRAL 3 USA Compute, 5’... Cmii. Ft Beiwou. AT 1N Ted s library

1 01,. , A13t Sect it the Army (R&D) I USAMERI) C Fr Bei,ouu ATTN SISFR 110

I Iu’ch Supiuuuu t Gic OJCS I USA Fog S.’h, Ft Bei,o,’ , ATTN L.t,r u.ry

USASA . A,iu,ur,touu A TIN tAR !) I USA Topographic Lab , Ft BrIms, ATTN I 1) TO S

(ISA PM). UI’., 0ru,Iu,um, A T TN I Ii’ Scu.’urcr”. 13., I LI SA T .t.oqrapt.ic I ui. Fr Hrdvu’.u AT TN STINI’O Cent’., 1’ IISARIIM N.,. ATTN SGRD UI CA I lISA 1 - ‘ , uuSI, ,iuh,, L ii’ Ft Be),,,.. ATTN (Ii. GSL

— I iruAl I.’ u I ..’. -. A t ? N ‘ it tt MI) A I I ISA tu. t.-ilu .j. ‘i. Cu. A Sd- i t II.,,, I.’.. .. AT1 N CII) MS

I LI SAIMA . It Itu.n~

u

~ Al IN A IS U Cli) (IN I USA I ,,,.‘I!,,irii,’

, i

~~

, P.. S.’). Ft Hu,,’t,,u, .u A I TN AIS d O—MS

I USAIMA . Ft B. ,etu i . AT TN . M,ir.tuat Liii I LISA i..teIi,u,encr- Ct. A Sc)., It Huilu,’hin, Al T N 4151- Ti

1 it S WAC Ct, & SrI,, Ft McC)eila.u ATTN. L it ’ I lISA t’uleiI,.uencr’ Ct ’ A Sc). Ft Hi,ul,d).,ui’a ATTN 4151 TEX CS

US WAC CI, & Sc). , Ft McClellan, A TTN : iou Dii I lI);,\ inleti-.ir’ruct’ Ct. & Sc)i . Ft H,, . ,,’i , ,,, ., ATtN ATSI CIS OR

I lISA Oua.teumasteu ‘I ii F u Lee . ATTN ATSM-T{ I LISA hi’ uu~’

nce Ct. & Sd’ Ft ) uaehuuca ATTN ATSi (‘.10 OT

I lr.teiiugence Material Des Plc EWI . It Hu,iai,,nI 1 USA ?‘uteliigenve Cii & ciii , Ft Huachuca . A1TN’ 4151 (‘113 CS

I USA SE Snt.i.u i 5,-i Ft (;.uru(nn, ATTN AlSO IA I lISA I,.teI)i~~’nc,’ Ct. & ccl’ Ft Huarhuuca. ATTN OAS ’S R()

I USA L ’h,u

~’? . -.’ Ci & ‘;‘ I’ 1,4Cc. ’ .. ‘I,..’ ATTN ATSC IF R~) I LISA i’ileli.up’nc.’ Cr ’ A Sit. It Huuarh,,. ... ATTN 41St TIM

1 I ‘ SAT SC)) . F’ Lust.’., All N I i,., A, ’

’.isu.r 1 USA I..tu’iisp’iics’ I’,. P. Sc’). Ft Hu,.,vhu,u-,i - At TN I 1,, a.’,

I USA Wa’ Cu.iIrip Cj,i,slu- 8.”. a.,k’. AT TN L.ti I CUR . HO Ft 14,, ,, i’,,, ., Al TN 1,’

,’). Re) O s

W RAIR , Nr-~~i ..i ’ s . 0 - -ii , D.~ 7 CUR . USA Elect.,.. ,- Prug ct ATTN ~~ I-FP Ml S

I DLI . 51)4 M- - ,‘.

~~ 1 HO. ICATA . ATTN Tech luh,aut,

I lISA Co,,re~

,t Aural A.

~

.y Bethesda. ATTN. MOC A MR I HO. TCATA . ATTN AT CAT OP 0 1114,.,,,)

1 USA Cor s ’etut Anal Agcy Bethesda. ATTN: MOCA-JF 1 lISA RfCitiituflQCmd. Ft SPreruulan ATTN USARCPM P

I (ISA A,ct .c Test CI’ . APO Se,stt i,’ , ATTN STEAC P1, MI I Si

~

nit~r Army Ads. OSAFAGOD ‘(AC. FIg..r Al’ Au, Fl,) N.. 9

I USA Arctic lest CD. APO S aIlis . ATTN: AMSTE-PL iS 1 HO. USARPAC . DCSPFR . APO SF 98558. Al TN: GPP( St

I uSA Armament Cm,) Redstone Arsen al. ATTN~ ATSK -TEM I SI,mson Lub Academy ol Health Scpnees . Ft Sam Hts,u ston

I LISA Armament C~ icI. Rock island , ATTN : AMSAR-TOC i Marine Cows inst . Al IN. Osan MCi

1 FAA-NA FEC , Atlant ic City . ATTN: Lib rary I 440. USMC. Commandant. A1TN~ Code MINT

1 FAA NAFEC , At ianti c Cuty , ATIN: Human Engr Br 1 HO. USMC . Com mand ant. ATTN Code MPi 20 26

1 FAA A.,onautscal Ct. . Oktaitoma City, ATTN: AAC 440 7 USCG Academy , New L ondon . AT TN: Admu,s.ous

2 USA rid A,t

~ Sc)., Ft Sill ATTN Libr ary 7 USCG Academy. New L ondon. ATTN I .tr,a’y

I (ISA Armo r Sc)., Ft Know , ATTN Library t (15CC Training C,, . N’( ATTN CO

I (IS A An..u,r Sri,. Ft Inn s , ATTN. ATSB Di 1 1 USd 3 T.a.nr.ig Ct. NY . ATTN E,4i,c Sue Otu ’

I USA Ainu,,, Sc I, I I P u. .,, ATTN A T’SR DI TP 1 (ISCG . (‘sychol Rec Br , OC , Al TN Ci’ I 62

I USA Am ,,.. Sc),, It ~

,ou.r , AITN ATSB CD AL) 1 HO Miii Raiuqe Br , MC I),t , Ouantucu, ATTN ra.s U...

D-1 

- - - 

‘1 

~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

- 

~~~~;~~~~~~~ _~~~~~~~~~~~~~~~~~ _~~, __ ~~~~~~~~~~~ -

~~
- - “~ ‘~ ~“““ “~~~

I US MM’,’.. 1’ .).’. 1. ,,,,,.,I)), , AMC . AIe.u~nrt.ua . ATTN AMCGS I I I).’) & Civil Inst ii) Ens,,o Medsc.urr . Canaul..

I IISA IRADiX: Ii Mini,. AIIM AT RO (0 1 AIR CRESS . Kan,ingtou. ATTN : I.’)o Sys Bn

s uSATRApOC , Ft “.1,,,,.- AtTN AIPH AD I M.lc raerpsykoloy.slt Tiuneste , Copenhagen

1 (JSAIRADOC Ft M...,.r... Al TN Airs FA 1 Mulicary Attache. Fu,’r,u). Embassy ATTN Do,: Sen

I USA Forces Cmli, F, M. Plum suu. , Alt N 1.1.-n ’ , 1 Mccli.,,.,, Clue) , C.E R.P A . Ansenal, TocuIr,,,, Nan..? F mc.’

2 (ISA Aunat ,ou. tes t Eli . F r Ru.. I’.’ AT TN S TI )l

~ P1,) 1 Pi.u. Sc.eu.tutir 0)1. ARId H,,,,, Engr R..ch 0.,. M,iuusIu’.

I (ISA Ag. Is., As’.,tu,cuc ‘.,ii,-I , I I Hi,, I.’. A I IN I twa’’. u,I Dc’l,,,se, New Dell,.

I USA Ag,’.).uu A..s..I,on SaI,’Iy , Ft Rucken , ATTN E,luc .

,

,,i,,’.. I Pers H’.,’). Ole Litiraty, AKA , Israe l Delco’.,’ I’o, - -, I USA Aviat,,,n Sc). I? Hs.u k u’ . ATTN P0 Drawer ~ I Minister., ca., Defenuie, 000P/KL AId Soc,aoi

I HOUSA A,,,,? -i. S~~ Ciu. sj St L’s, - ATTN AMSAV ?DR Psychologus,:he ?aken. T he Hague , Nelhi’riamN

2 l..i

~.A Ac. ..,. ’ ‘,,“. 1. - .t A.. F ui

~ i. Al H, All N SAY IF I

I USA i’ [Sr i)..). 1.51,’ . Al TN A ISA lI M

I )154 A, M i - l i , l

~ , ‘At ) . , LI. Mil l F I . ? , Al IN: SAVOL AS

I l’,A Asian ,,.. 5,,). H,”. T u g Mg) , Ft H,ucker , ATTN ATST I—P TM

I USA Avial,ou, Sc)., CO. Fr Rud er 41111 ATST - D A

I ISO I)ARCUM . A?r n, ,uiui,- ,. ATTN AMXCD TI

1 110 DARCPM Al....anctu,a, ATTN CUR

1 US Mulrtary Academy, West Pocu.I , A TTN Serials Unit

I 1) 5 MuI.ta;y ACaulu’m.. West Po,,,t ATTN Ole u-i) M~U L,lu’.),t,

‘‘5 M,I,Ia,y .3, ..-Iu’,r,’. ~

l, P ut ATTN MAOR

I USA Stsi.,I...,i, “ in Gnu , UK . (P0 NY , ATTN MASE crC

I Plc of Nasal Awl, A,i,. ’

,l,,u,, ATTN Cu.ute 452

3 ‘if - n) Naval R’.ch, Arlunqro.c . ATTN Code 4i,8

I O)c ut Nasal llsch. Arlington . ATTN: Coule 450

1 (II, it Na,.,i Rich , AiI.uuqti..., ATTN CivIc 441

I N.,,.,,l A m -si’, . Mr-u) Ret I.,), Ppnt,ai.,,la , ATTN Acous Sc). Ic-v

I N..,.,I 4

~

.,’.,~ Mi’,) 14,-c Lii, lk’uut,ic’i,l,i. AT TN Cu rIe 151

I TI.,,.,) Arr ,usc,c MeuI Ri’s lab. Perisacol.,. AITN. Cu,),’ LS

‘i .,I N,,~i’.-.s . A TTN Pe us-OR

I NAVAIR STA , Mouloik , ATTN: Safety Ct,

I N.,’. Ocsar.oguac iiu,c , IX’,. ATTN. Code 6251, Charts & Tech

I C. ‘Ie~ ul Naval A.ual, ATTN Doe CI.

I N.,,A,u SysCusm A 1’T N AIR 5313C

I Nas BuMed, ATTN / 1 3

I NavH.iicop(erSuhSryja 2, IPO SF 96601

1 AFHRL 1(T) W,)tiams MB

1 AFHRL ITT) Lowr y 4(8

1 AFHRL(AS) WPA FB .OH

2 AFHRI IDOJZ) Brooks AFB

1 AFHRL (DOJN) I ,..ckland MB

1 HOUSAF )INYSD)

I HOUSAF IDPXXA)

I A PVTG IRQ) Randolph 4(9

3 AMRL )HIIW PAFB OH

2 A l Intl of T el’ , WPAF B . OH, ATTN: ENE/SL

I MC IXPTDI R.un.ioii,h AFI’.

I LISAF A,’,.,M.”I I ‘c , )lr o’uks AFB ISUL 4/ , ATTN : DOC SEC

I Al (ISA ML? . Au i’.,gru...

I Al LuqCm.i M,-Cl.’I . i

~ At- B. ATTN ALC/ DPCRB

1 A,. F,,rce Acielem’., CU, ATTN: Dept ,u) Be) Sen

S Na,Pers & Dr-v Ct. San Dung’.

2 Navy Mo,l Neuuio) ,syci ’iatr.c Rsch Unit , Si,, Diego

1 N,.. Electtou,,u’ La!., Sari O.e.m . ATTN. Re.. Lab

I Na.. TriuqCu’n. Sin Uii’.~

o. ATTN Code 9000- Lil,

1 Ma~PuistGraSd c . Mont .’.’-y, AIIM: Colic 55Aa

I N.,..P’.sTC,,.aScF.. MurnIc’ui’y, ATTN Code 2124

1 N..vturuglguupCtr . Olianclo. ATTN Ted , Lub

1 US fletut 01 Labor . DC. ATTN Manpower Admin

I US Oet’t ~l Justice , OC. ATTh: Drug Entorce Admin

1 Nil 8w at Stanr)arr)s . DC, ATTN: Computer Into SecI.on

1 N.,) Cteam.ruq Hcuu.se lot MH Info, Rockvilte

I Deutuer Fed eral Ct, , I akewood , A’TTN: BLM

i’d Defense Documentation Center

4 Dir Psych , Army Hq, Russell Ofc~, Canberra

1 Scuer.t .)uc Adv sr , Mu) Bd, Army Hq, Russell Ofcs , Canberra

1 M,i and Air Attache, Austrian Embassy

I Cu t’,’ de Recherche Des Facleurs , Huma,ne cfr’ lit Drfen~~ NaI,nuiale, Bricciels

2 Cau,.u i)lan Ju-,,i.t Stall Wash,tuqtnn

1 C/A., Stall. Roy.iI C~nau ”an iF. ATTN Pert SI,) Au ta) Br

3 Clu e) Cesaduan Di.) A’.” St..)), ATTN: C/CRDS)W)

4 II. - ’ sc~ firS SIa)t . 10,t

~sl, .ni,as~y, Wasi.inqtor’

D’-2

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ ~~~~ - i.4~ -

~~- a .