Technical Report 435

AN ANNOTATED SELECTIVE BIBLIOGRAPHY

ON HUMAN PERFORMANCE IN FAULT

DIAGNOSIS TASKS

William B. Johnson, Sandra H. Rouse,

and William B. Rouse

0 Coordinated Science Laboratory, University of Illinois

MANPOWER & EDUCATIONAL SYSTEMS TECHNICAL AREA

)o,, '

U.S. Army

Research Institute for the Behavioral and Social Sciences

January 1980

Approved for public release; distribution unlimited.

80 7 10 012

L IIA'C
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

FRANKLIN A. HART

JOSEPH ZEIDNER Colonel, US Army

Technical Director Commander

NOTICES

DISTR IOU TI ON Prime(ry distribution of th~is report has been Made by AMt. Please address correspondene

concerning distribution of reports to. U. S. Army Research Institute for the Blehavioral and Social Sciences,

0 ATTN: PERI-TP, 5001 Eisenhower Avenue, Alexandria. Virginia 22333.

FINAL DISPOSITION This report may be destroyed whon it is no longer needed. Please do not return it to

the U. S. Army Research Insitute f or the Behavioral and Social Sciences.

I N=L The findings in this report are not to be construed as an official Department ml rhe Army Position.

unless so designated by other euthorized documents.

LL LIN"
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (When Data BamK )___________________

READ INSTRUCTIONS REPORT DOCUMENTATION PAGE BEFORE COMPLETING FORK

REPORTNUM2. GOVT ACCESSION NO. 3. RECIPI"T-S CATALOG NUMBER

Technical Report 435 &

TST ITYPf.

_%/ AN,*M$OTATED -jELECTIVE JIBLIOGRAPHY ON HUMAN

'PERFORMANCE IN FAULT DIAGNOSIS TASKS -N Final a# 8Jul#79

__ S. PERFORMING OR G.REPT UMR&

William B./Johnson Sandra H. /Rouse' and -- -/DAHCl9-78-G1 Z'-'

3PERFORMING ORGANIZATION NAME AND ADDRESS ) 10. PROGRAM ELEMENT. PROJECT, TASK

Coordinated Science Laboratory, College ofA&WOKUINMBR

Engineering . -. I j AR

University of Illinois7l Urbana, IL 61801

ii. CONTROLLING OFFICE NAME AND ADDRESS62R

US Army Research Institute for the Behavioral andli-jnNj~

Social Sciences (PERI-OK) 1.NM

5001 Eisenhower Avenue.,fAlexand _ VA 223'13 49

14 ONIWING AGENCY NAME & ADDESI diffn CroCntrolling Office) 1S. SECURITY CLASS. (of Ibis report)

~Unclassified T~a DCASFCATION/DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abstract entered I Block It, iffferent from Rep'o")

I*. SUPPLEMENTARY NOTES

Monitored technically by Leon H. Nawrocki, Richard M. Johnson, and Bruce W.

Knerr of the Manpower & Educational Systems Technical Area, 'RI

I9. KEY WORDS (Continuiie an revers, aide It nec.esar mnd Identity by block number)

Maintenance Training

Diagnostic Training

Troubleshooting Training

Fault Diagnosis

2j- 4kSTUACr Cmietwe eareerse ag N noseaey md identify by block numnber)

-2f/This is an annotated bibliography of 61 published works about human

performance in fault diagnosis. The sources are predominatly from the

disciplines of engineering, psychology and education. Computer searches were

conducted of the past 10 years of Psychological Abstracts and ERIC Documents.

Manual searches using references from reports, University of Illinois library

card catalog,0 and solicitation of references from experts in the field

extended' the search to include references dating from the early fifties .

DOR W~ 43 RIINTIOM OFI NOV %IS OBSOLETE Ucasfe

i SECURITY CLASSIFICATION OF THIS PAGE (Whent 0aq0 Eat~r
Technical Report 435

AN ANNOTATED SELECTIVE BIBLIOGRAPHY

ON HUMAN PERFORMANCE IN FAULT

DIAGNOSIS TASKS

William B. Johnson, Sandra H. Rouse,

and William B. Rouse

Coordinated Science Laboratory, University of Illinois

Submitted by:

James D. Baker, Chief MANPOWER & EDUCATIONAL SYSTEMS TECHNICAL AREA

Approved by:

Robert M. Sasmor, Director

BASIC RESEARCH

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

January 1980

Army Project Number Basic Research in Decision-Making

20161102B74F

AoprowAid for publmicle: dlnnbuton unlmid.

, iii
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv

A1

: ... : ... : ...• _ .,- .... ., .,.,' . t ,. - .:... ,, - : -
FOREWORD

The Manpower & Educational Systems Technical Area of the Army

Research Institute for the Behavioral and Social Sciences (ARI) performs

research and development in areas that include educational technology

and training simulation with applicability to military training. Of

special interest is research in the area of computer-based systems for

maintenance training. The development and implementation of such

systems is seen as a means of reducing training time and costs by

providing more highly individualized training than would be otherwise

possible, while at the same time reducing the need for operational

equipment for training.

The report provides an annotated bibliography of research directly

related to human problem-solving behavior in fault diagnosis (troubleshooting) tasks. It provides an information source to assist planners

of future research efforts in this area.

This research is responsive to the requirements of RDT&E Project

2QI61102B74F, "Basic Research in the Behavioral and Social Sciences."

JO INER

chnical Director

tI I'S 

,A&, J 

i r :C,r. .0 

XA~ 1

V ~ 7-x

* - .r." * . .. - - - - -~ - - -- -

* . **L
AN ANNOTATED SELECTIVE BIBLIOGRAPHY ON HUMAN PERFORMANCE IN FAULT

DIAGNOSIS TASKS

BRIEF

Requirement:

In conjunction with an ongoing program of research on the use of

computer simulations for fault diagnosis training, a thorough review of

the existing literature was necessary to provide historical input and to

identify current trends in fault diagnosis work. The results of this

search are presented here as a selective annotated bibliography.

Procedure:

The search procedure consisted of on-line computer search of the

past 10 years of Psychological Abstracts and ERIC Documents through

mid-1978. Manual searches used references from reports as well as from

the card catalog of the University of Illinois library. In addition,

experts in the area of fault diagnosis were sent the references resulting

from the computer and manual searches and asked to suggest any additional

references that should be included.

Findings:

The references revealed by the search procedure were predominately

from the areas of engineering, psychology, and education. Three major

categories of materials are presented. These consist of (1) research

reports, (2) manuals for fault diagnosis, and (3) material indirectly

related to fault diagnosis, such as medical diagnosis and mathematical

problem solving, or items directly related to fault diagnosis that are

not research reports.

Utilization of Findings:

This annotated bibliography will serve as an information source

for planners of future R&D efforts in this area.

vii
Page 1

INTRODUCTION

The authors and several colleagues at the University of

Illinois are investigating the use of computer simulations for

fault diagnosis training. This work runs the gamut from

context-free simulations and context-specific simulations into

the live environment of real equipment troubleshooting. This

research, supported by the U.S. Army Research Institute for the

Behavioral and Social Sciences, includes the development of

models of human fault diagnosis performance and ultimately an

intelligent interactive training simulator. To this end, we have

already run six experiments involving almost 160 subjects who

solved approximately 13,000 fault diagnosis problems.

As part of our work, we have prepared this annotated

selected bibliography on human performance in fault diagnosis

tasks. The sources in this bibliography are predominantly from

the disciplines of engineering, psychology, and education.

Although the primary emphasis is on the research literature, we

have included a representative number of actual troubleshooting

manuals from the automotive and aviation fields.

The literature search included several phases, involving

both on-line and manual searches. Computer searches of the past

ten years of Psychological Abstracts and ERIC documents through

mid-1978 were performed. Manual searches were conducted using

bibliographies from various reports. Using the card catalog at

the University of Illinois library, we obtained materials dating

from the early fifties.
PI

The third phase of this literature search involved the

solicitation of additional references from noted fault diagnosis

and/or computer simulation experts. Over twenty such individuals

were sent the list of references resulting from the first two

phases of our literature search. 'They were asked to skim the

list of references and suggest any additional references that

they felt to be essential for a selected bibliography on human

performance in fault diagnosis tasks. Approximately one-third of

those queried responded. We greatly appreciate their

contributions.

The materials in this bibliography are divided into the

following three major areas:

1. Research Reports

A. Primary Research

B. Secondary Research

2. Manuals for Fault Diagnosis

3. Materials Indirectly Related to Fault Diagnosis

Primary research reports include empirical and/or theoretical

studies aimed directly at human problem solving behavior in fault

diagnosis tasks. Secondary research reports refer to literature

reviews. As noted earlier, a representative set of training and

field manuals is included in the "manuals" category. Indirectly

related materials refer to studies of human behavior in related

domains (e.g. medical diagnosis or problem solving in

mathematics) -, reports that directly relate to fault diagnosis

L A "4 . "
Page 3

but do not provide research results of a type necessary for the

report to be categorized as a "research report." Since the number

of items that could be placed in the "indirectly related"

category was potentially enormous, we limited consideration tP

items that related to human behavior and particularly matched our

interests. Thus, for example, the large body of literature on

maintainability and automated fault diagnosis has been omitted

from this bibliography.

Where possible, we used the abstract that accompanied an

item. However, we usually had to supplement the abstract

considerably in order to make it self-explanatory. The indexing

of each item was based on a consensus of the authors.

We hope to periodically update this bibliography and

re-issue it. To that end, we would appreciate suggestions and

contributions. If possible, when submitting a suggestion for

inclusion of a new item, please include a copy of the item.

. gghh.. A4

,
Page 4

PRIMARY RESEARCH

Bond, N.A., Jr.; Rigney, J.W. (1966) "Bayesian Aspects of

Troubleshooting Behavior." Human Factors. 8:377-383.

Thirty-nine Navy technician trainees filled out a

symptom-malfunction matrix on a blocking oscillator

circuit. The particular sequence of checks used by each

man on each problem was combined with his

symptom-malfunction matrix, via a Bayesian algorithm, to

yield computer estimates of failure likelihoods for each

component. The computer program predicted actual

parts-replacement behaviors in about half of the cases.

Those technicians who start out with valid

symptom-malfunction matrices are more likely to resemble

the Bayesian processor. Subject's troubleshooting

behavior that was characterized as Bayesian correlated

with higher class standing and longer time between tests.

It is suggested that symptom-.malfunction matrix accuracy

could be a training criterion. Students could be trained

until they could produce a symptom-malfunction matrix to

a high standard of physical validity.

7 pages, 3 references

Brooke, J.B.; Duncan, K.D.; Marshall, E.C. (1978) "Interactive

Instruction in Solving Fault Finding Problems." International

Journal Man-Machine Studies. 10:603-611.

Thirty subjects were divided into two groups which solved

60 trials of fault diagnosis problems. One group was

given process information provided by the interactive

software system when an incorrect diagnosis was made. At

that point during the interaction, feedback regarding the

possible failures was given to the subjects. Also, if

subjects continued to ask for additional information when

the solution was obvious, the system informed subjects of

the fact that the failure could be identified. The

control group had no feedback and were not informed about

the correctness of their diagnosis. Analysis of variance

showed a significant effect of training on learning.

9 pages, 2 references
Page 5

Brown, J.S.; Burton, R.R.; Bell, A.G. (1975) "SOPHIE: A Step Toward Creating a Reactive Learning Environment." International Journal of Man-Machine Studies. 7:675-696.

This paper describes a fully operational AI-CAI system

which incorporates artificial- intelligence techniques to

perform question answering, hypothesis verification and

theory formation activities in the domain of electronic

troubleshooting. Much of its logical or inferencing

capabilities are derived from uses of simulation models

in conjunction with numerous procedural specialists. The

system also includes a highly tuned structural parser for

allowing the student to communicate in natural language.

Students can test components in an electronic circuit,

develop hypotheses regarding the fault, and obtain

feedback concerning why a hypothesis is incorrect.

Replacement of parts is allowed only when the student can

answer correctly queries about why he thinks the parts

should be replaced. Although the system is extremely

large it is sufficiently fast to be thoroughly exercised

in a training or classroom environment.

22 pages, 11 references

1 . -,- ' "',
Page 6

Brown, J.S.; Rubinstein, R.; Burton, R. (1976) "Reactive

Learning Environment for Computer Assisted Electronics

Instruction." Lowry Air Force Base, CO: AFHRL-TR-76-68.

The development of several new computer based strategies

for teaching .troubleshooting principles to electronics

technicians is described. The report documents an

experiment in which those materials were presented to

student technicians to determine their attitudes toward

the techniques, and to determine whether the resulting

training resulted in improvement of their technical

skills. Results in both cases were positive with

students responding very favorably to the materials and

with their performance improving quantitatively and

qualitatively after the instruction.

Seventeen college level students in an electronics

curriculum were subjects in an experiment for evaluating

alternative training and teaching methodologies on

troubleshooting behavior. Subjects were divided into

four groups, each group receiving a different sequence of

teaching and training activities. Most of the evaluation

is qualitative and based on extensive interviews and

questionnaires. However, the fourth group was

administered a pre-test for measuring basic understanding

of power supply troubleshooting. This pre-test also

included two faults to troubleshoot before the laboratory

games began. A post-test was administered to measure the

effect of computer-based activities on learning.

Students' scores reflected their ability to make correct

fault hypotheses given a set of symptoms. Improvement in

all subjects' scores was found to be significant at the

.05 level. Also, subjects experienced an average

decrease in task time by 41%, significant at the .05

level.

140 pages

!.' A I
Page 7

Crawford, A.M.; Crawford, K.S. (1978) "Simulation of

Operational Equipment with a Computer-Based System: A Low Cost

Training Technology." Human Factors. 20(2):215-224.

The feasibility and effectiveness of teaching performance skills using a computer-based training (CBT) methodology

were investigated. Graphic simulations of the appearance

and functions of a system in an anti-submarine aircraft

were presented to students within an instructional

framework. The objective was to determine whether CBT

could be used for low cost, part-task training. The

subjects were new co-pilot trainees for the S-3A

anti-submarine aircraft. The performance of the 22 CBT

students, as measured on a high fidelity simulator, was

compared to that of 22 students who had gone through conventional training consisting of workbook study and

hands-on practice in the simulator. Results showed that

CBT students performed the necessary skills as well

before practice in the simulator as conventionally

trained students could after this practice. Performance

measures were the number of tasks completed and the time

to completion. The authors felt that CBT was an

effective and low-cost means of part-task training of

certain performance skills.

10 pages, 11 references
Page 8

Crooks, W.H.; Kuppin, M.A.; Freedy, A. (1977) "Application of

Adaptive Decision Aiding Systems to Computer-Assisted

Instruction: Adaptive Computerized Training System (ACTS)."

Arlington, VA: U.S. Army Research Institute for the Behavioral

and Social Sciences. PATR-1028-77-1.

This report describes the Adaptive Computerized Training

System (ACTS) which combines the techniques of circuit

simulation, artificial intelligence, decision modeling,

and adaptive computer-assisted instruction to provide

training in decision making. The ACTS incorporates an

adaptive computer program which learns a student's

diagnostic value structure and uses th'is structure to

train the student in practical decision making. This

report describes the development and operation of the

ACTS as it is applied to training electronics

troubleshooting.

Six college students with electronics backgrounds served

as subjects in solving 45 fault diagnosis problems.

Subjects were divided into three groups receiving the

following initial aiding: 1) the expert's three best

alternatives were displayed to the subject and after the

subject's choice was made, additional feedback was

provided by identifying the expert's best alternative;

2) subjects first indicated alternatives before expert's

alternatives were displayed; and 3) no feedback or

indication of expert's choice. Results indicate that

those who received aiding or feedback solved problems at

lower costs than subjects who received neither aiding nor

feedback. Practice was shown to have a positive effect

for all groups in terms of reducing costs.

54 pages
Page 9

Elliott, T.K. (1966) "A Comparison of Three Methods for

Presenting Procedural Troubleshooting Information." Aerospace

Medical Research Laboratories, Wright-Patterson Air Force Base,

Ohio. Report No. AMRL-TR-66-191.

The effects of subject aptitude and performance aide mode

of presentation on the performance of procedural,

between-stage troubleshooting tasks on a real piece of

electronic equipment were studied. The study used

nondecison aids presented in three modes, namely, an

automatic retrieval of visual information, an automatic

retrieval of audio information, and a manual retrieval of

visual information. Two aptitude groups (Air Force

electronic index 40-60 and 75-90) with no previous

electronic training or experience were given from 3 to 5

hours of task training before the experiment. Each

subject solved 26 actual, and 11 synthetic problems. The

study indicated no difference in effectiveness of aids

using visual representations. Both were superior to

audio presentations. The two aptitude groups were

equally effective in performing troubleshooting tasks

using nondecision aids. In comparison with a previous

study using decision aids, the study showed nondecision

aids produced superior performance on the same

between-stage troubleshooting problems using similar

subjects.

38 pages, 10 references

ri

i:

I
Page 10

Elliott, T.K.; Joyce, R.P. (1971) "An Experimental Evaluation

of a Method for Simplifying Electronic Maintenance." Human

Factors. 13(3):217-227.

Two groups of subjects solved the same set of 13

troubleshooting and repair problems in seven

solid-state-circuit modules which contained as many as

five stages each. Both groups used the same hand tools

and test equipment. One group was composed of 41

conventionally trained Air Force 5- and 7-level

technicians who normally maintain such equipment as part

of their jobs. The technicians used the same

troubleshooting techniques they ordinarily used on their

jobs, and they were provided with a performance aid

resembling an Air Force technical order. The other group

was composed of 20 high-school students with no prior

training or experience in electronics. Their training

for this study consisted of a 12-hour course in the use

of hand tools and test equipment and in the use of the

proceduralized troubleshooting aid evaluated in this

study. The aid indicated which check to make based upon

the outcome of previous checks. Using the proceduralized

troubleshooting aid, the high-school students took

significantly less troubleshooting time than did the

experienced technicians using normal techniques;

however, the technicians requirdd significantly less

repair time and made significantly fewer errors than did

the students. These differences may be accounted for in

part by differences in the tasks and scoring methods for

the two groups, and there were no differences between the

two groups on a number of other measures. However, the

difference in training time and, therefore, cost of training between the two groups was so great as to

suggest the possibility that job-relevant training and

proceduralization of the task can introduce substantial

savings, even after the cost of developing the special

performance aids required by proceduralized

troubleshooting is subtracted.

11 pages, 10 references I

r ., -,- ., 
Page 11

Ely, J.H.; Hall, N.B.; Van Albert, C.E. (1960) "Coding

Electronic Equipment to Facilitate Maintenance." IRE Transactions

on Human Factors in Electronics. HFE-1(2):66-69.

The aim of this study was to improve maintenance of

electronic equipment by detcrmining what information to

place on the equipment and developing techniques for its

display. Trips to various communications installations

were made in order to examine equipment, observe

maintenance activities, examine mAnuals, and interview

personnel. Detailed data were collected showing the

marked variation between observed test point readings and

those called for in maintenance manuals. Recommendations

were developed concerning information to be displayed on

prime electronic equipment, covering 1) designation of

functional groupings, 2) identification of signal paths,

3) identification and sequence for using test points, and

4) presentation of historical information for each test

point. These recommendations were evaluated by

incorporating them on an oscilloscope and evaluating the

performance of 18 technicians. Comparing this "coded"

scope with an identical uncoded one showed that

troubleshooting time was reduced on the average to

approximately one-half that for the uncoded equipment.

4 pages, 3 references

Finch, C.R. (1971) "Troubleshooting Instruction in

Vocational-Technical Education via Dynamic Simulation."

Pennsylvania Department of Education, Bureau of Vocational,

Technical and Continuing Education. ERIC Document Reproduction

Service ED 055 204.

This study used a commercial real equipment simulator to

train automechanics for fault diagnosis. The research

also explored the attitude of the students and the

instructors towards the simulator. The experimental

group received simulator practice, while the control 4 group received no treatment. Those students who

practiced on the simulator had better troubleshooting

performance on four of the six criteria measures. The

instructors were satisfied with the simulator, and the

students believed that it was at least as good as

traditional classroom or laboratory instruction.

107 pages, 21 references

I

-. - .-. - -- .
Page 12

Finch, C.R. (1972) "The Effectiveness of Selected

Self-Instructional Approaches in Teaching Diagnostic Problem

Solving." The Journal of Educational Research. 65:219-223.

This study examined the effects of three

self-instructional approaches of teaching diagnostic

problem-solving (troubleshooting). Two of the approaches

attempted to simulate student-equipment interaction via

programmed and text instruction while a third approach

utilized actual equipment. After forty-five community

college students received an orientation to equipment

operation, fifteen were randomly assigned to each of the three treatments. Criterion variables included

troubleshooting knowledge, troubleshooting performance,

and attitude toward instruction received. Analysis of

covariance indicated that the equipment approach was

superior to non-equipment simulation approaches in terms

of troubleshooting performance scores. Correlational

analyses of criterion variables provided some evidence

that relationships between attitude and behavior were

dependent upon the instructional approach used.

5 pages, 10 references

Freedy, A.; Crooks, W. H. (1975) Use of an Adaptive DecisioQ

Model in Computer-Assisted Tr-aining on Electronics

Troulesh53ting. Woodland Hills, CA: Perceptio--ns Inc., Apri

1975.

The report discusses a system for Computerized Decision

Training (CDT). This system has the capability to

compare subjects' troubleshooting decisions with the

optimal as defined by "experts." The goal was to train

for quality decision making rather than cookbook

troubleshooting. The simulations are computer generated

problems applicable to electronic circuits. The problems

are presented on a CRT and permit the subject to: check

symptoms, measure a component value, replace a module, or

declare circuit to be fault free. The adaptive decision

model was found to converge on the decision behavior of

the students who quickly settle into consistent decision

strategies. The authors' conclusion notes that there is

a need for considerably more research in this area.

9 pages, 17 references

!I

* K

~ ..
Page 13

Glaser, R.; Damrin, D.E.; Gardner, F.M. (1954) "The Tab Item:

A Technique for the Measurement of Proficiency in Diagnostic

Problem Solving Tasks." Educational and Psychological

Measurement. 14:283-293.

This work describes a method of fault diagnosis training

relying on the "tab item." The tab item is a foldout

cardboard flow chart which allows the troubleshooter to

gather information by pulling tabs. This system provides

the following: A symptom, check procedures, and a list

of possible failures. This system therefore attmepts to

reproduce the troubleshooting situation on paper.

Measures for scoring troubleshooting performance which

are discussed are: number of tests performed, number of

redundant tests, test relevancy, and test adequacy.

Although this article offers no empirical evidence of the

instructional value of the tab item, it does speculate as

to its potential value across a broad spectrum of

technical training. Addit-ional uses of the Tab Test can

be found in Standlee (1956) and Fattu (1956).

10 pages

* "~
Page 14

Glass, A.A. (1967) "Problem-Solving Techniques and

Troubleshooting Simulators in Training Electronic Repairmen."

Doctoral Thesis. Columbia University.

The purpose of this project was to study the practical

and operational application of problem-solving concepts

by introducing into the regular training of military

electronic repairmen two experimental methods of teaching

troubleshooting and to evaluate their effectiveness in

promoting success. These two experimental methods were:

1. a series of recorded lectures dealing with problem

solving methods, and 2. a series of paper and pencil

troubleshooting simulators to provide the opportunity to

apply problem solving techniques. There were two

simulators, one commercial and the other custom-built.

The main criteria for evaluation of the experimental

variables were performance tests on actual electronic

equipment (N:240). The method for administering and

scoring the performance tests was the result of extensive

pilot studies. The essence of the scoring method was to

score for sequence of logic in troubleshooting, time, and

solution success. The conclusions were: 1. Subjects

receiving problem-solving lectures were superior on

performance test to subjects not receiving them. 2.

Subjects using simulators were superior on performance

criteria tests to those not using the devices, but only

when the custom-built device was used. 3. Subjects

receiving a combination of lectures and devices were

superior on performance criteria tests to subjects not

receiving the combination. 4. The use of the

custom-built simulator produced superior performance

criteria test scores as compared to subjects using the

commercial version, only when the lectures were used as

well.

136 pages, 54 references

* ~*.

*; I

.74
Page 15

Goldbeck, R.A.; Bernstein, B.B.; Hillix, W.A.; Marx, M.A.

(1957) "Application of the Half-Split Technique to

Problem-Solving Tasks." Journal of Experimental Psychology.

53(5):330-338.

An experiment was designed (1) to test the hypothesis

that the half-split instruction increased efficiency for

simple problems and (2) to determine half-split

efficiency for more complex problems. A second

experiment was designed to determine whether additional

practice, different methods of instruction, and more

complex training materials would produce an improvement

in the subject's ability to arrive at a set of

alternatives, and, if so, whether the half-split method

would then be more efficient.

In experiment I, 90 male university students participated

as subjects and were instructed to replace a faulty unit

in a troubleshooting task. Thirty were assigned problems

to each of three complexity levels. Based on scores

obtained from a problem solving test, subjects were

randomly assigned to a control group and a group which

received explicit instructions regarding the application

of the half-split technique. Solution time for the

half-split group was significantly longer than the

control group. Increasing complexity resulted in an

increase in the number of tests only for those subjects

who scored low on the problem solving test. Results

suggest that the half-split technique is superior under

certain conditions, such as with diagrams of low

complexity where subjects apparently had little

difficulty in finding the trouble-possible units, and

where subjects avoided incorrect replacements.

Experiment II was an attempt to improve the effectiveness

of the half-split method by improving instruction and

giving extended practice. Only problems of the highest

level of complexity were used. Forty male-university

students participated and were randomly assigned to four

training groups. The deductive half-split group was like

the previous half-split group but given more practice.

The instructions and practice given the deductive

half-split group were more effective in reducing the

number of tests than were the less extensive instructions

* in Experiment I. Results of both experiments indicate

that the empirical superiority of the half-split

technique is a function of the nature of the task, and

the degree and type of training given the troubleshooter.

From these experiments, it is now clear that future

research on training methods should emphasize the more

basic deductive phase rather than the relatively

mechanical half-splitting aspect of the process.

9 pages, 3 references
Page 16

Landa, S. (1972) CATTS: Computer-Aided Training in

Troubleshooting. (RO18-PR) Santa Monica, CA: Rand Corporation.

ERIC Document Reproduction Service No. ED 066 888.

The Rand Corporation's Programmer-Oriented Graphics

Operation (POGO) was used in the design, implementation

and testing of a computer-assisted instruction course to

train airmen in malfunction diagnosis - CATTS (Computer

Aided Training in Troubleshooting). The design of the

course attempted to reduce the problems of computer

graphics for both instructor and student. The

observations of Air Force instructors using the system

suggest that the system is workable. No empirical

evidence was gathered. The report emphasizes the POGO

software more than troubleshooting.

108 pages, 11 references

Mallory, W.J.; Elliott, T.K. (December 1978) "Measuring

Troubleshooting Skills Using Hardware-Free Simulation." Air Force

Human Resources Laboratory, AFHRL-TR-78-47.

The hypothesis formulated was that troubleshooting

performance on simulation would be predictive of the same

tasks on real equipment. Sixteen instructors and fifteen

beginning students in a precision measurement equipment

course were given the same problems with two treatment

levels. Problems were solved for criterion of job sample

test (JST) and simulation or symbolic performance test

(SPT). The task in each case was to locate the fault at

the stage level rather than component level. Performance

measures included mean percent correct solution, time to

complete problems, and number of steps to completion.

Statistically significant correlations indicate a high

degree of similarity between the job sample test and

symbolic performance test. An algorithm is presented which is used to evaluate problem difficulty.

Correlation of the hypothesis problem difficulty with

empirical problem difficulty (obtained by using test

results of subjects) did not produce significant

agreement. It was noted that the problem difficulty

algorithm was insensitive to feedback loops which

occurred in some of the problems and would be modified in

the future.

120 pages, 25 references
Page 17

Mills, R.G. (1971) "Probability Processing and Diagnostic Search: 20 Alternatives, 500 Trials." Psychonomic Science.

24(6):289-292.

Ss' capabilities for processing probability information

in performing a diagnostic search task similar to

troubleshooting is explored. Six Ss searched (queried)

among 20 components in a series circuit to isolate a

single component failure per trial. Failures were

generated according to 10 alternative

probability-of-component-failure distributions (D), each

N = 500. Prior to performing their search on a trial, Ss

predicted the failure for that trial. Of primary

interest was the influence of varying D on predictions

and search performance. Results indicate that: (1)

distributions of predictions tend-to match actual D in

shape and location but exhibit certain characteristics;

(2) probability of correct prediction is low and is

influenced by D; (3) search strategies are suboptimal

(The optimal solution required an average of 14% fewer

tests.) but "logical" and to some extent predictable;

and (4) in some instances, search strategy appears to be

related to predictions.

4 pages, 13 references

Osborn, W.C. (1970) "An Approach to the Development of Synthetic

Performance Tests For Use in Training Evaluation." presented at

the 12th Annual Military Testing Association Conference, French

Lick, Indiana.

This paper describes an approach to the selection of

inexpensive alternatives to fully job-relevant

performance tests. Based on the concept of content

validity, the approach is to utilize the skill and

knowledge required of the trainee as a criterion for

assessing the cost and job-relevance of fabricated

alternatives to performance tests. Two job tasks are

analyzed to illustrate the concept.

6 pages

4.

AUX

IL. I
Page 18

Rasmussen, J.; Jensen, A. (1974) "Mental Procedures in

Real-LIfe Tasks: A Case Study of Electronic Troubleshooting."

Ergonomics, 17(3):293-307.

An experiment was conducted at a nuclear research

institute to investigate troubleshooting approaches used

by skilled electronics repairmen during on-the-job tasks.

A verbal record of the repairmen's strategies was

collected for data analysis. Codes were applied to

describe the heuristics employed: 1) topographic search

- use the topographic location of measuring points; 2)

functional search - use the normal functional relation

between a feature in the system's response and a specific

part in the system; and 3) search by evaluation of a

fault. This last heuristic implies the human is using a

mental model of the system relating changes to internal

signals, parts or components to the changes observed in

the system response. Results suggest that procedures

considered unsystematic in a laboratory study may turn

out to be rational and systematic when one listens to the

individual's rationalization. Also, it was found that

verbal records appear to be useful for recording rapid

sequences of simple decisions based upon informationally

redundant observations.

15 pages, 11 references
Page 19

Rasmussen, J. (1978) Notes on Diagnostic Strategies in Process

Plant Environment. Roskilde, Denmark: Riso National Laboratory,

R iso-M-T83.

In the report are discussed some aspects of state

identification and diagnosis in process plant control

which must be considered in connection with automatic

disturbance analysis and man-machine interface systems.

The content of the diagnostic process depends upon the

overall goal of the diagnostician - whether it is to

protect the plant, to maintain operation or to repair,

and it may not necessarily include a determination of the

cause itself. Important aspects of diagnosis include

critical variables and causal flow paths which are

intimately related to the paths along which events and

changes propagate through the system, i.e. to the flow

of energy, matter and information which together form

complex, interacting flow structures.

In the process plant environment, a diagnostic task

implies a search to identify a change from a normal or

planned plant state. Several elementary strategies can

be identified. In the report a distinction is drawn

between two main groups - topographic search strategies,

performed as search through the system with reference to

a model of normal plant state; and symptomatic search

strategies, performed as a search through a library of

abnormal state models with reference to the actual plant

state.

Typical properties of the different strategies are

discussed such as processing capacity requirements and

dependence upon a priori analysis.

The role of the elementary strategies in the overall

diagnostic task is not discussed since this is considered

possible only with reference to selected scenarios

describing realistic and comnplex real life situations

arising from disturbed plant operations. The aim of the

report is to identify and formulate some of the issues

which should be taken into account when creating such

scenarios based on careful field studies and analysis of

incident reports. Based on these scenarios, different

overall diagnostic strategies can then be formulated and

tested by simulator experiments.

36 pages, 12 references

* I
Page 20

Rigney, J.W. (1973) "A Discussion of Behavioral Technology

Laboratories CAI Projects in Relation to a CAI Test-Bed Concept."

Behavioral Technology Laboratories, Report No. 71, University of

Southern California, Los Angeles, CA.

The elements of an instructional system are discussed and.

some literature bearing on the functions of these

elements is reviewed, with the objectives of stimulating

thought about an instructional system as a CAI test-bed

and of pointing out some results of laboratory research,

particularly in cognitive psychology, that are believed

to be worthy of the instructional technologist's

attention.

In the second part of the report, selected parts of CAI

projects underway at the Behavioral Technology

Laboratories are described in relation to instructional

system elements, to give an overview of this research.

The projects include: work on computer grahics to (1)

simulate front-panel topography and functional

organization of circu-its (2) facilitate interpretation

of relative motion, and (3) create visual analogies for

abstract concepts and processes to serve as the basis for

students to develop their own internal representations;

work on new methods for recording cortical evoked

potentials and correlating them with learning and memory

processes; work on a dynamic programming model for

adaptive control of problem-solving types of CAI; and

work on three-dimensional mock-ups of electronic

equipment to be placed on-line with a CAI system.

40 pages, 46 references

r .
Page 21

Rigney, J.W.; Towne, D.M. (1969) "Computer Techniques for

Analyzing the Microstructure of Serial-Action Work in Industry."

Human Factors. 11(2):113-122.

Three computer-based techniques for analyzing and

simulating serial-action tasks are described. The first,

called BETS (a Bayesian model), measured the efficiency,

in terms of -expected information, of tests made by

technicians who -were troubleshooting malfunctioning

electronic circuits. It computed efficiency ratios for a

technician's test sequences in comparison to optimal

sequences. The second method, ARMAN, generated detailed

time-and-motion analyses from gross descriptions of

serial-action tasks and man-machine interfaces and computed the time costs of these tasks.

The third technique incorporates a general model of the

action-goal structure of serial-action work. It operates

on this structure with special functions composed in a

symbol manipulation language. This program, called

TASKSYM, can generate all alternative correct ways to

accomplish serial-action work and can track a subject

through the performance of this work. The model includes

an anti-goal structure which identifies action sequences

leading to catastrophic error, e.g., electrical damage to

the equipment or shock hazard to the performer. TASKSYM

is extended with LISP to develop TASKTEACH which provides

feedback on tests and errors, and offers various levels

of help.

10 pages, 14 references

L++%• ),
Page 22

Rigney, J.W.; Towne, D.M. (1970) "TASKTEACH: A Method for

Computer-Assisted Performance Training." Human Factors.

12 (3 ): 285-296.

A method for using a computer time-sharing system to

assist the learning of serial tasks, from operating

equipment through troubleshooting, is described. The

method is based on mediational theory. The current

resurgence of interest in mediational theory in

psychological research is noted, and parallels between

experimental strategies and instructional strategies for

evoking and manipulating mediating processes are pointed

out. Categories of processes which mediate performance

of serial tasks are described. Procedures for

facilitating the learning of these processes are

implemented by a computer program, called TASKTEACH. The

program sustains the student's performance of complex

serial tasks by giving him variable amounts of support

while helping him learn and organize the processes which

mediate his performance of these tasks. The program

generates output to the student during the learning

session by processing short lists and the prior responses

of the student. The lists, which are input to the

program replace the conventional frame-by-frame

description of an instructional sequence written in a

"CAI language."

12 pages, 21 references

* 1'. i
Page 23

Rigney, J.W.; Towne, D.M.; King, C.A.; Moran, P.J. (1978)

"Field Evaluation of the Generalized Maintenance

Trainer-Simulator: I. Fleet Communications System." Behavioral

Technology Laboratories, Report No. 89, University of Southern

California, Los Angeles, CA.

The Generalized Maintenance Trainer-Simulator (GMTS) is a

concept for giving students in Class C schools intensive

practice in troubleshooting equipment and systems taught

in those schools. It can be used for any device in which

signal paths and their relationships to controls,

indicators, and test points can be defined. The GMTS

uses generative CAI. That is, it generates the

interaction with the student by referring to his last

inputs and to its stored history of interactions with him

up to that point. To the extent that these individual

student histories constitute models of individual

students, GMTS constructs a model of each student to

interact with that student. In addition to these

features, GMTS is uniquely suitable for use in Class C

school training. The computer program that implements

the instructional system is indifferent to the specific

equipment being taught. What specific equipment that is

simulated by the GMTS is determined by loading two data

bases for that equipment: one describing essential

internal features of the equipment and the other

describing the external features.

This is a report of a field evaluation of the GMTS

applied to systems level troubleshooting in the UHF

communications side of the Fleet Communications System.

Twenty Class A school students waiting to enter C schools

practiced solving thirty-five troubleshooting 'problems.

Results were generally positive. With respect to

practice, the students became uniformly more fluent at

troubleshooting; mean times to solve a problem were

decreased by a factor of two,.and standard deviations of

these time were decreased by a factor of five. Similar

reductions existed, for average number of extra

replacements per problem. Also, students' attitudes

toward the trainer were generally favorable. An

additional field trial is underway, using the AN/SPA-66

radar repeater as the subject matter.

85 pages, 3 references
Page 24

Robinson, J.E.,Jr.; Deutsch, W.E.; Rogers, J.G. (1970) "The

Field Maintenance Interface Between Human Engineering and

Maintainability Engineering.,' Human Factors. 12(3):253-259.

Field maintenance actions reported in a deployed air

defense system were reviewed to determine and quantify

(1) the nature of operator and maintainer influences on

system malfunctions, and (2) the nature of man-machine

problems being encountered by system personnel in actual

diagnosis and repair activities. Maintenance events

described in 213 problem reports from the field were

categorized according to three probable sources of

malfunction: primarily human, combination human and

equipment, and primarily equipment. Human involvement in

system malfunctions was clearly established in 25% of the

events reported. Additional information contained in 21

documents analyzing system maintenance was examined. The

nature of human influence on system malfunction is

described by examples of "unfortunate" field maintenance,

"ordinary" field maintenance, and troubleshooting

deficiencies. Troubleshooting deficiencies were cited as

a frequent problem observed by equipment contractors who

had more knowledge and understanding than the technical

personnel. The nature of reported man-machine problems

highlighted the fact that human engineering participation

in establishing maintainability design requirements can

serve to make equipment that will be suitable to the

presence and actions of human users.

7 pages, 5 references
Page 25

Rouse, W.B., (1978) "Human Problem Solving Performance in a Fault

Diagnosis Task." IEEE Transactions on Systems, Man, and

Cybernetics. SMC-8(4T -58-27-.

It is proposed that humans in automated systems will be

asked to assume the role of troubleshooter or problem

solver and that problems which they will be asked to

solve in such systems will not be amenable to rote

solution. The design of visual displays for problem

solving in such situations is considered, and the results

of two experimental investigations (N=20) of human

problem solving performance in the diagnosis of faults in

graphically displayed network problems are discussed.

The effects of problem size (9,25,and 49 components),

forced-pacing, computer aiding, and training are

considered. Results indicate that human performance

deviates from optimality as problem size increases.

Forced-pacing appears to cause the human to adopt fairly

brute force strategies, as compared to those adopted in

self-paced situations. Computer aiding substantially

lessens the number of mistaken diagnoses by performing

the bookkeeping portions of the task. Further, for

self-paced situations, skills developed with

computer-aided training are transferred to unaided

situations.

14 pages, 52 references

Rouse, W.B., (1978) "A Model of Human Decisionmaking in a Fault

Diagnosis Task." IEEE Transactions on Systems, Man, and

Cybernetics. SMC-8(5T-77-3b1.

Utilizing elementary concepts from the theory of fuzzy

sets as well as several nonfuzzy heuristics, a model is

presented of human decisionmaking in the task of

troubleshooting graphically displayed networks. The

performance of the model is compared to the results of

two previously reported experimental studies. The

ability of the model to represent human decision making

as a function of network size, forced-pacing, and

computer aiding is considered. The model is used to

support the hypothesis that humans tend to discount the

value of information about what has not failed in

searching for the sources of malfunctions.

5 pages, 3 references

i "

- j
Page 26

Rouse, W.B. (1979) "A Model of Human Decision Making in Fault

Diagnosis Tasks that Include Feedback and Redundancy." IEEE

Transactions on Systems, Man, and Cybernetics. SMC-9(4):237-21.

A previously reported model of human decision making in

fault diagnosis tasks is extended to include situations

where feedback and component redundancy are important

considerations. The model is based on concepts from the

theory of fuzzy sets. The results of an experiment with

human subjects are reported and used to estimate the

parameters of the model. The model is used to illustrate

the notion that subjects who discount the importance of

feedback loops tend to have considerable difficulty when

they encounter problems having ajlarge number of loops.

5 pages, 7 references

Rouse, W.B. (1979) "Problem Solving Performance of Maintenance

Trainees in a Fault Diagnosis Task." Human Factors.

21(2):195-203.

Forty trainees in an FAA certificate program participated

in an experimental study of troubleshooting of

graphically displayed networks. The effects of network

size, computer aiding, and training were considered. It

was found that performance degraded as network size

increased, improved with the use of computer aiding, and

that skills developed with computer aiding were

transferred to the unaided situation.

9 pages, 15 references

Rouse, W.B. (1979) "Problem Solving Performance of First

Semester Maintenance Trainees in Two Fault Diagnosis Tasks."

Human Factors. 21(5).

Forty-eight first semester trainees in an FAA certificate

program participated in an experimental study of

troubleshooting of two different types of graphically

displayed networks. The effects of network size, redundancy,

feedback, computer aiding, and training were considered. It

was found that performance degraded as network size

increased, degraded as the level of feedback was reduced,

improved with the use of computer aiding, and that skills

developed with computer aiding in one task were transferred

to the other task.

4 references

LIS W~k
Page 27

Rouse, W.B.; Rouse, S.H. (1979) "Measures of Complexity of

Fault Diagnosi3 Tasks." to appear in IEEE Transactions on System,

Man, and Cybernetics, SMC-9(11).

The literature of complexity is reviewed and the

distinction between perceptual complexity and problem

solving complexity is discussed. Within the context of

two particular fault diagnosis tasks, four measures of

complexity are considered. These measures are evaluated

using data from two previously reported experiments which

employed eighty-eight subjects. It is shown that two

particular measures of complexity, one based on

information theory and the other based on the number of

relevant relationships within the problem, are reasonably

good predictors of human performance in fault diagnosis

tasks. The success of these measures is explained by the

fact that they incorporate the human's understanding of

the problem and specific solution strategy as well as

properties of the problem itself.

54 references

Saltz, E.; Moore, J.V. (1953) A Preliminary Investigation of

Troubleshooting. Technical Report 53-2. Lackland Air Force

ease, Texas: Human Resources Research Center.

Troubleshooting was investigated using radar,

reciprocating engines, and remote control turrets as the

hardwa-e for malfunction analysis' observation. The

discrete types of equipment were chosen in an effort to

promote generalizability. The following hypotheses were

formulated and tested:

1. Good troubleshooters know more about the

functioning of the equipment upon which they

work than do Ooor troubleshooters.

2. Good and poor troubleshooters differ in

previous experience.

3. Good and poor troubleshooters differ in

intelligence.

10 pages, 4 references

L W fU _ Ail i ' 
Page 28

Saupe, J.L. (1954) Troubleshooting Electronic Equipment: An

Empirical Approach to the Identification of Certain Requirements

of a Maintenance O-cu--ation. Urbana, IL: University Of Ttliiiois.

The study attempts to identify the criteria for

proficiency in a troubleshooting situation. The study

provides "leads for the construction cf proficiency tests

which are important aspects of the efficient selection,

training, and distribution of maintenance personnel." The

goals of the research were:

1. to develop a psychological model by which

to view troubleshooting behavior.

2. to test certain hypotheses taken from this model

relative to aspects of the troubleshooting

process

3. to illustrate an empirical approach to the

identification of important objectives

of training programs

Subject performance was evaluated on a simulated radio

receiver and on a paper and pencil test. Empirical

results were presented.

Standlee, L.S.; Bilinski, C. (1966) "Note on Simulated Vs.

Actual Electronic Troubleshooting Performance." Perceptual and

Motor Skills. 23:532.

This is a brief note on electronic troubleshooting

performance. It discusses the two basic approaches to

measuring this performance (1) measurement of performance

on simulated and (2) actual electronic equipment. 114

subjects (Naval electronics technicians) were divided

into two groups, one group troubleshooting with the

simulator, and the other with actual equipment. Results

indicated poor troublshooting performance with the

simulator and proficient performance with the actual

equipment.

1 page

..... ' ' - * . , -.. . ,. :". .
Page 29

Steinemann, J.H., (1966) "Comparison of Performance on Analogous

Simulated and Actual Troubleshooting Tasks." U.S. Naval

Personnel Research Activity, Memorandum SRM 67-1, July 1966.

This investigation compared the performance of a group of

subjects assessed on a simulated troubleshooting task and

on the identical actual troubleshooting task using real

equipment. Subjects were 14 students in the experimental

training program for Electronics Technicians, conducted

by the Navy Training Research Laboratory, San Diego.

Analysis of results revealed that the simulated

performance measure did not provide a valid estimate of

performance proficiency on the actual task. Obtained

negative inter-test correlations indicate that simulated

test results would actually be misleading in terms of

estimating actual performance scores. In addition to

performance score discrepancies, there were observable

differences in specific performance procedures and

overall troubleshooting strategy attributable to the

differences in test mode, Although the research design

was weak, the evidence strongly -suggests caution in

assuming that a simulated performance measure, even with

considerable face validity, will provide a valid estimate

of actual performance on a common task.

12 pages

C -
Page 30

Stolurow, L.M.; Bergum, B.; Hodgson, T.; Silva, J. (1955)

"The Efficient Course of Actioh in 'Troubleshooting' as a Joint

Function of Probability and Cost." Educational and Psychological

Measurement, 15(4):462-477.

This research focuses on the development of a

probabilistic model of troubleshooting behavior which

prescribes an efficient course of action in locating

defects based on probabilities of failures and work time

given particular symptoms. Data were obtained from

maintenance records for 46 four-engine aircraft to obtain

the probability of each possible cause of a particular

set of diagnostic signs. Mean repair times were also

completed for each type of repair. A complete method and

a simplified method are described for determining an

efficient course of action to be followed in reducing a

set of possible causes of malfunction to the actual

cause. The simplified method was proposed as an

alternative to overcome the cumbersome calculations of

the complete method. The simplified method uses the

ratio of mean repair time to probability of failure to

determine the course of action to follow in reducing the

list of possible troubles. Results of the analyses of

maintenance records were sufficiently encouraging to

suggest that further research of this type be done for a

variety of complex devices.

16 pages, 4 references
Page 31

Towne, D.M. (1979) "The Automated Integration of Training and

Aiding Information for the Operator/Technician." presented to the

Third Biennial Maintenance Training and Aiding Conference.

Orlando, Fla.

A responsive and intelligent device which can provide

both training information and job performance aiding can

be extremely efficient in utilization of hardware,

software, and so-called course-ware. The hardware

components useful for presenting automated instruction

are essentially the same as. those required to aid an

operator or technician, viz., a versitile and fast

microprocessor CPU, ample random access memory, ample

bulk storage, and appropriate input-output media. While

the software driving such a device may be a collection of

relatively special-purpose routines designed to satisfy

particular training or aiding needs, a large portion of

this software may be common utility functions which

respond to specific requests of the user or serve to

manipulate data about the target equipment or data about

the operator/technician. Finally, the data base which

characterizes a target equipment can be designed in such

a way that both training and aiding requirements are met

without duplication or inefficiency of preparation

effort.

Under a contract with Naval Training Equipment Center,

Behavioral Technology Laboratories has assembled the

hardware elements of such a system, designed the target

equipment, and programmed routines for training novice

operator/technicians and aiding equipment operation and

maintenance.

21 pages, 5 references

r1

' t :'"'
Page 32

Wescourt, K.T.; Hemphill, L. (1978) "Representing and Teaching

Knowledge for Troubleshooting/Debugging." Institute for

Mathematical Studies in the Social Sciences, Report No. 292,

Stanford University, Stanford, CA.

An extensive review of the troubleshooting/debugging

literature is covered from an artificial intelligence

perspective. The authors were interested in measuring

the effect of training, that is, exposure to specific

debugging heuristics, on problem solving performance.

Twelve subjects with relatively little programming

experience participated. They were currently enrolled in

a BASIC programming course. Subjects were randomly

assigned to two groups (aided and unaided) based on a

computer programmer aptitude battery. Aided subjects

could refer to a text tutorial during problem solving

sessions. The chronology of subjects' actions were

judged according to 5 criteria relating to implementation

of heuristics contained in the tutorial guidelines and

success of attempted solution. Successful debugging

grammars that would capture the subjects' strategies were

not developed. Subjective judgments of participants on

the research team were necessary to identify the extent

to which each measure applied to subjects' strategies.

Results indicated poor performance of both groups in

solving problems. After the experiment large differences

were found in-post test scores attained on the computer

programmer aptitude- battery, and thus by chance, subjects

assigned to the non-aided group had become better

programmers since completing their course in BASIC. In

conclusion, there were no strong results to show that the

tutorial aid affected implementation of heuristics. It

is suggested that a pedagogy is needed for teaching

procedural rather than declarative knowledge.

141 pages, 31 references

Ow..
Page 33

SECONDARY RESEARCH

Czeh, R.S. (1957) "Studies of Complex Behavior and their

Relation to Troubleshooting in Electronic Equipment." Alexandria,

VA: Human Resources Research Organization.

This report is a review of the literature concerned

either directly or .indirectly with the troubleshooting of

electronic equipment. Specifically, this review presents

the most pertinent findings in the areas of problem

solving and concept formation and indicates the manner in

which these findings have impinged upon, or could impinge

upon thinking about troubleshooting. The literature on

troubleshooting is discussed and the final section

outlines a research task designed to develop an

efficient, generally applicable procedure for

troubleshooting electronic, and to develop methods for

teaching the procedure.

23 pages, 59 references

Edwards, J.; Norton, S.; Taylor, S. Weiss, M.; Dusseldorp, R.

(November 1975) "How Effective is CAI? A Review of the

Research." Educational Leadership, 33: 147-153.

This article summarizes many research studies that

involved Computer Assisted Instruction. The CAI modes

discussed are: drill and practice, problem-solving,

simulation, and tutorial. The areas of CAI research

discussed are:

1. CAI as a summplement to traditional instruction

2. CAI as a substitute for traditional instruction

3. Effectiveness of different modes of CAI

4. Comparison of CMI with other nontraditional methods

of instruction

5. Compression of time

6. Retention of learning with CAI

7. Effectiveness according to ability levels

This literature review referred to 9 studies that showed

that the combination of CAI and traditional instruction

was distinctly better than traditional instruction only.

As a substitute for traditional instruction, the findings

were mixed with 9 studies finding CAI superior and 8

studies finding traditional instruction better. In all

cases CAI was effective in reducing training time.

7 pages, 33 references
Page 34

Fattu, N.A. (1956) A Catalog of Troubleshooting Tests. Indiana

University: Institute of EducatioinalResearch.

This catalog contains examples of various kinds of

troubleshooting tests. The justification for a catalog

of troubleshooting tests is that it summarizes and

illustrates materials assembled from widely scattered

sources and makes them available to individuals who need

the information but do not have time to assemble it.

This catalog attempts to be representative rather than

exhaustive.

Tests are reported in the following groups:

1. on-the job measures

2. performance tests

3. simulator tests

4. ARC or "tab" tests

5. pencil-paper and oral-interview tests.

The major difference between the troubleshooting research

of the fifties and the troubleshooting research of the

seventies is the use of computers for data gathering in

the later efforts.

130 pages, 43 references

: .; ,I
Page 35

Fink, C.D.; Shriver, E.L. (July 1978) "Simulators for

Maintenance Training." Air Force Human Resources Laboratory, AFHRL-TR-78-27.

This report reviews past and present applications of

simulation to maintenance training. Emphasis is on

describing issues, problems and areas for future research

as identified by recent authors. There is little

evidence from the literature regarding the degree to

which simulators promote tradsfer of training to the

field. Thus far, transfer research has demonstrated that

simulators can increase proficiency during and at the end

of resident training. A variety of research has also

demonstrated that major variations in resident training

cannot be correlated with job proficiency after a year or

more in the field. It still remains to be shown

conclusively that transfer of training affects

performance during the first weeks and months on the job. If transfer does occur, it should have its most important

and noticeable impact during these weeks. There is a

need, therefore, for studies which examine the

relationship between resident training, especially that

supported by simulators, and on the job proficiency

during the first one to three months on the job.

Allocating the position of simulators and actual

equipment systems in maintenance training requires

research investigating the optimal sequence of tasks and

training aids for maintenance training.

60 pages, 71 references

I
Page 36

Foley, J.P., Jr. (1977) "Overview of Advanced Systems Division

Criterion Research (Maintenance)." Air Force Human Resources

Laboratory, AFHRL-TR-77-77.

A prime but seldom considered cause of the current high

maintenance cost of DOD hardware, and thus the high

ownership cost of systems, is the current criteria used

by personnel systems to select, train, assign, and

promote maintenance personnel. The current criteria

emphasize the ability to obtain high scores on

paper-and-pencil theory and job knowledge tests. This

paper summarizes the many studies which indicate that

such tests have little demonstrated relationship to

ability to perform job tasks. Table 1 summarizes the

current research on performance measurement and indicates

for each citation the low correlations obtained in those

experiments for various maintenance tasks and

paper-and-pencil theory tests, job knowledge tests, and

school grades. Based on an experiment conducted in the

past by AFHRL, it was shown that graphic symbolic tests

were more promising than symbolic tests in measuring job

task performance. A maintenance task taxonomy is

discussed and a hierarchy of dependencies is developed

for electronic maintenance tasks. This conceptualization

is considered necessary before identifying the relevant

criteria in job task performance tests.

22 pages, 38 references

King, W. (1978) "New Concepts in Maintenance Training." Aviation

Engineering and Maintenance, 2 (6): 24-26.

This article discusses the present status of Navy maintenance training in light of recent developments in

CAI (Computer Assisted Instruction). The advantages of

2-D computer simulators and 3-D real equipment simulators

for troubleshooting is the main focus of example. The

author presents monetary comparisons between actual

equipment trainers (AET) and simulators of both the

computer and real equipment type. The best example is

the cost comparison for the training of radio room

technicians for the Trident submarine. The AET costs 20

million dollars compared to a 2 million dollar computer

simulator. King believes that the utilization of

computer simulations for maintenance training has the

potential to surpass the savings offered by simulators

for flight training.

3 pages
Page 37

Standlee, L.S.; Popham, W.J.; Fattu, N.A. (1956) A Review of

Troubleshooting Research. Indiana University: Tnstitute -3

Educational Research.

This report is an annotated bibliography of the

troubleshooting literature of the fifties. It contains a

brief description of the work of nearly one hundred

authors. The report is indexed with the following terms:

general discussion, proficiency assessment techniques, training equipment or materials, analysis of job

activities, evaluation of maintenance personnel,

experimentation, bibliography. This report is invaluable

to persons interested in the early efforts of fault

diagnosis research.

127 pages, 110 references

Am

!-1

'?'

};~

*
Page 38

MANUALS OF ACTUAL FAULT DIAGNOSIS OR TRAINING IN FAULT DIAGNOSIS

AVCO Lycoming. (1975) "AVCO Lycoming Reciprocating Engine

Troubleshooting Guide." Williamsport, PA: SSP-475.

A manufacturer published list of problems, causes, and

solutions that represents the more common and recurring

problems encountered by mechanics when troubleshooting.

The list is general and refers to turbocharged and

normally aspirated engines. This manual is written for

the professional mechanic and assumes that shop manuals

and proper tools are available.

47 pages

Bean, R. (1974) Petersen's Basic Automotive Troubleshooting.

Los Angeles,CA: Petersen Publi s11 ng Go.

This manual is a very basic guide for the novice or

do-it-yourself automotive enthusiast. It includes an

easy to follow symptom-cause format with illustrations.

The book is divided into such chapters as the following:

Troubleshooting the Brakes; Troubleshooting the Ignition

System; Troublshooting Automatic Transmissions; Engine

Oil Pressure Problems; and Troubleshooting with a Vacuum

or Compression Gauge. Simple logic trees are included

for the starting, ignition and fuel systems.

144 pag'es

k4

!J
Page 39

Chilton Book Co. (1972) "Troubleshooting." Chilton's Mustang

Repair & Tune-Up Guide. Radnor, PA: Chilton Book Company;

A section of the manual is designed to aid in the rapid

diagnosis of engine problems. The systematic format is

used to diagnose problems ranging from engine starting

difficulties to the need for engine overhaul. It is

assumed that the user is equipped with basic hand tools

and test equipment (tach-dwell meter, timing light,

voltmeter, and ohmmeter). Troubleshooting is divided

into two sections. The first, General Diagnosis, is used

to locate the problem area. In the second, Specific

Diagnosis, the problem is systematically evaluated. The

format presents a malfunction system followed by a list

of tests and procedures. The proper results of the test

are provided in an effort to aid the troubleshooter.

15 pages

Glenn, H.T. (1971) Glenn's Auto Troubleshooting Guide.

Philadelphia, PA: Chilton Book Co.

This manual is another example of a popular book written

for the automotive enthusiast rather than the experienced

mechanic. The specific areas that it covers are:

emergencies, hard starting, engine failure, excessive oil

consumption, engine noises, overheating, fuel system

trouble, electrical system failure, ignition system

failure, drive, transmission, and rear axle trouble, and

power brake and power steering failure.

98 pages

Howard W. Sams & Co. (1973) Tune-up Service with Testing, Diagnosis and Troubleshooting Indianapolis, IN Ho d W. ams

& Co., Inc. 2nd Edition.

This manual covers the latest emission control systems,

electronic ignition systems, electronic test equipment,

and conventional engine analysis equipment. The

troubleshooting section presents the familiar

symptom/cause format.

72 pages

l .. .... ,. .... . ...
Page 40

Mitchell Manuals, Inc. (1977) Diagnostic Guide. San Diego, CA:

Mitchell Manuals, Inc.

The book contains information designed to aid the

professional mechanic in diagnosis and repair of unusual

automotive problems. Since many malfunctions are caused

by incorrect, inoperative, defective or missing

components, attention is focused on these problems and

conditions. All information presented has been carefully

compiled, primarily from factory service bulletins,

pinpointing the cause of specific malfunctions. This

diagnostic guide is designed to help the mechanic avoid

potential problems and save time.

The book is arranged for quick reference in finding

particular information. The book's main divisions are

done by vehicle manufacturers: American Motors, Buick,

Cadillac, etc. Within each of the divisions there are

three areas of information:

1) troubleshooting and diagnosis

2) jacking, hoisting and towing

3) hood latch and safety release locations

142 pages

General Motors Corporation. (1978) Diagnosis and Repair Manual -

Easy to Follow Graphic Troubleshooting Procedures. Southfield,

MI: General Motors Corporation.

This manual contains graphic mode diagnosis and repair

charts. They are a relatively new approach to

systematic diagnosis and repair. The charts are easy to

use since they require minimal reading. The sketches of

exact engine parts allows the mechanic to proceed to

work without reading a manual, although on complex

repairs the book emphasizes that a manual should be

used.

These diagnosis procedures apply to various automotive

systems in most 1970 thru current GM cars and light-duty

trucks-, with some procedures applying to even earlier

model years. The procedures are based on the fact that

the vehicle worked right at one time and the problem is

due to time, wear, dirt or other causes.

201 pages

..
Page 41

MATERIALS INDIRECTLY RELATED TO FAULT DIAGNOSIS

Berner, W.S.; Hamilton, L.A.; Best, W.R. (1974) "A New

Approach to Evaluating Problem-Solving in Medical Students."

Journal of Medical Education. 49:666-672.

This paper describes the development of a new technique

to assess clinical. problem-solving skills in medical

students. The procedure enables each step in the

solution of clinical problems to be evaluated

independently by providing feedback to students at each

decision point. There are several unique features,

including the incorporation of some aspects of the

problem-oriented medical record. Students were required

to read an extensive data base and construct problem

lists, order diagnostic tests, and plan for the

management of the patient. The technique was designed to

(a) be administered to large groups of students, (b) be

easily scorable, and (c) overcome some of the

difficulties inherent in previously developed means of

assessment.

7 pages, 11 references

I...
Page 42

Brown, J.S.; Burton, R.R. (1978) "Diagnostic Models for

Procedural Bugs in Basic Mathematical Skills." Cognitive Science.

2(2):155-192.

A new diagnostic modeling system for automatically synthesizing a deep-structure model of a student's

misconceptions of bugs in his basic mathematical skills

provides a mechanism for explaining why a student is

making a mistake as opposed to simply identifying the

mistake. This report is divided in four sections: The

first provides examples of the problems that must be

handled by a diagnostic model. It then introduces

procedural networks as a general framework for

representing the knowledge underlying a skill. The

challenge in designing this representation is to find one

that facilitates the discovery of misconceptions of bugs

existing in a particular student's encoding of this

knowledge. The second section discusses some of the

pedagogical issues that have emerged from the use of

diagnostic models within an instructional system. This

discussion is framed in the context of a computer-based tutoring/gaming system developed to teach students and

student teachers how to diagnose bugs strategically as

well as how to provide a better understanding of the underlying structure of arithmetic skills. The third

section describes our uses of an executable network as a

tool for automatically diagnosing student behavior, for

automatically generating "diagnostic" tests, and for

judging the diagnostic quality of a given exam. Icluded

in this section is a discussion of the success of this

system in diagnosing 1300 school students from a data

base of 20,000 test items. The last section discusses

future research directions.

38 pages, 30 references
Page 43

Ellery, G.L. (1977) "The EE's Role in the Garage Diagnosis of

Vehicle Faults." Proceedings of 1978 IEEE Conference on Vehicular

Technology, Denver.

The complexity of modern automobiles, trucks and other

internal combustion engines demands more sophisticated,

computerized diagnostic equipment. The requirement of

high fuel economy and low hydrocarbon emissions further

highlights this need. The author discusses the fact that

automobile manufacturers are increasing their involvement

in the design and construction of automated test

equipment. The R & D for such equipment is the combined

effort of the EE, ME, and Automotive Service Engineer.

Computer controlled test equipment which relies on

micro-processor technology is already found in many

automotive repair facilities, but the sophistication of

such equipment is rapidly increasing. The author offers

examples of electrical analysis of starter-motor-current

oscillation and analysis of ignition spark amplitude to

determine engine compression.

3 pages

McGuire, C.H.; Babbott, D. (1967) "Simulation Technique in the

Measurement of Problem-Solving Skills." Journal of Educational

Measurement. 4:1-10.

Employing simulation technique analogous to that utilized

in management games and military exercises, the authors

have directed the development of a series of branched

problems in patient management requiring sequential

analysis and decision. These problems are designed to

measure aspects of behavior'defined by a criterion group

as essential components of clinical competence. The

materials are suitable to either paper and pencil or

on-line application. The validity and reliability of

simulations is also discussed.

10 pages, 13 references

K
Page 44

McGuire, C.H.; Solomon, L.M.; Bashook, P.G. (1976).

Construction and Use of Written Simulation. United States of

America: The Ts-ycho-gic-ai Corporation.

This work is a comprehensive discussion of written

simulations. The discussion ranges from basic

definitions through construction, use and evaluation of

simulations. The book is additionally applicable to

computer simulations, or real equipment simulations.

Sample problems are presented throughout the book.

Murray, T.S.; Cupples, R.W.; Barber, J.H.; Dunn, W.R.; Scott,

D.B.; Hannay, D.R. (1977) "Teaching Decision Making to Medical

Undergraduates by Computer-Assisted Learning." Medical Education.

11:262-264.

This paper describes a computer-assisted learning (CAL)

program that is used on an open-acess basis by students

at the University of Glasgow medical school. The authors

report an experiment in which the experimental group

(n=12) was presented with on-line case studies in which

they had to decide on patient treatment. Upon completion

of the experiment, the experimental group scored higher

on a factual knowledge test and had improved decision

making skills based on comparison of their scores to

those of medical experts. The subjects felt that CAL

enabled them to consider and discuss clinical decisions

in a way that is both more realistic and more instructive

than is possible in the more orthodox clinical teaching.

3 pages, 9 references
Page 45

Swain, A.D. (1961) Factors Affecting Degree of Automation in

Test and Checkout Equipment. Stamford, Con-n.: Dunlap an

Tissocia-e-s, Inc.

This report deals primarily with the human factors

problems attendant to the increasing automation of weapon

system test and checkout actions. The purpose of the

report is to identify unfavorable tactical situations

resulting from less-than-optimum allocation of test and

checkout actions, 'from inadequate design of automatic

checkout equipment (ACE), or from particular management

or other practices related to the utilization of ACE.

These cases suggest ways of avoiding such situations in

the future. While most of the examples are taken from

Air Force weapon systems, some are taken from Army and

Navy weapon systems and equipment; and the principles

and guidelines are generally applicable to all the

services. It is only the application which should be

different, as considerable differences often occur

between services in such areas as training, manpower

selection, environmental conditions, and logistics

support (to name only a few differences in system

criteria).

224 pages

¢
Page 46

Wollmer, R.D.; Bond, N.A., Jr. (1975) "Evaluation of a

Markov-Decision Model for Instructional Sequence Optimization."

Behavioral Technology Laboratories, Report No. 76, University of

Southern California, Los Angeles, CA.

Two CAI programs in electronics and trigonometry were

written to test the Wollmer Markov Model for optimizing

hierarchical learning; calibration samples totalling 110

students completed these programs. Since the model

postulated that transfer effects would be a function of

amount of practice, half the students were required to

complete one practice problem successfully before moving

to the next stage; the other half-had to do two practice

problems successfully.

All students completed the courses satisfactorily.

Practice effects were small; students who had one

success in each stage did about as well as those who had

two successes. The Wollmer model was thus not suitable

for optimizing instruction, in terms of minimizing

overall time, in these particular courses. Perhaps the

main reason for this result was that, as the student

works up to the top of the hierarchy, the sheer number of

subskills involved in the final task becomes a major

determinant of performance time, and number of practice

trials has a relatively minor effect, unless a very large

number of practice trials is given. To see if a much

larger amount of practice would affect time to perform,

three additional subjects were run with the electronics

problems under the condition that 5 correct solutions

were required at each of 11 levels. It was found that a

much larger training time was required to get a 2-3

minute improvement at the final level. Because of this

relative inefficiency, the Wdllmer model would not

prescribe such extensive additional practice.

35 pages, 8 references

IL4
47

AUTHOR INDEX

Avco Lycoming. ......... 38

Babbott, D .. ............ 43

Barber, J.H .. .......... 44 IBashook, P.G .. ......... 44

Bean, R .. .............. 38

Bell, A.G .. ........... 5

Bergum, B .. ........... 30

Berner, W.S .. .......... 41

Bernstein, B.B .. ........ 15

Best, W.R .. ........... 41

Bilinski, C .. .......... 28

Bond, N.A., Jr. .. ....... 4, 46

Brooke, J.B....... .... . .. . . . 4 4

Birown, J.S.. .......... 5-6 4?

Burton, R.R. .. ......... ,4

Chilton Book Co. .. ....... 39A

Crawford, A.M. .......... 7

Crawford, K.S .. ........... 7

Crooks, W.H .. .......... 8, 12

Cupples, R.W. ........... 44

Czeh, R.S .. ........... 33

Damrin, D.E. .. ...........13

Deutsch, W.E .. ........... 24

Duncan, K.D .. .......... 4

Dunn, W.R........................44

Dusseldorp, R. .. ........ 33

Edwards, J. ........... 33

Ellery, G.L .. .......... 43

Elliott, T.K .. ......... 9-10, 16

Ely, J.H .. ........... 11

Fattu, N.A .. ............ 34, 37

Finch, C.R. ............ 11-12

Fink, C.D .. ........... 35

Foley, J.P., Jr ......... 36

Freedy, A .. ........... 8, 12

Gardner, F.M ............ 13

General Motors Corporation . . . 40

Glaser, R.............13

Glass, A.A.o........ .........14

Glenn, H.T .. ............ 39

Goldbeck, R .A .. ......... 15

* Hall, N.B...............o

Hamilton, L.A... .......... 41

* Hannay, D.R .. ............ 44

Hemphill, L .. ............ 32

Hillix, W.A......... ..... 15

~w.,
Hodgson, T .. ............ 30 48

Howard W. Sams & Sons............39

Jensen, A .. ........... 18

Joyce, R.P .. .......... 10

King, C.A .. ........... 23

King, W.. ............. 36

Kuppin, M.A. ..............8

Landa, S .. ........... 16

Mallory, W.J .. ......... 16

Marshall, E. . .......... 4

Marx, M.A. .. ........ .. .. 15

Mcguire, C.H ........... 43-44

Mills, R.G .............. 17

Mitchell Manuals, Inc. .. ...... 40

Moore, J.V .............. 27

Moran, P.J ............ 23

Murray, T.S .. .......... 44

Norton, S .. ........... 33

Osborn, W.C .. .......... 17

Popham, W.J .. .......... 37

Rasmussen, J ........... 18-19

Rigney, J.W .. .......... 4, 20-23

Robinson, J.E., Jr .. ...... 24

Rogers, J.G. ........... 24

Rouse, S... ............ 27

Rouse, W.B .............. 25-27

Rubinstein, R. .......... 6

Saltz, E...............27

Saupe, J.L..............28

Scott, D.B ............ 44

Shriver, E.L ........... 35

Silva, J..........................30

Solomon, L.M ........... 44

Standlee, L.S. .......... 28, 37

Steinemann, J.H. ......... 29

Stolurow, L.M. .......... 30

Swain, A.D .............. 45

Taylor,.................33

Towne, D.M .. .......... 21-23, 31

Van Albert, C.E. .. ....... 11

Weiss, M..............33

Wescourt, K.T. .......... 32

Wollmer, R.D ........... 46

ea.~.
49

SUBJECT INDEX

Artificial intelligence .... 5-6, 8, 12, 21-22, 32, 42

Automated test equipment . ... 43, 45

Complexity ... ........... .. 16, 27

Context - automotive ........ .. 11-12, 27, 38-40, 43

Context - aviation ......... .. 7, 16, 30, 36, 38

Context - computer programming 32-33

Context - electronics .... 4-14, 16-18, 23, 27-29, 33, 46

Context - mathematics ..... 33, 42, 46

Context - medicine ......... .. 41, 43-44

Context - process control . . . 4, 19

Heuristics ... ........... .. 8, 15, 17-19, 25-26, 30, 42

Human error ..... .......... 24

Manual .... ............. .. 38-40

Measures - attitude ........ .. 6, 11-12

Measures - cost ... ........ .. 8, 30

Measures - errors ......... 4, 6-7, 9, 11, 13-14, 25-26

Measures - number of tests . . . 4, 13, 15, 17, 23, 25-26

Measures - subjective assessment 6

Measures - task time ........ .. 6-7, 11, 14, 23, 25-27

Measures - verbal protocol . 18

Model ...... ............ .. 4, 25-26, 46

Proceduralization .. ....... .. 9-10

Simulation - computer ...... 4-8, 12, 16, 20-23, 25-27, 31,

33, 35-36, 41-42, 44, 46

Simulation - other ......... .. 11, 15, 17

Simulation - paper and pencil . 4, 12-14, 16, 35, 43-44

Simulation - real equipment . 10-12, 14, 27, 29, 35-36

Task analysis . . . ...... 21, 30, 35

4
51

DISTRIBUTION

ARI Oist'ibution List

4 OAS (M&RAI 2 HOUSACOEC, Ft Ord, ATTN: Library

2 HODA (OAMI-CSZ) 1 HOUSACOEC. Ft Ord. ATTN: ATEC-EX-E-Hum Fc

I HODA (OAPE-PBRI 2 USASEC, Ft Beniamin Har.on, ATTN: Library

1 HODA (DAMA-ARI 1 USAPACDC Ft Benjamin Harrison. AiTN: ATCP-HR

1 HODA (OAPE-HRE-POI I USA Comm-Elect Sch. Ft Monm uth. AiN: ATSN-EA

1 HODA (SGRO-IO) 1 USAEC. Ft Monmouth. ATrN: AMSEL-CT-H0P

I HODA IOAMI-DOT-C) 1 USAEC, Ft Monwmoth. AiN: AMSEL.-PA1 HODA (OAPC-PMZ-AI 1 IUSAEC. Ft Monmuth. ATIN: AMSEL-SI-CS

I HODA (OACH-PPZ-A) 1 USAEC. Ft Monmuth. ATTN: C. Fad ow Br

1 HODA (OAPE-HREI I USA Materials Sys Anal Apy, Aberlen. ATTN: AMXSY-P

I HODA fDAPE4PO-C) I Edg-wuod Atned. Aberdeen, ATTN: SAREA-O1L-H

1 HODA (OAPE-OW) I USA Ord Ctr & Salo. Abserln, ATTN: ATSL-TM-C

1 HODA (OAPE-HAL) 2 USA Hn Engr Lob. Aberdeen. ATN: Librrv/Dir

1 HODA (OAPE-CPS) I USA Combat Arms Tng Sd. Ft Banning ATTN: Ad Supevisor

1H ODA (DAFO-MFA) I USA Infantry Hum Rich Unit, Ft Benning AWTN: Chief

I HODA IOARD-ARS-P 1 USA Infantry Sd. Ft BeninL ATTN: STISC-TE-T

1 HODA IOAPC-PAS-A) I USASMA. Ft Slim, AirN: ATSS-LRC

1 HODA (OUSA-OR) I USA Air Dof Sefi, Ft Slim. ATTN: ATSA-CT-ME

1 HODA IOAMO-ROR) I USA Air DeE Sch. Ft Slim, ATTN: Tecs Lib

1 HOOA (OASG) 1 USA Air Oaf Bd. Ft Bliss. ATIN: FILES

I HOA (DA 10-PtI I USA Air Dof Sld. Ft Bliss, ATTN: STEBD-PO

I Chief. Consult Div IDA-OTSG). Adelphi. MO I USA Cmd & General Stf College. Ft Leaanwsmth, AWTN: Lib

SMil Asst. Hum Res. OODOR&E. OAO (EILS) 1 USA Cand & General St College. Ft Leavenworth. ATTN: ATSW-SE-L

1 HO USARAL. APO Seattle. ATTN: ARAGP-R I USA Cmd & General Stf College. Ft Leavenworth. ATTN: Ed Advisor

I HO First Army, AWN: AFKA-Ol-TI I USA Combined Arms Cmbt Dey Act. Ft Lawwwrth. ATTN: OeCdr

2 HO Fifth Army. Ft Sam Houston I USA Combined Arms Cmbt Oev Acts Ft Leavenworth. AWYN: CCS

1 Dir. Army Stf Studies Off. ATrN: OAVCSA (DSP) I USA Combined Arm Cmbt 0ev Act. Ft Leweworth. AITN: ATCASA

1 Off: Chief of Stf. Studies Oft: I USA Combined Arms Cmbt ow Act. Ft Lomemoush, ATTN: AYCACO-E

1 DCSPER. ATYN: CPS/OCP 1 USA Combind Arms Cmbt De Act. Ft Lwnwoet, ArN: ATCACC-CI

1 The Army Lib. Pentagon, ATTN: RSS Chief I USAECOM. Night Vision Lab. Ft Belvoir. AIN: AMSEL-NV-SO

I The Army Lib. Pentagon, ATTN: ANRAL 3 USA Compuser Sys Cad. Ft Befvoir. ATTN: Treh Library

1 Oft, Ant Soct of the Army (R&D) 1 USAMEROC. Ft Belv-ir. ATTN: STSFS-OO

1 Tech Support Ofc. OJCS I USA Eng 5ch. Ft Beloir. ATN: Library

1 USASA. Arlington. AT'N: IARO-T 1 USA Topographic Lab. Ft Selvoir. ATIN: ETL-TO-S

I USA Rsch Oft. Ourham, ATTN: Life Sciences Dir 1 USA Topographic Lab, Ft Selvmr. ATIN: STINFO Center

2 USARIEM, Natik. ATTN: SGRO-UE-CA I USA Topographic Lab. Ft Beai ar. ATITN: ETL-GSL

I USAT'C. Ft Clayton. ATTN: STETC-MO-A 1 USA Intelligence Cs, & Sch, Ft Huacwca. A"TN: CTO-.MS

1 USAIMA. Ft Bragg, ATTN: ATSU-CTO-OM I USA Intelligence Ctr & Slt. Ft Huachuca. ATTN: ATS-CTD-MS

1 USAIMA, Ft Bragg, ATrN: Manuat Lib 1 USA Intelligence Ctr & Sch. Ft Huachuca. ATN: ATSI-TE

1 US WAC Ctr & Sch. Ft McClellan. ATTN: Lib 1 USA Intelligence Ccr & Sch. Ft Huachuca. ATT'N: ATSI-TEX-GS

1 US WAC Ctr & Sch. Ft McClellan, ATTN: Tng Oir I USA Intelligence Ctr & Sch. Ft Huachuce. ATIN: ATSI-CT"-OR

1 USA Duarternaster Sth. Ft Lee. ATTN: ATSM.TE 1 USA Intelligence Car & Sah. Ft Huascuca. ATTN: ATSI-CTM-OT

1 Intelligence Material oev Oft. EWL. Ft Holabird I USA Intelligence Ct & Sch, Ft Huachuca. AiN: ATSI-CTD-CS

I USA SE Signal Sth, Ft Gordon, ATTN: ATSO-EA I USA Intelligence Ctr & Sch. Ft Huehum. ATIN: O*/SRO

1 USA ChaWlain Cm' & Sch. Ft Hamilton, ATTN: ATSC-TE-RO I USA Intelligence Ctr & Sch. Ft Huschuce. ATTN: ATSI-TEM

1 USATSCH, Ft Eustis. ATTN: Educ Advisor 1 USA Intelligence Car & Sch. Ft Huschuce. ATIN: Library

I USA War College. Carlisle Barracks. ATrN: Lib I CDR. HO Ft Huachuce. ATrN: Tech Ref Div

2 WRAIR. Neuropsychiatry Div 2 CDR. USA Electronic Ism Grd, ATTN: STEEP-MT-S

1 DLI. SOA. Monteray I HO. TCATA. ATTN: Tech Library

I USA Concept Anal Agcy. Belhsda. ATTN: MOCA-MR I HO. TCATA, ATTN: AT CAT-OPO. Ft Hood

I USA Concept Anal Apty, Bethesda, ATTN: MOCA-JF 1 USA Reeng Cmd, Ft Sheridan. ATTN: USARCPM.P

I USA Arctic Tes Ctr. APO Seattle, ATTN: STEAC-PL-MI 1 Seor Army Ad., USAFAGOOITAC, 1W AF Aux Fld No. 9

I USA Arctic Test Ctr. APO Seattle. AIiN: AMSTE-PL-TS I HO. USARPAC. 0CSPER, APO SF 9ML ATTN: GPPE4E

1 USA Armament Cmd. Redstone Arsenal. ATTN: ATSK-TEM I Stimwl Lib. Aceo of Health Sciee's, Ft Sam Hoston

1 USA Armament Cmd, Rock Island. ATrN: AMSAR.TDC I Marine Corpe Inst.. ATTN: 0ean-MCI

I FAA.NAFEC. Atlantic City. ATIN: Library I HO. USMC. Cormendant. ATIN: Code MTMT

1 FAA-NAFEC. Atlantic City. AT N: Human Engr Or I HO. USMC. Commanlent. ATTN: Code MPI-20-20

I FAA Aeronautical Ctr. Oklahoma Citv, ATITN: AAC-440 2 USCG Academy. New London. ATTN: Admiseion

2 USA FId Arty Sh. Ft Sill. ATTN: Library 2 USCG Academy. New London. ATIN: Library

I USA Armor Sh. Ft Knox, ATTN: Library I U9CO Training Ct, NY, ATTN: CO

1 USA Armor Sch, Ft Knox. AT1'N: ATSS-01-E I USCO Tralinng Car. NY. ATIN: Educ Svc Off

I USA Armor Sch. Ft Knox, ATTN: ATSS-OT-TP 1 USCG. Psat Ra Br. DC, ATTN: GP 1/62

I USA Armor Sch. Ft Knox. AITN: ATSB-CD-AO I HO MId-iag Or. MC Det. Quantico AiN: Pil Div

-I,. -

A
52

IUS Marne Corps Liaison Ofc. AMC. Alexandria. ATTN-. AMCGS-F I Def & Civil Instaf Envireo diimeCnada

1 USATRAOOC. Ft Moonro. ATTN: ATRO-ED 1 AIR CRESS. Kensington. ATTN: Info Sys Br

6 USATRACOC. Ft Monroe. ATTN: ATPR-AO I Milit@gqgykli' TioneMs. Copenhagen

1 USATRAOOC. Ft Monroe. ATTN: ATTS-EA 1 Military Attache. Frend Emassy. ATTN: Doc Sac

I USA Forces Cand. Ft McPherson. ATTN: Library I Misdex Chef. C.E.R.A.-Arunal. Toulon/Navsll France

2 USA Aviation Test ad. Ft Rucker. ATTN: STESO-PO 1 Prin Scientific Off, AO Hum Engr Rich Div, Mintry

iUSA Agcv tar Aviation Satety, Ft Rucker. ArTN: Library of Dfense. Now DeWN

1 USA Agley for Aviation Safety, Ft Rucker. ATTN: Educ Advisor I Pairs Rath Ofc Library. AKA, Israel Defense Forces

1 USA Aviation Sch, Ft Rucer. ATTN: PO Draw 0 1 Ministaris van Defenue. OOOPI'KL Afd Socieel

I ISOUSA Aviation Sys Cmd. St Louis. ATTN: AMSAV-ZDR Psyrchologiedie, Zaken. The Hague. Netherlands

2 USA Aviation Sys Test Act.. Edwards AFS. ATTN: SAVTE-T

I USA Air Oaf Sch. Ft Sis., ATTN* ATSA TEM

1 USA Air Mobility Rech & 0ev Lab. Moffatt Fid. ATTN: SAVOL-AS

1 USA Aviation Sdi. Rev Trig Mgt, Ft Rucer,. ATTN: ATST-T..RTM

I USA Aviation Sci. CO. Ft Rucsker. ATTN: ATST-D-A

I MO. OARCOM. Alexand*a ATTN: AMXCO-Ti.

I MO. DARCOM. Alesandria. ATTN: CDR

I US Military Academy. West Point. ATTN: Sarials Unit

I US Military Mademy. West Point. ATTN: Ofic of Milt Ldrahp

1 US Military Acaemy. Wort Point. ATTN: MAO.

1 USA Standardizebio Go. UK. FP0 NY. A1TM:- MASE-OC

I Oft of Naval Rudh. Arlington. ATTN: Cadsi 412

3 Oft of Noval Rids. Arigton. ATTN: Cods 466

1 Oft of Navel Rids, Arlington. ATTN: Cods 450

I Oft of Naval Rsds. Arlington, ATTN: Cods 441

1 Naval Aaoope Mad Res Lab. Pensacola. ATTN: Acous Sch Div

1 Naval Aeraocp Mod Res Lab. Pensacola, ATTN: Coda LI

I Naval Aerospc Med Rev Lab. Pensacola. ATTN: Cods LS

1 Chief of Never, ATTN: Pon-.OR

1 NAVAiRSTA. Norfolk, ATTN: Safety Ctr

1 New Oceanographic. DC. ATTN: Cods 6251. Charts & Techt

1 Canter of Naval Anal. ATTN: Doc Ctr

I NavAirSyvom ATTN: AIR-5313C

I Nav Su~ed. ATTN: 7 13

1 Nov~elicptrSubiqua 2. FPO SF 9W60

I AFHRL IFT) Williami API

1 AFMRL lITT) Lowry AFB

I AFMRL 4ASI WPAFB. ON

2 AFIIRL (OOUZ) Brooks API

1 AFINRL (00*41 Lacktland API

I MOUSAF IINYSO)

I MOUSAF (DPXXA)

1 AFVTG IRDI Randolph API

3 AMRL (ME) WPAFI. ON

2 AF Int of Tech, WPAFB. OH. ATTN: ENE/SL

1 ATC (XPTDI Randolph API

1 USAF AeroMad Lib. Brooks API (SUL-41. ATTN: DOC SEC

I AFOSR 411L1.. Arlington

I AF Log Cmd. McClellan AFI. ATTN: ALCIOPCRB

1 Air Force Academy. CO, ATTN: Dept of Bill Scsi

5 HasPars G Dev Ctr. San Diego

2 Navy Med Nesuropsychiarric Rach Unit. San Diego

1 Nay Electronic Lab. San Diego. ATTN: Rev Lab

I Nay TrngCon. San Diego. ATTN: Cods 9000-Lib

I NevPostGreSch, Monterey. ATTN: Cods 5IAa

1 NavPostGradch. Montsrey, ATTN: Cods 2124

I NavTrngfquipCtr, Orlando. ATTN: Techt Lib

1 US Dept of Labor, DC. ATTN: Manpower Admin

1 US Dept of Justice. OC. ATTN: Drug Enforce MAdin

1 Nat But of Standards. DC. ATTN: Computer Info Secton

I Nat Clearing House for MM-Info. Rockiville

1 Denver Federal Cir. Lakewood, ATT'N: OLM

12 Defense Documentation Canter

4 Dtr Psych. Army Ho. Russell Ofcs. Canberra

I Scientific Advsr. Mil 8d, Army Hp, Russell Ofcs. Canoerra

I Mil and Air Attache. Austrian Embassy

I Contre do Recherche Des Factsurt. Humaine dis Deferne

Nationals. Brussels

2 Canadian Joint Staff Washington

1 C'Air Staff. Royal Canadian AF. ATTN: Pori Std Anal St

3 Chief. Canadian Daf Rich Staff. ATTN: C:'CRDSlWI

4 Britisti Def Staff. Bintih Embassy, Washington