*AR Research Note 87-71 FILE COPY

PROBLEM SOLVING IN A NATURAL TASK

O0 AS A FUNCTION OF EXPERIENCE

* 00

Juliana S. Lancaster and Janet L. Kolodner

Georgia Institute of Technology

Ifor - 2 -LECTE

~JAN 2 5 D

Contracting Officer's Representative

Judith Orasanu

BASIC RESEARCH LABORATORY

Michael Kaplan, Director

I

I U. S. Army

Research Institute for the Behavioral and Social Sciences

December 1987

I

I

*w --* ' -'-~ . .- -, -~ --. U. S. Army

Researh Insitute Seh orth iralan $oia Sciences
AK

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WM. DARRYL HENDERSON

EDGAR M. JOHNSON CO)L, IN

Technical Director Cfuadn

Research accomplished under contract

for the Department of the Army

Georgia Institute of Technology

* Technical review byDan Ragland K

.... j.. 

Ths eprt y h a cntacor sbmttd hs ee ceaed1, elas DfeseTehncl ifrainCne (ITC oc m l ih tusoy eutr nt.I isbe ienn rm r itiuio 1itta o.i

an il bet availubmled bny th conaraiC or ohr befencervces sucheas t Detonl Technical Information n

Service (NTIS). The viws. e-pinirns. aned/ot findings containerd in this report ate those of the authot(5l arid

should not be constrejaj as on oflcioa COepatment of the Army position. policy, or decision. unless so desig nated

by other official documentation.

IL
UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered)

DOCUMENTATION PAGE READ INSTRUCTIONS REPORT BEFORE COMPLETI7G FORM

. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

ARI Research Note 87-71

4. TITLE (and Subttle) S. TYPE OF REPORT & PERIOD COVEREO

Problem Solving in a Natural Task Interim Report

July 86 - July 87

S. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(@) S. CONTRACT OR GRANT NUMBER(&)

Juliana S. Lancaster and Janet L. Kolodner MDA903-86-C-0173

S. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT. TASK

School of Information and Computer Science AREA A WORK UNIT NUMBERS

Georgia Institute of Technology 2QI61102B74F

Atlanta, GA 30332

11. CONTROLLING OFFICE NAME AND ADDRESS 12. 'REPORT DATE

U.S. Army Research Institute for the Behavioral December 1987

and Social Sciences, 5001 Eisenhower Avenue, is. NUMBEROF PAGES

Alexandria, VA 22333-5600 12

14. MONITORING AGENCY NAME A ADDRESS(if dilferent ' m Controlling Office) IS. SECURITY CLASS. (of this report)

* -Unclassified

ISa. DECLASSI FI CATION/ DOWNGRADI NG SCHEDULE

__-_-_n/a

16. DISTRIBUTION STATEMENT (of thle Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abetct entered In Block 20, It different fron RePort)

IS. SUPPLEMENTARY NOTES

Judith Orasanu, contracting officer's representative

IS. KEY WORDS (Continue an reverse elde It necemary and Identify by block nmber)

Cognitive Psychology

Problem Solving

Expertise

Experience

21L ADSTRACT CM; -. m,- , (I N -- 4y -- l" if by block ,mb.bm) ii This research note investigates the effects of experience on problem solving

behavior and the knowledge base of workers in an applied setting -- that of

automobile mechanics.

The automobile is a highly complex system with many interconnected subsystems. Problem descriptions presented to a mechanic who needs to diagnose a car are

usually quite sketchy, however. Novices are less able than experts to diagnose

any but the most obvious problems. This research note concerns itself with

(OVER)

DD" lFo~ IM V'O F O 9IS0SL

w 3 UNCLASSIFIED

* i SECURITY CLASNIFICATtOWt OF" THIS PAGE 1tran Date Entered)

*%*% %* % . %
SCCURITY CLASSIFICATION OF T0IS PAGE(Wh',n Dole e d)

ARI RESEARCH NOTE 87-71 (Continued)

20. Abstract (Continued)

identifying the qualitative differences between mechanics with different

levels of expertise. In this research note, three student mechanics are

observed in a post-secondary technical school, each at a different level of

expertise, diagnosing six problems introduced into cars in the school.

Collected protocols are then analyzed to find the knowledge and strategies

used in solving each problem. Series of protocols for each student were also

analyzed to find the changes in knowledge and strategies used in solving later

problems as compared to earlier problems. Differences were seen in both the

knowledge used by the subjects and in their general approach to diagnosis. As

a result of experience, the student mechanics seemed to improve in three areas:

1) their knowledge of the relationships between symptoms and possible failures

was augmented, 2) their causal models of the car's systems were augmented, and

3) their general troubleshooting procedures and decision rules were much

improved.

I

4'

i

UNCLASSIFIED

SECURITY CLASSIfiCATSON OF TOIS PAOGElmen De £nt.red)

ii

.. . . - - -,Im i a l l l l l l N i ... ..
Problem Solving In a Natural Task as . Function of Experience*

Julian S. Lancaster

Janet L. Kolodner

School of Information and Computer Science

Georgia Institute of Technology

Atlanta, GA 30332

Abstract

Problem solvig is known to vary in some predictable ways as a function of experience. In this study, we have

investigated the effects of experience on the problem solving behavior and knowledge base of workers In an applied

setting: automobile mechanics. The automobile Itself Is a highly complex system with many Interconnected subsystoms. Problem descriptions (i.e., symptoms) preented to a mechanic who needs to diagnose a car, however, are

usually quite sketchy, requiring the collection of more Information before solutio. Novices are le" able than experts

to diagnose any but the obvious problems, and we are InWested in Identifying the qu@litivo dIfferences between

mechanics at different levels of expertise. In the study reported, we observed three student mechanics in a postsecondary technical school, each at a different level of experte, diagnose a& problems Introduced Into crs in the

school. We then analyzed the protocols we collected to find the knowledge and strategles used in soMng each problem. We also analyzed the seies of protocols for each student to find the changes In knowledge and strategles used

SIn solving lter problems as compared to erlier problems. Differences were seen in both the knowledge used by the

subjects and in their general approach to diagnosls. As a result of experlenos, the student mechanics seemed to

Improve In three areas: (1) their knowledge of the relationships between symptoms and possible lures was augmented, (2) their causaJ models of the car's system were augmented, and (3) their general troubleshooting procdures and decision rules were much Improved.

1. Introduetlon

Problem solving Is known to vary In some prodictablo ways as a function of expertise. When the process of

problem solving first come under srutiny by psychology and computer science researchers. the problems studied

were In knowledge-lean domains In which wel-defined situations have known solutions (Reed, Ernst, & Baneril, 1974;

Reed & Johnson, 1977; Reitman, 1976; Simon. 1975). In that work, the behavior of Interest was generally a varlable

such as number of steps to completion or number of correct soluions. Recenty however, Interst In problem solving

." -. has leaned more toward problems in knowledge-rich domains such as physics (Chi, Glaser,& Ro, 1982; Smon &

Simon, 1978), thermodynamics (Bhaskar & Simon, 1977), architecture (Akin, 1980). ard political science (Voss,

Greene, Post, & Penner, 1983; Voss & Tyler, 1981). Wilhin these domains, researchers have continued to look at the

steps and plans generated In coming to a solution, but they have also developed a further interest in the nature or , organization of the knowledge used in the process of problem solving. A major question regarding the nature or

organization of knowledge has boen how that knowledge and its changes Influence performance. Our knowledge of the differences between novices and experts has reached the point where several general

statements can be made. Fist, exr In any field are more able to recognize and remember typical conditions within

their oa of expertise. Second, experts generaly organize the.i knowledge by functional characteristics of problems

O. whle novices are more ikely to use surface features to characterize problems.- There have not been a lot of explicit

conclusions, however, about the particler knowledge structures used by epe and novi. Nor has there been

work dscrbing the particuisr changes In knowledge and processing behavior that happen as a result of a single

experience.

* This research Is supported In part by tho Army Research Institute for the Behavioral and Social Sciences under Contact No. MDA-903-86-C-173. Thaks to Ken Allison and GIt Ragarajan, who

, provided representations for the paper and Ides about analyzing the protocols. **See Chl, sf al (1982) "d Onow (I988) for more dscuson Of ovtepern differences.

*1

O- wvwr
-2-

Our primary goal is to discover the changes that hdkvidual experlences have on a problem solver. In order to

achieve that goal, we first have to id out what knowledge the problem solver stalrts with before solvng any problem

and what knowledge he has later to solve a sklr problem. While earlier work has Idicated that 'good" diagnostic

ability I functon more of knowledge about the problem area beig diagnosed than of general diagnostic skils

(Miller, 1975), we find that the diagnostic skills of novices and experts also differ, and therefore also observe Initial

s'ategles of problem solvers and hoe used after a particular experlence.

In the particular experlment to be discussed, we had two goals. Our first wes to d out what knowledge subjects at different levels of expertise had anid to be able to Vle the problem solvng strategies used by suJects at

varying levels of aqcrtilee. This, we flt, would gv us a good Idea of what things experlence Neches. Based on our

previous work on memory and problem solvhig (Kolodner, 1985; Kolodner & Simpson, 1984; Kolo iner & Kolodner,

1987), we expected that differences would be in both the amount known and acoseslbity (or organkation) of known

knowledge. Our scond goal we. to identy partcular changes over time In each Idividual's handling of specific probisns and types of problems. The squence of problems presented to the subjecl wes derived such that this would

be possible.

The "k domain we have chosen to look at, diagnosis of automoive problems, is hWilng for sar reason. The automobile engine le a highly complex entity. It consists of a number of iteracting systems octng to produce the car's motlon. Failures I any component or system of the engine usually produce noticable symptoms or

changes I the car's performance, but the failures themselves are seldom obvious to the amateur. In addition, a given

symptom can indicate numerous possible failures wlthi the engine. The person who comes to the shop with a probkilm descrbes a symptom or set of symptoms to the mechanic, and It Is the mechsn's job to further Investigate the

car to fnd out which of the many possble problems that could cause the reported symlom(s) Il in fact rsponsble for

ft. Experts are much better Vn novice at determiinig th causes of automotive problems. (As the old story goes;

it's Ion cents for the screw and twenty doliars for knowing which one to replace.)

The domain Is knowledge-rich, and the depth of knowledge end abity to use t are both Imporhit In making a

good diagnosis. Schools each about cars In general, but sice there are so many different kinds of cars, each of

which have thelr own peculiarties, lbook and schools o'1 tMach everything. Diagnosing a oar with a given set of

symptoms may depend as much on the age and type of engine as on the symptoms prmented. A given failure can be

a common cause of a plrticulr symptom in one engie and not poesble I another. Experience with different t of

cars and different types of problems Il thus eential I gaining expertise. Furthermore, tre are too ainy typoe of

-'"cars (most models change at least a 1ttie every year) and too much In the sele of manuals for individual owe for a

mechanic to know everythig about waery car. Thus, ft is esental for th exmport mechanic to draw his own generallzations about rs that allow him to orgnbia and a knowleg g appropriate to aWy partioular war end problem he

Is iooklng at.

In the work repod here, three student mecaics were observed while diagnosing c failures. 8ix problems

were presented at weekly Intervals and think-aloud protocols were collected while the students worked and were tans"rbed eWd coded for liter analysis. Each week the Instructor demonstrated the correct or optimum Iroubleshootling

sequence for diagnosis of the failure dtr all subjects were finished. Thus, each student had an opportunity for feedbeck and an explanatlon of the car* problem whether or not he had diagnosed It correctly. Each failure w Intoduced Into the wr deliberately and each problem was caused by only one filed part. Analysis of the data focussed on

the knowledge and otategies used by stms at different levels of traning, how ti knowledge w orgolked, and

how theai knowledge and straegles changed wlth experience.

*We expected that the more experienced student would solve more problems and would give evidence of having

a more organized knowledge base than the lms uperanced students. In addhilon, we mpected that Individuals would

show evidence over the serie of problems of acqubig new diagnostic skls and new knowledge and connw-tone

* , .' wlthin their knowledge.

.N'. ,

%5,e,

A

0: - ,:. . " ,."..' . -*- -. : c -; ? " ?, ," . ' C=' ',' w,'. , ,, .
t .3-

2. Method

2.1. Subjeets

Three students atea post secondary technical school volunteered to participate In the project. The technical

program Is a two-year, sight-quarter program. Durig much of the second yeaw, the students work In a shop settig

wthin the school. Cars belongig to school personnel and friends of the students and hIelutor are diegiosed end

repsh'ed by students. In addition, the school owns several oars that can be used In leaching students to teach about

specific problems.

Each of the three student volunteers was at a different poit I the program. The novice student was In his first

quarter of the program end had no prior trainig or experience. The Itermediate student was at the beginnig of hie

second year In the program. The advanced student was near the end of the second year and held a pert-time job as

a mechaic outside of school. Each student worked on at lest tour of ale problems.

2.2. Proeedure

Subjects wers obeerved oac a weekc whim diagnosig en actual problem In a oar. The problems used wer

selcted by en Iastructor i the program in consultation with the experimenter. The problem end the Information

given as the customer's complinht are desod in Table 1. Each fault was Itroduced Ito a oar by the Istructor or

by a student not I the study under the direction of the Istructor. The cars used were all owned by the school with

one exception: a new car brought i by a school official that haed symptoms we had been presenting to the students In

previous weeks. In evary case, a sigle complint was given and a sigle fault could be traced to account for the

complaint. Students were told to track down the fault, but not to fix it unless repair was necessary to confirm the diagnosls.

In each session, the student was led to the car and, with the experimenter posig as a customer, told that the

car was exhibitig a particua symptom. The student wsthen slowed to perform any sets desired an the car and

Its engine, with the exception of a drivng road test, prohblted primarily by the symptoms prsented by the car. The

student was Istructed to thikc aloud as he worked to fid tie failed component In the wa. Hle comments ware tape

*recorded by the experimenter, who also served as an assstant to the student when necesary.

Table I

Faults and their compleits as presented to subject

Problem Complaint (symptom) Fault

I canrs but wi niot start sedment or other blockage In gas Ile

2 crarnks slowly when staring bad cell In batery-wINl not hold charge

3 oarks but will not start bad corviaction behid fuse penel and

fuel pump fuse

4 orankis but wi not start os ground whos from Electronic

Control Module (computer)

5 cankts but wil not start open tach circuit

6 detonation on acceleration poorly adjusted timig

2.3. Coding

Atear al protocols were treinscribed, each statement was coded Ito one of six categories, shown In Table 2

with emples Statlements coded as hypotheses we those hi which a specific system or component was first

nameod as a possible source of the falure or I which the system or component was accepted or rejected as the
.4-

Table 2

Codig Categories for Protocols

Category Additi nal Specifications Examples

Hypotheses Nunber and Status Could be starved for go& (N-PI)

It could be, could be the starter (N-

__________________________ P2)

Rules Topic(Failure, normal funcionig, or Fuel Pump should come on for 3

troubleshootig) aeconds (I-P4)

First of all, I have to locate the

connector to the bac* of the fuel

pump (A-P3)

Information Gatherig Source of Information obtained Boefors look I the book, I'm goig to

check the fuse (A-P3)

Obeervatlon Topic (hypothesa(nurnter) or What we don't have is fuel to the

complait) throtle body (A-P3)

I don't believe I heaw It runninig (A-P3)

Restatements Topic (complait or summary of to rephras that-the throttle body Is

________ I_ observations) not Ijectig fuel (A-P3)

source of the fallure. Hypotheses we numbered i order of appearance and. each time one was mentioned, Its

* status was noted. Its status could be open, accepted, confirmed, or rejected. Rules were statements givng known,

contatnt Infornation about an engie or about the process of diagnosis. Statements coded as information gathering

were generally desciptions of the actions beig taken by the subject at the time. Such actons could elici or obtain

Inforrnatlon from the customer, from a book, or via a procedure or tet applied to the angina. Obeervatlona we

* statements givig the Iformation obtained from the action takcen. Restatements were repetitions of previously stated

* or collected Information rather than new Iformnation. Each statement faling ito one of the iast three categorles was

Identlled with a specific hypothesis by it numober I poesible. All other statements we uncodable and were maurked

* as such.

2. Results and Dilseusslen

As sipecta, the ablity of the students to correctly diagnose the problems changed substantially between the

novice level and the hitermedle and advanced levels. The diagnoses given by each subject and the number of

hypotheses considered are shown i Table 3. The novice correctly diagnosed only one of four problems attempted,

while the Itermediate student correctly diagnosed three of s&x and the advanced student thres of four. In addition, the

number of hypotheses considered Icreased with expertise. The novice generated a mean of 3.0 hypotheses per

problem and the Itermediate and advanced students generated 6 and 5.0 hypothese per problem respectively.
- -5-

F D'o a N b Table 3

Final Dignoes and Number of Hypotheses Considered by Each Subject

Problem Novice Intermediate Advanced

I not getting fuel(4) dogged fuel ine(3)

2 dead battery cell(5) strter(4) dead battery cell(6)

3 -- fuel pump reby(S) fuel pump fus(5)

4 fuel pump(3) no dlagnoels(6) Injector solenold(g)

5 no dlagnosle(O) open tech crcult(9)

a --- bad tlming(1O) bad tmning(2)

3.1. Knowledge Struotures end Knowledge Organlzalon

In general, the diagnostic behavior we saw was elmilr to that reported by other rmearchers (Hunt, 1981;

Rasmussen, 1978; 1979; Rasmussen & Jensen, 1974). Students generated one or more poeb hypotheses for the

failure Immedistely after observing the symptom(s). These hypotheses were then rteed In a fairly sysien, tic (albeit

sometimes dlosynoratic) way either by observation of the Inputs to and outputs from specic components and sysmtes or by performance of specific diagnostic tets. In succmful oe"s, a slingle diagnosis ultimately Was given,

accompanled by an explanation of how or why that failure would generate the oberved symptom(s).

* We Interpret this process as being Indicative of an Interaction between two types of knowledge stuctures. The

first, a causal model of the ear's engine, contains knowledge about Individual components and their Inputs, outputs,

and normal behavior; relates components withIn a system to one another; end deacrbes the relationships and connec-

% tions between systems. It is uead to evaluate hypotheses In light of the evidence obtained from tho failed engine and

to lInd the mechanic through the engine to the source of te problem in a systemtc way. The cusal model Is generally quite large, and the second type of knowledge structure, symptom-fault sets, is used to Index Into t causal

model at appropriste places. Symptom4-ul tsf represent the reistlonshlps between perticular symptoms or set of

symptoms and failures. For example, given the symptom "the car crarks but will not start", the symptom-fault sets will

Identify three systens s possible locations for the fallure: the fuel system, tie ar Intae system, and the ignition sysmte. Within each of these systems, additional symptom-fault sets will Identify Individual components that may cuse

th symptom(s). For 1h fuel system, tim would be a failed fuel pump, en empty gas Wink, or a blocked fuel line.

For the Ignition system, these would be a bad dietributor, bed spark plug wise, or bad sperk plugs. These

symptom-aut sets are used to derive Initil hypotheses, diocting te mechanic to Ioc at only appropriate pieces In

the causal model.

If, In fact, mechanics re using these two types of knowledge structurea during roubleshooting, then we can

predict several changes we should expect to see In thee structures as a remut of experience, and from those, we can

predict the processing differences that would result from these changes. Fist, we predict that through experience, a

mechanic's set of symptom-hult set Inrmeas end that the et he aready knows become more accurate. As a

result of these changes, the mechanic should have better ways to Index Into the causal model, leading to more efficient searches for the correct failure. Second, the causal model should become more flled out with experience, both

through addition of oomponents end/or systems that were previously unknown and through addition of relationships

and dependencies between te known components. The causal model, Ike symptom-hult set, should also become

O.. more accurate. As a result of having a better causal model, a mechanic should be better able to systematically reason

*- ,,)about th way the war wafs, allowing him to bird engine flailures more systemrtitally and In more case.

" We did, In lact, - lear differences between stunts at different levls of experience reflecting xctly these

_ changes In their knowledge sructures. Fist, we saw evidence that both the organization and number of symptomfault sets Increased with experience. The advanced student seemed to know more syrmptom-ault sets than the

O novice, as evidenced by the larger number of hypotheass he was able to generate for each problem. In addition, the

,. advanced student seemed to organize his symptom-fault seats differently than the novice, evidenced by the more systenatic procedure he used for generating and tating hypotheses. The advanced student's procedure was to zero In

, .V
on one of the enghe's subsystems and then to consider which component of that system was faulty, while the novice

v did not differentiate between systems and components of systems In diagnosis. While for the novice, all faults are

equal and an hypothesis at the component level was as likely to be selected as the first to Ivestigate as an hypothesis

at the system level, the more advanced Iroubleshooter seemed to organize his symptom-fault sets Into two categories,

each used for different purposes. One set poited to faulty subsystems within the car (e.g., fuel system, electrical

systems) and was used early I diagnosis to zero in on the faulty subsystem, while t second set poited to faulty

components of these systems (e.g., the fuel pump, the battery) and was used to diagnose the problem within that

system. Such a change requires that t mechanic also reorganize his knowledge about the car's engine i a more

hierarchical way that differentiates between systems and components of systemns. Figure 1 shows a portion of the

r novice and advanced student's organizations of the causal model of the engie.

NOVICE

Level of Abstraction Values

Highest Car EnoIc----

Component Level Battery Coll Distributor SperkcPlug Gaa-Tar* Fuel-Pump Carburtor AJr-Filtu

ADVANCED

Level of Abstraction Values

Highest Car Engine

System Level Ignition Fuel Air Intake

Component Level Battery Coll Distributor Spark-Plug Oss-Tarel Fuel-Pumnp Carburtor Air.PFtmar

Figure 1

Novice and Advanced Student Representations of the Car's Engine

We also sew evidence that content of the causal model changed with experience. The causal model of the

more advanced students contaied niot only more knowledge about Inividual components, but also more knowledge

about the hIterconnected nature of the engine's systems. The behavior of the students durig troubleshooting Mluswates thes f'idhtgs. Consider, lot sample, the behavior of t1he advanced student I Problem 4. His reasonig went

is folows:*

The ftat thig you want to do, which Is the easiest thig to do, 1s lock and wee If we have any fuel.

because you gotle hae" fuel, air, and hat... Dent have fuel-The first thig I want to do Is check

the fus..."hy~re OK... hook this jmper lead to the bypass to the fuel pump ...tOe fuel pump Is

runn Ing... check and sa our connection up here to the energizer ... going from fth ECM up to the

linjecitor Is OK..try Io energize this solenoid by hand ... check to see N we got any gss..Ah the Onies

are alright ... got gas to the throttle body... my diagnosis Is the solenoid Is bad because everythinig

else checks out.

The hypothese generated by this student are i an order that reflects the multi-level and highly Itegrated

For a tuli protocol ot the session, whe to te first author.

%j %'
.- 7-

organization of both his causal model and his symptom-ault sets. He first determined which of three possble sysims of the engine was affected and then Investigated Its components and others that could impinge on the behavior

of the system under focus. In fact, his primary focus was on the electronic (or computer controlled) influences on the

behavior of the fuel pump and fuel Injectors. This reasoning showed an awareness (reflected In the student's causal

model) of the Interdependencles between subeystems. His reasoning shows that he knows that systems (such as the

fuel and electronic systems) ny intersect at several points and that an apparently or possbly failed component In one

system nay rect an action, or lack of a.tion In another system.

In contrast, the novice generated reistively few hypothess for a given problem. His protocols Indicate that

this is because he has little knowledge about the relationships between given symptoms and thei causes and also

becuse his cusal model is Inadequate. In oming the same problem the advanced student was working on above,

the novice reasoned:

This problem could be in the fuel system, Ignition system...we know t's not In the starting system

because the car will crank ovm...One small drop of fuel...n ta bowl...Ao I's In the fuel

system...the fuel pump's.., supposed to turn for 10 to 15 econds...I can'I hear it...lt might just be

a bad fuel pump.

We can s It evidence of an Integrated hierarchy of levels In his organization of symptom-fault set. While his

hypotheses were sometimes at the system level 0.9., fuel system) and sometimes at the component level (.9., fuel

pump s bad), in only one problem (this one) did he clearly consider first a system and then a component within that

system. More commonly, he generated hypotheas at both levels and then Investigated only specific components.

Furthermore, he showed a amilar lack of Integration In his causal model. Speclicaltly, he never considered the possibIlity that one system could affect the behavior of another. His knowledge appeared to stop at the Indiv dua

component's behavior and did not include the possbillty that the actions of another system (the electronic system)

could be affecting the behavior of the component he was cxnsidering (the fuel pump).

While the novice knew about nany of the components of the car's angIna and about what their connections

were within a single system, he did not know how the systems and the components In different systems were Interrelated. The advaned student, on the other hand, knew both the con:nections between components nd the connections between systems. Thus the advanced student had a more integrated and complete undersiondng of the car's

engine, while the novice's understanding seemed to be highly disjoint. Figure 2 shows our Interpretation of what the

novice and advanced students knew about the fuel pump, for example.

PUMP Source: a container

Substance: a substance in the container

Conduit: a pipe

Destination: a container

Energy-Source: an energy device

NOVICE ADVANCED

FUEL PUMP ISA PUMP FUEL PUMP ISA PUMP

Source: gas tank Source: gas tefk

Substance: gasoline Substance: gasoline

Conduit: hose Conduit: hose

Destination: carburetor Destination: carburetor

Energy-Source: elocriosl ystem

Figure 2

Novice and Advanced Student Represenlltons of a Fuel Pump

%.,."".
~-8-

Note that the general Inornmstion about pumps is avalilSbe to both the novice and the advanced student. However, the Information that the fuel pump requires an energy source which is the electrical system of the car is not part

of the novio's representetion of the fuel pump. If asked "What makes the fuel pump run?", the novice is able to construct the appropriate answer by using the more general Information about pumps, but he does not use this

knowledge during problem solving. The same pattern is probably true of knowledge about systems and components.

The novice can undoubtedly MRl an Inquirer what system of the engine a particular component resides In, but he does

not maintain this Inforrmation where It is readily usable during problem solving.

We also saw wthin-subject changes in the knowledge structures over the course of the experiment. These

changes were most evident In the intermediate student. Two examples will serve to demonstrate changes across

problems. In working on problem three, the Intermediate student made a long and protracted search for the fuel pump

relay using both written reference materials and extended visual examination of the engine. While working on problem

four, he was able to Immediately locate and check the same part. This component, and its physical relationship to

others, had been Incorporated Into the causal model during or following problem three. Similarly, the symptom-fauht

sts *tanged as new Information was acquked. For example, the first hypothes the Inttrmediate student checked at

t component level for problem four was the fuel pump fuse, which was the correct diagnosis for problem thres. He

made the point as he worked that he was checkking this possibility out first because of the previous case. ('I'm gonn

check the fuel pump fuse first [this lime].")

3.2. Dlegnotlo Strategies

*In addition to the changes experience makes in knowledge structures and organization, we also saw differences

In diagnostic style. Diagnostic strategies seemed to be used differently by subjects at different levels of expertise and

evaluation criteria changed significantly with experience. Some of these changes are due to the development of better

strategies for testing and confirming hypotheses with experience while others appear to result from the differences In

the knowledge available for diagnosis as a mechanic gets more experlenced.

The change In how the mechanics tested and confirmed hypotheses was striking. As the example above

showed, the novios sudent was willing to accpt an hypothesis when preliminary evidence could be Interpreted as

congruent with that hypothesis and not pursuing the tesk any further (I.e. "can't hear the fuel pump"). In contrast, the

advanced student sought, for each hypothesis, specifically confirming or disconfkming vcidence that was part of a

causal explanation. While he was willig to select an hypothesis to pursue on the basis of preiminary evidence, he

would not accept or rejoct it without causally based informetIon (.. "the fuel pump's not running, now we have to fi

out why").

The changes in diagnostic strategies that reulted from changes n tinh knowledge structures wers re r

" apparent In the efficiency of dlagnosls. As the causal model gets filled out, it should allow the mechanic to pursue a

. .. longer systematic search through the engine and also allow him to evaluate kIformation in more detall and with more

concern for the real effects of the behavior observed. At the same time, as the number and complexity of symptomfault sets increases, long searches should become less necessary, because the mechanic is able to idex into his

model In more, and more effective, locatione.

" These two types of changes In the mechanic's diagnostic strategies work together to produce the results we

saw. As the mechanic gaine exprience with nking correct and Incorrect diagnoses, he gains a sens of what kind

and how much Information is "enough" to be sure of his opinions. In addition, as his causal model and symptom-faut

-set become more complete and accurate, he Is more able to select hypofth for Investigation appropristely and to

. continue tIvgestigating a problem to the point that only one hypothesis remains as a possible diagnosis. Coneequently, tho conditione under which he will scept an hypothes as a final diagnosis will become more accurate and

the path by which he reaches his diagnosis will become more efficint.

This result Is clearly evident In protocols of the novice and advanced students. When the novice's working

hypothesis was a that a perticuisr component was faulty, he sither accepted It or rejected It as the cause of the symp-

* tom. He never Investigated other effects on or inputs to that component. For sample, in problem 2, the falure was a

dead battery cell which caused the car to crank very slowly. The novice based his diagnosis on the following informs-

. .

0 '% % ° % = " % % " " % " % " " " " " " = '" = % - " - - " ' - " - % % " " % ,
First of all, well have to chec tie battery ...lt could be the starter... It could be the alternator ... it

could be a voltage loss ... could be a deao caln th ue oattery...we've only got 10 volta I the battery-

-each battery cell is 2 volta arnd ther's Soca I the battery, so dead battery cell.

Here we see the novice generating both system and component level hypotheses but, because his knowledge is not

hierarchcally organized, not pursuing them in that order. Rather, he locks first at the battery charge. Because NIt s

low, ha accepts the hypothesis of a dead cell. His diagnostic strategy does not require that ha consider any

* hypotheses relatig to why the battery might be low, such an a mlfunction In another system.

In contest, the advanced atudent generally collected more Iformation before glvkg a diagnosis. If possible, he

confirmed his diagnosis by visually fidig the condition that created ths symptom O.e., the disabled fuss panel connection I Problem 3). When that was not possible, he Justified his diagnosis wthin his causal model. For iamnpe, In

Problem 2, the failure could not be confirmed by visual evidence. Instead. the advanced student reaches his diagnosis with the followig Iformation:

...check the starter dr'aw ... It's puling enough down to get the starter to go ah'ght ... We put the betlary under load, you can see the amps rising and It's charging the, battery, ..So the alternator's

working OK ... what I belleve we hae le the cell Is dead In the battery ..Try to test on the VAT..Ns

you see on the Idicator le also showing that It needs chargig for tie battery Is bad ... So what we

have here is a battery with a couple of cells dead, and It's a maled battery and you cannot check

tie specific gravity with a hydrometer to check and m which one's dlead.

He reached and Justified hle diagnosis by elimiatig all other possiblties from his symptom-fault set arid the causal

model. In other words, he tested and verified norml functionig of both the staring system ( lit's puling enough

O down to get the starter to go aright") and the charging system ("So the alternator's working OK"). These are the only

two systems, other then accessories such as headlights and radio, that affect the level of charge In the battery. Consequently, accordig to the student's causal model, I tie battery's charge le low and tie starting and charging systems are functionig correctly, the only remaiing component In which the failure can be located is the battery Itself. In

some types of batteries, this conclusion con be tested directly, but I the car used i tie problem, the battery Is

meeled. Therefore, the mechanic must stop with his explanation rather than attempt to verily the diagnosis any further.

in comparison to the novice, he selected his hypofthoem more efficIently, lirst eliminatig con Onkg systems from conelderatlon. In addition, he based his acceptance of the diagnosis on a full causal expiarutlon rather then on superficial

evidence.

'.4 4. Ceneluslene

The results are as predicted by our Iterpretation of the diagnostic behavior as en Iteraction between seversal

knowledge structurs. Both the causal modal and the symptom-fault setls chage with wcperlkmc, end we hae seen

some examples of exactly what changes ocwu. In tie causal model, the most notable change Is the hIceasig coin-

* pladly of the model, reflected I the growig awareness of the hIterconnecteclhess of systems witthin the engine. The

-4. novice le clearly unaware of the possibility that electronic failrs can affect tihigs Ike fuel delivery, since he knows lltgoe about the dependencies between the fuel system and tie electrical system, while tie more advanced mechanic not

only knows that such relatioships udat, he considers them a highly common source of falluires. Simlarly, the

number, organization, and accuracy of the symptom-faul sets changes withi icreasing experience. Ultimautely, they

* arwe able to represent a complex, hierarchical system of relationships. The date suggest that componenIs are organ4 baed hierarchically undesr their respective systems and are never dhectl considered unless tiek system is detarmined

to house the failure, or at least to be the source of Information crucial to locatig the failure.

6 Building pertly on these changes I tie knowledge structures, and pertly on Idependent effects of experience

on decision processes, the mechanic's procedures midl guideinies for acceptig hypotheses as diagnoses als change.

- ~ The processes or procedures used becom a crasingly focussed on iformation that wi allow a causal hItepretation

-. of the behavior observed. At tie same time, the develphg knowledge structures allow the mechanic to search for

and squire more, and more accurate, Information from his symptom-fault set and his causal model. The Iteraction

* of tiem changes I both knowledge and process led to the more accurate and eficent problem solving men I
-10-T 2: VTV . .

Thus, we a that exprenoe le providing fth mechanic with three things. His overall level of knowledge Is

Increasng; the organization and Itegration of his knowledge structures, both the symptom4autt sts and the causal

model, are hasaing; and his processes and criteria for reachling diagnoses are becoming more accurate, more efficient, and more focussed on causal lInformation.

5. Reference*

Akin, 0. (1980) Models of arc hitectural knowledge London: Pion.

Bhaskar. R. & Simon, HA. (1977) Problem solving i emantically rich domainsa: An example from engineering themodynamics. Cognitive Science. 1. 193-215.

Chi, M.T.H., Glaser, R., & Rees, E. (1982) Expertise In Problem Solving. In R.J. Sternberg (ad) Advances In the

Psychology of Human intelligence Hillsalle: Lawrence Eribaum.

Glaser, R. (1985) Thoughts an expertise. Technical Report #8. Leaning Research and Development Center. Univerally of Pittsburgh, Pittsburgh, PA 15260.

Hunt,.RAM. (1981) Human pattern recognition and Iormaion seeking I simualated thought diagnosis taks. Report

#T-1 10, Coordinated Science Laboratory, University of Ilinois at Urbana-Champaign.

Koiodiner.J.L. (1985) Experlential processes I natural problem solving. Technical Report #GIT-CS-85/23, School of

Information and Computer Science, Gsorgia Institute of Technology, Atlanta, GA 30332.

0 Koiodier,J.L. & KoloctrerARM. (1987) Using experience i clinical problem solving: Introduction and framework. IEEE

Transactions on Systems, Mon. and Cybernetic.

* -Kolodiier,J.L. & Slmpson,R.L. (1984) Experience an Problem Solvng: A Framework. Proceedings of the Sixth

Annual Conference of the Cognitve Science Society.

Miller, P.5. (1975) Strategy selection i medical diagnosis, Project IAAC. Report #TA-163, Massachusetts Institute of

Technology, Cambridge, MA.

Rasmussen, J. (1978) Notes on diagnostic strategies I process plant environment Also National Laboratory Report

#RISO-M-1983, Rosilide, Denmark.

Rasmussen, J. (1979) On the structure of knowledge-A morphology of mentl models I a man-nachie context.

Alec National Laboratory Repor #AISO-M-2192, Roskilde, Denmnark.

Rasmussen, J. and Jensen, A. (1974) Mentel procedures i real *s teaks: A case study of elecironic troubleshooting.

Ergonomics, 17(3), 293-307.

Reed, 5.K., Ernst,. OW., & Banerjl, A. (1974) The role of analogy I transfer between simliar problem states. Cognilive Psychology, 6. 436-450.

Read, S.K. & Johnson, J.A. (1977) Memory for problem solutions. In O.H. Bower (ed) The Psychology of Learning

and Moffvatlonk New York: Academic Press.

Reltman, J.S. (1976) Skilled perception I go: Deducing memory structures from Inter-reesp times. Cognitive

Psychology, 8. 336-358.

Simon. H.A. (1975) The functional equivalence of problem solving skills. Cognitive Psychology, 7. 268-286.

Simon, D.P. & Simon, HA. (1978) Indvdi differences i solving physics problems. In R. Siegler (Ed.) Chlldren's

Thinking: What Develops? Hillasdale, N.J~awronce Erlbaum

Voes, J.F., Greeine, T.R., Post, TA., A Penner, B.C. (1983) Problem solving skcill I fth sodial sciences. In G. Bower

(Ed.), Trhe psychology of learning and motIvaton: Advances in research theory. New York: Academic Press.

-Voss, J.F. & Tyler, S. (1981) Problem solving in an social science domain. Unpublished flaiiuaalpt. University of

Pittsburgh, Learning Research and Development Center.

1