ARI Research Note 86-98

NOVICE RULES FOR ASSESSING IMPORTANCE B

SIN SCIENTIFIC TEXTS k

Diana Dee-Lucas and Jill H. Larkin

Carnegie-Mellon University

for

Contracting Officer's Representative

Judith Orasanu

BASIC RESEARCH LABORATORY

Milton R. Katz, Director

- % .

DTlC

FEB 7 1A

U. S. ArmyE

Research Institute for the Behavioral and Social Sciences%

December 1986

Approved for public release; diStributiT)n unlimilea.

87 2 25 169
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WM. DARRYL HENDERSON *

EDGAR M. JOHNSON CoL, nIN %

Tcchnical Director Cmadn

Research accomplished under contract

for the Department of the Army

NTIS -

Carnegie-Mellon University

Technical review by I ~.~

Steve Kronheim

%

INSPECTED I

This report. as submitted by the contractor, has been cleared for releise to DfneTcnclIfrainCne

and will be available only through DTIC or other reference services such as the National Technical Information

Service (NTIS). The vicwis. opini'~ns, and/or findings contained in this report are those of the author(s) and

should not be construeuas an officia; Cepaitmeni of the Army position, policy, or decision, unless so designated4. ..

by other official documentation. ,..

% %, 01,~4
UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE MWNien Data Entered)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

BEFORE COMPLETING FORM

1. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER .6 A%"IN

Research Note 86-98

4. TITLE (and Subtitle) 5. TYPE OF REPORT & PERIOD COVERED

NOVICE RULES FOR ASSESSING INPORTANCE Interim Report

IN SCIENTIFIC TEXTS January 86 - July 86 6. PERFORMING ORG. REPORT NUMBER " % ,.. %

7. AUTHOR(*) S. CONTRACT OR GRANT NUMBER(&)

Diana Dee-Lucas and Jill H. Larkin MDA 903-85K-0180

W% 9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASK .. " .

AREA & WORK UNIT NUMBERS

Department of Psychology,

Carnegie-Mellon University 2Q161102B74F

Pittsburgh, PA 15213 ._... __ 11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

U.S. Army Research Institute for the Behavioral December 1986-."" .... -

and Social Sciences, 5001 Eisenhower Avenue, 13. NUMBEROF PAGES ..-

Alexandria, VA 22333-5600 36 14. MONITORING AGENCY NAME & ADDRESS(If different from Controlling Office) IS. SECURITY CLASS. (of tis report) .. . .

Unclassified

15a. DECLASSIFICATION/DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of thile Report)

Approved for public release; distribution unlimited. %

17. DISTRIBUTION STATEMENT (of the abetract entered In Block 20, It different from Report) .J, e

S. SUPPLEMENTARY NOTES..

Judith Orasanu, contracting officer's representative

19. KEY WORDS (Continue on reveree aide if necessary and Identify by block number) R 'S

Text Learning ,:% %

Expert-Novice Differences %

2a.ABSTRACT (Cutiu an reverse ef*b If noeeeara" mdIdentifr by block number) ~ % 

This study complements research indicating that content-area novices judge the

importance of what they read in texts on the basis of sentence type (e.g.

whether sentences are definitions or statements). Subjects varying in expertise

judged the importance of sentences in physics texts which were presented as

definitions or as statements of fact. Definitions and statements were identical

in substantive content.

Those in the categories of expert and novice judged the variants as equal(ovpr - '""',

DO I ,o-AA n 1473 ETION, OF I MOVS S IS OBSOLETE UNCLASSIFIED

i SECURITY CLASSIFICATION OF THIS PAGE (When Date Entered)

%... ". "-'-
UNCLASSIFIED

SECURITY CLASSIFICATION OF T0IS PAGE(W'Im Data Kneted)

ARI Research Note 86-98

20. Abstract (continued)

'in importance. Beginning physics students, however, judged the definitions as

more important. These results suggest that sentence form is a salient text

feature for beginning-level students. Beginners lack the knowledge necessary

to judge the importance of content directly, but they have developed general

rules about what types of information are often important in physics. By contrast, sentence form is not relevant for people with no training in physics

-- they have no expectations at all regarding what types of information should

be important; and experts pay little attention to sentence form because they

have rich content schemas which enable them to judge importance directly.

These results have theoretical implications for understanding content schema

development and also have practical implications for the writing of textbooks.

le

UNCLASSIFIED

iiSECURITY CLASSIFICATION OF THIS PAGE(W?,.n Date E..w-sd)

-. * -- . .5'. - 5.- 5~5

U S - -. 5 S-" - -

'S *." -- 5- S . S
Novice Rules for Assessing Importance in Scientific Texts:

Executive Summary

In learning from text, finding the important information is crucial, particularly for a novice

reader (i.e., one who is unfamiliar with the content domain). Such information serves as a

focus around which to organize related details. This research examined novice strategies for

determining what is Important in texts containing unfamiliar content. The research reported

here provides the theoretical background for related research which demonstrated that

novices' inability to correctly identify important text information can influence their learning in

adverse ways.

This study specifically examined how novices process definitions and facts, two types of

information that are both common and important in formal domains. The results Indicate

that in judging importance, novice readers tend to focus on definitions to the exclusion of

facts. This study had novices judge importance in passages which contained definitions and

facts that were (through experimental manipulation) identical to each other in content. Even

though the definitions and facts contained the same information, novices judged the

definitions to be more important. This was not true of experts, who judged definitions and

facts containing the same content to be equal in importance.

These findings suggest how novices begin to develop a "content schema" for expository

knowledge domains. Novices learn that certain easily identifiable types of Information (e.g.,

definitions) are more important...a reasonable general rule. However, they judge importance

on the basis of information-type category without regard for content, considering the same

information to be more important if it is expressed in the form of a definition. Thus novice

readers can be very sensitive to the form in which information is presented, in that minor

variations in wording (i.e., saying something "is defined as") can influence novices'

perception of how important that information is. Accordingly, writers can unintentionally

signal particular Information as important through wording selections of this kind if they are

unaware of novice preconceptions about the types of information that are important. On the

other hand, knowledge of novice rules can be used in conjunction with rhetorical indicators

of importance (i.e., underlining, signaling, structural changes, etc.) to emphasize important

content and de-emphasize less critical information. Additionally. knowledge of novice content

schemas and how they differ from those of experts gives us a basis for designing instruction

in a way that helps novices develop better text-learning skills.
Novice Rules1

Abstract

This study complements research Indicating that content area novices judge importance

In texts on the basis of sentence type (e.g., whether sentences are definitions or facts).%

Subjects varying In expertise judged the importance of sentences In physics texts which

were presented as definitions or facts. The definitions and facts were Identical in

substantive content. Experts and subjects without physics training judged these variants as

equal In Importance. However, beginning physics students judged definitions as more

Important. These results suggest that sentence form is a salient text feature for beginninglevel students. They lack the knowledge necessary to judge the importance of content

directly, but have developed general rules about what types of Information are often

Important In physics. Sentence form is not relevant for people without physics training--they

do not have expectations regarding what types of Information are important. Sentence form

also has little Influence on experts, who have rich content schemas allowing them to judge

Importance directly. These results have theoretical implications for understanding content

schema development, and practical implications for textbook writers.

.

a-L m, 4-f CC -
",,

Novice Rules 2

Novice Rules for Assessing Importance in Scientific Texts

Diana Dee-Lucas & Jill H. Larkin

Carnegie-Mellon University

Scientific textbooks are typically densely packed with complex information, including

equations, symbols, and specialized terms. Consequently, it can be very difficult for

students who are unfamiliar with scientific subject matter to distinguish the important content

from the elaborative information when reading this type of text. The purpose of the present

research was to investigate rules used by novice readers (i.e., readers who are unfamiliar

with the text content domain) In determining what is important In scientific texts.

There are various sources of information that novices readers could use in assessing

Importance. Most research has focused on how text-based indicators of importance such as

text structure and signaling devices (e.g., underlining, adjunct questions, staging,

typographical cuing, etc.) influence novice readers' attention. These textual manipulations

are "content-free," In that their effects should not depend on the nature of the text content

or the expertise of the reader, In contrast, a "content-specific" source of information for

assessing Importance In texts is the reader's "content schema" (Kieras, 1985). A content

schema consists of domain-specific knowledge about how information in a content area is

typically organized, including what is important.

Past research has largely ignored the role of content-based schemas in the

comprehension of novice readers. This is because it has typically been assumed that novice

readers who are unfamiliar with the content domain of a text would lack such a schema

(see Kieras, 1985, for discussion). However, recent work by Dee-Lucas and Larkin (1986)

SL,;gests that novices do develop a rudimentary "content schema" for scientific content

domains. This content schema consists of rules specifying the types of information (i.e.,

definitions, facts, etc.) that are important to that domain. In their research, Dee-Lucas and

Larkin compared the importance judgements of expert and novice physicists for different

..- ..° .o .. . o o ... . .. . . ., i - , ° ,. , o .- °. . , - ° ' " ° q° " ' ' = ' ° " °" * ° " " " ' " ' ' "" '' ° " ° " -

%.o . ° •.D =. .. l~l • •• - • ° • - =.l . • - % • - ." •.q' l=! =I= 4 .I.• ="n o''4, °. '. -.. o*.-.
Novice Rules 3

types of information in physics texts. They found that both groups judged definitions to be

more Important than facts. However, the novices were even more likely than the experts to

judge definitions as important and facts as unimportant, suggesting that they had 4'

"overgenerallzed" this rule. The experts discriminated between important and unimportant

Information within the type categories to a greater degree than the novices, indicating a

more articulated content schema. Thus novices are Inaccurately identifying the important

text content on the basis of these overgeneralized rules.

This research comparing the judged Importance of definitions and facts did not,

however, control for content differences between the two sentercri types. Thus novices in

these studies may have been basing their importance judgements on some feature of the

content that differed between definitions and facts, rather than on the definition/fact

distinction per se. For example, novices may have considered definitions to be more

Important than facts because the definitions may have contained more new terms. If this

were the case, then the rule used by the novices would be that statements with new terms

are more Important than statements involving known terms, rather than a rule that definitions

are more Important than facts. Ile

The current study provides a strong test of the hypothesis that novices are using a '

rule Involving a pure sentence type distinction; in other words, that novices consider

definitions to be important simply because they are definitions, regardless of their content. 'p

This was tested by comparing novices' and experts' relative sensitivity to the form (i.e.,

definition or fact) in which information is presented in physics passages. Experts and

novices judged the Importance of target sentences in physics texts when they were

presented as definitions or facts. These target sentences contained information which could

be stated as facts or definitions through minor variations in wording which did not alter the

primary sentence content. In this way, the content of the definitions and facts remained

constant and only the form in which the information was presented varied. Thus unlike

- .° ." "=l .. .° . = . • - •• • o l • • • '-
Novice Rules 4

earlier research, this study controlled for possible sentence-type effects due to content

differences between definitions and facts. Two versions of two physics passages were used,

each version being identical except for the form in which the target sentences were

presented. The importance judgements of the novices and experts for the faczt and

definition versions of the target sentences were compared to see to what extent the

sentence form influenced the perceived Importance of the information.

Method

Stimulus Materials

One passage was about work and energy and one dealt with fluid statics. Each was

about 50 sentences long. One contained 9 target sentences and one had 11 target

sen.ences.

The definition and fact versions of the target sentences differed in that definitions

always Included "is defined as," and thus were signaled as being definitions. In the fact

versions, 'Is defined as" was dropped or replaced with "is represented as," "is calculated

as," or "is Indicated by." Thus the facts were "non-definitions" in that in place of

definition signaling they contained phrases indicating that the sentence was presenting ,5"

attributive Information about the sentence topic (as opposed to criterial attributes defining the

sentence topic). Examples of the definition and fact versions of some of the target

sentences are shown in Table 1. There were two versions of each passage. In one

version, the odd-numbered target sentences were definitions and the even-numbered were

facts; in the second version this was reversed.

Insert Table 1 about here

Each of the target sentences was classified according to its level in the hierarchical

structure of the passage. The procedure used for the structural analysis is reported in Dee-

%-'._4 J.' .- == .' 4"=P o ° .'' O ". .""°"".-"°. °".r'" " * - ." " " . -,. .. . ° . . . . . . . •-,*
Novice Rules 5

Lucas and Larkin (1986). This analysis produced a hierarchy with the main topics or

concepts occurring at the highest levels and modifying information occurring at the lower

levels. Modifying Information consisted of examples, attributes and properties, derivations

(i.e., Information implied by or derived from higher level information), explanations, sub-topics,

and preconditions (i.e., necessary conditions for a rule, principle, or fact to hold true). The

hierarchical analysis was performed at the sentence level. There were 7 sentences at level

1 (the most superordinate level), 6 sentences at level 2, and 7 sentences at level 3.

Hierarchical level was Included as a variable In the data analyses to see whether perceived

Importance was Influenced by level, and If this variable Interacted with sentence form (i.e.,

definition or fact).

Subjects

The novices were 14 u ndergraduates with 2 or 3 semesters of college physics.

Novices with this level of physics training were selected to insure that the novice group had

had enough exposure to physics to have developed information-type rules, but had not

approached the expert level in training. The 24 experts had completed at least one year of

graduate study in physics.

Two control groups were also run in the experiment to see if expert-novice differences

In the perceived importance of the target sentences were due to differences in educational

level (i.e., undergraduate vs. graduate level training) as opposed to differences in physics

koiowledge. The two control groups were selected so as to differ in their educational level

In the same manner as the two experimental groups. However, none of the control group

subjects had taken any college-level physics, so they were similar to each other in terms of

their physics knowledge. The undergraduate control group consisted of 24 undergraduates:

the graduate student control group consisted of people who had completed at least 1 year

of graduate training in the humanities or social sciences. Although this group will be

referred to as the graduate student control, it included some post-doctoral researchers and

~ . . -.

- %Y ----
Novice Rules 6

faculty. This was also true of the corresponding expert experimental group.

Although the control groups were specifically selected to control for educational

differences, they would also indicate differences In perceived importance due to age,

maturity, and verbal ability. In the case of verbal ability, it is reasonable to assume that%

graduate students In the social sciences/humanities would be as high in verbal ability as

graduate students in physics. Similarly, It is likely that undergraduates attending the same

university are roughly equivalent in verbal ability.

Procedure

The subjects were given one version of each passage. They were told to read each

passage carefully, then rate the importance of each sentence on a scale from 1 (most

Important) to 5 (least important), and then indicate the 10 most important sentences in each

passage. The Instructions for the rating task Indicated that each rating should be used at

least once. All of the sentences were rated, but only the ratings for the target sentences

were analyzed. The order in which the passages were read and the versions of the

passages received were counterbalanced.

The novice and the undergraduate control groups were told that in completing the

tasks, they were to indicate which sentences they thought would be most important to learn

If they were going to be tested on the passage content. The expert and the graduate

student control groups were told to pretend that they were teaching a course and indicate

which sentences they thought were most Important for their students to learn. These

instructions match those used by Dee-Lucas and Larkin (1986) in their initial research on

expert-novice differences in perceived Importance. The instructions were designed to

compare what novices think they should learn with what experts (their instructors) think

novices should learn.
Novice Rules 7

Results

The data from the two dependent measures were analyzed in two ways. The ratings

data were analyzed using a multiway frequency analysis. This analysis fits a loglinear model

to categorical data. The number of responses in each rating category (1 through 5) for

each sentence type (definition and fact) occurring at each level (1 through 3) was tabulated

for each of the subject groups. The multiway frequency analysis was performed on the total

number of responses occurring in each of these cells.

The data from the sentence selection task (i.e., select the 10 most important

sentences) were analyzed with a logistic regression The variai'les entered in the analysis

for each target sentence for each subject were sentence type (definition or fact), level (1

through 3), and subject group (novice or expert). The dependent measure was whether or

not the sentence had been selected as one of the most important.

The data from the two control groups were submitted to identical analyses. The

results of these analyses were compared to the results of the corresponding analyses of the

experimental group data to determine if expert-novice differences were also reflected in

differences between graduate students and undergraduates who had had no advanced

physics training.

Ratings Data

Experimenta/ groups. The multiway frequency analysis of the ratings data indicated

that the best-fitting model was a hierarchical model including the type x group interaction

and the main effect of level (A2

2=23.80, df=30, p<.78). The mean ratings predicted by the

model for the type x group interaction are shown In Figure la. The predicted means for

the novices are 1.67 when the sentence was in the form of a definition and 1.89 when it

was in the form of a fact. For experts, the predicted ratings are 1.79 for definitions and

1.82 for facts. This interaction indicates that novices were influenced in their importance

11i li~ " " " • " r • , • - ." . .° , o - - ,' , s'
N

Novice Rules8

ratings by the form in which the information was presented. They considered the same

content to be more important if it was stated as a definition as opposed to a fact. The

experts, on the other hand, did not appear to base their ratings on sentence form; there is

very little difference in their predicted mean ratings for definition and fact versions of the '.

target sentences.

Insert Figure 1 about here .5

The parameter estimates for the main effects and interaction for the complete model

are shown In Table 2. Because of the usual constraints placed on the model, all parameter

estimates for each main effect and interaction are constrained to sum to zero. Therefore,

for all effects the magnitude of the parameter estimates for each variable are the same but

In the opposite direction. The ratios of the estimates to the standard errors indicate the

degree to which the parameter estimates differ from zero.

Insert Table 2 about here

The parameter estimates for the main effect of group show little difference between

the experts and novices In their use of the five rating categories. The largest differences

occurred in the use of rating 5 (the lowest rating) and rating 3 (the middle rating). The

novices tended to use the rating 3 category more often than the experts (as indicated by -5

the positive parameter estimate), while the experts tended to use the lowest rating more

often than the novices. This suggests that the experts rated the target sentences lower in

importance than the novices.

The parameter estimates for the main effect of type indicate that the largest difference

occurred in the rating 3 category (the middle rating). The negative parameter estimate

Indicates that this rating was used more often with facts than definitions. There were also

°-
- ~ -- - -~ K"

Novice Rules 9

smaller differences In the use of the first two rating categories, with the definitions rated 1I

or 2 more often relative to facts. This indicates that definitions were rated higher overall in

Importance than facts.

The type x group interaction estimates indicate that the greatest differences between4.'

experts and novices in rating definitions and facts occurred in the first two rating categories.

The novices were more likely than experts to give a target sentence a rating of 1 If It was

In the form of a definition, and somewhat more likely to rate it a 2 if It was In the form of

a fact. The opposite was true for the experts, relative to the novices.

Level had a very strong influence on the target sentence ratings, as shown by the

large parameter estimates for this effect. Level 1 target sentences tended to be rated as

most Important, Indicated by the large positive estimate for the rating 1 category. The

ratings for level 2 target sentences were spread over the categories without any strong

clustering In any one category; none of the parameter estimates for level 2 sentences .

differed from zero by more than two standard errors. Level 3 target sentences tended to

be rated as 3 or 4 in importance, indicated by the positive parameters for these ratings

categories. Level did not interact with type or group in influencing the ratings (i.e.,

Including these effects reduced the fit of the model).

Control _gro us. The muitiway frequency analysis of the control group data indicated

that the best-fitting model Included only the main effects of group and level (A2=28.64,

df-40, p<.91). The inclusion of the type x group interaction or the main effect of type

reduced the fit of the model to the data set. The mean ratings predicted by the model

with the type x group Interaction Included are presented in Figure lb for comparison with

the corresponding experimental group means. As Figure lb shows, there was no difference

between the undergraduate and graduate controls in the influence of sentence type on the

mean ratings of the target sentences. Additionally, the lack of a main effect of sentence

type Indicates that the form in which the target sentences were presented did not influence
'1-jo ce Rules 10

the control group ratings (i.e., they did not consider definitions to be important than facts).

The parameter estimates for the loglinear model including the main effects of group

and level are shown in Table 3. The estimates for the main effect of group indicate that

the undergraduates and graduates differed primarily in their use of ratings 1 and 4. The

positive estimate for the undergraduates for rating category 1 indicates that they gave the

target sentences a rating of 1 more often than the graduate controls. The opposite was

true for the rating 4 category, with the graduates using this rating more often relative to the

undergraduates. This indicates that undergraduates rated the sentences as more important

overall than the graduate controls. This effect is also apparent in Figure lb.

Insert Table 3 about here

The pattern of parameter estimates for the main effect of level is similar to that

obtained with the experimental groups. Level 1 target sentences were most likely to receive

a rating of 1, indicated by the large positive parameter estimate for that rating. The ratings

given to level 2 target sentences were spread over the categories, with the strongest

clustering In the rating 1 category (though this parameter estimate was much smaller than

the rating I estimate for level 1 sentences). Level 3 target sentences were most likely to

receive a rating of 4, with 3 as the next most frequent rating category for this level.

Summary. The results of the multiway frequency analyses indicate that novices base

their judgements of the importance of text information on the form in which the information

is presented. They specifically rate the same content as more important when it is

presented as a definition as opposed to a fact, as is shown in Figure la. Expert physicists

are not influenced by sentence form in rating importance, presumably basing their

Importance judgements on the nature of the sentence content. Similarly, subjects without

advanced physics training are not influenced by sentence form, so that they appear to

behave like experts in rating importance. This can be seen in Figure lb. This is most

'p
Novice Rules 11

likely because these subjects have no strong expectations about the relative importance of

definitions and facts In physics texts, and thus are not Influenced by this text feature. The

lack of a type x group interaction in the control group data indicate that expert-novice

differences in the perceived importance of definitions and facts are not due to differences in

educational level.

Sentence Selection Data

Experimental aroups. The sentence selection data were analyzed using a logistic

regression. The regression analysis indicated that a good fit to the experimental group data

was provided by a hierarchical model including the group x type interaction and the main

effect of level (X2=8.69, df=7, p<.2 7 6). This is the same model found to provide the best

fit for the ratings data from the experimental groups. The predicted mean proportion of

target sentences selected for the type x group interaction are shown in Figure 2a plotted on

a loglt scale. This Interaction is very similar to the type x group Interaction obtained with

the ratings data. It shows that the novices were more likely to select a target sentence as

important when it was presented in the form of a definition than a fact, while the experts

were relatively unaffected by sentence form In their selection of the important sentences.

Insert Figure 2 about here

The parameter estimates for the complete logistic regression model are shown in Table -

4. Unlike the estimates for the multiway frequency analysis, the logistic regression estimates

show the size of the difference between the means for the two variables in an effect.

Therefore only one parameter estimate is presented for each main effect (two for the

Interaction) and the ratios indicate the size of the difference between the two variables in

each effect and interaction.
Novice Rules 12

Insert Table 4 about here

The negative estimate for the main effect of type indicates that Subjects tended to

select more target sentences when they were presented in the form of a definition. This is

consistent with the main effect of type found with the ratings data from the experimental

groups. The estimate for the main effect of group indicates little difference between the

experts and novices in the number of target sentences selected as important. The negative

estimate indicates that the novices selected more target sentences than the experts.

However, the parameter estimates for the two groups differed by less than two standard

errors.

The type x group Interaction estimates indicate that novices were more likely to select

target sentences as Important when they were in the form of a definition rather than a fact.

Relative to novices, the experts were more likely to select the sentences when they were in

the form of a fact. These findings are shown by the positive estimate for definitions and

the negative estimate for facts.

The very large parameter estimates for the main effect of level indicates that this%

variable had a strong effect on which target sentences were selected as important. The

negative estimate indicates that target sentences from the upper levels of the passage

hierarchy (level 1) were selected more often than the target sentences from the lower levels

(level 3). This finding is also consistent with the strong levels effect found in the ratings

data. As with the ratings data, there was no Indication that level interacted with sentence

type or group in influencing sentence selection.

Control groups. The logistic regression for the control group data Indicated that the

best-fitting model was a hierarchical mode) including the main effect of group and the type

x level Interaction (A~, _1.78, df-5, p<,879). The inclusion of the type x group interaction

reduced the fit of the model to the data. Thus there was no evidence that the importance
Novice Rules 13

judgements of the undergraduate and graduate control groups differed in the degree to

which they were Influenced by sentence type. This can be seen in Figure 2b, which shows

the predicted mean proportion of definitions and facts selected by the two control groups

plotted on a logit scale.

The parameter estimates for the regression model including the effects of group and

the type x level Interaction are presented In Table 5. Unlike the experimental groups, level

did not have a linear effect on the number of target sentences selected by the control

groups. It was therefore entered into the analysis as a categorical (as opposed to a linear)

variable, and separate parameter estimates were obtained for each level. The parameter

estimates presented for the main effect of level represent the size of the difference between

levels 1 and 2, and levels 2 and 3. For the type x level interaction, the parameter

estimates Indicate the size of the difference between definitions and facts at each level.

Insert Table 5 about here

The negative parameter estimate for the main effect of type indicates that the control

groups were more likely to select a target sentence when It was in the form of a definition

than a fact. However, the type x level Interaction Indicates that the effect of type varied

with level. The negative parameter estimate for level 1 indicates that facts were selected

more often than definitions at the top level, while the positive parameters for levels 2 and 3 "-

show that definitions were more likely to be selected than facts at the lower levels. The

predicted cell means for this Interaction are presented in Figure 3. Figure 3 shows that the

main effect of sentence type is due primarily to a very large sentence form effect at level 3.

Insert Figure 3 about here

-- -

This finding suggests that subjects without physics training may tend to judge details (low-

'S .. . .. .. . . . . . . . ..
Novice Rules 1

level Information) in physics texts as being more important when they are presented as

definitions than facts. However, this sentence type difference is based on very few data

points, as most level 3 sentences were not selected as important. The mean number of

sentences from level 3 selected by the control groups were 2.3 for the undergraduates and

1.6 for the graduates. Additionally, this type x level interaction was not found in the ratings

data for the control groups. Therefore, it is possible that this particular effect is not

replicable.

The negative parameter estimate for the main effect of group indicates that

JIM

undergraduates selected more target sentences as important than the graduate control

group. This Is consistent with the finding in the ratings data that undergraduates tended to

rate the target sentences higher in importance relative to the graduate controls.

The positive parameter estimates for the main effect of level indicate that the number

of sentences selected as Important decreased with level. The size of the estimates show

that the drop in the number of target sentences selected as important was much greater

between levels 2 and 3 than between levels 1 and 2. This can also be seen in Figure 3.

Summary. The results of the sentence selection task analyses are consistent with the

findings from the ratings data. Sentence form influenced the importance judgements of

novices, with novices selecting more target sentences when they were presented In the form

of a definition (see Figure 2a). Sentence form had very little effect on the sentences

selected by experts. There again was no type x group interaction in the control group

data, as shown In Figure 2b, indicating that expert-novice differences are not due to

differences In educational level. Sentence form did have some influence on the judged

importance of sentences for the control groups. The regression analysis indicated that the

control groups tended to select more sentences from level 3 when they were in the form of

a definition than a fact. Thus it Is possible that there is a bias towards considering lowlevel definitions as important in subjects who do not have scientific backgrounds.

• . • #m, .. % - o .. ., * .. - °. . . .- " , . " " .•. . . ,, . . . " . " . .
Novice Rules 15

Discussion

Although much research has examined the knowledge representations of experts,

relatively little Is known about the knowledge structures of novices. This is most likely

because novice representations are assumed to be more variable than those of experts, and

thus are studied less. However, how novices represent unfamiliar content domains and the

nature of the changes occurring In these representations has Important educational

Implications for facilitating learning from text. Characterizations of the knowledge structures

used by novices to govern text processing Indicate how text should be structured, both

globally and locally, to maximize the probability that novices will learn the Important text

content.

This study examined one feature of novice knowledge representations-rules for

assessing Importance In unfamiliar scientific domains. it Investigated the nature of novice

scientists' content schema for physics subject matter, specifically testing the hypothesis that

novices judge importance in physics texts on the basis of sentence form. This was done by

using minor wording changes to transform sentences into definitions or facts, and looking at

how this altered their perceived importance. Novices judged the same information as more

Important when It was presented as a definition, indicating that they based their judgements 4

on sentence form. The form of the information did not affect the importance judgements of

experts. The experts presumably judged the importance of the information according to its

content, rather than its form. This type by group interaction was not found with

undergraduate and graduate controls, so this interaction is not due to differences in the

educational level of the physics experts and novices.

These findings indicate that novices who have had some college physics develop a

rule that definitions are more important than facts. This rule is not inappropriate-experts .

also consider definitions to be more important than facts in physics passages in which

content differences between the two sentence types have not been controlled (Dee-Lucas &
N'VV N 6-WVV V'v WW

Larkin, 1986). These expert data suggest that definitions in physics textbooks may contain

Information that is particularly important for understanding these texts. However, the results

of the current study Indicate that novices judge importance on the basis of sentence type

category without regard for sentence content. They judged the same information as more

Important when it was presented as a definition, indicating that they consider definitions to

be Important regardless of their content. This suggests that novices are applying this

Information-type rule too rigidly, in that they are not distinguishing between important and

unimportant Information within the definition and fact categories. In this sense, they have

"overgeneralized" the rule. This overgeneralization results In novices systematically misIdentifying the Important text information by missing Important facts and focusing on

unimportant definitions.

These findings parallel those on expert-novice categorization of problem types in

physics and math (Chi, Feltovich, & Glaser, 1981; Schoenfeld & Herrmann, 1982). This

research on problem identification has found that novices are Influenced by surface-level

features in their early categorization schemes, but move to a more expert-like representation

with Instruction. In the current study, the novice rules are also based on superficial

characteristics (e.g., category type membership). However, the resulting importance

judgements based on these rules are not radically different from those of experts, Indicating

that this feature Is related to some degree to the underlying content structure defining

Importance within the domain of physics.

Information-type rules of the kind demonstrated here can be thought of as the

foundation for a content schema. A content schema includes knowledge about what ""

Information Is Important In a content area. The results of the current study suggest that an

early stage in the development of a content schema for scientific subject matter may be the

specification of rules indicating what types of information are important in that content

domain. These information-type rules have important learning consequences. Previous

%i
Novice Rules 17

research has found that novices spend more time on definitions than facts when reading

physics passages, recall more definitions than facts afterwards, and include more definitions

than facts In their summaries of physics texts (Dee-Lucas & Larkin, 1985. 1986). Thus

these information-type rules appear to Influence novice readers' attentional processes during

reading, as well as the macrostructure they develop for physics texts.

Unlike novices, the importance judgements of the control subjects with no college

physics were largely unaffected by sentence form. These subjects therefore appear similar

to experts. This is most likely because they have no strong expectations about what types

of content should be important in physics texts. Thus the control subjects were not

Influenced by sentence form because they lack a physics-relevant content schema; the

experts were not Influenced by form because they have developed a much more refined

schema which Indicates the importance of specific information within the domain of physics.

The experts' content schema includes a finer-grained analysis of the importance of the

Information than that captured by sentence-type classifications.

This suggests an inverted U-shaped relationship among naive, novice, and expert

physicists In their sensitivity to sentence form. Both naive and expert subjects are not '.

Influenced by sentence form; they judge the same content equal in importance regardless of

Its form. Novices, on the other hand, are 'distracted" by form in their Importance

judgements, In that they alter their judgements according to whether the information is

presented as a definition or a fact. This Is because their knowledge of physics has caused

them to develop a general expectation that definitions are more important than facts (an

expectation lacking in naive subjects), but they do not have sufficient knowledge to judge

the Importance of specific content within these categories (as experts can), and thus rely on

this general rule based on sentence form.

A similar U-shaped relation has been found among beginning, intermediate, and expert

radiologists In diagnosing x-ray pictures. Lesgold, Feltovich, Glaser, & Wang (1981) found
Novice Rules 18

that beginning residents and experts were better than intermediate-level radiologists in the

diagnostic reading of certain classes of x-ray films. This is because accurate diagnosis'"'

involves an interaction between the physical features found on the x-ray and the radiologist's

knowledge of the relevant contextual features which constrain the possible alternative "4

diagnoses. Beginning-level radiologists base their diagnoses on the physical features of the

x-ray, and are accurate when there happens to be a match between their interpretation of

those features and the actual pathology; experts use their schematic knowledge to interpret

the physical features in the context of other relevant Information (such as the patient's

medical condition), and thus are systematically accurate in their diagnoses. Intermediatelevel radiologists possess some schematic knowledge, but this knowledge is not refined or

flexible enough to provide accurate diagnoses--it 'distracts" them from the direct physical

features used by beginners, and is not elaborate enough to allow them to pinpoint the

appropriate alternative In the same manner as experts.

Thus both with physicists and radiologists, there is a stage in the development of

expertise in which novices possess a primitive schema incorporating very general information

about the domain (i.e., what types of information are typically important for understanding .4

physics; what contextual features are relevant for interpreting an x-ray). These schemata are

not sufficiently refined to reliably allow accurate performance, and actually impair novice

performance (relative to that of naive subjects) or certain tasks. Similar devolopmental

trends have been found in the performance of children on a variety of cognitive tasks (see

Richards & Siegler, 1982). These findings suggest that a short-term decrement in certain

aspects of task performance may be a necessary consequence of the early forms of

schematic knowledge that develop in the course of acquiring expertise in a variety of

content domains.

This study was a strong test of the hypothesis that novices are influenced by sentence

form in judging importance, In that all variables other than sentence form were held constant

.... . .. .. . .. ... . . . . -- . ... . . - •. %.: . . . ..,. ., , ! !
Novice Rules 19

In the target sentences. The differences in the perceived importance of definitions and facts

obtained for novices were small in absolute terms. This is to be expected given the nature

of the manipulation. The only difference between the definition and fact versions of the

target sentences was In the use of the words 'is defined as"-the substantive content of the

sentences and their location in the texts remained the same for both sentence versions.

Thus the contextual and content matter constraints on importance were identical for both

versions. These constraints reduce the potential difference in the importance of the two

versions by limiting the range of Importance to that which is plausible given the relationship

of the sentence content to the passage as a whole. The results of this study show that in

addition to the usual contextual and subject matter constraints which influence all skilled

readers, novice scientists are Influenced in a systematic and reliable manner by the form in

which the content Is presented when judging importance in scientific texts.

These findings suggest that certain variations in wording which do not affect the

reading of experts or domain-naive readers will have essentially a "signaling" effect on

novice attention and recall. Signaling involves the use of non-content words, such as "more

Importantly," 'note that,' etc., to emphasize particular text information. Signaling has been

found to alter readers' attention, as indicated by differences in recall patterns for signaled

and unsignaled texts (Loman & Mayer, 1983; Meyer, 1983). The results of the current study

show that there are particular alternative wordings (e.g., "is defined as" vs. "is represented

as') which do not function as signaling phrases for most readers but have a signaling effect

for novice readers because they reflect the categorization scheme used by novices to assess

importance. Therefore authors can unintentionally signal particular information as important

through wording selections of this type if they are unaware of novice preconceptions

regarding the text content domain.

On the other hand, an author can use knowledge of novice importance rules in

conjunction with signaling techniques to guide readers' attention to the appropriate text

I.

___
Novice Rlules

.4,

content. Writers can use signaling devices to emphasize important content within the

sentence-type categories that novices consider unimportant, and de-emphasize less critical

Information In the categories that novices judg. as important. Meyer (1975) describes tour

types of signaling techniques: (1) providing cues as to the text structure, as in "the

problem is .the solution Is," or "first ... second ... third;" (2) paraphrasing important text content

before It is presented, such as "the important points in the following discussion are ... (3)

summary statements of key ideas presented after the relevant text; and (4) pointer words

emphasizing specific statements In the texts, as In "more importantly,' "it is notable that,"

-unfortunately," etc. These devices can be used independently of the overall hierarchical

structure of the passage to emphasize specific content within each level, a technique that

Meyer (1983) terms a *differential emphasis plan." These techniques have been found to

be effective In altering recall patterns of readers of varying backgrounds and abilities (Loman

& Mayer, 1983; Meyer, 1983).

Signaling devices can be used by writers of physics texts to help guide novices'

attention to the Important content. For example, the current study indicates that definitions

are a type of content that is particularly salient to novices. Therefore, it would be helpful

to novices If writers would differentiate the less important definitions from those that are

central to the text. This could be done by including structural cues or preliminary

summaries that emphasize the main topics or content that the aLthor wants the reader to

abstract from the text, and thus in effect "de-emphasize" unimportant definitions. This

would help novices distinguish between the definitions that are to be learned (i.e., the main

points) and those that are there to simply aid in comprehension or elaborate on the

important points. Facts, on the other hand, are not particularly salient for novices, and

earier research indicates that novices may be missing important content of this type (DeeLucas & Larkin, 1986). In this case, the use of underlining and pointer words (e.g.. note

that, It Is Important to understand, this leads to the important conclusion that, etc.) could be

V. N .. . .
Novice Rules 21

used to draw attention to important facts.

Another technique that could be used to help novices distinguish between important

and unimportant information is to teach novices a general learning schema that can be

applied to texts to assess importance. This approach to manipulating attention and learning

is 'strategy-based' rather than "text-based." The ability of students to acquire general

skills for learning from scientific texts has been demonstrated by Larkin and Reif (1976).

They taught students a skill for understanding quantitative relations in physics texts by having

students work through a series of training materials that required them to read physics texts

and answer a prespecified set of questions. The subjects were able to learn to use this

question set as a strategy for acquiring an understanding of new relations, and performed

appreciably better on tests of understanding than subjects who had not acquired this

learning strategy. In the case of the current study, the goal of the learning schema would

be to help readers distinguish between important and unimportant definitions and facts. This

would Involve teaching students what types of definitions and facts should be learned. For

example, Larkin and Reif were able to specify a subset of facts (i.e., those dealing with

units and typical magnitudes) that are particularly important for understanding quantitative

relations. This type of schema could be used by novices in reading scientific texts to aid

in identification of the important content within categories of information (like facts) that they

generally assume are unimportant. This schema would hold true for a broad content area

within the domain of physics, but would most likely vary to some extent with subfields of

study In physics.

The present study indicates that novice readers can be sensitive to very minor changes

In wording at the sentence level in a text, in that these changes can be relevant to

distinctions made in their content schema for the text domain. This suggests that writers

need to be aware of the manner of presentation of information at a fairly local level in a

text. In particular, experts writing for a novice audience will be most effective in enhancing

'N.

* . .*- .*J....

- . " . ". '. .'2 ' ., ' "," ," "' ' • .' . '- " ." -.- , . .. . ... . . , ' , ' " • ll' " - t , -" ,~l' -" " ",',"', ", " ", ",- "-, , - -, .'.' . .,.. ,. ..
Novice Rules 22

learning of the Important text content if they use text-based indicators of importance in

conjunction with knowledge of their audiences' content schema. In this way, writers can use

techniques such as signaling to provide novices with clear signals of importance which will

help them distinguish between Important and less critical content.

0

.0

%

,e.

S.

.5

.*./.-. . 4?* .* d . .* , . : ,* '''. . ."' ".. .. ' .. .-. ....... 
Novice Rules 23

References

Chi, M., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics 0

problems by experts and novices. Cognitive Science, 5, 121-152.

Dee-Lucas, D., & Larkin, J.H. (1985). Attentional strategies for studying scientific texts (Tech.

Rep.). Paper presented at the annual meeting of the American Educational Research

Association, Chicago.

Dee-Lucas, D., & Larkin, J.H. (1986). Novice strategies for comprehending scientific texts.

Discourse Processes, 9, 329-354.

Kleras, D.E. (1985). Thematic processes in the comprehension of technical prose. In B.K.

Britton and J.B. Black (Ed.), Understanding expository text. Hillsdale, N.J.: Lawrence

Erlbaum.

Larkin, J.H., & Reif, F. (1976). Analysis and teaching of a general skill for studying

scientific text. Journal of Educational Psychology, 68, 431-440. '

Lesgold, A.M., Feltovich, P.J., Glaser, R., & Wang, Y. (September, 1981). The acquisition of

perceptual diagnostic skill in radiology (Tech. Rep. PDS-1). Learning Research &

Development Center, Univ. of Pittsburgh.

Loman, N.L., Mayer, R.E. (1983). Signaling techniques that increase the understanding of

expository prose. Journal of Educational Psychology, 75, 402-412.

Meyer, B.J.F. (1975). The organization of prose and the effect on recall. Amsterdan

North-Holland.

Meyer, B.J.F. (1983). Text dimensions and cognitive processing. In H. Mandl, N. Stein, &

T. Trabasso (Ed.), Learning and understanding from text. Hillsdale, N J. Lawrence

Erlbaum.

Richards, D.D., & Siegler, R.S. (1982). U-shaped behavioral curves! It's not whether you're

right or wrong, it's why. In S. Strauss (Ed.), U-Shaped Behavioral Growth New York.

NY: Academic Press.

"",- J.~d* ~g~pjr
Novice Rules 24

Schoenfeld, A.H., & Herrmann D.J. (1982). Problem perception and knowledge structure in

expert and novice mathematical problem solvers. Journal of Experimental Psychology, 8.

484-494.

%a
Novice Rules 25

Table 1: Examples of definition and fact versions of target sentences.N

1. Asolue smplytheactul pessue i defnedas pessue a a pint

1.Absolute pressure is ee s simply the actual pressure at a point.

2. Specific gravity is defined as the ratio of the density of a substance to the

Specific gravity Is Indicated densiy by the ratio of 

of the density ateI of a substance to the

3. In terms of this notation, the work A W done by a force F In moving an object

through a displacement Ar is defined as

AW= F- Ar = F(ArcosO)

In terms of this notation, the work A W done by a force IF in moving an object

through a displacement Ar is represented as

A W F Ar= F (A rcos 0)

4. Specifically, the total amount of work done In changing the motion of a particle

Is defined as the sum of the works done by each of the Individual forces acting

on the particle.

Specifically, the total amount of work done In changing the motion of a particle

Is equal to the sum of the works done by each of the Individual forces acting

on the particle.

5. Intrinsic quantities are defined as quantities which are independent of volume.

Intrinsic quanitities are Independent of volume.

6. The unit typically used for measuring work, the joule, is defined as the work FNI

done by a unit force (one newton) acting on a unit distance (one meter).

The unit typically used for measuring work, the joule, Indicates the amount of

work done by a unit force (one newton) acting on a unit distance (one meter).

7. Pressure Is defined as the magnitude of a fluid force divided by the area of the

surface on which it acts.

Pressure can be calculated by dividing the magnitude of a fluid force by the

area of the surface on which It acts.

% %.

;Val . A %1 -2
Novice Rules 26

Table 2: Parameter estimates, standard errors, and ratio of estimates to standard

errors for a loglinear model of the ratings data from the experimental groups. Asterisks indicate coefficients that differ from zero by more than 2 standard errors

Ratio: Effect Coeff. St. Error Coeff./St. Error

(a)Group: novice estimates (expert estimates are opposite)

Rate 1: .045 .066 .69

Rate 2: .074 .075 .98

Rate 3: .094 .090 1.04

Rate 4: .019 .118 .16

Rate 5: -.232 .182 -1.27

(b)Type: definition estimates (fact estimates are opposite)

Rate 1: .073 .067 1.10

Rate 2: .082 .075 1.10

Rate 3: -.224 ° .090 -2.48

Rate 4: .064 .115 .56

Rate 5: .005 .192 .03

(c)Type x Group: definition estimates (fact estimates are opposite)'

Rate 1:

Novice (+) .133* .066 2.01

Expert ()

Rate 2:

Novice 1-) .098 .075 1.30

Expert (+)

Rate 3:

Novice (+) .003 .081 .04

Expert (-) .

Rate 4:

Novice (+) .014 .117 .12

Expert (-)

Rate 5:

Novice (-) .052 .180 .29

Expert (+)

'Parameter estimates indicate the size of the effect; positive and negative symbols

Indicate the direction of the effect for each group.

5I

-,',,. ., .,. .. ,. , . . ,. . . . . ,. .. . .. ,.. . .. ... . . . . . .".

' "; '-,'-:':.' -';,,:;-.;,.,, .-.. ./,.:. , -,. , -,.-.:. .:......:.....:.. .--.-..-.. :..-..-....-....-.-..-......-..-...1.
Novice Rules 27

(d)Level

Level 1:

Rate 1 .917' .121 7.57

Rate 2 .214 .135 1.59

Rate 3 -.454'* .185 -2.46

Rate 4 .504 * .250 -2.01

Rate 5 -.173 .351 -.49

Level 2:

Rate 1 -.042 .110 -.38

Rate 2 .052 .120 .44

Rate 3 .140 .148 .95

Rate 4 .165 .193 .86

Rate 5 -.315 .320 -.98

Level 3:

Rate 1 -.875* .098 -8.97

Rate 2 -.265* .104 -2.56

Rate 3 .314' .128 2.46

Rate 4 .339* .169 2.00

Rate 5 .488 .251 1.94

%p
- p. -jqi.WWL-WWUW-WrVIV1 U ,'!-' .7T % -R.-

Novice Rules 28

Table 3: Parameter estimates, standard errors, and ratios of estimates to standard

errors for a loglinear model of the ratings data from the control groups.

Asterisks Indicate coefficients that differ from zero by more than 2

standard errors.

Ratio:

Effect Coeff. St. Error Coeff./St. Error ,6

Group: undergrad estimates (grad estimates are opposite)

Rate 1: .323" .065 4.98

Rate 2. .083 .074 1.12

Rate 3: -.070 .089 -.79

Rate 4: -.323 .117 -2.76

Rate 5: -.010 .172 -.06

Level

Level 1:

Rate 1 .695 .105 6.62

Rate 2 .141 .119 1.18

Rate 3 -.302 .156 -1.94

Rate 4 -.492" .220 -2.24

Rate 5 -.042 .282 -.15

Level 2:

Rate 1 .128 .102 1.25

Rate 2 .013 .118 .11

Rate 3 .133 .137 97

Rate 4 -.047 .188 -.25

Rate 5 -.226 281 -.80

Level 3:

Rate 1 -.822* .092 -8.92

Rate 2 -.154 .099 -1.56

Rate 3 .169 .119 1.42

Rate 4 .539 .155 3.48

Rate 5 .268 .226 1.19

.
Novice Rules 

29

Table 4: Parameter estimates, standard errors, and ratios of estimates to standard

errors for a logistic regression model of the sentence selection data from the

experimental groups.

Ratio

Effect Coeff. St. Error Coeff./St. Error

Type: -.414 .099 -4.18

Group: -.186 .099 -1.88

Type x Group: .236 .099 2.38

Definition (+)

Fact (-)

Level: -1.016 .088 -11.49

'.d

'°9
Novice Rules 30

Table 5: Parameter estimates, standard errors, and ratios of estimates to standard

errors for a logistic regression model of the sentence selection data from the

control groups.

Ratio:

Effect Coeff. St. Error Coeff./St. Error

Type: -.220 .098 -2.24

Type x Level:

Level 1: -.448 .139 -3.22

Level 2: .014 .139 .10

Level 3: .462 .140 3.30

Group: -.590 .098 -6.02 .

Level:

Level 1-2 dliff: .476 .139 3.42

Level 2-3 ditt: 1.316 .140 9.40 p-.

4."

p

r.

I

",I

4'

1'4.

.

.4..

~ :.........
I.1

Novice Rules 31

Figure 1: Predicted mean ratings for the type x group interactions in the

ratings data.

1.55

.~1.60

1.65 Definitions

,.65-

C 1.70

:1.75

1.80

1.85

1.9o0

Facts

1.95.

2.00-

2.05

2.10 Novice Expert

Group

(a) Experimental Groups

1.55 a,

. 1.60

S1.65

1 1.70

: 1.75

S1.80

j1.85

1.90

1.95 Definitions

2.00

2.05 Facts

2.10 Undergrad Grad

Group

(b) Control Groups

ON'1
Novice Rules 32

Figure 2: Predicted proportion of target sentences selected as important for

the type x group interactions In the sentence selection data.

...=,

4 Definitions

.-..

.88 Facts

64

.76

.72 Novice Expert

Group

(a) Experimental Groups

1.64Z

UQ

.60

S.568

-. 52-

.48 .48 Definitions .

.44.

Facts

Undergrad irad

Group

(b) Control Groups

',

"4
Novice Rules 33

Figure 3: Predicted proportion of target sentences selected as important for

the type x level interaction in the Sentence selection data for the control

groups.

.70

4L

)b Uefinibofls

Facts

20

1 2 3

L evei