34
£25 : ia
a [—o— Trin (passing)

|—— Cate (passing)

| —k— Trmnt (non-passing)
g 2 |—@— Cuil (non-passing)
Ry 4

‘
15 4-7

Pre Post

Figure 4. Focus Scores by Condition and Initial Writing Aptitude

Sequence. Students in the treatment condition improved 17% on the sequence score,
about 12% more than those in the control group. Percent gains for passing-treatment,
passing-control, non-passing treatment, and non-passing control sub-groups were 9%,
3%, 27%, and 13% respectively. The condition X writing aptitude X repeated measure
interaction shown in Figure 5 was significant [F(1, 848) = 5.82, p = .016] although again
its associated effect size was relatively small (x? = .007). The condition X repeated
measure and writing aptitude X repeated measure interactions accounted for 4% and 8%
of the score variance respectively.

3 —-—_—_—_-_
2 ; -
a 5
s 2 —@— Tmt (passing)
3 |W — cot (passing)
A |—se— Trmnt (non-passing)
3 2 |—@— cut! (non-passing)
= F
154

Pre Post

Figure 5. Sequence Scores by Condition and Initial Writing Aptitude

14


3.2_Instruments
Persuasive mode writing samples were collected from each student in the sample to

baseline initial writing ability and to quantify instructional gains. The writing prompts
came from the National Assessment of Educational Progress (NAEP) and were used with
permission. Using prompts that had been field-tested by a national study concerned with
longitudinal measurement of achievement in writing guaranteed both equivalency and test-
retest reliability, as well as the absence of regional bias.

A 1-6 point holistic scoring guide, formulated by the authors, gave focus to the general
impression scoring of the professional raters. This guide reduced the amount of
subjectivity in the assessment by giving the raters a set of constraints and a list of features
by which to determine a writer's achievement in a persuasive piece. Because the guide
was an abstract description of criteria, raters were trained through the discussion of
anchor papers chosen as representative types. As a result, holistic assessments were
standardized against a set of papers with known scores.

Because the papers were judged on a rather broad set of features, the holistic assessment
carried little diagnostic information. Therefore, a second rubric was designed to assess the
more specific set of cognitive operations emphasized in R-WISE 1.0. This rubric (more
fully described in the Discussion) outlined four subskills necessary but not sufficient for
good persuasive writing. The subskills were: 1) abstraction, 2) elaboration, 3) focus, and
4) sequence. Each of these subskills was scored on a 1-4 point scale.

3.3_ Procedure
The study took place during the second semester of the 1992-1993 school year or from

late January to late May of 1993.

Software Design. R-WISE 1.0 was developed and refined over a period of 12 months by
an inter-disciplinary team of practitioners, researchers, and programmers supported by the
Air Force’s Armstrong Laboratory, Brooks, AFB, TX. A simple explanation of the
composing process that has gained wide acceptance in pedagogy depicts writing as having
three central stages: prewriting or invention, writing or drafting, and revision or editing.”
Therefore, the three workspaces of R-WISE 1.0 were created to mirror this partitioning.
The teaching comes from the powers of reification (or representing complex processes as
manipulable objects on the computer screen). Each of the workspaces (1) accommodates
deficiencies and thereby reduces frustration for a weak writer, (2) emulates some of the
crucial functionality of paper copy, (3) enriches the environment and thereby sustains
motivation, and (4) models robust behaviors.

Study Design. A quasi-experimental contrasted groups design was used to guide data
collection. It was not possible to randomly assign schools to treatment or control
conditions or to randomly assign students within these groups. As a result, the groups
were not necessarily equivalent at pre-test.


Turning to the realm of the cognitive tool in writing instruction, conventional wisdom
suggests that more adept users -- that is those with more general aptitude and better
domain-specific skills -- can make better use of the device and, therefore, learn more from
the interaction. The converse of this reasoning prompts a belief that less adept students
will benefit less from a cognitive tool. Kozma (1991) summarizes the cautions voiced in
regard to using computerized instruction for novice writers.

Tools, by their nature, depend heavily on the skills of the users, yet novices
bring fewer skills to the task. Furthermore, any technological intervention
targeted specifically at novices must not only be effective but must also
increase efficiency enough to compensate for the added cognitive burden of

its own use. (p. 2)

Results from our study did not support the contention that the tool is only as good as its
user. Certainly it is possible that persons with good generalized spatial and verbal ability
can navigate through intricate computer interfaces more swiftly than those who must
puzzle over the manipulations. Furthermore, it is probable that more adept writers have
less of a learning curve in mastering a writing tool because they bring significant domain
knowledge to the task and are better able to infer patterns of usage for the tool. All of
these observations do not take into account the fact that integrating both principled
instructional design and an active pedagogy into the tool creates an interactive device that
guides the student through difficulties and grows with the learner's increased abilities.

6.0 Conclusions

The current study shows persons in the treatment group of lower writing abilities
achieving greater gains than more adept writers. Several explanations can be suggested
for this result. First, R-WISE 1.0 has an effective instructional design founded in theory.
It is transparent to the point that fledgling writers are not bogged down or required to
divert cognitive energies learning to manipulate the tool itself. Second, R-WISE 1.0
worked as intended and provided the scaffolding that enabled weaker students to practice
- in a meaningful fashion -- the higher-order mental activities of good thinking. Third,
motivation probably played a role in all the gains, but is more of a factor for the lower end
where novelty translates into an increased focusing of activities and improved attention
span. We are not prepared to make an argument for any of these three comments based
on the data from the current study. Nevertheless, these issues suggest questions of
interest for the design and direction of future research. .

18


LIST OF FIGURES
Page
Figure 1. Holistic Scores by Condition and Writing Aptitude ...........:-:csssssessseeenterterens 10
Figure 2. Abstraction Scores by Condition and Writing Aptitude ...........ccccceeenees 12
13

Figure 3. Elaboration Scores by Condition and Writing Aptitude .............-.-:sceeeee

Figure 4. Focus Scores by Condition and Writing Aptitude

Figure 5. Sequence Scores by Condition and Writing Aptitude ............--s:-sseeeeeesee 14
LIST OF TABLES

Page

wl

Table 1. Descriptive Statistics for Holistic and Analytical Assessments .........-.:+-1e+++


5.0 Discussion

5.1 R-WISE and Writing Performance
Holistic score trends (i.e., the significant condition X repeated measure interaction)

suggest that not only does R-WISE 1.0 help students become better writers, but use of R-
WISE 1.0 effectively supplements and extends classroom instruction in writing.

Our software follows the concept of a microworld (Papert, 1980). In other words, we
conflated powerful mental operations foundational to writing into a set of manipulations
and placed the visualizations in a constrained context so as to model the elaboration, state
transitions, and configurations of the knowledge structures being worked within the
problem space (Newell & Simon, 1972). Learners interact with the computerized
environment in rich but highly defined ways. The microworld mediates learning by making
choices explicit, by helping to manage the cognitive load, and by encouraging reflection
during the writing process.

R-WISE 1.0 contains a visual nomenclature for writing whose components become
synoptic overviews that trigger sophisticated cognitive actions such as formulating
inferences about relationships, evoking strategies to facilitate reasoning, and prompting
metacognition, or self-regulation for deploying, adapting, or abandoning sets and subsets
of mental manipulations based on awareness of the situation. Rather than trivializing the
task, the three tools in R-WISE 1.0 mediate four significant areas for improving
instruction in writing.

Fluid Task Representation and Adaptive Planning. Teaching novice writers to
construct graphical representations for the components of a writing task helps
them to gain intellectual control over a process that would otherwise remain both

obscure and opaque.

© Mediating between Idea Generation and Text Production. In a composing task,
ideas exist in two very different forms -- as fluid thoughts and as concrete text.
Teaching students to use mediating prompts during the drafting stage increases
interaction between content and rhetoric subgoals (on a cognitive level) and
encourages the kind of inferencing, interpretation, and logical connections among
ideas that result in higher quality student writing .

¢ Schema Building for Task-Types to Facilitate Text Production. Expert writers
have powerful strategies for gaining intellectual control over a body of seemingly
discrete data. Unfortunately, many students are never taught to develop such
schema-driven strategies; they are never encouraged to extract patterns for writing
tasks from one assignment to the next. Having no schema forces the student to re-
invent the writing process with each new tasking. Because of this deficiency,
novice student writers cannot make efficient use of their time; they frequently
construct patterns based on misconceptions; and they are, understandably,
frustrated by the rigors of the writing process.

15


Miller, K. F., & Stigler, J. W. (1991). Meanings of skill: Effects of abacus expertise on
number representation. Cognition and Instruction, 8(1), 29-67.

Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs, NJ:
Prentice-Hall.

Odell, L. (1977). Measuring change in intellectual processes as one dimension of growth
in writing. In C. Cooper, & L. Odell (Eds.), Evaluating writing: Describing,
measuring, judging (pp. 107-132). Urbana, IL: National Council of Teachers of
English. i

Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York:
Basic Books.

Pea, R. D. (1985). Beyond amplification: Using the computer to reorganize mental
functioning. Educational Psychologist, 20(4), 167-182.

Perkins, D. N. (1985, August). The Fingertip effect: How information-processing
technology shapes thinking. Educational Researcher, \1-17.

Pike, K. L. (1964). A linguistic contribution to composition: A hypothesis. College

|
Perkins, D. N. (1986). Thinking frames. Educational Leadership, 43, 4-10. |
Composition and Communication, 15, 82-88.

Rose, M. (1984). Writer's block: The cognitive dimension. Carbondale, IL: Southern
Illinois University Press.

Ryba, K., & Anderson, B. (1990). Learning with computers: Effective teaching
strategies. Eugene, OR: International Society for Technology in Education.

Salomon, G. (1988). AI in reverse: Computer tools that turn cognitive. Journal of
Educational Computing Research, 4(2), 123-139.

Salomon, G. (1993). On the nature of the pedagogical tool: The case of the writing
partner. In S.P. Lajoie, & S. J. Derry (Eds.), Computers as cognitive tools (pp. 179-
196). Hillsdale, NJ: Lawrence Erlbaum Associates. :

Salomon, G, Globerson, T., & Guterman, E. (1989). The computer as a zone of proximal
development: Internalizing reading-related metacognition from a reading partner.
Journal of Educational Psychology, 81 (4), 620-627.

Salomon, G., Perkins, D.N., & Globerson, T. (1991). Partners in cognition: Extending
human intelligence with intelligent technologies. Educational Researcher, 20(3), 2-9.

21



linguistical markers as evidence that certain mental operations have taken place (Pike,
1964). Odell (1977) extends the idea that specific linguistic features are evidence that
corollary types of mental operations occurred. More recently, Tamor and Bond (1983)
have developed a text-analysis system that links “. . . task, cognitive system, and
performance” (p. 113). In formulating their method, these researchers argue that
explanations of text production (i.e., the cognitive dimensions) must be drawn from an
examination of the text itself because -- after all -- this artifact is the item of interest in the

measurement.

Our study dealt with 1,704 writing samples. Sheer volume precluded our adopting a fine-
grained scrutiny of text features -- as any of the above methods would require. Therefore,
we devised our second scoring instrument to directly assess text features as indicators of
cognitive development. This second rubric draws from the methods of primary trait
analysis (White, 1985): it has the specificity and narrowed focus of analytical scoring but
the convenience of guided-holistic scoring. The instrument takes into account how well
the writer understood the rhetorical situation and then deployed appropriate cognitive
operations to develop a prose performance to meet these needs. Like all primary trait
scoring guides, this rubric defines performance capacity as a combination of cognitive and
affective maturity. We believe our rubric has merit and could be offered as an instrument
for future research in which process is inferred from product.

5.3 R-WISE and Initial Writing Aptitude

‘When the present sample was partitioned by initial achievement in writing, a reliable and
significant aptitude-treatment interaction emerged across the scoring criteria. Low
aptitude students using R-WISE 1.0 showed, by far, the most impressive improvements in
writing performance. Gains for the upper segment of the treatment group were in the
order of 9-11% while for the lower segment, the same gains were between 27-34%.
Similarly, gains made by the upper segment of the control group were 1-3% while gains
for the lower segment were 6-16%. Improvements made by the lower segment of the
treatment group on all four analytical scales were at least double those made by both the
upper segment of the treatment group and the lower segment of the control group.

In short, it appeared that students on the lower end profited more from the computer
intervention than did student on the upper end. Nevertheless, in all cases but one, both
segments of the treatment group outperformed both segments of the control group (on all
five measures) by a factor of three. Thus, R-WISE 1.0 helped most ninth-grade writers in
the sample to improve, but it was especially effective for those needing the most
improvement.

With any form of intervention, persons of different abilities are differentially impacted: it is
axiomatic that not all students benefit equally well from any form of treatment. Aptitude-
treatment interaction (ATI) research has produced some solid results and useful insights,
and will continue to grow in importance as computer-mediated instruction improves in its
ability to adapt to individual learning needs and preferences (Kyllonen & Shute, 1989).

17


Repeated measures ANOVA was used to test the relationships between experimental
variables. Based on the research hypotheses, the condition (i.e., treatment, control) X
repeated measure (i.€., pre- and post scores), the condition X writing aptitude (i.e.
passing, non-passing), and the condition X writing aptitude X repeated measure
interactions were of experimental interest. For holistic scores, two of the three
interactions were significant. The condition X repeated measure [F(1, 848) = 80.15, p<
.001] and the writing aptitude X repeated measure [F(1,848) = 191.58, p< .001]
interactions were both significant. The 3-way condition X writing aptitude X repeated
measure interaction was not significant however [F(1,848) = 2.65, p = .104]. The
condition X repeated measure interaction explained about 9% of the observed variance
(17 = .086) while the writing aptitude X repeated measure interaction accounted for

approximately 18% of the score variance (r= .184).

4.2. Analytic Scores
Cognitive growth, as measured by the analytical scales, was less enigmatic. Table 1 (pg

11) shows descriptive statistics for the analytical assessments.

Abstraction. Subjects using R-WISE 1.0 gained almost 21% on this measure compared
to the 6% improvement made by control subjects over the same pre- to post-test period.
Percent gains for the passing-treatment, passing-control, non-passing treatment, and non-
passing control sub-groups were: 11%, 3%, 34%, and 16% respectively. As suggested by
these data and as shown in Figure 2, abstraction scores displayed interactions similar to
those observed for holistic scores. For abstraction scores, however, the 3-way condition
X writing aptitude X repeated measure interaction was significant [F(1, 848) = 6.12, p=
.014] although it accounted for only a small proportion of the observed variance (7? =
.007). As before, both of the 2-way interactions were significant with the writing
aptitude X repeated measure interaction accounting for more of the score variance (of? =

.094 vs .046).
—O— Trt (passing)
| Cotrl (passing)
WA |—a— Trmnt (non-passing)
7 |—@—Catrl (non-passing)

1.25 +-———.—
Pre Post

2.75 7

N
i
a

yy
a

Mean Abstraction Score

Figure 2. Abstraction Scores by Condition and Initial Writing Aptitude

12


processes, and constraints. What is needed, according to these researchers, is a set
of visual workspaces that help the novice writer to gracefully move through the
various state-transitions inherent in moving from thoughts to finished text.

e Provide embedded strategic models for higher-order cognitive activities (such as
discerning patterns in bodies of information, decision-making, staged problem
solving, analysis, synthesis, and inferencing).

Pea (1985), Perkins (1985), and Salomon (1988, 1993; Salomon, Perkins, & Globerson,
1991) make a compelling case that some types of computer applications not only facilitate
a task's accomplishment but also help the user to internalize profound strategies for later
performance of the same or similar tasks, without the presence of the technology.

Pea (1985) makes a distinction between amplification and reorganization in examining
the effects of cognitive technologies. Defining cognitive technology as "any medium that
helps transcend the limitations of the mind, such as memory, in activities of thinking,
learning, and problem solving" (p. 168), Pea starts with an historical perspective and
reviews the effects of literacy on cultures. Certain highly-designed artifacts of literacy
have the power -- over a broad expanse of time -- to change human patterns of thinking.
Goody (1977) and Eisenstein (1979) show that evolving textual conventions and the wide-
spread availability of printed materials facilitated sophisticated mental operations, such as
comparison and contrast.

From a less sweeping perspective, however, not all cognitive amplifiers have a lasting
effect on the user. Citing Cole and Griffin (1980), Pea notes that a child can extend her
short-term memory with paper and pencil by writing down a long list of words. The
short-term memory is amplified in this single instance, but the child's mental capacity has
not been improved or altered -- unless the paper and pencil somehow prompt the child to
chunk the words in more easily processed clusters. In short, cognitive tools that have
been carefully designed to move beyond mere conveniences will teach strategies for
mental activities and -- rather than deskilling -- leave their users better off for having

engaged the tool.

Two specific examples make these claims for the efficacy of the cognitive tool more
definable. Pea (1985) uses the electronic spreadsheet as". . . an illustration of computer
technologies that can reorganize, and not merely amplify, mental functioning . . ." (p. 170).
Therefore, examining the cognitive design features of an electronic spreadsheet provides
us with a point of departure. First, the electronic representation recaptures all the features
and functionality of the paper ledger. A two-dimensional array displays categories and
attributes, creating individual cells for the placement of data. Even this static ordering
enables various forms of complex intellectual activity -- such as inferencing and
categorical reasoning. Placing this representation in electronic form adds new dimensions:
now the user can dynamically manipulate data in each cell and watch the impact of change
on other elements of the system. This, contends Pea, changes. the level of engagement
between the information and the user. Instead of merely entering data, the user is


Beyond Word Processing: Using an Interactive
Learning Environment to Teach Writing

1.0 Process Models of Writing and Cognitive Development

The past three decades of inquiry into the writing process have yielded a greater
understanding of prose composition as a cognitive act. Hayes and Flower pooled their
talents as cognitive psychologist and rhetorician (respectively) to author a series of
groundbreaking articles that present a flowchart model of composing based on hierarchical
decision-making, deconstruction of the tasks, and feedback loops that monitor and
regulate the interactions among (1) task environment, (2) long-term memory, and (3)
goals and plans (Flower & Hayes, 1980, 1981; Hayes & Flower, 1983). Other
researchers have also made significant contributions to a descriptive model of the
cognition involved in writing. Bereiter and Scardamalia (1987) draw upon years of
empirical research to extract the notion of intentional instruction, or explicitly teaching
writing as a process of setting goals and achieving those goals through schema-driven
strategies and metacognitive awareness. Collins and Gentner (1980) champion a fully-
specified theory of composing so that the model can be extended into computer
technology, particularly artificial intelligence. Rose (1984) as well as Beaugrande (1984),
on the other hand, argue for a less hierarchical, more situational model similar to the
notions of contingency management and the idea of opportunism presented by the Hayes-
Roth's (1979) investigations into the nature of planning.

Although there is no unified field-theory of composing and no single model has gained
enough acceptance to displace others, any process description of writing takes as a given
that good writing is the product of good thinking and that good thinking consists of
powerful, insightful, and systematic forms of intellectual activities. While dismissed by
some traditional rhetoricians (e.g., Steinmann, 1982) and minimized by proponents of the
social interaction view of composing (e.g., LeFevre, 1987), instruction that emphasizes
cognitive development as the foundation for writing gained support in the 1980s
(D'Angelo, 1980; Flower, 1981; Hillocks, 1982). Though differing in some of their
features, these pedagogies essentially wed the forms of inquiry to the formtis of discourse --
or the methods of formulating thoughts with requirements for textual presentation of
ideas. Recognizing both the cognitive and affective sides of learning, these approaches
foster analytical reasoning as well as creative reconstruction of information and are quite
complex in the fullness of their concerns. For example, cognitive models of composition
instruction encourage interactive rather than passive learning by guiding the student
through higher-order considerations about writing, such as allocation of attention,
inferential reconstruction of previously known facts and observations, inductive and
deductive reasoning, interpretive elaboration, and the like.

For this study, we draw from the research of Bereiter and Scardamalia (1987) to give a
cogent description both of the process and of the resultant product of novice and expert
writers. Inexperienced writers practice a form of writing that Bereiter and Scardamalia



PREFACE

R-WISE ( Reading and Writing in a Supportive Environment) is part of a 7-year Air Force
effort -- the Fundamental Skills Training project -- to design, build, evaluate, and
transition advanced computer-aided instruction to the education community. This
research was done while Patricia Carlson was an associate at the Armstrong Laboratory,
Human Resources Directorate, Intelligent Training Branch, Brooks AFB, TX, on leave
from Rose-Hulman Institute of Technology. This article does not necessarily reflect the
opinions or policies of the U.S. Air Force or any other government agency.

This report describes R-WISE 1.0 and presents the results of an initial evaluation of the
software. Many people contributed to the development of R-WISE. The authors express
their gratitude to the high school teachers who served as subject matter experts, the
programmers who developed the software; and the research assistants who tabulated the
data for the pilot study. We especially acknowledge the generous sharing of time and
talent and continued support of the following individuals: Dr. Wes Regian, Senior
Scientist for the Intelligent Training Branch (AL/HRTI); LtCol Jim Parlett, AL/HRTI
Branch Chief; Dr. Kurt Steuck, the Fundamental Skills Training Project Manager, and Ms
Teri Jackson, who oversaw the implementation of R-WISE at 10 sites in five different

states.



Elaboration. Treatment students showed a 22% pre- to post-test increase on elaboration
scores while the control group realized a 4% improvement over the same period. Percent
gains for passing-treatment, passing-control, non-passing treatment, and non-passing
control sub-groups were 13%, 1%, 34%, and 11% respectively. All 3 of the relevant
interactions were significant. The 3-way condition X writing aptitude X repeated measure
interaction shown in Figure 3, [F(1,848) =11.33, p = .001] accounted for about 1% of the
variance. This time the condition X repeated measure interaction was more predictive
than the writing aptitude X repeated measure interaction (1? = .104 vs .081).

2.75 7

|—@— Trmnt (passing)
|—W— Cntr! (passing)
|—dk— Tent (non-passing)
|—@— Cntr! (non-passing)

ya
rv

~
WN

1.25 4
Pre Post.

Figure 3. Elaboration Scores by Condition and Initial Writing Aptitude

Focus. R-WISE 1.0 subjects gained around 19% on focus scores. Control students
showed a 5% improvement over the pre- to post-test period. Percent gains for passing-
treatment, passing-control, non-passing treatment, and non-passing control sub-groups
were: 11%, 3%, 29%, and 16% respectively. The 3-way condition X writing aptitude X
repeated measure interaction shown in Figure 4 was not significant [F(1, 848) = .94, p=
measure [F(1, 848) = 46.09, p< .001] and the writing

.332]. The condition X repeated
aptitude X repeated measure [F(1, 848)] = .94.28, p < .001] interactions were significant
however. For focus scores, the writing aptitude X repeated measure effect was stronger

than the condition X repeated measure effect (1? = .100 vs .052).

13



on the paragraph level by indicating relationships or connections among the individual
sentences making up the discourse unit. Common patterns are problem-solution, cause-
effect, or chronological order. Sequence can also be signaled by transitional devices that
integrate sentences in such a way as to orient the reader and indicate a movement of
meaning in a systematic way.

2.0 Mediating Cognitive Development through External Representation

Extending psychological models of writing into effective classroom enactments has proven
difficult. Though models help to identify the components of writing, the process is not
linear and teaching is not as simple as providing instruction for each component and then
putting the parts together. At the heart of the matter seems to be a kind of paradox.
Accomplished writing requires considerable finesse in higher-order intellectual activities:
critical thinking as a precursor to knowledge; judgment as the foundation for decision
making; and self-confidence as the initiator of self-monitoring. Unfortunately, novice
writers are so overwhelmed by the heavy demands of the process that they seldom are able
to exhibit -- even on a rudimentary level -- these higher-order activities. Thus, they are
precluded from working with the very same intellectual activities that would make them
better writers.

2.1 Computers and Intelligence Amplification
‘What is needed to help weak writers handle the cognitive overload is an instructional

system that serves as a procedural facilitator. This term is used by Vygotsky (1978) to
explain the cognitive mentoring and developmental dynamics that occur between master
and apprentice and between peers during collaboration. Salomon (1988) and
Zellermayer, Salomon, Globerson & Givon (1991) use the term to indicate that computer
technology can serve as a peripheral brain for the fledgling student and provide the
scaffolding that allows the novice to practice the more robust problem-solving behaviors
of an expert. In brief, such a procedural facilitator for composition would:

e Ease demands on short-term memory and help to focus attention on strategically
important aspects of writing.

© Guide the inculcation and self-initiation of higher-order processes (metacognition)
which the novice writer is unlikely to activate without prompting.

© Model explicit, strategic intellectual processes so that the fledgling student avoids
what Collins & Gentner (1980) have termed downsliding, or becoming
increasingly entangled in lower and lower levels of mental actions, finally
concentrating all attention on such things as spelling, grammar, and sentence
construction to the exclusion of larger concerns in the process.

¢ Mediate transitions from abstract thoughts to symbolic representations. Smith &
Lansman (1989) conceptualize composition as an activity that takes place in three
modes: thinking, organizing, and adapting. Each mode has its own set of goals,



References

Beaugrande, R. de (1984). Text production: Toward a science of composition.
Norwood, NJ: Ablex.

Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition.
Hillsdale, NJ: Lawrence Erlbaum Associates.

Bloom, B.S. (Ed.). (1956). Taxonomy of educational objectives, Handbook I: Cognitive
domain. New York: David McKay.

Christensen, F. (1967). Notes toward a new rhetoric. New York: Harper & Row.

Cole, M., & Griffin, P. (1980). Cultural amplifiers reconsidered. In D. R. Olson (Ed.),
The social foundations of language and thought: Essays in honor of Jerome S.
Bruner (pp. 343-364). New York: Norton.

Collins, A.M., & Gentner, D. (1980). A framework for a cognitive theory of writing. In
L. W. Gregg & E. Steinberg (Eds.), Cognitive processes in writing (pp. 51-72).
Hillsdale, NJ: Lawrence Erlbaum Associates.

D'Angelo, F. J. (1980). Process and thought in composition (2nd ed.). Cambridge, MA:
Winthrop.

Eisenstein, E.L. (1979). The printing press as an agent of social change. New York, NY:
Cambridge University Press.

Flower, L. (1981). Problem-solving strategies for writing. New York: Harcourt Brace
Jovanovich.

Flower, L., & Hayes, J. R. (1980). The dynamics of composing: Making plans and
juggling constraints. In L. Gregg, & E. Steinberg (Eds.), Cognitive processes in
writing (pp. 31-50). Hillsdale, NJ: Lawrence Erlbaum Associates. _ *

Flower, L., & Hayes, J.R. (1981). Plans that guide the composing process. In C.H.
Frederiksen, & J. F. Dominic (Eds.), Writing: The nature, development, and
teaching of written communication: Volume 2, Writing: Process, development and
communication (pp. 109-123). Hillsdale, NJ: Lawrence Erlbaum Associates.

Gentner, D., & Gentner, D. (1983). Flowing waters or teeming crowds: Mental models of
electricity. In D. Gentner, & A. L. Stevens (Eds.), Mental models (pp. 99-129).
Hillsdale, NJ: Lawrence Erlbaum Associates.

Gentner, D., & Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Lawrence Erlbaum
Associates.

19



ALJHR-TR-1996-0090

‘

BEYOND WORD PROCESSING: USING AN
INTERACTIVE LEARNING ENVIRONMENT
TO TEACH WRITING

Patricia A. Carison

HUMAN RESOURCES DIRECTORATE
TECHNICAL TRAINING RESEARCH DIVISION
7909 Lindbergh Drive
Brooks AFB, TX 78235-5352

19961223 009

Todd M. Miller

Mei Technology Corp.
8930 Fourwinds Drive #450
San Antonio, TX 78239

Dre Warrry

TUBES a

September 1996

Interim Technical Report for Period January 1995 -July 1995

Approved for public release; distribution is unlimited.

<DOAYPAOWFr AZODAVSAY

AIR FORCE MATERIEL COMMAND
BROOKS AIR FORCE BASE, TEXAS nese



conserving and allocating mental energies, and orchestrating steps in staged problem-
solving techniques. In the realm of problem representation, John Hayes (1981) notes the
utility of visualizations (such as matrices) to delimit the problem space and to facilitate
search in a complex situation. As an example, Jones, Amiran, & Katims (1985) found that
using a grid to encourage name-and-attribute clustering aided in recall and systematically
produced effective compare-and-contrast type essays ina study of young adults. The rich
body of research into mental models and instructional design (Gentner & Stevens, 1983;
Kieras, 1988) suggests that highly-complex, multi-dimensional intellectual activities can be
represented so as to help the novice activate and amplify specific expert strategies. For
example, Gentner & Gentner (1983) found that the types of thinking people could do
about electrical circuits depended on what kind of analogy (teeming crowds or flowing
water) was used to represent the system and -- consequently -- the inferences these

metaphors enabled.

Our software -- Reading and Writing in a Supportive Environment (R-WISE 1.0) --
provides scaffolding and visual algorithms that gently guide the writer through multi-
staged intellectual activities. The software encourages students to practice powerful
strategies for composing in a computer-mediated environment that fosters guided-
inductive learning and ensures mindful engagement in the task. R-WISE 1.0 enables the
internalization and self-initiation of higher-order processes and metacognition, which the
novice writer is unlikely to activate without external prompting. The entire R-WISE 1.0
learning environment is made up of three sets of adaptive workspaces, each mediating the
entire writing process but focusing on the strategies and self-regulative awareness
characteristic of one of three domains (1) finding ideas, (2) transitioning ideas into prose,
and (3) refining both ideas and text into final documents.

Cubing ~The Invention Tool: Loosely based on Bloom's taxonomy (1956), this
workspace uses a graphical representation (a six-sided cube) to depict complex mental
operations. (The idea of a cube as a mnemonic for power perspectives on a subject show
up in many traditional classroom instructional techniques.) This guided brainstorming tool
helps student writers to generate and focus ideas while they work through a systematic
exploration of their subject. Students generate a series of notecards for each selected
perspective (side of the cube): describe, compare, associate, apply, analyze, and argue.
This process modeler helps to bootstrap comprehension by reifying a process for (1)
extracting meaning, (2) formulating inferences, interpretations, and reinterpretations, and
(3) engaging passive knowledge and connecting it with the current situation.

Idea Board —The Drafting Tool: This workspace mediates a major cognitive shift in the
writing process -- moving from the macro structures of thought to the micro structures of
socially accepted, connected prose. Using a sequence of integrated thinking frames, this
workspace breaks the process into four complex yet structured states: (1) planning, (2)
visual outlining, (3) constructing paragraphs, and (4) structuring the entire paper. By
foregrounding activities at appropriate times and relegating others to less prominence, the
visual presentations teach student to manage the cognitive load of drafting.


empowered to perform financial modeling, forecasting, and other forms of systemic and
predictive thinking. Recent empirical studies support Pea's claims that using the tool
elevates the user's understanding of the system in ways that both endure and generalize,
even without the presence of the tool (Guster & Batt, 1989; Ryba & Anderson, 1990,

Schiffman, 1986).

Our second example, the abacus, demonstrates how a non-western, non-computerized
device can mediate cognitive reorganization. Stigler, Chalip, & Miller (1986) review a
series of four studies showing that ". . . abacus training result[s] in qualitative changes in
children's representations of mental calculations through development of a 'mental abacus.'
... Abacus training was found to affect both calculation skills and conceptual knowledge
of the numeration system" (p. 447). In other words, these studies suggest that achieving a
high level of skill (both speed and accuracy) in performing arithmetic calculations with an
abacus results in enhanced cognitive abilities that can be attributed to practice with the
device rather than factors such as socioeconomic status, ability, and previous mathematical
knowledge.

Miller and Stigler (1991) studied abacus use as a example of research questions inherent in
a whole category of representational systems. The abacus has utility in that one
manipulates the device to complete specific, well-defined mathematical calculations.
However, in mastering the tool, one also gains insights into the conceptual dimensions of
the domain. In other words, students who had reached a level of proficiency with the
abacus are also better able to answer sophisticated questions about number theory than a
set of matched controls who had no exposure to the abacus. Within a wider context, the
abacus as a reorganizer of cognition could lead to questions of minimalism: How small
and simple can a device be and still leave a cognitive residue by reorganizing mental
abilities? Abacus studies show that even within non-computer examples, cases exist of a
simple but eloquent teaching device not only having a pragmatic result but also a
conceptual effect.

2.2 The Cognitive Architecture of R-WISE 1.0
Perkins (1986) extracts the notion of the thinking frame from the more abstract concept

of schema theory. Perkins offers the following definition and explanation:

[A] representation intended to guide the process of thought, supporting,
organizing, and catalyzing that process. This representation may be verbal,
imagistic, even kinesthetic. When well-practiced, it need not be conscious. A
thinking frame, in order to organize our thinking, includes information not only
about how to proceed but when to proceed in that way (p. 7, italics in the

original).
In practice, thinking frames occur in a number of different domains. Their form spans a

gamut from simple (but powerful) mnemonic devices for extending the working memory
to rich mental models that foster expert behaviors by invoking appropriate strategies,



call knowledge telling (1987). Their inadequacy is not necessarily a result of poor
spelling, vocabulary, grammar, verbal fluency, syntax, paragraphing, or other such
production skills. Knowledge telling is characterized by (1) a simple task execution
involving only limited planning and mental engagement, (2) production methods adapted
from oral abilities, (3) organization patterns based on almost free-association or simple
narration, (4) development that contains large chunks of irrelevant information or
elaborations based on simple descriptions, and (5) an egocentric perspective. The
antithesis, knowledge transformation, as practiced by good writers, is characterized by (1)
guided planning and situational diagnostics, (2) rich mental representations of text
possibilities for a wide range of scenarios, and (3) a robust executive control program or
metacognitive self-prompting for allocating mental resources and for handling the
tremendous cognitive load of verbal composition. From Bereiter and Scardamalia's
(1987) descriptions both of the process and of the products for novice and expert, we
have extracted four features of writing -- each of which represents a cluster of intellectual
activities that manifest themselves as qualities of good writing. Although general, these
features are sufficiently explicit to be testable.

Abstraction -- Experienced writers cluster specifics within a propositional frame-of-
reference based on similarities, differences, likely outcomes, or proximity in time and
space. Much of writing requires the writer to move from part to whole, or to draw from a
body of evidence some inference or enumerative generalization about the body of
observations. Though related to inductive reasoning, these intellectual operations might

also be described as classificatory reasoning, synthesis, or inferencing.

Elaboration -- While experienced writers may have a variety of strategies for discovering
appropriate content for the paper, they also have filtering mechanisms for selecting only
those ideas appropriate for the audience and the purpose of the paper. A less-
accomplished writer may lack the rhetorical awareness necessary for pruning and placing
ideas within the paper. These writers practice a kind of memory dump when generating
discourse. While they may include a number of separate ideas in their paper, the text has a

superficial, canned, or random quality to it.

Focus -- Advanced writers are adept at establishing a purpose while at the same time
providing a rich texture of detail. They do this by appropriately foregrounding ideas of
interest while backgrounding contextual elements. Effective writing is more than correct
lexical and syntactic manipulations. Connected discourse produces in the reader a sense of
centeredness. This sense of aboutness is derived from the writer's ability to integrate
concepts in the text. Focus is a product of intentionality (a goal or motive underlying the
discourse) and of sustained point-of-view (a consistent perspective on a topic).

Sequence -- Expert writers formulate relationships of ideas in their minds and then convey
these connections over large blocks of text through devices for coherence and cohesion

that may be as simple as using transition words or as complex as interweaving a given in j
one sentence with the new in a subsequent sentence. This degree of logical organization is |
one feature that separates written text from spoken discourse. Organization is established



Situational Awareness and Metacognitive Guidance. Writing is a complex, multi-
dimensional activity analogous to a contingency management problem. Good
writers use an opportunistic approach; they constantly measure the emerging text

against a set of expectations, while at the same time recognizing and capitalizing
on serendipitous gains, weaving these discovered possibilities into a new rendition
of the overall plan and product. Helping students to assess their audience and to
understand their purpose makes it easier to calibrate among external demands,
internal resources, and desired outcomes.

5.2_R-WISE and Cognitive Growth
Introducing R-WISE 1.0 into ninth-grade English classes produced significant gains over

and above traditional instruction on all four of the analytic measurements. If these
measurements can be taken as indicators of cognitive development, then R-WISE 1.0
appears to facilitate cognitive growth above and beyond traditional methods of writing

instruction.

In this study we inferred cognitive development from an examination of textual artifacts.
The more common means of process research for composing are ethnographic in nature:
protocol analysis, personal observation and notation, videotaping, and structured
interviews, to name the more prevalent. All involve direct observation for identifying
covert thinking activities that manifest themselves through specific, observable behaviors.
Such methods are not without their problems. Because they are labor-intensive, sample
sizes are kept small. Additionally, the procedure for scoring or interpreting episodes in the
transcripts of sessions may be problematic -- as an example, how do we know for certain
that a pause indicates that the writer is reflecting? And finally, the standard criticism of
direct observation is that the method is intrusive and alters the very phenomena it seeks to

objectively record.

On the other hand, inferring cognitive growth from text samples has a fairly long history in
the study of writing. Hunt's (1977) use of T-units to assess sentence complexity is based
on the notion that syntactic sophistication and cognitive growth are one and the same.
Other researchers question T-units as a measure of research and point out the limitations
of quantitative enumeration of surface features without concern for larger issues such as
contextuality and meaning (Williams, 1979). Francis Christensen's generative approach to
paragraph structure has close parallels to whole-text discourse analysis in that it accounts
for units of meaning larger than a single sentence. Through a system of layering
representations, Christensen (1967) devises a notation for measuring hierarchical
reasoning and the logical connections through which ideas emerge and are merged into

connected prose.

Other researchers also make a case for text analysis that link forms of discourse with
forms of inquiry (or patterns of writing with patterns of thinking). Kenneth Pike (who, in
collaboration with colleagues Richard Young and Alton Becker, advocated both a theory
and a pedagogy of writing as a problem-solving process) makes a case for specific

16


Schiffman, S.S. (1986). Productivity tools for the classroom. Computing Teacher, 13(8),
27-31.

Smith, J. B., & Lansman, M.C. (1989). A cognitive basis for a computer writing
environment. In B. K. Britton, & S. M. Glynn (Eds.), Computer writing
environments: Theory, research, and design (pp. 17-56). Hillsdale, NJ: Lawrence

Erlbaum Associates.

Steinmann, M. (1982). Speech-act theory and writing. In M. Nystrand (Ed.), What
writers know: The language, process, and structure of written discourse (pp. 154-
172). New York: Academic Press.

Stigler, J. W., Chalip, L., & Miller, KF. (1986, August). Consequences of skill: The case
of abacus training in Taiwan. American Journal of Education, 447-478.

Tamor, L., & Bond, J.T. (1983). Text analysis: Inferring process from product. InP.
Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and
methods (pp. 99-138). New York: Longman.

Vygotsky, L. (1978). Mind in society: The development of higher psychological
processes. Cambridge, MA: Harvard University Press.

White, E. M. (1985). Teaching and assessing writing: Recent advances in
understanding, evaluating, and improving student performance. San Francisco, CA:
Jossey-Bass Publishers.

Williams, J. (1979). Defining complexity. College English, 40(6), 595-609.

Zellermayer, M., Salomon, G., Globerson, T., & Givon, H. (1991). Enhancing writing-
related metacognitions through a computerized writing partner. American Educational

Research Journal, 28(2), 372-391.

22


TABLE OF CONTENTS

1.0 Process Models of Writing and Cognitive Development............-..+-s1-sereneee

2.0 Mediating Cognitive Development through External Representation............-...+------ 3

2.1 Computers and Intelligence Amplification ...

2.2 The Cognitive Architecture of R-WISE 1.0..........csssssssseseessss sess erentsssecens 5

3.0 Methods

3.1 Subjects... eeesseeccsseeessecsneeeeeenneesessesesenensennsnsseccanaetsonneseesesseenasssnnnseeseessets 7

3.2 Instruments seceesseesteeseeeetenesresseeeseeses 8

3.3 Procedure ..csccccscssssssssevessvsvssssssssscedeceesceseeessensseetsssssssnssssseccnnsseeteeseessenscecessee B

4.0 Results .....ccccscccssessessessessessessecssersesecsscsecsnssucenseenacsecsnssecsseasenecancnasansnecascasensesgesgnsteses 9
A.1 Holistic SCOreES.......c.cccsssessecesseceesesesseeeeeeseeeeneresscerenestsnserescasanensasensecececssensntss 9

4.2 Analytical Scores

5.0 Discussion sesuesssssessecsessesesneesnesserseseessseee 15

5.1 R-WISE and Writing Performance
5.2 R-WISE and Cognitive Growth............::cs-sssessscsenseeeeseseneseseeneesatenenncnntanenes 16

5.3 R-WISE and Initial Writing Aptitude . vessstescesneteegaeeecesecessneeeene LT

6.0 Conclusions vesssansssnsssesessssecenanissesseesssseessssneee 1B

iii



Experimental Groups. The control group received traditional classroom instruction in
prose composition. As mandated by Texas curriculum guidelines, emphasis was given to
1) responding appropriately to purpose and audience, 2) organizing ideas effectively, 3)
providing development for a central idea, and 4) conforming to standard syntax, usage,
and mechanics.

The treatment group received similar instruction in writing, augmented by sessions on the
R-WISE 1.0 tutor. The software was used in a computer lab, with 30 networked PCs
arranged around the room. Each student had his or her own workstation and access to the
teacher for questions. Trips to the lab substituted for classroom time periods of 55
minutes each, and were interspersed throughout the semester for a total of 8 hours in the
lab-- approximately 14% of the total classroom time. On average, students went into the
lab on rotations equal to once every 9 class days. In practice, more challenging
assignments might have required two contiguous class periods. In any case, time in the
lab was interspersed during the 16-week semester. As much as possible, time on the tutor
was divided equally among the three workspaces. For more complex assignments,
workspaces were conjoined; that is, the product of the drafting workspace might be read
into the revision workspace. For other assignments, students took intermediate products
(e.g., notecards and drafts) back to the classroom to finish the written product.

Data Collection. Subjects were asked to respond to two paper-and-pencil writing
prompts, once at the beginning of the semester and once at its conclusion. Trained
proctors provided students with standardized instructions and were present in the
classroom at all times during testing.

The writing samples were scored by Psychological Corporation's Writing Assessment
Center. Each paper was read at least 10 times: once by two separate raters for a holistic
score and four times by two separate raters for assessment of specific subskills of the
writing. Standard practice in scoring writing samples requires that two readers differ by
no more than 1 point. If they do, then a third expert rater is called in to resolve the
discrepancy (White, 1985). Using this criterion, composite interrater reliability scores for
both pre- and post-tests ranged from .95 to .99. If we take a more rigorous approach and
calculate reliabilities using the two initial ratings, interrater reliabilities drop, but are still
acceptable: holistic = .79, abstraction = .66, elaboration = .63, focus = .58, and
sequence = .64.

4.0 Results

4.1 Holistic Scores

Mean holistic scores, as shown in Table 1 (pg 11), revealed an unexpected trend. While
treatment group scores rose by approximately 9% from pre- to post-test, control group
scores dropped by about 9% over the same period. Several checks were made and
additional analyses performed to identify the source of this anomaly.



“aBueyo o1008 389}-3S0d 0} -o1d st 7 ‘WOHeIASp psepueys pue ueoU ojdutes ore s pue Ww ‘9}0N,

or=V st =V 1T=v 6r=V oF SI
£9'=s 99'=s 19°38 £9'=8 Ivss o1'=s ws tL=8  STT=8 (90 T=8
C9 OPT=UE EO Z=W ZHU Spa PE THU EPH PT T=UES6'T=UE I r'f=ur UD
0s'=V 9'=V 99'=V w9'=V wav
L9'=8 ou'=s 19'=8 L9'=s Loss 69°58 $L'=s pL=S  OTT=S—s ATT =ps
pot PTTL THU GTZ 6ST EG T= pS T= EG I=W Ty E=Ul  L67=W uA,
0d od 0g oid s0d stg 0g Br og od
WOHeZUUSIQ osodang Jusmdojaaoq WOHIEAISGY DHSHOH

Syvoulssossy eorATeuy pue ONSHOH Joy sonsterg SANduosaqd “1 91921,

i


REPORT DOCUMENTATION PAGE Teen otae

Publi reporting burden fr tig collection of information is estimated to average 1 hour pe fesponse, including the time for redwing instruons, searching enstng data sources, gathering
other asf

See eae ee caret at cence cients Sacer uc beaten Spore apes Te San Sate gy, Sie
aoe Arnyton, VA Saat aa0n, and to ‘of Management and Badges Paporwerk Recucton Project (07040168), Washington, DG 20503, *
1. AGENCY USE ONLY (Leave blank) | 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
September 1996 Interim - January 1995-July 1995
4, TITLE AND SUBTITLE 5. FUNDING NUMBERS
Beyond Word Processing: Using an Interactive Learning Environment to Teach C-F33615-91-D-0651
Writing PE - 66205F
PR-1121
6. AUTHOR(S) TA-09
Patricia A. Carlson wu-79
Todd Miller
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
REPORT NUMBER
Mei Technology Corp.
8930 Fourwinds Drive #450
San Antonio, TX 78239
9, SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING
‘AGENCY REPORT NUMBER
Amnstrong Laboratory
Human Resources Directorate
Technical Training Research Division AL/HR-TR-1996-0090
7909 Lindbergh Drive
Brooks Air Force Base, TX 78235-5352
41, SUPPLEMENTARY NOTES
Armstrong Laboratory Technical Monitor: Patricia A. Carlson (210) 536-2034.
12a, DISTRIBUTION/AVAILABILITY STATEMENT 12b, DISTRIBUTION CODE

Approved for public release; distribution is unlimited.

13. ABSTRACT (Maximum 200 words)

This study examines whether computer-aided instruction that explicitly models the process of composing for basic writers
is more effective than traditional classroom instruction. Three objectives guided the research: (1) to determine the basic
quality of essays from the treatment and the control group using standard, holistic rating methods, (2) to infer cognitive
development by measuring improvement along four separate measures using an analytic scale, (3) to determine whether
initial aptitiude was a factor in performance differences. Eight-hundred and fifty-two ninth-grade English students (423 in
the control group and 429 in the treatment group) completed a 40-minute transactional writing sample at the beginning and
at the end of a 16-week semester. The results show that the group using a computerized “cognition facilitator" outperformed
the group taught only in the traditional classroom both on the holistic and on the analytical measures. Additionally, when the

ion was partitioned to reflect initial ability, the treatment group in the lower segment showed marked improvement,
whereas the high-end segment of the treatment group produced no significant gain. When partitioned in the same manner,
the high-end for the control group degraded in performance, while the lower-end control improved both on the holistic and
the analytic measures.

14, SUBJECT TERMS ae 15. NUMBER OF PAGES
Cognitive tools Strategy acquisition in writing 30
Eaitical learning. g environments 16. PRICE CODE
17. SECURITY CLASSIFICATION | 18. SECURITY CLASSIFICATION | 19. SECURITY CLASSIFICATION | 20. LIMITATION OF
OF REPORT OF THIS PAGE OF ABSTRACT ABSTRACT
Unclassified Unclassified Unclassified UL
-01-. ‘Standard Form 208 (Rev 2-00) Prescnbed by ANSI Std 2-30-16

NSN 7540-01-280-5500 298-102 COMPUTER GENERATED



Goody, J. (1977). The domestication of the savage mind. Cambridge, UK: Cambridge
University Press.

Guster, D., & Batt, R. (1989). Cognitive and affective variables and their relationship to
performance in a Lotus 1-2-3 class. Collegiate Microcomputer, 7(2), 151-156.

Hayes, J.R., & Flower, L. (1983). Uncovering cognitive processes in writing: An
introduction to protocol analysis. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds.),
Research on writing: Principles and methods (pp. 206-220). New York: Longman.

Hayes, J. R. (1981). The complete problem solver. Philadelphia, PA: The Franklin
Institute.

Hayes-Roth, B., & Hayes-Roth, F. (1979). A cognitive model of planning. Cognitive
Science, 2, 275-310.

Hillock, G. (1982). Inquiry and the composing process: Theory and research. College
English, 44(7), 659-673.

Hunt, K. W. (1977). Early blooming and late blooming syntactic structures. In C.
Cooper, & L. Odell (Eds.), Evaluating writing: Describing, measuring, judging_(pp.
91-104). Urbana, IL: National Council of Teachers of English.

Jones B. F., Amiran, M. R., & Katims, M. (1985). Teaching cognitive strategies and text
structures within language arts programs. In J. Segal, S. F. Chipman, & R. Glaser
(Eds.), Thinking and learning skills: Relating instruction to research. Vol. 1 (pp.

259-295). Hillsdale, NJ: Lawrence Erlbaum Associates.

Kieras, D. E. (1988). What mental models should be taught: Choosing instructional
content for complex engineered systems. In J. Psotka, L. D. Massey, & S. A. Mutter
(Eds.), Intelligent systems: Lessons learned (pp. 85-118). Hillsdale, NJ: Lawrence

Erlbaum Associates.

Kozma, R. B. (1991). The impact of computer-based tools and embedded prompts on
writing processes and products of novice and advanced college writers. Cognition

and Instruction, 8(1), 1-27.

Kyllonen, P. C., & Shute, V. J. (1989). A taxonomy of learning skills. InP.L.
Ackerman, R. J. Sternberg, & R. Glaser (Eds.), Learning and individual differences

(pp. 117-163). New York: Freeman.

LeFevre, K. B. (1987). Invention as a social act. Carbondale: Southern Illinois
University Press.

20


NOTICES

This report is published as received and has not been edited by the technical editing staff of the
Armstrong Laboratory.

When Government drawings, specifications, or other data are used for any purpose other than
in connection with a definitely Government-related procurement, the United States
Government incurs no responsibility or any obligation whatsoever. The fact that the
Government may have formulated or in any way supplied the said drawings, specifications, or
other data, is not to be regarded by implication, or otherwise in any manner construed, as
licensing the holder, or any other person or corporation; or as conveying any Tights or
permission to manufacture, use, or sell any patented invention that may in any way be related

thereto.

The Office of Public Affairs has reviewed this report, and it is releasable to the National
Technical Information Service, where it will be available to the general public, including

foreign nationals.

This report has been reviewed and is approved for publication.

‘JAMES W. PARLETT, Lt Col, USAF ; BRUCE GOULD

Chief, Intelligent Training Branch Technical Director
Technical Training Research Division

Cems lL. bEkaohmtr .

JAMES B. BUSHMAN, Lt Col, USAF
Chief, Technical Training Research Division

Please notify this office, ALIHRPP, 7909 Lindbergh Drive, Brooks AFB TX 78235-5352, if your
address changes, or if you no longer want to receive our technical reports. You may write or call
the STINFO office at DSN 240-3877 (or 240-2295) or commercial (210) 536-3877 (or 536-2295).


Triple Vision —The Revision Tool: This workspace encourages the student to re-see a
completed draft at three levels of focus: sentence, paragraph, and whole paper. Thinking
frames foster strategic editing such as improving style, adding to or subtracting from the
content, rearranging ideas, or completely rewriting. These more global, deep-structured
editing acts are associated with higher-order intellectual activities (e.g., discerning patterns
in bodies of information, exercising judgment, analysis, and synthesis).

In this study, we were interested in exploring the notion that a computerized process
modeler as powerful and as eloquent as the abacus could be designed, built, and fielded for
the writing process using ninth-grade basic writers as test subjects. Three research
questions guided the study:

© Does a computer-mediated process-modeler improve the writing performance of
ninth-graders when compared to traditional classroom instruction? Based on a
standard six-point holistic scoring rubric for measurement, would the treatment
group demonstrate improved abilities to produce quality writing? In short, have
they become better writers?

© Does a computer-mediated process modeler improve cognitive development, as
measured by four dimensions on an analytical scale when compared to traditional
classroom instruction? Can we infer from textual features of the treatment group's
writing that using the tool has reorganized rather than merely amplified the
treatment group's abilities to perform complex, sophisticated thinking as a
foundation to composing? In short, have they become better thinkers?

e Is the effect of the treatment influenced by initial writing ability? Is the computer-
mediated treatment sufficiently transparent to be accessed by all subjects, or does
the data indicate that measures of effect depend upon pre-existing abilities in such
a way as to suggest that this cognitive facilitator requires a certain threshold of
expertise to be useful? In short, is the tool only as good as its user?

3.0 Methods

3.1 Subjects
The sample consisted of 852 ninth-grade students studying English at two

demographically similar high schools in San Antonio, TX. One school (with 9 teachers,
26 classes, and 429 participating students) was designated the treatment group. The other
school (with 6 teachers, 22 classes, and 423 participating students) was designated the
control group. Neither group contained honors or enriched classes. Nor did the sample
contain classes specifically designated as remedial. However, correlated language arts
students (CLA), who in previous years would have been placed in a special class, were
mainstreamed and accounted for approximately 10% of the sample.



First, test and implementation plans were reviewed to ensure uniformity across
experimental units. We could find nothing in the design or implementation that accounted
for the unexpected drop: all students received the same instructions; the same proctors
were used at both test sites; and tests were given on separate but contiguous days at each
school. Additional investigation ensured that each school covered basically the same
curriculum; that is, they used the same texts and gave about the same number and types of
writing assignments during the semester. Therefore, it is unlikely that either test
administration procedures or instructional equivalency were at fault.

Next, holistic score means were calculated by teacher to determine if one or more teachers
accounted for the drop in scores. Although effect sizes varied from teacher to teacher
within each school, they exhibited distinct patterns across schools. The students of
treatment teachers displayed no change or showed significant, positive pre- to post-test
gains while the students of control teachers showed no change or significant score losses.

Finally, student aptitude differences were examined as a potential cause. Many states have
instituted periodic testing of writing skills as proof of competency. In most cases, the
half-way mark on a holistic scale is taken as the pass/non-pass demarcation. Therefore,
we selected a similar criterion as an indicator of initial writing aptitude in this study. Thus,
for this sample of Texas students, dividing the total sample based on pre-test holistic
scores yields two groups, roughly equivalent to those who would have failed the TAAS
(Texas Assessment of Academic Skills) and those who would have passed.

Figure 1 shows mean pre- and post-test holistic scores for all combinations of condition
(i.e., treatment, control) and writing aptitude (.e., passing, non-passing). Passing
students, regardless of condition group membership, lost ground from pre- to post-test
(2% vs. 14%). Non-passing students, on the other hand, showed pronounced score gains
(23% vs. 6%). In both cases, treatment students significantly outperformed control

students.

44
2 3.54
8
a == Trmat (passing)
B 34 Cuil (passing)
= |—te—Trmat (non-passing)
g —@— Catrl (non-passing)
* 254

24

Pre Post

Figure 1. Holistic Scores by Condition and Initial Writing Aptitude

10