CANADIAN JOURNAL OF EDUCATIONAL COMMUNICATION

Volume 23, Number 3, Winter 1994

ISSN 0710-4340

The Effectiveness of Instructional Orienting Activities in
Computer-Based Instruction

Richard F. Kenny

A Case Study: Development of Interactive Multimedia Courseware

Benolt David

Arto Demirjian

Steven Shaw

Educational Technology Adoption and Implementation:
Learning from Information Systems Research

Karen Lee Jost

Scott L. Schneberger

Authentic Activity as a Model for Appropriate Learning Activity:
Implications for Emerging Instructional Technologies

David G. Lebow

Walter W. Wager

An AMTEC Publication
EDITORIAL BOARD

Farough Abed
Central Connecticut State University

Cheryl Amundsen
McGill University

Gary J. Anglin
University of Kentucky

Jon Baggaley
Concordia University

Robert C. Branch
Syracuse University

Robert M. Bernard
Concordia University

Katy Campbell-Bonar
State University of New York

Bruce Clark
University of Calgary

Dan Coldway
Athabasco University

Thomas M. Dully
Indiana University

Rodney S. Earle
Brigham Young University

D. J. Engel
University of Alberta

George L. Gek
OISE

Edward S. Holpem
AT&T Bell laboratories

Michael Hannafin
Florida State University

Denis Hlynka
University of Manitoba

W. J. Hunter
University Of Calgary

Robert Jones
Sheridan College of Applied
Arts and Technology

Lorne Koroluk
University of British Columbia

James J LaFollette
University of Alberta

Richard F. Lewis
University of Windsor

David A. Mappin
University of Alberta

Earl R. Misanchuk
University of Saskatchewan

Ronald Owston
York University

Beverley A Park
Avalon Consolidated School Ed. of
St John's

Lauran Sandals
University of Calgary

Louise Sauve
Tele-Universite

Richard F. Schmild
Concordia University

R. J Schmidt
Strathcona County Board of
Education

Richard A Schwier
University of Saskatchewan

Steven Shaw
Concordia University

Mariela Tovar
Concordia University

Rolland Viau
Universite de Sherbrooke

Clayton R. Wright
Grant MacEwan Community
College

AMTEC BOARD OF
DIRECTORS

President
Allen LeBlanc
Sault Ste. Marie School Board

Past President
Ross Mutton
Carleton University

President Elect
Gary Karlsen
Sharp's Audiovisual Ltd.

Secretary / Treasurer
Lillian Carefoot
Curriculum Resources Centre

Director
Bob Christie
Christie Communications

Director
Danielle Fortosky
University of Saskatchewan

Membership
Genevieve Gallant
Cabot
Membership Form

AMTEC c/o Genevieve Gallant
3-1750 The Queensway, Suite 1818
Etobicoke, Ontario, Canada M9C 5H5

name _______________ title _______________

address _______________

city _______________ prov/postal code _______________

phone (w) _______________ phone (h) _______________

Fax _______________ Netmail _______________

☐ I am willing to have pertinent information from the above included in the next Members' Handbook

ALL RENEWALS COME DUE ON EITHER JANUARY 1ST OR JUNE 1ST

If this is a renewal then please indicate your Membership Number _______________

☐ Student of Educational Technology (includes GST) $32.10

☐ Retiree (includes GST) $32.10

☐ Individual (includes GST) $80.25

☐ Commercial (includes GST) $80.25

☐ Subscription Only - Canada $42.80

☐ Subscription Only - USA $40.00 U.S.

☐ Subscription Only - Outside North America $55.00 U.S.

*Students must have this form signed by your Department Head: _______________

GST # 122939853

Special Interest Areas:

___ Media Production

___ Media Management

___ Telecommunications

___ School Libraries

___ Distance Education

___ Media Literacy

___ Other _______________

___ Micromputer Applications

___ Instructional Development

___ Mass Communications

___ Educ. Tech. Research

___ Curriculum Implementation

___ Interactive Video

Work Environment

___ Elementary

___ Secondary

___ College

___ University

___ Other _______________

___ Business

___ Government

___ Health Care

Please duplicate this form.

CJEC is typeset on an Apple
Macintosh™ in Aldus PageMaker™.
Galley proofs to authors and final
camera-ready impressions are output on
a Qume CrystalPrint™ Publisher.

Concordia University Printing Services

Acknowledgement

The *Canadian Journal of Educational Communication* is supported by a grant from the Social Sciences and Humanities Research Council of Canada.
# The Effectiveness of Instructional Orienting Activities in Computer-Based Instruction

Richard F. Kenny

**Abstract:** The growing interest in interactive multimedia has created an increased demand for appropriate instructional design strategies. In response, the ROPES+ meta-model (Hannafin & Rieber. 1989) was proposed for the design of instructional prescriptions for computer-based instruction (CBI). The '0' in ROPES refers to 'Orienting' activities, an activity which acts as a mediator through which new information is presented to the learner. Included in this category are attention-gaining techniques, lesson objectives, prequestions and instructional organizers. This paper reviews the research literature on instructional organizers and provides a comparative analysis of their effectiveness with CBI. The review considers what evidence there is that any of these techniques affect learning or retention and examines how they have been used with CBI. Three cognitive theories, Ausubel's (1963) Subsumption Theory, Wittrock's (1974) Generative Learning Hypothesis and Schema Theory (Anderson, 1977; Rumelhart & Orteny, 1977), are compared to discern which most strongly predicts the effectiveness of these techniques.

**Résumé:** L'intérêt grandissant que suscite l'approche multimédia interactive a crée une demande croissante pour l'élaboration de strategies d'éducation adaptees. Pour répondre a cette situation on a propose l'utilisation du meta-mode1 ROPES+ (Hannafin & Rieber. 1989) pour la conception de méthodes pédagogiques dans l'enseignement informatise (Computer-based instruction ou CBI). Le «O» dans ROPES fait référence à l'expression «Orienting» (orientation), une activité qui tient le rôle de médiateur a travers lequel la nouvelle information est présentée à l'apprenant. Cette catégorie comprend des techniques qui «attirent l'attention» des apprenants, des objectifs de leçons, des préquestions et des structures de formation.

Dans cet article, nous examinons la recherche qui a été faite sur les structures de formation, et nous faisons une analyse comparée de l'efficacité de ces structures en matière d'enseignement informatise (CBI). L'étude pose un regard critique sur les indices existants démontrant que ces techniques affectent l'apprentissage ou la rétention et examine comment ces techniques ont été utilisées dans l'enseignement informatise (CBI). Trois théories cognitives ont été comparées : Ausubel's (1963) Subsumption Theory, Wittrock's (1974) Generative Learning Hypothesis et Schema Theory (Anderson, 1977; Rumelhart & Orteny, 1977). Cette comparaison vise a identifier laquelle des trois théories peut le plus efficacement predire le rendement de ces techniques.
The relatively low cost and accessibility of the microcomputer has placed the power and flexibility of computing into the hands of the individual and opened a new range of instructional possibilities. Included among the latter are computer-based instruction (CBI) and its more recent derivative, interactive multimedia instruction. While a precise definition of multimedia has eluded even technology experts (Galbreath, 1992), the term interactive multimedia instruction essentially refers to a computer-controlled system which provides the possibility of varying combinations of digitized audio, graphics and text, analog and digitized video, accessed through the computer itself and/or a variety of peripheral devices such as videodisc players, compact disc players and music synthesizers (cf. Schwier & Misanchuk, 1993; Gayeski, 1993). Defined in this way, interactive multimedia instruction can be seen to encompass other more specific terms, including computer-based interactive video (CBIV) and hypermedia. While the use of CBI, as a general class of technology, has been promoted for its capability to individualize instruction, interactive multimedia instruction extends the power of the computer to support student interaction by adding the richness of the various audio and visual media. To use Jonassen's (1984) description of CBIV, interactive multimedia instruction can be seen both as adaptive and interactive. Jonassen defines adaptive as "the ability to adapt or adjust the presentation sequence, mode or sign type to meet a variety of instructional requirements", a capability requiring the availability of various media, while interactivity is seen as "the program engages the learner to participate in a variety of ways that utilize learner responses." (1984, p.21). Hannafin (1989) further details interactivity, noting that it encompasses various instructional capabilities such as confirmation of response, learner control of pacing and lesson sequencing, inquiry (glossaries and libraries) and elaboration, techniques which allow learners to combine known with to-be-learned lesson information.

Computer-based instruction, and especially interactive multimedia, while providing great flexibility, is not without problems. One is the potential for learner disorientation, the loss of one's sense of location or of the structure of the material. Navigation is the most commonly identified user problem in hypermedia (Jonassen, 1989; Kinzie & Berdel, 1990; Rezabek & Ragan, 1989). Learners can easily become lost and frustrated and may give up without acquiring any information from the program. Another potential, and related, problem with interactive multimedia is cognitive overload. Jonassen (1989) also notes that the exponentially greater number of learning options available to learners places increased cognitive demands upon learners that they are often unable to fulfil. Tripp and Roby (1990) claim that disorientation leads to the expenditure of more mental effort to maintain a sense of orientation in the program which in turn reduces the mental resources available for learning.

A major challenge for teachers and instructional designers is to learn how to make effective use of the capabilities of such interactive learning systems
to assist people to learn while avoiding the inherent problems. To guide research on the design of instructional prescriptions for CBI, the ROPES+ meta-model (Hooper & Hannafin, 1988; Hannafin & Rieber, 1989) was proposed. ROPES+ refers to Retrieval, Orientation, Presentation, Encoding, and Sequencing, “+” the influence of contextual factors. The “0” in ROPES+, then, refers to any form of “Orienting” activity which acts as “a mediator through which new information is presented to the learner” (Hannafin & Hughes, 1986, p. 239). Included are attention-gaining techniques, lesson objectives, pre-questions and advance organizers.

It is this category of orienting activities that may suggest methods for alleviating the problem of disorientation and cognitive overload in CBI. Tripp and Roby (1990), for instance, suggest that it has been the advance organizer which has traditionally been used as a device to orient students to content. However, other related instructional organizers might also be useful. Among these are the structured overview graphic organizer (Barron, 1969) and the pictorial graphic organizer (Hawk, McLeod & Jonassen, 1985), both derivatives of the advance organizer.

The purpose of this paper is to review the research literature pertaining to the use of these three forms of instructional organizer and to provide a comparative analysis of their effectiveness with CBI. The review will first consider what evidence there is that any of these techniques have an effect on learning or retention. Second, it will examine the relevant research on the use of such orienting techniques with CBI. Third, the paper will compare two cognitive theories, Ausubel's (1963) Subsumption Theory and Wittrock's (1974) Generative Learning Hypothesis, to try to discern which most strongly predicts the effectiveness of these techniques. A third theory, Schema Theory (Anderson, 1977; Rumelhart & Ortony, 1977; Jonassen, 1989), and a related instructional orienting technique, the hypermap (Jonassen, 1989; Reynolds & Dansereau, 1990; Reynolds et al., 1991), are suggested as an alternative for further investigation.

## INSTRUCTIONAL ORGANIZERS

The advance organizer, first proposed by David Ausubel (1960, 1963), is meant to facilitate the retention of meaningful verbal information. It is introduced in advance of the learning material itself and presented at a higher level of abstraction, generality and inclusiveness (Ausubel, 1963). Since its main function is to bridge the gap between the learner's cognitive structure and the material-to-be-learned, the advance organizer must be stated in terms familiar to the learner.

The graphic organizer was first advanced as a "structured overview" by Barron (1969) as a modification of the advance organizer and later renamed (e.g. Barron & Stone, 1974). It is a tree diagram which introduces the new vocabulary to be used in the material-to-be-learned and uses the spatial
characteristics of diagrams to indicate the relationships and distances between key terms (Hawk, McLeod and Jonassen, 1985). It is unlike the advance organizer, however, because it is written at the same level as the to-be-learned material and uses lines, arrows and spatial arrangement to depict text structure and relationships among key vocabulary (Alverman, 1981). Hawk, McLeod and Jonassen (1985) further developed Barron's modification. Their form of organizer is a more pictorial, visual, or graphic presentation than the two previous organizers. Pictorial graphic organizers take one of two forms: participatory organizers, in which students participate in the completion of the organizer, and final form organizers, in which they do not.

## RESEARCH ON ADVANCE ORGANIZERS

### Ausubel's Studies

Probably the most-cited research supporting the effectiveness of the technique has, not surprisingly, been provided by the author of the technique, David Ausubel, and his associates (Ausubel, 1960; Ausubel & Fitzgerald, 1961,1962; Ausubel & Youssef, 1963). Ausubel (1960) tested the learning of undergraduates from a 2500 word passage on metallurgy and produced statistically significant results in favour of the expository advance organizer group. Ausubel and Fitzgerald (1961) compared the effects of an expository advance organizer and a comparative organizer on learning from a 2500 word passage on Buddhism. The comparative organizer group significantly outperformed the expository group on a posttest given after three days, but there was no significant difference between the expository and control (descriptive passage) groups. A posttest given after 10 days indicated that both organizer groups retained significantly more of the material to be learned than the control group.

Ausubel and Fitzgerald (1962) also compared the effects of an expository advance organizer on learning from two sequential passages on endocrinology. No significant main effect was shown for either passage. However, a significant main effect was demonstrated for subjects in the lower third subgroup of a test of verbal ability as predicted by advance organizer theory. Finally, Ausubel and Youssef (1963) compared the effects of a comparative advance organizer on learning material from a passage on Zen Buddhism to a control group. They reported a significant main effect for the organizer treatment when both verbal ability and knowledge of Christianity (to which Buddhism was compared in the organizer) were controlled.

A recent detailed analysis of these four studies (McEneany, 1990), however, calls these results into question. McEneany claims no consistent evidence across the four studies in support of advance organizers nor for predicted interactions with verbal ability. He suggests that "a sound operational definition of an advance organizer eludes even Ausubel himself (p. 95), a claim previously advanced by other writers (e.g. Hartley & Davies, 1976;
Lawton & Wanska, 1977; Macdonald-Ross, 1978; Clark & Bean, 1982). Nor was McEneany (1990) the first to dispute the effectiveness of advance organizers. Hartley and Davies (1976) reviewed the technique and found conflicting evidence. Regardless, they were able to conclude that advance organizers facilitated both learning and retention'. Barnes and Clawson (1975) were less generous. They rated 32 studies and found that non-significant results prevailed 20 to 12, leading the investigators to judge the technique ineffective. The Barnes and Clawson (1975) review, however, was itself strongly criticized on methodological grounds (Ausubel, 1978; Lawton & Wanska, 1977; Mayer, 1979a).

Based on the above reports, the early research on the advance organizer was, at best, inconclusive. Certainly, these studies and reviews did not provide sufficiently strong evidence to support the use of the technique in instruction. Later analyses and reconceptualizations, however, proved to be more positive.

## The Advance Organizer and Assimilation Encoding Theory

Mayer (1979a) reinterpreted subsumption theory in terms of assimilation encoding theory. This theory predicts that the organizer will facilitate both the transfer of anchoring knowledge to working memory and its active integration with the received information. Encoding theory also predicts that the advance organizer may have no effect if the content and instructional procedure already contains the needed prerequisite concepts, if the content and instructional procedure are sufficiently well-structured to elicit the prerequisite concepts from the learner, or if the organizer does not encourage the learner to actively integrate the new information. Further, if the learner already possesses a rich set of relevant past experiences and knowledge and has developed a strategy for using it, the advance organizer would not be effective (for example, a high ability learner).

Thus, Mayer stipulates the following characteristics for constructing advance organizers:

1) Short set of verbal or visual information.

2) Presented prior to learning a larger body of to-be-learned information.

3) Containing no specific content from the to-be-learned information.

4) Providing a means of generating the logical relationships among the elements in the to-be-learned information.

5) Influencing the learner's encoding process (Mayer, 1979a, p. 382).

Mayer (1979b) also reviewed 27 published advance organizer studies which contained either an advance organizer group and a control group or a post organizer group. He concluded that, when used, there was usually a small but consistent advantage for the advance organizer group. As well, he claimed that advance organizers more strongly affected performance when material was poorly integrated, that they more strongly aided inexperienced learners and that they facilitated transfer more than specific retention of details.
*Meta-Analyses of Advance Organizer Research*

Summary reviews such as those discussed above have been strongly criticized as overly subjective (e.g. Wolf, 1986). Two later reviews of the advance organizer research (Luiten, Ames & Ackerson, 1980; Stone, 1984), however, use meta-analysis, a technique which permits quantitative reviews and syntheses of the research issues (Wolf, 1986), and Glass' effect size statistic (E.S.) in particular (e.g. Glass, McGaw & Smith, 1981). The E.S. allows the comparison of studies which vary in design, sample selection and setting in order to form conclusions and, because it is based on standard deviations, also permits an assessment of degree of effect. Thus, for t-tests of independent means, an E.S.'s of 0.20 could be considered of mild strength, an E.S. near 0.50 moderate and those 0.80 and above as strong (Cohen, 1988, p.25-26). The E.S. will be used in this paper for the analysis of studies reported since the publication of the Luiten, Ames & Ackerson, (1980) and Stone (1984) reviews.

The first meta-analysis (Luiten, Ames & Ackerson, 1980), examined 135 studies and reported mean E.S. of 0.21 for learning and 0.26 for retention. They concluded that advance organizers had a small, facilitative effect upon learning as well as retention. The effect on learning, however, runs contrary to predictions based both on Ausubel's subsumption theory and Mayer's assimilation encoding theory. Luiten et al. (1980) also found advance organizers effective for all ability levels but especially for high ability learners (mean ES. of 0.23) which also contradicts both theories.

Stone's (1984) meta-analysis (Table 1) more closely followed Mayer's model. The review included only studies with a control group or a post organizer group and those in which a posttest was administered one week or later after the treatment (retention only). Consequently, Stone's results were consistently higher than those of Luiten et al. (1980). She reported a mean effect size of 0.66 indicating that advance organizers facilitate the long term retention of new, unfamiliar material. However, she also compared "true" advance organizers (those acting as subsumers) to those which were at the same level as the material-to-be-learned. The mean E.S. for the subsuming organizers (0.75) was only slightly larger than that for non-subsuming organizers (mean E.S. of 0.71). As well, Stone (1984) found no special facilitation for low ability learners. While generally supportive, these results contradict two main assumptions of both subsumption theory and assimilation encoding theory.
<table><caption>**Table 1**
A Comparison of Effect Sizes for the Luiten et al. and Stone Meta-Analyses</caption><thead><tr><th>Variable</th><th>Luiten et al.</th><th>Stone</th></tr></thead><tbody><tr><td>learning</td><td>0.21</td><td>---</td></tr><tr><td>retention</td><td>0.26</td><td>0.66</td></tr><tr><td>low ability</td><td>0.13</td><td>0.26</td></tr><tr><td>medium ability</td><td>0.08</td><td>0.64</td></tr><tr><td>high ability</td><td>0.23</td><td>0.34</td></tr></tbody></table>

*A Review of More Recent Advance Organizer Research*

A comparison of several more recent studies of advance organizers reflect the same variable results (Table 2) and are discussed below. Four studies (Carries, Lindbeck & Griffin, 1987; Tripp & Roby, 1990, 1991; Kenny, 1992) used advance organizers with CBI and will be discussed in a later section. Most of these studies produced positive E.S.'s but these ranged from very mild (0.14) to very strong ((1.25). Three studies (Corkhill, Bruning & Glover, 1988; Kloster & Winne, 1989; Kenny, in press) report negative effect sizes.

*Positive results.* Corkhill et al. (1988) conducted six experiments to investigate retrieval context set theory. This theory holds that re-reading advance organizers before the posttest will aid retrieval of the information from long-term into working memory. The investigators supplemented the advance organizers in three experiments with additional activities such as paraphrasing. Advance organizers were used unsupported in the other three. The mean effect sizes for the latter are 3.75 for the cue condition (re-reading the organizer before the test of learning) and 2.24 for the no cue condition. Since the cue condition may have added a practice effect to the presumed subsumption function of the advance organizer, only the results for the no cue condition, which represents the use of advance organizers as specified by Ausubel and Mayer, are reported in Table 2. As the measure of learning in all three experiments tested retention, the average E.S. of 2.24 is strong evidence for the facilitating effect of this instructional technique.

Tajika, Taniguchi, Yamamoto and Mayer (1988) used pictorial advance organizers with fifth grade Japanese mathematics to produce strong results (E.S.'s for the integrated organizer treatment of 2.04 for learning and 4.08 for retention). The treatments compared two types of pictorial advance organizer to a control: (a) an integrated organizer, presenting two geometric figures divided into component parts in an organized manner and (b) a fragmented organizer, presenting the same shapes in a disorganized way. The students studied the organizers before reading a 550 word passage about an imaginary land emphasizing geometric shapes and were assessed using free recall tests of learning and retention. Effect sizes were highest for the retention test as predicted by theory.
Lenz, Alley and Schumaker (1986) investigated the effects of the delivery of an advance organizer prior to each lesson on Learning Disabled (LD) students' retention and expression of information from a given lesson. Learning was assessed by an after-class interview recording the number of statements made by the student related to the lesson in which the organizer was used. Results indicated improvement both after teacher training and again after student training on taking notes from the organizers. The first improvement can be attributed to the use of the advance organizer per se and conflicts with Mayer's theory which indicates that it should not be effective for learning. The student training result could be ascribed as much to the generative activity of note-taking as to the advance organizer. Studies by Gilles (1984) with surgical nursing content (E.S. = 0.33 for retention) and Doyle (1986) with college mathematics (E.S. = 0.74 for learning and E.S. = 1.03 for retention) indicated expository advance organizers affected both learning and retention with stronger support for the latter, thus supporting assimilation encoding theory.

Negative results. Three studies fall into this category. Corkhill, Bruning and Glover (1988) also compared the effects of concrete and abstract advance organizers on students' recall of prose (i.e. learning). The concrete organizer was hypothesized to function as a comparative advance organizer and the abstract organizer as an expository advance organizer. The abstract organizer treatments produced a mean effect size of -0.62 while the concrete organizer treatments had a mean effect size of 2.25. These results provide support for comparative but not expository advance organizers, yet both should be effective and for retention, not learning.

Kloster and Winne (1989) randomly assigned 227 eighth grade mathematics students to four treatment groups: (1) expository advance organizer, (2) comparative advance organizer, (3) outline and (4) unrelated passage (control). A mean E.S. of -0.18 was obtained for the expository advance organizer and -0.15 for the comparative advance organizer indicating a slightly negative effect for advance organizers. Finally, as will be detailed below, Kenny (in press) compared the use of an advance organizer to that of participatory and final form graphic organizers with a CBIV program on cardiac nursing. In this study, the final form graphic organizer was the most effective treatment.
<table><caption>**Table 2**
A Comparison of Effect Sizes for Recent Advance Organizer Studies</caption><thead><tr><th>Study</th><th>Learning</th><th>Retention</th></tr></thead><tbody><tr><td><em>Carnes, Lindbeck & Griffin</em> (1987)</td><td>0.49</td><td>0.14</td></tr><tr><td><em>Corkhill et al.</em> (1988)</td><td></td><td></td></tr><tr><td>Expt. 3 (after 1 week)</td><td>--</td><td>1.96</td></tr><tr><td>Expt. 4 (after 24 hours)</td><td>--</td><td>2.85</td></tr><tr><td>Expt. 5 (after 2 weeks)</td><td>--</td><td>1.91</td></tr><tr><td><em>Corkhill, Bruning & Glover</em> (1988)</td><td></td><td></td></tr><tr><td>Concrete Organizer - Expt. 1</td><td>1.56</td><td>--</td></tr><tr><td>Concrete Organizer - Expt.2</td><td>2.93</td><td>--</td></tr><tr><td>Abstract Organizer - Expt. 1</td><td>-1.02</td><td>--</td></tr><tr><td>Abstract Organizer - Expt. 1</td><td>-0.21</td><td>--</td></tr><tr><td><em>Doyle</em></td><td><strong>0.74</strong></td><td>1.03</td></tr><tr><td><em>Gilles</em></td><td>0.015</td><td>0.33</td></tr><tr><td><em>Kenny</em> (in press)</td><td></td><td></td></tr><tr><td>Adv. Org. > Partic. Graph Org.</td><td>0.45</td><td><strong>0.95</strong></td></tr><tr><td>Adv. Org. > Final Form Graph Org.</td><td>-1.17</td><td>-0.45</td></tr><tr><td><em>Kloster & Winne</em> (1989)</td><td></td><td></td></tr><tr><td>Comparative Organizer</td><td>--</td><td>-0.15</td></tr><tr><td>Expository Organizer</td><td>--</td><td>-0.18</td></tr><tr><td><em>Lenz, Alley & Schumaker</em> (1986)</td><td></td><td></td></tr><tr><td>After teacher training</td><td>1.03</td><td>--</td></tr><tr><td>After student training)</td><td>2.93</td><td>--</td></tr><tr><td><em>Tajika, Taniguchi, Yamamoto & Mayer</em> (1988)</td><td></td><td></td></tr><tr><td>Fragmented Pictorial</td><td>0.078</td><td>1.49</td></tr><tr><td>Integrated Pictorial</td><td>2.04</td><td>4.08</td></tr><tr><td><em>Tripp & Roby</em> (1990)</td><td>1.25</td><td>--</td></tr><tr><td><em>Tripp & Roby</em> (1991)</td><td>0.33</td><td>--</td></tr><tr><td>Mean</td><td>0.76</td><td>1.16</td></tr></tbody></table>
Based on these reviews, the research evidence concerning any facilitative effect of advance organizers upon learning and retention is unclear. Much of the evidence appears positive, and yet, it is quite variable. Effect sizes for the more recent studies range from -1.02 to 2.04 for measures of learning and from -0.18 to 4.08 for tests of retention. It may be that this range represents the distribution of study means around the true effect size mean for this instructional technique. The variability could reflect differences in experimental design, subject selection and methodology. Given the frequent discussion of the difficulty in constructing an advance organizer (e.g. Hartley & Davies, 1976, McEneany, 19901, one likely source of error is treatment fidelity. Much of this variability may reflect the lack of clarity about the definition and design of an advance organizer.

If advance organizers do affect learning, how do they achieve this effect? The inconsistent support for subsumption theory (Ausubel, 1963) and for assimilation encoding theory (Mayer, 1979a) calls their ability to predict into question. Corkhill, Bruning and Glover (1988) suggest that concrete (comparative) advance organizers may furnish ideational anchorage in terms already familiar to the learners, that is, to provide meaning by association with existing schema. They may assist the learner to visualize the content of the organizer more readily. These authors also stress the importance of ensuring encoding of the organizer using techniques such as paraphrasing, generative techniques to be discussed below.

In summary, the effectiveness of the advance organizer as a pedagogical strategy and the validity of the theory on which it is based to predict its effectiveness are both still in question. What, then, of the other variations on the advance organizer concept? The next section extends this discussion to include two of its progeny: the structured overview form of advance organizer and the pictorial graphic organizer.

# RESEARCH ON THE USE OF GRAPHIC ORGANIZERS

## *The Structured Overview Form of Graphic Organizer*

The graphic organizer was first presented as a variation on the advance organizer. However, as indicated previously, key characteristics of this technique vary from its predecessor (See Alverman, 1981). Since it is not an advance organizer, does independent evidence exist that this strategy affects learning or retention? A meta-analysis by Moore Readance (1984) reported an average effect size of 0.22. They also noted an average effect size of 0.57 for graphic post organizers constructed by the instructor with the class or by the student alone. They concluded that the structured overview form of graphic organizer does have an effect, especially for university students, that vocabulary learning is most positively affected and that post-organizers benefit learners more than advance organizers.

More recent studies appear to support these conclusions (Table 3). Alvermann (1981) found that partially complete advance graphic organizers
had an effect on ninth grade students' comprehension and retention of text. The results indicated the strongest effect for the less well organized text. Two studies (Boothby & Alvermann, 1984; Alvermann & Boothby, 1986) found the graphic organizer to be effective as a strategy for facilitating fourth graders comprehension and retention of social studies text. While the second (Alvermann & Boothby, 1986) study did not test for retention, it did indicate that graphic organizers had an effect on a transfer of learning task.

Bean et al. (1983) reported a study in which tenth grade world history students were divided into three groups: those taught to construct summaries and post graphic organizers, those taught to construct post graphic organizers only, and those taught to build outlines only. Results indicated a small effect for combined organizer and summary training group (0.16) but a negative effect for the organizer only group (-0.11). Because a true control group was not used in this study, effect sizes were calculated using the outlining group as a control.

Finally, Carr and Mazur-Stewart (1988) found that avocqlary overview guide (a multi-page booklet) which included a graphic organizer was significantly superior to a traditional form of instruction in improving the vocabulary comprehension and retention of college students. While the results are neither extensive nor consistent, taken overall, these studies do indicate that the structured overview form of graphic organizer — especially the post organizer type — does affect learning and retention.

<caption>Table3</caption>

A Comparison of Effect Sizes for Recent Structured Overview Graphic Organizer Studies

<table><thead><tr><th>Study</th><th>Learning</th><th>Retention</th></tr></thead><tbody><tr><td><strong>Alvermann (1981)</strong></td><td></td><td></td></tr><tr><td>Descriptive Passage</td><td>1.26</td><td>1.76</td></tr><tr><td>Comparative Passage</td><td>0.06</td><td>0.41</td></tr><tr><td><strong>Boothby & Alvermann (1984)</strong></td><td>0.98</td><td>0.99</td></tr><tr><td><strong>Alvermann & Boothby (1986)</strong></td><td></td><td></td></tr><tr><td>Passage 1</td><td><strong>-0.34</strong></td><td>--</td></tr><tr><td>Passage 2</td><td><strong>0.45</strong></td><td>--</td></tr><tr><td>End of chapter test</td><td><strong>0.41</strong></td><td>--</td></tr><tr><td><strong>Bean et al. (1986)</strong></td><td></td><td></td></tr><tr><td>Organizer and Summary Training</td><td>0.16</td><td>--</td></tr><tr><td>Organizer Only</td><td>-0.11</td><td>--</td></tr><tr><td><strong>Carr and Mazur-Stewart (1988)</strong></td><td>0.89</td><td>1.23</td></tr><tr><td>Mean</td><td><strong>0.42</strong></td><td>1.10</td></tr></tbody></table>
## The Pictorial Graphic Organizer

A review of the pictorial graphic organizer research is reported in Table 4. Two experiments by Jonassen and Hawk (1984) tested teacher-constructed participatory graphic organizers in regular classrooms. The results indicated a stronger effect for learning than for retention. Two studies (Hawk, McLeod & Jeanne, 1981; Hawk & Jeanne, 1983, cited in Hawk, McLeod, & Jonassen, 1985) reported statistically significant results in favour of participatory pictorial graphic organizers. However, insufficient data were available to calculate effect sizes. A more recent study by Hawk (1986) also found this technique to be effective in facilitating retention for above average students studying life science in the sixth and seventh grades.

Darch, Camine and Kameenui (1986) compared the cooperative (group) and individual completion of participatory pictorial graphic organizers to a more traditional directed reading approach. The graphic organizers, especially for the cooperative learning approach, were found to be more effective in facilitating retention than the traditional approach. Learning was not tested.

<caption>Table 4</caption>

A Comparison of Effect Sizes for Pictorial Graphic Organizer Studies

<table><thead><tr><th>Study</th><th>Learning</th><th>Retention</th></tr></thead><tbody><tr><td><strong>Jonassen & Hawk (1984)</strong></td><td></td><td></td></tr><tr><td>Experiment 1</td><td>1.17</td><td>0.67</td></tr><tr><td>Experiment 2</td><td>1.82</td><td>0.77</td></tr><tr><td><strong>Hawk (1986)</strong></td><td>--</td><td>0.64</td></tr><tr><td><strong>Darth, Carnine & Kameenui (1986)</strong></td><td>--</td><td></td></tr><tr><td>Cooperative Graphic Organizer</td><td>--</td><td>1.59</td></tr><tr><td>Individual Graphic Organizer</td><td>--</td><td>0.72</td></tr><tr><td><strong>Alvermann (1988)</strong></td><td></td><td></td></tr><tr><td>Self-perceived High Ability</td><td>-0.64</td><td>--</td></tr><tr><td>Self-perceived Low Ability</td><td>3.94</td><td>--</td></tr><tr><td><strong>Kenny, Grabowski, Middlemiss, & Van Neste-Kenny (1991)</strong></td><td>0.59</td><td>-0.07</td></tr><tr><td><strong>Kenny (in press)</strong></td><td></td><td></td></tr><tr><td>Participatory Graph. Org. &gt; Adv. Org.</td><td>-0.45</td><td>-0.95</td></tr><tr><td>Final Form Graph. Org. &gt; Adv. Org.</td><td>1.17</td><td>0.45</td></tr><tr><td>Mean</td><td>1.09</td><td>0.48</td></tr></tbody></table>
Alvermann (1988) investigated the effects of a final form graphic organizer designed to induce students to look back in their texts for missed information. The organizer facilitated learning for self-perceived low ability students (those whose ability as measured by a standardized achievement test matched their own perception of their ability) compared to a control group of self-perceived low ability students who only read the passage. The organizer appeared to interfere with the learning of self-perceived high ability students compared to the equivalent control group. For this study, the organizer was designed as a road map to guide students back to sections in the text in order to answer posttest questions.

Finally, studies by Kenny, Grabowski, Middlemiss and Van Neste-Kenny (1991) and Kenny (in press) used pictorial graphic organizers with CBIV and found evidence to support the technique. Again, these are discussed in the section on organizers in CBI. In summary, the evidence reported above, while once again inconsistent, is generally positive. Pictorial graphic organizers of both forms appear to facilitate the learning and retention of information, at least for younger learners. Learning, though, seems to have been more consistently affected than longer-term retention of information.

Taken overall, there is positive evidence for the effectiveness of all three of these techniques. The question posed by this review, however, is whether or not these results generalize to instruction delivered by CBI. Can one expect any of these techniques to help alleviate such potential problems as disorientation and cognitive overload? Research specific to the use of instructional organizers in CBI is discussed next.

## INSTRUCTIONAL ORGANIZERS WITH CBI

While the majority of studies reviewed used text-based advance organizers, the strategy can be produced in other formats. Mayer (1979a) noted that advance organizers could be verbal or visual material and used a pictorial advance organizer in a later study (Tajika, Taniguchi, Yamamoto & Mayer, 1988). Therefore, it is not too large a leap of faith to presume that they might be effectively incorporated into CBI, a class of technology which can effectively combine both text and visuals. Hannafin and his associates have published a series of papers, based on the ROPES+ meta-model discussed above, on orienting activities in CBI generally and CBIV in particular (Hannafin, Phillips, Rieber & Garhart, 1987; Hannafin, Phillips & Tripp, 1986; Hannafin, 1987; Rieber & Hannafin, 1988). Hannafin, Phillips, Rieber and Garhart (1987) found that both behavioral and cognitive orienting activities, used in CBI with university students, improved factual learning. The cognitive orienting activity was: "designed to provide an integrative method for establishing meaningful relationships, while also serving as a subsumer of lesson detail" (p.80). This is labelled an advance organizer, however, the one example they give, "In the next section, you will be presented information
about: The importance of studying cultures." (p.77), seems insufficient to affect encoding.

Similarly, Hannafin, Phillips and Tripp (1986) used a one sentence cognitive orienting activity in a CBIV lesson on artists and art periods with 80 volunteer college students. They noted a significant interaction between the orienting activity and processing time but no main effect. Another study (Hannafin, 1987) compared the effects of orienting activities, cuing and practice on the learning of material on space voyages by ninth grade students. A significant interaction was found between the cognitive orienting activity and practice but the orienting activity alone was not a significant component. Again, the cognitive orienting activity provided, "The Next Section Presents the Following Concepts: ==> Unique lighting found throughout the solar system (and) ===> Matter found throughout the solar system", based on Mayer's (1979a) characteristics of an advance organizer, seems meagre. Rieber and Hannafin (1988) also studied the effects of textual and animated cognitive orienting activities on a CBI lesson explaining Newton's laws of motion to 111 elementary students. Three types of orienting activities (one minute duration each) were used: a text only, one sentence summary of the particular basic concept, an animated graphic sequence, or a combined text and graphic sequence, all presented throughout the lesson before each basic concept. There was also a control (no activity) group. The authors found no statistically significant effects.

These orienting activities all meet three of Mayer's (1979a) characteristics: (a) a short set of verbal or visual information, (b) presented prior to the material-to-be-learned and (c) containing no specific content from that material. However, none seem of sufficient scope to meet the other two conditions; (d) generate logical relationships among elements in the to-be-learned material, or (e) to sufficiently influence the learner's encoding process. The organizing activities used in these studies were probably not advance organizers.

Some studies, however, did use true advance organizers in computer-based instruction. Carnes, Lindbeck and Griffin (1987) used a computer-based tutorial on kinematics with 100 high school students. There was no statistically significant difference between the advance organizer treatment group and a non organizer group (which read a related passage designed not to act as a subsumer). However, effect sizes of 0.49 for a test of learning and 0.14 for the retention test indicate a mild positive effect by the advance organizer.

Krahn and Blanchaer (1986) tested the use of an advance organizer to improve knowledge application by medical students in a computer-based simulation. Post-test scores showed a statistically significant difference between the experimental and control group, both for the total scores and particular questions designed to test far transfer, as predicted by assimilation encoding theory. The test, however, was given immediately after completion of the simulation and consisted of only six questions. No validity or reliability
data were provided. Insufficient data were reported to allow the calculation of effect sizes and, hence, this study was not included in the previous table (Table 2).

Tripp and Roby (1990,1991) reported the use of an advance organizer in two studies with a Japanese-English hypertext-based lexicon. They measured immediate recall and assessed rote learning, which runs contrary to assimilation encoding theory. The authors reasoned, however, that the advance organizer, rather than act as a bridge to existing cognitive structures, would convey the structure of the database and so contribute to meaningful learning.

The results of the first study were not statistically significant (p < 0.171) yet the E.S. was 1.25, indicating a probable lack of power in the study. As well, a significant negative effect was reported for the interaction between the advance organizer and a visual metaphor for half the group. In the second study, the advance organizer was rewritten to provide a metaphorical structure hypothesized to be congruent with the visual metaphor treatment. The authors reported a significant main effect for the advance organizer treatment, but not for the visual metaphor, nor was there a significant interaction. The E.S. of 0.33 for this study may be more accurate given the larger number of subjects used. Regardless, these studies clearly contradict advance organizer theory since they demonstrated that the technique facilitated rote, rather than meaningful, learning.

Kenny, Grabowski, Middlemiss and Van Neste-Kenny (1991) compared participatory graphic organizers to the identical final form versions on the learning of nursing students from a CBIV program on nursing elderly patients with pulmonary disease. The participatory graphic organizer group substantially outperformed the final form group on a test of learning, scoring an average of 1.77 points higher on an 18 question multiple choice test (E.S. = 0.59). The difference, however, was not statistically significant. As well, there was only a slight difference on the retention test, in favour of the final form version (E.S. = -0.07). Considerable extraneous note-taking by subjects in both groups likely confounded the differences in generativity between the two treatments.

Kenny (in press) next compared the use of an advance organizer to that of participatory and final form graphic organizers with a CBIV on cardiac nursing. In this study, the final form graphic organizer was clearly most effective, garnering the highest mean scores on both tests of learning and retention. The participatory graphic organizer group had the lowest mean scores while the advance organizer group fell in the middle. The difference between the final form and advance organizer group means was statistically significant at the p < 0.05 level for learning (E.S. = 1.17) but was not significant for retention (E.S. = 0.45). The difference between the advance and participatory organizer group means was statistically significant at the p < 0.10 level for retention (E.S. = -0.95) but not significant for the test of learning (E.S. = -0.45). The effect sizes reported here (and in Table 4) assume the advance organizer as a control. Extraneous note-taking was controlled.
However, the cardiac program used a guided discovery approach design (unlike the pulmonary program) which demanded considerable interaction on the part of the learner and may have interacted with the organizer treatments.

Overall: there appears to be mild evidence to suggest that advance organizers and pictorial graphic organizers could be effective if incorporated in instruction based on CBI. Furthermore, this literature is congruous with the research on the use of these techniques in instruction in general, that is, the evidence of their effectiveness is mostly positive, if somewhat variable. Why are these generally positive results not reflected more often in the research? In the case of the studies focusing on CBI, many of the organizers did not appear to be properly constructed, that is, they were insufficient to produce a subsumptive effect. Again this may reflect the theme sounded by McEneany (1990) that a sound operational definition for the construction of advance organizers is lacking. The format and construction of the pictorial graphic organizer is similarly unclear and often (in the experience of the author) difficult.

Perhaps there is also a problem with the theory underlying instructional organizers. If subsumption theory and assimilation encoding theory are not effective in predicting when instructional organizers will be effective, will another theory be more accurate? In a discussion of the psychological underpinnings of hypermedia, Borsook and Higginbotham-Wheat (1992, p.62) note that as "we explore some of the ideas of what we know about how we learn and apply knowledge, it becomes obvious that activity as well as interactivity [emphasis added] are integral components of both theory and its application in the technology of hypermedia." Cognitive principles suggest that learning is an active, constructive process in which learners generate meaning for information by accessing and applying existing knowledge (Borsook & Higginbotham-Wheat 1992, p.64). The authors point out that Wittrock's (1974) generative learning theory incorporates such principles. Generative learning theory is considered next.

## THE GENERATIVE LEARNING HYPOTHESIS

In Wittrock's (1974, p.88) view, "it is the learner's interpretation of and processing of the stimuli, not their [the stimuli's] nominal characteristics, which is primary". Learners must construct their own meaning from teaching (Wittrock, 1985). This meaning is generated by activating and altering existing knowledge structures to interpret new information and encode it effectively for future retrieval and use. Further, generative learning involves not only generating meaning, but overt activities as well, such as generating associations among words, and generating pictures (Doctorow, Wittrock & Marks, 1978). These learning activities require the learner to relate new information to an existing knowledge structure and depend on complex
cognitive transformations and elaborations that are individual, personal and contextual in nature. Information is transformed and elaborated into a more individual form making it more memorable as well as more comprehensible (Borsook & Higginbotham-Wheat, 1992).

Of the instructional organizers discussed above, only the participatory form of the pictorial graphic organizer (Hawk, McLeod and Jonassen, 1985) and the student-constructed form of structured overview (Moore & Readance, 1984) elicit, by design, a generative response from the learner. In fact, Hawk, McLeod & Jonassen (1985) recommended the participatory version because it was their belief that "the more generative the nature of student participation, the more likely it is that transfer and higher level learning will be affected" (p.179). These organizers elicit generative activity because they require the learner to actively search a body of material to select information to complete the organizer. Advance organizers, on the other hand, cannot necessarily be considered to be generative since they invoke a covert response rather than specifically engage the learner in overt, active learning. Perhaps this explains why the research on instructional organizers is so variable. Given how difficult advance organizers are to construct, some investigators may have unknowingly developed organizers or other activities that engaged the learners in generative learning.

Moore and Readance (1984) note superior results for structured overview graphic organizers constructed after reading the learning passage. In support of Wittrock's hypothesis that familiar words facilitate the learners' generation of meaning for the passage (Marks, Doctorow & Wittrock, 1974, Wittrock, Marks & Doctorow, 1975), Corkhill, Bruningand Glover (1988) demonstrated the advantage of concrete advance organizers over more abstract ones, suggesting that the former furnish ideational anchorage in terms already familiar to the learners. Even the Alvermann (1988) study of final form graphic organizers directed the learners to engage in what can be argued to be a generative activity by asking them to use the organizer as a map in an active search back in the text for question answers. Thus, some of the most impressive results were garnered when students were actively engaged in the learning process.

## Generative Instructional Organizers with CBI

Generative learning theory as applied to CBI, at least as pertains to instructional organizers, has yet to be widely tested. The few studies completed have not provided strong evidence to suggest that generative learning activities can be successfully applied to computer-based media. Two studies described earlier (Kenny, Grabowski, Middlemiss & Van Neste-Kenny, 1991, Kenny, in press) obtained mixed results. Kenny, Grabowski, Middlemiss and Van Neste-Kenny (1991) obtained a mild effect size in favour of a participatory pictorial graphic organizer used with CBIV. Kenny (in press), however, found the final form graphic organizer with CBIV to be the most effective on both measures of learning and retention. The group using the hypothetically
generative participatory organizer achieved the lowest mean scores. However, analysis of interview data indicated that the guided discovery design of the CBIV program may have interfered with the generative nature of the participatory graphic organizer. The learners were already engaged in a demanding learning activity and the normally generative organizer, rather than helping to make learning more meaningful, likely contributed to, rather than alleviated, cognitive overload.

Harris (1992) compared the use of learner-generated summaries to the completion of multiple choice questions with tutorial courseware delivered by CBIV. Contrary to predictions, the control group, which received the treatment considered to be least generative (multiple choice questions completed at the end of each module), achieved the highest mean score on a test of learning (given right after completion of the modules). Effect sizes were -0.44 for the learner-generated summaries without feedback and -0.12 for those with feedback. None of the differences were statistically significant.

Finally, Jonassen and Wang (1992), conducted three studies on acquiring structural knowledge from hypertext. Structural knowledge is that of how concepts in a particular domain are interrelated (Jonassen & Cole, 1992). In the second study, Jonassen and Wang (1992) tested the use of a generative activity with a HyperCard-based text. The control treatment consisted of referential links embedded in the cards, while the generative treatment asked the learners to classify the nature of the relationship between the node they were leaving and the one they were traversing to. The control group scored higher on average on a test of recall (E.S. = -0.64) and on 2 of 3 tests of structural knowledge (E.S. for relationship proximity = 0.36, E.S. for semantic relationships = 0.43 and E.S. for analogies = -0.15). Again, none of the differences were statistically significant.

Once again, this group of studies is not conclusive. Three of these studies use small sample sizes and may have been underpowered. Only the first provided any evidence for the effectiveness of generative activities in CBI. While, theoretically, the application of generative learning theory to the use of instructional organizers with CBI seems to hold promise, there has been little evidence to demonstrate that it more effectively predicts the effectiveness of instructional orienting activities than the theories considered previously. Given this, is there any theory that can be shown to provide guidance? Borsook and Higginbotham-Wheat (1992) suggest a number of theories, generative learning among them, which may provide insight about how and why hypermedia (that is, interactive multimedia instruction) might be used to deliver effective instruction. Perhaps most prominent among these cognitive theories is schema theory (Rumelhart & Orteny, 1977, Anderson, 1977).

## Schema Theory and the Hypermap

Jonassen (1989) claims that it is schema theory that describes the organization of human memory, not Ausubel's hierarchical, or subsumptive, model. A schema for an object, event or idea is comprised of a set of attributes,
that is, associations that one forms around an idea. Schema are in turn
arranged into semantic networks, sets of nodes with ordered relationships
connecting them (Jonassen, 1988). Kiewra (1988) indicates that an outgrowth
of schema theory has been applied research on the effectiveness of spatial
learning strategies. Such strategies involve the reorganization of information
into some form of spatial representation that clarifies the inherent relation-
ships among ideas or concepts. Such representations allow information to be
more readily processed since they reflect cognitive structure and provide
multiple retrieval cues for accessing it. As the graphic organizer is one form
of spatial representation, schema theory may help explain the effectiveness
of the technique. In the study by Kenny (in press), for instance, the filled-in
(Final Form) version of the pictorial graphic organizer was most effective and
may have acted as a form of cognitive map.

To reflect an individual's knowledge structure, then, an instructional
organizer, rather than being constructed at a higher, more abstract level as
is an advance organizer, might be designed to reflect either novice or expert
schemata. Jonassen (1989), for instance, advocates the use of hypermaps, or
graphical browsers, an instructional orienting technique similar to the
graphic organizer. A hypermap provides a graphical view of the program
structure. The user may select a node on the hypermap and be taken
immediately to that part of the program. Hypermaps represent a graphical
interface between the user and a hypertext that is designed to reduce
navigation problems (Jonassen & Wang, 1992). They can be expected to be
effective because they should enhance the learner's structural knowledge (the
knowledge of interrelationships between ideas) of the information in the
program (Jonassen, 1989).

Research on hypermaps, however, has been even more scant than that on
generative techniques in CBI. In the first study described above, Jonassen
and Wang (1992) compared the effectiveness of a hypermap to a control
treatment consisting of referential links embedded in the cards. Again, the
control group significantly outperformed the hypermap group on a recall test.
Tests of structural knowledge acquisition, however, showed no significant
differences between the techniques. In the third study, Jonassen and Wang
(1992) provided learners with either a control treatment as described above
or a hypermap to use for navigation. They informed half the learners in each
group that they would be responsible for developing a semantic network
(essentially their own hypermap) after completing the program. While this
activity was ceased after only a few minutes in order to control for time-on-
task, those given the semantic network task performed significantly better on
the semantic relationships scale and the graphical browser/semantic network
group was significantly better on the analogies subscale. In effect, the
instructions to the semantic networks groups may have been sufficient to lead
them to actively engage the material, that is, to elicit generative learning. The
analogies subscale result, then, is one that could be explained by either
Wittrock's (1974) theory, by schema theory, or both. These two studies appear
to be the only ones to date testing this form of orienting activity, although a variation of the knowledge mapping system (e.g. Dansereau et al., 1979) has been tested.

Like the graphical browser, the knowledge map represents multi-dimensional knowledge in associative networks akin to semantic nets or schemata. Nodes denote the type and importance of the content and the spatial properties of the map clarify the organization of the domain (Reynolds and Dansereau, 1990). Perhaps the greatest distinguishing feature of the knowledge map is the use of 8 specified link types to label linkages between nodes while Jonassen's version uses flexible labelling for links. Two recent studies (Reynolds and Dansereau, 1990, Reynolds et al., 1991) have presented this technique to learners in computerized form as a hypermap. Two variations of a statistics package were developed: a "standard" hypertext version and a version in which all the concepts were represented in hypermap versions of the knowledge map, that is, no standard textscreens were provided. Since these studies used this format for the main body of the learning material rather than as an orienting activity, these studies are not reviewed here.

## CONCLUSION

As presented, the research pertaining to the use of instructional organizers in CBI has not been extensive. Much has been conducted based on the ROPES+ meta-model proposed by Hannafin and his associates. While advance and graphic organizers are included in the orienting activities category, it is doubtful that the cognitive orienting activities used by these theorists (e.g. Hannafin, 1987, Hannafin & Hughes, 1986, Hannafin, Phillips, Rieber, & Garhart, 1987) have represented forms of these techniques at least as described by the originators of the strategies (e.g. Ausubel, 1960, 1978; Mayer, 1979a). Where such organizers have been constructed according to original guidelines, they have been somewhat more successful. In general, there is substantial, if variable, evidence that these strategies can influence learning and retention. Based on these results, some tentative recommendations for the effective use of instructional organizers in CBI can be made.

### Advance Organizers

The designer should be clear about the reasons for incorporating this strategy. Advance organizers were proposed as a means to facilitate the retention of meaningful verbal information. The technique is founded on the premise that the structure of memory is hierarchical in nature. Based on the above review, advance organizers are probably best used with at least some of Mayer's (1979a) conditions in mind:

1. Use advance organizers in CBI intended to present information that is hierarchical in nature and for which more general or more abstract subsuming material can be designed.
2. Be clear that the advance organizer encourages the learner to actively integrate the material to come. Will it truly provide a means for generating logical relationships among the elements in the material to come and thereby influence the learner's encoding process? The wording or design of an advance organizer for CBI can be difficult if one also wishes to limit the number of screens of text in which it is presented. Ausubel typically used advance organizers of about 500 words. Such length may be unacceptable for CBI, particularly if learners have the choice whether or not to read the organizer.

3. Explain the purpose of the organizer to the learners before they encounter it whether they have the choice or not to use it.

4. Be certain to word the organizer in terms familiar to the learner. It is intended to be a form of prior knowledge to which the coming material is to be related and must be meaningful.

5. Use an advance organizer if the CBI is poorly organized or lacks the prerequisite concepts contained in the organizer.

6. Do not use an advance organizer if the CBI also makes use of other learning strategies that engage the learner in overt, generative activities.

## Graphic Organizers

By definition, graphic organizers can contain material from the main CBI program and are not advance organizers. However, they can be used as a pre-instructional strategy, that is, presented at the beginning of a program. Again, based on the research presented in the above review, graphic organizers are also probably best used under certain conditions in CBI:

1. Design the organizer so that it elicits generative activity from the learner either as a part of the on-going interaction with the CBI or right after the learner has completed the program (or a substantial part thereof).

2. Do not use a graphic organizer if the CBI also makes use of other learning strategies that engage the learner in overt, generative activities.

3. If you intend to use a graphic organizer as a pre-instructional activity, design it to reflect the structure of the material such that it provides a conceptual overview of the material to come.

4. Consider the structured overview form of graphic organizer if it is important or helpful to present new vocabulary to learners prior to their encountering the material to come.

## The Role of Theory

As this review has indicated, there is substantial evidence to indicate that instructional organizers can be effective learning strategies but the evidence has been variable. Of the three instructional organizers discussed, only the
advance organizer is founded on a strong theoretical base (subsumption theory and assimilation encoding theory), and yet, these theories have not consistently predicted the effectiveness of the strategy. However, neither has the limited research testing instructional organizers on the basis of two other theories, generative learning theory and schema theory, been strongly indicative of their predictive value. Given the evidence presented here, it may be that no one theory is sufficient in and of itself to predict the effectiveness of instructional orienting techniques and to guide their design. Rather, it may be necessary to consider various theories acting in concert. If the results of the third Jonassen and Wang (1992) study, which combined a hypermap treatment with a generative activity, are indicative, the design of instructional organizers may have to be based on features from various theories, each of which describes a different aspect of human cognition. While further research on both the application of generative learning and schema theory to CBI is needed, perhaps this will be most fruitful if it combines the two. Simple tests of theory may not be sufficient.

## REFERENCES

Alvermann, D.E. (1981). The compensatory effect of graphic organizers on descriptive text. *Journal Of Educational Research, 7, 44-48.

Alvermann, D.E. (1988). Effects of spontaneous and induced lookbacks on self-perceived high and low-ability comprehenders. *Journal of Educational Research, 81*(6), 325-331.

Alvermann, D.E., & Boothby, P.R. (1986). Children's transfer of graphic organizer instruction. *Reading Psychology, 7*, 87-100.

Anderson, R.C. (1977). The notion of schemata and the educational enterprise. In R.C. Anderson, R.J. Spiro, W.E. Montague (Eds.), *Schooling and the acquisition of knowledge*. Hillsdale, N.J.: Lawrence Erlbaum Associates.

Ausubel, D.P. (1960). The use of advance organizers in the learning and retention of meaningful verbal material. *Journal of Educational Psychology, 51*(5), 267-272.

Ausubel, D.P. (1963). *The psychology of meaningful verbal learning*. New York: Grune and Stratton.

Ausubel, D.P. (1978). In defense of advance organizers: A reply to the critics. *Review of Educational Research, 48*(2), 251-257.

Ausubel, D.P., & Fitzgerald, D. (1961). The role of discriminability in meaningful learning and retention. *Journal of Educational Psychology, 52*(5), 266-274.

Ausubel, D.P., & Fitzgerald, D. (1962). Organizer, general background, and antecedent learning variables in sequential verbal learning. *Journal of Educational Psychology, 53*(6), 243-249.
Ausubel, D.P., & Youssef, M. (1963). Role of discriminability in meaningful parallel learning. *Journal of Educational Psychology, 54*(6), 331-336.

Barnes, B.R., & Clawson, E.U. (1975). Do advance organizers facilitate learning? Recommendations for further research based on an analysis of 32 studies. *Review of Educational Research, 45*(4), 637-659.

Barron, R.F. (1969). The use of vocabulary as an advance organizer. In H. Herber & P.L. Sanders (Eds.), *Research in reading in the content areas: First year report*. Syracuse, NY: Syracuse Reading and Language Arts Center.

Barron, R.F., & Stone V. F. (1974). The effect of student-constructed graphic post organizers upon learning vocabulary relationships. In P. L. Nacke (Ed.), *Interaction: Research and practice in college-adult reading. The 23rd yearbook of the National Reading Conference*. Clemson, SC: Courier-Journal Lithographing Co.

Bean, T.W., Singer, H., Sorter, J., & Frazee, C. (1983). The effect of metacognitive instruction in outlining and graphic organizerconstruction on students' comprehension in a tenth-grade world history class. *Journal of Reading Behavior, 18*(2), 153-169

Boothby, P.R., & Alvermann, D.E. (1984). A classroom training study: The effects of graphic organizer instruction on fourth graders' comprehension. *Reading World, 23*, 325-339.

Borsook, T.K., & Higginbotham-Wheat (1992). The psychology of hypermedia: A conceptual framework for R&D. In M. R. Simonson & K. Jurasek (Eds.), *14thAnnual Proceedings of Selected Research and Development Presentations* (pp. 60-69). Washington, DC: Association for Educational Communications and Technology.

Bull, R.L., & Wittrock, M.C. (1973). Imagery in the learning of verbal definitions. *British Journal of Educational Psychology, 43*,289-293.

Cames, E.R., Lindbeck, J.S., & Griffin, C.F. (1987). Effects of group size and advance organizers on learning parameters when using computer tutorials in kinematics. *Journal of Research in Science Teaching, 24*(9), 781-789.

Carr, E.L., & Mazur-Stewart, M. (1988). The effects of the vocabulary overview guide on vocabulary comprehension and retention. *Journal of Reading Behavior, 20*(1), 43-62.

Clark, C-H., & Bean, T.W. (1982). Improving advance organizer research: Persistent problems and future directions. *Reading World, 22*(1), 2-10.

Cohen, J. (1988). *Statistical power analysis for the behavioral sciences* (2nd ed.). Hillsdale, N.J.: Lawrence Erlbaum Associates.

Corkhill, A.J., Bruning, R.H., & Glover, J.A. (1988). Advance organizers: Concrete versus abstract. *Journal of Educational Research, 82*(2), 76-81.

Corkhill, A.J., Bruning, R.H., Glover, J.A., & Krug, D. (1988). Advance organizers: Retrieval context hypotheses. *Journal of Educational Psychology, 80*(3), 304-311.
Dansereau, D.F., McDonald, B.A., Collins, K.W., Garland, J., Holley, C.D., Diekhoff, G.M., & Evans, S.H. (1979). Evaluation of a learning system. In H. F. O'Neil, Jr., & C. D. Spielberger (Eds.), *Cognitive and affective learning strategies*. New York: Academic Press.

Darth, C.B., Carnine, D.W., & Kameenui, E.J. (1986). The role of graphic organizers and social structure in content area instruction. *Journal of Reading Behavior, 18(4)*, 275-295.

Doctorow, M., Wittrock, M.C., & Marks, C.B. (1978). Generative processes in reading comprehension. *Journal of Educational Psychology, 7(2)*, 109-118.

Doyle, W.H. (1986). Using an advance organizer to establish a subsuming function concept for facilitating achievement in remedial college mathematics. *American Educational Research Journal, 23(3)*, 507-516.

Galbreath, J. (1992). The educational buzzword of the 1990's: Multimedia, or is it hypermedia or interactive multimedia, or.. ? *Educational Technology, 32(4)*, 15-19.

Gayeski, D.M. (1993). Making sense of multimedia. In D.M. Gayeski (ed.). *Multimedia for learning: Development, application, evaluation*. Englewood Cliffs, NJ: Educational Technology Publications.

Gilles, D.A. (1984). Effect of advance organizers on learning medical surgical nursing content by baccalaureate nursing students. *Research in Nursing and Health, 7*, 173-180.

Glass, G.V., McGaw, B., & Smith, M.L. (1981). *Meta-analysis in social research*. Beverly Hills, CA: Sage Publications.

Hannafm, M.J. (1987). The effects of orienting activities, cuing, and practice on learning of computer-based instruction. *Journal of Educational Research, 81*(1), 48-53.

Hannafm, M.J. (1989). Interaction strategies and emerging instructional technologies. *Canadian Journal of Educational Communications, 18(3)*, 167-179.

Hannafin, M.J., & Hughes, C.W. (1986). A framework for incorporating orienting activities in computer-based interactive video. *Instructional Science, 15*, 239-255.

Hannafin, M.J., Phillips, T-L., & Tripp, S.D. (1986). The effects of orienting, processing, and practicing activities on learning from interactive video. *Journal of Computer-Based Instruction, 13*(4), 134-139.

Hannafin, M.J., Phillips, T.L., Rieber, L.P., & Garhart, C. (1987). The effects of orienting activities and cognitive processing time on factual and inferential learning. *Educational Communications and Technology Journal, 35*(2), 75-84.

Hannafin, M.J., & Rieber, L.P. (1989). The psychological foundations of instructional design for emerging computer-based instructional technologies: Part II. *Educational Technology: Research and Development, 91-101.*
Harris, B.R. (1992). The effects of integrating a learning strategy in CBI. In M. R. Simonson & K. Jurasek, (Eds.), *14th Annual Proceedings of selected Research and Development Presentations* (pp. 294-304). Washington, DC: Association for Educational Communications and Technology.

Hartley, J., & Davies, I.K. (1976). Preinstructional strategies: The role of pretests, behavioral objectives, overviews and advance organizers. *Review of Educational Research, 46*(2), 239-265.

Hawk, P.P. (1986). Using graphic organizers to increase achievement in middle school life science. *Science Education, 70*(1), 81-87.

Hawk, P.P., McLeod, N.P., & Jonassen, D.H. (1985). Graphic organizers in texts, courseware, and supplemental materials. In David H. Jonassen (Ed.), *The technology of text, volume two: Principles for structuring, designing, and displaying text*. Englewood Cliffs, NJ: Educational Technology Publications, Inc.

Hooper, S., & Hannafin, M.J. (1988). Learning the ROPES of instructional design: Guidelines for emerging interactive technologies. *Educational Technology, 28*(7), 14-18.

Jonassen, D.H. (1984a). Developing a learning strategy using pattern notes: A new technology. *Programmed Learning and Educational Technology, 21*(3), 163-175.

Jonassen, D.H. (1985). Generative learning vs. mathemagenic control of text processing. In D. H. Jonassen (Ed.), *The technology of text, volume two: Principles for structuring, designing, and displaying text*. Englewood Cliffs, NJ: Educational Technology Publications, Inc.

Jonassen, D.H. (1988). Designing structured hypertext and structuring access to hypertext. *Educational Technology, 18*(11), 13-16.

Jonassen, D.H. (1989). *Hypertext /hypermedia*. Englewood Cliffs, NJ: Educational Technology Publications, Inc.

Jonassen, D-H., & Cole, P. (1992). Learner-generated vs. instructor-provided analysis of semantic relationships. In M. R. Simonson & K. Jurasek, (Eds.), *14th Annual Proceedings of Selected Research and Development Presentations* (pp. 294-304). Washington, DC: Association for Educational Communications and Technology.

Jonassen, D.H., & Hawk, P.P. (1984). Using graphic organizers in instruction. *Information Design Journal, 4*(1), 58-68.

Jonassen, D.H., & Wang, S. (1992). Acquiring structural knowledge from semantically structured hypertext. In M. R. Simonson & K. Jurasek, (Eds.), *14th Annual Proceedings of Selected Research and Development Presentations* (pp. 294-304). Washington, DC: Association for Educational Communications and Technology.

Kenny, R.F., Grabowski, B.L., Middlemiss, M., & Van Neste-Kenny (1991, November). *The generative effects of graphic organizers with computer-based interactive video by global and analytic thinkers*. Paper presented at the annual meeting of the Association for the Development of Computer-Based Instructional Systems. St. Louis, MO. (ERIC Document Reproduction Service No. ED 341378).
Kenny, R.F. (in press). The generative effects of instructional organizers with computer-based interactive video. *Journal of Educational Computing Research*.

Kiewra, K.A. (1988). Notetaking and review strategies: Theoretical orientations, empirical findings and instructional practices. *Research and Teaching in Developmental Education, 4*(2), 5-17.

Kinzie, M.B., & Berdel, R.L. (1990). Design and use of hypermedia systems. *Educational Technology: Research and Development, 38*(3), 61-68.

Kloster, A.M., & Winne, P.H. (1989). The effects of different types of organizers on students' learning from text. *Journal of Educational Psychology, 81*(1), 9-15.

Krahn, C.G., & Blanchaer, M.C. (1986). Using an advance organizer to improve knowledge application by medical students in computer-based clinical simulations. *Journal of Computer-Based Instruction, 13*(3), 71-74.

Lawton, J-T., & Wanska, S.K. (1977). Advance organizers as a teaching strategy: areplyto Barnes and Clawson. *Review of Educational Research, 47*(1), 233-244.

Lenz, B.K, Alley, G.R., & Schumaker, J.B. (1986). Activating the inactive learner: Advance organizers in the secondary content classroom. *Learning Disability Quarterly, 10*, 53-67.

Luiten, J., Ames, W., & Ackerson, G. (1980). A meta-analysis of the effects of advance organizers on learning and retention. *American Educational Research Journal, 17*(2), 211-218.

Marks, C.B., Doctorow, M.J., & Wittrock, M.C. (1974). Word frequency and reading comprehension. *Journal of Educational Research, 67*, 259-262.

Mayer, R.E. (1979a). Can advance organizers influence meaningful learning? *Review of Educational Research, 49*(2), 371-383.

Mayer, R.E. (197913). Twenty years of research on advance organizers: assimilation theory is still the best predictor of effects. *Instructional Science, 8*, 133-167.

Macdonald-Ross, M. (1978). Language in texts. *Review of Research in Education, 6*,229-275.

McEneany, J.E. (1990). Do advance organizers facilitate learning? A review of subsumption theory. *Journal of Research and Development in Education, 23*(2), 89-96.

Moore, D-W., & Readance, J.E. (1984). A quantitative and qualitative review of graphic organizer research. *Journal of Educational Research, 78*(1), 11-17.

Novak, J.D., & Gowan, D.B. (1984). *Learning how to learn*. New York: Cambridge University Press.

Rieber, L.P., & Hannafin, M.J. (1988). Effects of textual and animated orienting activities and practice on learning from computer-based instruction. *Computers in Schools, 5*(1/2), 77-89.

Reynolds, S. B., & Dansereau, D.F. (1990). The knowledge hypermap: An alternative to hypertext. *Computers in Education, 14*(5), 409-416.
Reynolds, S. B., Patterson, M.E., Skaggs, L.P., & Dansereau, D.F. (1991). Knowledge hypermaps and cooperative learning. *Computers in Education, 15*(2), 167-173.

Rezabek, R.H., & Ragan, T.J. (1989, February). *Elaborated resources: An instructional design strategy for hypermedia*. Paper presented at the annual meeting of the Association for Educational Communication and Technology. Dallas, TX. (ERIC Document Reproduction Service No. ED 300 787).

Rumelhart, D.E., & Ortony, A. (1977). The representation of knowledge in memory. In R.C. Anderson, R.J. Spiro, W.E. Montague (Eds.), *Schooling and the acquisition of knowledge*. Hillsdale, N.J.: Lawrence Erlbaum Associates.

Schwier, R.A., & Misanchuk, E.R. (1993). *Interactive multimedia instruction*. Englewood Cliffs, NJ: Educational Technology Publications.

Stone, C.L. (1984). A Meta-analysis of advance organizer studies. *Journal of Experimental Education, 51*(4), 194-199.

Tajika, H., Taniguchi, A., Yamamoto, K., & Mayer, R. (1988). Effects of pictorial advance organizers on passage retention. *Contemporary Educational Psychology, 13*, 133-139.

Tripp, S.D., & Roby, W. (1990). Orientation and disorientation in a hypertext lexicon. *Journal of Computer-Based Instruction, 17*(4), 120-124.

Tripp, S.D., & Roby, W. (1991). The effects of congruent orienting strategies on learning from hypertext bilingual lexicon. *Journal of Hypermedia and Multimedia Studies, 2*(2), 6-12.

Wittrock, M.C. (1974). Learning as a generative process. *Educational Psychologist, 11*(2), 87-95.

Wittrock, M.C. (1985). Teaching learners generative strategies for enhancing reading comprehension. *Theory into Practice, 24*(2), 122-126.

Wittrock, M.C., & Alesandrini, K (1990). Generation of summaries and analogies and analytic and holistic abilities. *Arnetian Educational Research Journal, 27*(3), 489-502.

Wittrock, M.C., Marks, C., & Doctorow, M. (1975). Reading as a generative process. *Journal of Educational Psychology, 67*(4), 181-189.

Wolf, F.M. (1986). *Meta-analysis: Quantitative methods for research synthesis*. Beverly Hills, CA: Sage Publications.

Wolf, F.M. (1990). Methodological observations on bias. In KW. Wachter & M.L. Straf (Eds.). *The future of meta-analysis*. New York, NY: Russell Sage Foundation.
NOTES

1. The term “learning” will be used throughout this document to mean a measure of learning completed by the learner within 24 hours of having studied the material-to-be-learned. This is consistent with the use of the term in the research literature pertaining to advance organizers and is not meant to imply solely the immediate recall of facts or verbal information. “Retention” will refer to any measures of learning given 24 hours or more after the completion of the lesson.

AUTHOR

Richard F. Kenny is Assistant Professor of Educational Education, University of Ottawa, Ottawa, ON KIN 6N5.
# A Case Study: Development of Interactive Multi-Media Courseware in Dentistry

Benoit David
Arto Demirjian
Steven Shaw

**Abstract:** This article details a project undertaken at the School of Dentistry at the Université de Montréal concerning the development of multi-media computer-based instruction for clinical and anatomical subjects in dentistry. Under the direction of Dr. Arto Demirjian, six courses were developed on Macintosh and PC platforms in three languages (English, French and Spanish) and packaged on CD-ROM. The courses, produced between January 1992 and December 1993, are intended primarily for dentistry students but are also being marketed as a vehicle for continuing professional education of practicing dentists. The article describes the contents and design features of the various courses, outlines the human and technical resources which were implicated in their development and production and relates the approaches which were taken to promote their use.

**Résumé:** Cet article décrit un projet entrepris par la Faculté de médecine dentaire de l'Université de Montréal. Il s'agit de la mise en place d'un programme d'enseignement multimédia informatise pour les études cliniques et anatomiques du domaine dentaire. Sous la direction du Dr. Arto Demirjian, ce projet de six cours a été mis au point en trois langues (française, anglaise et espagnole) pour les systèmes Macintosh et PC, sur progiciel CD-ROM. Les cours, développés entre janvier 1992 et décembre 1993, s'adressent d'abord aux étudiants en médecine dentaire mais ils sont également adaptes à la formation professionnelle continue des dentistes praticens. Cet article donne un aperçu du contenu et des caractéristiques de conception des différents cours, il présente les ressources techniques et humaines qui ont contribué à leur élaboration et à leur production, et relate les démarches entreprises pour la promotion et l'utilisation de ceux-ci.

## INTRODUCTION

A two year research project in multi-media courseware development was undertaken at the Faculty of Dentistry of the Université de Montréal, from January 1992 until December 1993, under the direction of Dr. Arto Demirjian, professor of anatomy.

Funding for this project allowed us to assemble a diversified team of human resources, and provide up-to-date equipment and software, for both Macintosh and PC/Compatible platforms. The development team involved
several people with different backgrounds, providing a wide variety of re-
sources: content expert, designer, programmers and medical illustrator.

The courseware developed was intended primarily as a learning tool for
undergraduate students at different stages of the dentistry program. How-
ever, it was also envisaged that it would serve as a continuing education tool
for dentists who are practicing their profession. Some, of the courseware is
already in use by students and it is being evaluated in different parts of the
world, mainly in the United States, but also in Canada, Europe, Australia and
South America.

In this report we are going to describe the courseware, then address the
context in which the project was completed, the methods employed to achieve
the goals, and finally the promotion and diffusion of our work.

COURSEWARE

The following courses, presented in a chronological order of production,
are the tangible results of our efforts. Each successive module was treated as
an opportunity to design and test new interfaces and evaluate various
refinements concerning the interactive features.

Anatomy & Anaesthesia of the Mandibular Nerve (AAMN)

Written for dentists, professionals and students alike, this course, the
first to be produced during the project, describes the anatomy and anaesthesia
of the mandibular nerve as well as other anatomical structures in the area. It
contains three chapters of theoretical information (Osteology, Dissection and
Clinic) and a Quiz section.

The original version was developed prior to this project, by Dr. Demijian
and Richard Tanguay, with the authoring software SuperCard. That initial
version served as a prototype for the development of a second one, created with
Macromedia's Director software. Basically, the object was to improve the
interface and graphic design, while the content was mostly left untouched.

The content comprises more than 100 photographs, diagrams and medical
illustrations. The standard screen layout (Figure 1) presents an image on the
left, accompanied by a descriptive text on the right. The text contains
highlighted words that double as buttons. When clicked on, these button
activate the corresponding structure in the image, in a particular manner,
through animation or highlighting.

A location and access map of the contents is always available through the
use of a Where Am I?" button (Figure 2). This brings up a flowchart of the
different sections of the document with the present location of the user
highlighted. The user can also access any part of the content by clicking on
the corresponding section of the flowchart.

The Quiz section was originally one module consisting of multiple choice
questions. It was modified by adding a second module, an anatomical puzzle
<figure><img src="image_1.png" /><figcaption>**FIGURE 2.**
AAMN Access and Location Map</figcaption></figure>

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /><figcaption>**FIGURE 4.**
AAMN — Puzzle</figcaption></figure>

<figure><img src="image_4.png" /></figure>
of the cranial bones. The two modules work as follows:

« The multiple choice questionnaire (Figure 3) presents 68 questions in random order based on 10 photographs of dissection material, each having six to eight different anatomical structures highlighted in some way. The user must identify each of these structures from a list of possible answers that is provided. After two incorrect tries the correct answer is provided. The user's score is compiled and displayed in the bottom part of the screen.

« The puzzle (Figure 4), presents the bones of a skull dispersed about the screen. The object is to reassemble the skull by putting each bone back into its correct position. The software verifies if the chosen position is correct. A response is then given accordingly. If the position is not correct, the bone is returned to its original position.

## Dental Development

This interactive tutorial for the evaluation of Dental Age is based on the Bio-Age tutorial/database package, a product also developed at the School of Dentistry prior to this project.

The new Dental Development package was designed to offer a high-quality presentation of the findings of a 25 year research project on child growth and development. Aimed at dentists, orthodontists and pediatricians, the information given is divided into five sections: the Demirjian System, training modules for beginner and advanced users, a bibliography, a sample of longitudinal x-rays, and a clinical evaluation utility for practical application of the Demirjian System.

The section entitled "Demirjian System" presents the Dental Age Evaluation System developed by Dr. Demirjian. Each developmental stage is explained and illustrated with x-rays, photographs and diagrams (Figure 5).

Once the user has assimilated the theory, s/he can test himself or herself with the two self-assessment modules provided in the Training section: Beginner and Advanced. In both cases, the user is presented with dental X-rays on which specific teeth are pointed out. The object is to determine the stage of dental development reached by those teeth, out of three to four possibilities displayed in the lower part of the screen.

In the Beginner's module, the user chooses one dental x-ray out of 54 (nine x-rays for each of six patients, three boys and three girls, taken longitudinally from age six to 14) (Figure 6). Then he or she has to evaluate the dental age of the specified tooth. The information made available to the user includes the gender and age of the patient. The correct answer can be accessed through the use of a "Solution" button. The user can view all x-rays of the patient at the same time, and change x-ray or patient at any time (Figure 7).

With the Advanced module, the difficulty level was raised as follows: 378 questions are available in random order, and the only information displayed are the gender and age of the patient, the x-ray on which a tooth is pointed out, and the possible answers.
<figure><img src="image_5.png" /><figcaption>**FIGURE 6.**
AAMN — Dental Development X-ray Selection in the Self-Evaluation Section</figcaption></figure>

<figure><img src="image_6.png" /></figure>
<caption>FIGURE 7.</caption>

Dental Development - Beginner's Module in the Self-Evaluation Section

<figure><img src="image_7.png" /></figure>

The Bibliography section contains over 800 pages of text, tables and graphs, which represents the main body of literature published by Dr. Demirjian and others on the subject of dental development. Articles can be selected through an index or via abstracts, and are presented in scrollable fields (Figure 8), with their accompanying tables and graphs. There is a print function to generate hard copies. An external function (XFCN) was developed for that purpose since the authoring software used (namely Macromedia's Director) did not offer an elegant way to accomplish this task. When necessary, external functions like these are developed for both Macintosh (XFCN) and PC/Compatible (DLL Dynamic Linked Library) platforms.

The Sample X-Bays section shows a longitudinal collection of dental x-rays taken from the same patient from one and one-half to nineteen years of age. The images displayed were digitized with x-ray scanners, to present the user with the finest quality possible.

Finally, in the last section, the Clinical Evaluation, two calculation tools are made available as practical applications of the theory. These enable the users to calculate the dental age and maturity score of a patient, provided that x-rays of that patient are available. External functions were specifically developed for those tools, for both Macintosh and PC-compatible platforms.
<caption>**FIGURE 8.**</caption>

*Dental Development — Browsing Through Articles in the Bibliography Section*

<figure><img src="image_8.png" /></figure>

*Anatomical Puzzles*

The anatomical puzzle of the cranial bones, originally designed as part of the AAMN courseware, was supplemented with a hand/wrist area puzzle and questionnaire, to form a stand-alone instructional game, to offer different, fun and challenging ways to learn.

As previously described in the puzzle of the cranial bones, with the hand/wrist puzzle the user is asked to place the previously dispersed bones in their correct configuration. The software verifies if each bone is properly placed, and reacts accordingly. However, with the hand/wrist area, a puzzle and a questionnaire are offered as two different methods of learning:

* The puzzle asks the student to position the bones of each of the three groups — Phalanges, Metacarpals and Carpals — in their correct locations (Figure 9). While similar to the puzzle on the cranial bones, this puzzle does not verify if the bones are dropped in their correct location. Instead, an outline is made available to facilitate the positioning of each bone. The user can hide or show that outline in order to modify the difficulty level.

* With the questionnaire, on the other hand, hand and wrist bones are displayed correctly assembled, and the student is asked to click on specific bones, one after the other, randomly, until there is none left (Figure 10). The user is given three chances to correctly identify each bone, before the correct answer is revealed. The score is compiled and displayed on the left side of the screen.
<figure><img src="image_9.png" /><figcaption>**FIGURE 10.**
Anatomical Puzzles — Hand Questionnaire</figcaption></figure>

<figure><img src="image_10.png" /></figure>
## Masticatory Muscles

This courseware was prepared for first year medical and dental students, but it is also intended as a continuing education tool for health professionals. Three groups of muscles, the infra-hyoid, the supra-hyoid and the Masticatory group per se, are defined and an explanation of their origin, insertion, innervation and function is given. The contents of the course includes 21 high-quality dissection photographs (most were obtained from the Bassett Collection, at Stanford University) and several medical illustrations, which were commissioned specifically for this document (Figure 11).

<caption>FIGURE 11.</caption>

Masticatory Muscles — Typical Screen layout with Medical Illustration

<figure><img src="image_11.png" /></figure>

As in the AAMN courseware, the standard screen layout shows an image on the left, with an accompanying text on the right in which highlighted words double as buttons (Figure 12). When clicked on, those buttons identify in some way the corresponding anatomical structures in the image, with special effects created to emphasize their characteristics. Basically, muscles and bone structures are animated, or "moved", while veins, arteries and nerves are literally "highlighted". Digital movies are also included at one point to illustrate some of the muscle group's functions. In this courseware, we tried scrollable text fields (Mac version), instead of flipping pages. The text window is relatively small and we wanted to see if it would be less confusing to the user, since there would be fewer screen changes. Again, as in the AAMN courseware, a location and access map is made available by a click of the mouse.
<figure><img src="image_12.png" /><figcaption>**FIGURE 12.**
Masticatory Muscles — Typical Screen Layout with Dissection Material</figcaption></figure>

The Quiz section was divided into two modules: one with multiple choice items, and the other with short answer questions. These questionnaires were developed with Macromedia's Authorware Professional authoring software. The first module, a multiple choice questionnaire based on dissection photographs, is basically the same as the questionnaire in the AAMN courseware. The second one, requesting short answers, was designed to have the student use the keyboard to enter responses to randomly asked questions. While very interesting, we found that this kind of self-assessment tool implies a lot more preparation, and is longer to develop and test, because of the larger number of different good answers that can be given.

*Electronic Encyclopedia on Maxillo-facial, Dental and Skeletal Growth*

Due to its complexity, this document has been under continuous development from almost the start of the project.

The Encyclopedia on Skeletal Growth supersedes the Biological Age Tutorial/Database (containing over 2000 x-rays) which was developed by Dr. Demijian prior to this project. It is a new collection of hand/wrist, dental and cephalometric x-rays of children aged zero to nineteen, acquired during the twenty-five year research project on the subject of child growth and development conducted by Dr. Demijian at the Human Growth Center of the de Montreal, referred to earlier. This collection contains over 4000
x-rays (providing over 450 MB of data).

The Encyclopedia is based on a custom built relational database, developed by Richard Cheng, which is designed to answer complex inquiries rapidly. Multiple parameters (dental, skeletal, chronological age, height, weight and sex) may be combined together to create either simple or very complex search requests, which may be completed either longitudinally (to compare growth of children at different ages) or cross-sectionally (comparing different children of the same age or at different ages) (Figure 13).

Once a query has been launched, the result of the search, when successful, is presented to the user as a list of files responding to the parameters that have been set (Figure 14). The user can select any file from the list, and view its data (numerical and visual) by clicking on one of the following categories: skeletal (hand-wrist area), cephalometric or dental (panorex) (Figure 15). At that point the user can change category at will, and can always come back to modify or create a new search request.

All the x-rays have been digitized at high resolution and are displayed in 256 grays, to provide highly accurate pictures. An on-line help (always available) is provided to inform the user regarding what he or she sees on the screen: the different parameters, their definition, what they are used for.

<caption>FIGURE 13.</caption>

Electronic Encyclopedia Selection of Search Parameters

<figure><img src="image_13.png" /></figure>
<figure><img src="image_14.png" /><figcaption>**FIGURE 15.**
*Electronic Encyclopedia — Visualization of Subject's Data and X-rays*</figcaption></figure>

<figure><img src="image_15.png" /></figure>
## *The Temporo-mandibular Joint (TMJ)*

This document is currently under development. It is being produced with Macromedia's Director on the Macintosh to be played back on both Macintosh and PC/Compatible platforms. This courseware is being developed with all the positive aspects of the previous documents, plus new options (Figure 16). The newest feature provides monitoring of user access throughout the courseware. The information collected is recorded to external files under a user specified personal code. This allows the user to view what he or she has seen and, by the same token, know what he or she has yet to see. The Access and Location map, always available, will display this information through flowcharts of the different sections of the document. Those sections will be marked in the following manner: what is underlined will show the user where he or she is, what is italicized will show what he or she has seen, and what is bold will show what is left to see (Figure 17). This user code will also be used to record "bookmarks" inserted by the user at specific locations for later reference. The user will also be able to take notes, which will be recorded externally either for later reference or to be printed.

Review Questions are provided in the Main Menu as an advisement strategy to the user. The user has the choice of reviewing the questions with or without answers. If the questions cannot be answered correctly, it is assumed that the user will go back to the corresponding chapters for review.

<caption>FIGURE 16.</caption>

<figure><img src="image_16.png" /><figcaption>TMJ — Typical Screen Layout</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>FIGURE 17. TMJ -Access and Location Map</figcaption></figure>

Different types of quizzes are provided in the Self-assessment section to test the knowledge acquired in the tutorial. Two content questionnaires, one Multiple Choice (Figure 18) and the other requiring Short Answers (Figure 19) are provided. There is also one visual Multiple Choice questionnaire, based on dissection room material and x-ray images. A FileMakerPro template has been created to produce the questions and answers included in the questionnaires. Once they are completed, external files are exported which are then read by the self-assessment module. This particular facility allows easier management of quiz material, and facilitates the modification of the contents of the questionnaires as desired.

OBJECTIVES

As a research and development project, the primary objectives were as follows:

* To design and develop computerized teaching or learning tools. Subject-matter was selected from Dr. Demijian's own teaching and research material, and prepared to be developed into computerized documents.

* To search for, test, select and evaluate the different tools necessary to produce high quality documents. Keeping in mind that a wide variety of media can be used to communicate information, whether visual or audio, still
<caption>FIGURE 19.</caption>

TMJ — Mu/tip/e Choice Questionnaire in the Self-Assessment Section

<figure><img src="image_18.png" /><figcaption>FIGURE 19.</figcaption></figure>

TMJ — Short Answer Questionnaire in the Self-Assessment Section

<figure><img src="image_19.png" /></figure>
or animated, textual or graphic, we needed to find the appropriate hardware and software to capture and incorporate the necessary media into the courseware in an efficient manner.

To assess the implications of producing such tools, as well as managing them, in terms of human and technological resources. Even if this project was to be terminated at a known point in time, we wanted to work as if it was a regular activity of a permanent service inside the faculty. We had to be careful not to get caught by the "one shot deal" syndrome, in order to develop a standard and practical procedure to produce interactive documents.

To promote technology-based learning inside Dentistry Programs. It was important to expose people to our work; not only students and faculty members within our own environment but also individuals outside the boundaries of the school, in other learning environments around the world.

## RESOURCES

### Human Resources

Many people were involved in the course of this project, with specialties that complemented one another.

* Project Director and Content Expert. He managed the project, and acted as the primary content expert. A professor of Anatomy at the Faculty of Dentistry of the Universite de Montreal, he thought out, defined, and wrote the content of the courseware. Other experts were also consulted when necessary.

* Multi-media Specialist. An industrial designer by training, his tasks were: provide support to the expert for the preliminary gathering and organization of the information; evaluate the human and technical resources needed; conduct instructional, interactive and navigational design; conduct analysis for the scripting on the Macintosh platform; interface design; production of the Macintosh versions; supervision of the production for the PC versions. In parallel to the actual production of the courseware, he managed the Macintosh network and the production of support material, such as the user's booklet, brochures, and disk labels. He also participated actively in public events to promote the courseware at conferences and expositions.

* Authoring Programmer. Responsible for the production of the PC versions of the courseware, using Toolbook and Authorware Professional.

* Programmer. Responsible for the production of the Electronic Encyclopedia on Maxillo-facial, Dental and Skeletal Growth including the development of the relational database with db Vista, and the front-end for the PC version with Toolbook.

* Medical Illustrator. Responsible for the production of medical illustrations of subjects difficult or impossible to photograph.

* Scanning Technicians. To digitize the x-rays (over 5000) to be included in the Electronic Encyclopedia, we required the services of the equivalent of
one full time technician for a year and a half.

*Programming Consultant*. Some part of the courseware development required the services of a programming consultant, mainly to develop and produce external functions for the Macintosh versions (XFCN), and to assist in their development for the PC versions (DLL).

*Translators*. Having taken care of the English and French translations ourselves, we required the services of two Spanish speaking dentistry students to produce the Spanish versions of three courses. Their work was of course verified and adjusted by Spanish speaking content experts.

*Technical Support*. Using new technologies one inevitably must rely on various sources of technical support including hotlines provided by software developers and hardware manufacturers, after sales support services from the dealers, and exchange of information through the Internet network.

## Technological Resources

Having to develop for both Macintosh & PC/Compatible platforms, we organized and structured the department to maximize the exchange and transfer of information among people and workstations.

An ethernet network was first implemented to link all workstations. Two file servers were included on the network: one dedicated for Macintosh development, the other linking all platforms (Macintosh, PC/Compatible and UNIX). Considering the large number of documents that would be created and manipulated, a backup scheme was organized to prevent any loss of data.

Through the development stages, many different configurations of workstations were used to ensure compatibility between courseware and hardware: various peripherals, accessories, and various CPU performance and clock speed (MHz) were integrated. On the Macintosh side, we used a IIsi as the lower end, up to the Quadra 900 for higher performance. As for the PC/ Compatible platform, it ranged from an aging IBM 386 running at 20 MHz, to more recent 486 models running at up to 66 MHz. Many peripherals were intentionally acquired from different manufacturers to diversify the stations' configurations, thus giving us more chance to experiment and to detect and address any incompatibility of platforms with our documents.

As with most multi-media documents, our courseware needed to incorporate various types of information presented in as many different media: artwork, photographs, slides, x-rays, video, sounds. The following peripherals were used to acquire that information: reflective and transparency scanners, slide scanners (Figure 20), 35mm digital camera, 8mm S-VHS video camera, microphones, CD-ROM players.

Once a unit of courseware was finally completed, meaning that the English, French and Spanish versions of both Mac and PC platforms were finished and verified, all documents were written to a compact disk (Figure 21), in hybrid format, tested, and if necessary, sent to a production plant for mass reproduction.
<figure><img src="image_20.png" /><figcaption>**FIGURE 21.**
*Compact Disk Writer*</figcaption></figure>

<figure><img src="image_21.png" /></figure>
DEVELOPMENT, DESIGN & PRODUCTION

The first step taken was to define, structure and prepare the content of the courseware-to-be (Figure 22). That part was achieved by Dr. Demirjian himself, the content expert, with the occasional input of other experts to verify the correctness of the information to be included. We found that it is important to finalize the written content, translate it if necessary, and get it approved before the design process is engaged in order to prevent as much as possible any delay in production once a project is under way.

<figure><img src="image_22.png" /><figcaption>FIGURE 22.
Sample Now Chart of Comfort</figcaption></figure>
Once approved, the content was given on paper to the multi-media specialist. Private tutoring sessions were provided for him so that he could assimilate as much as possible the nature of the material. The expert and the multi-media specialist then evaluated existing media such as slides, artwork or video, and discussed new media to be obtained, like medical illustrations, new pictures from the dissection room, or 3-D models to be animated in some way.

Then the design process was put into motion, starting with information and navigational design (based on the information flowchart), which defined the bases for interactive navigation, as well as determining what kind of options would be offered to the user. This naturally brought about the interface design: what the user would see, what would the items displayed on the screen tell him or her, how those items would react upon the user's actions.

To create this prototype of the courseware, a small part of the content was used (typically one chapter) which provided typical situations representative of the whole document with respect to, for example, maximum length of titles, or maximum number of buttons on the screen at the same time. That prototype was then evaluated in respect of the design objectives, and tested for bugs in the scripting (programming).

Each unit of courseware was used as a new opportunity to test various interactive options and interfaces, to try out different production approaches (hardware and software) and to determine the best way to produce multi-platform, multi-language courseware.

## Tools

The original intent was to develop the courseware using Macromedia's Director, and take advantage of the Windows Player utility to port the finished documents over to the PC for immediate playback. But we had to take a different approach since that utility was not to be fully reliable for another year and a half after we began production.

As a general rule, the design and development of each courseware was done on the Macintosh, using primarily Macromedia's Director authoring software. Of course, a variety of other software were used in the process, for example Adobe's PhotoShop for image manipulation and retouching, or Macromedia's SoundEdit Pro for sound grabbing.

Once the prototype was approved, the first fully functional version was completed (usually the English version for Macintosh), tested, verified and approved as to its functionality and content. The other versions (translations) for the Mac were then produced, while the PC versions were created from scratch with other software, using the Macintosh documents as a guideline.

To create the PC versions, we had to choose another authoring software that would come as close as possible to Director's performance, while waiting for Macromedia to improve their Windows Player utility. We first used
Toolbook, but its lack of visual or special effects kept us on the lookout for another solution. Then we tried Authorware, because of its cross-platform transferability. The original documents were developed with Authorware for the Macintosh, then transferred to the PC and re-opened with Authorware for Windows. The transfer results were good, but we were dissatisfied with Authorware's overall speed, its restricted scripting environment, and its lack of visual effects (again we were comparing it to the Director version on the Macintosh). It's positive aspect was its built-in functions and variables which prove to be very useful when creating a document to be used as an evaluation tool, like a self-assessment, questionnaire or quiz. But since it was possible to achieve the same thing with Director, we were still not fully satisfied.

When Macromedia released a reliable version of the Windows Player in September 1993, we re-oriented our strategy toward developing a Mac version transferable over to the PC, for immediate playback. It meant changing some design aspects, because of minor incompatibilities between platforms, but we felt it was worth it because the overall result was very acceptable, and there were important savings in production time.

Finally, these efforts brought us to emphasize the use of Director first to design and develop the prototype, then to use it with the Player software (for both Mac & PC) to produce the tutorial part of the courseware. At the same time we retained Authorware as a tool to develop the self-evaluation modules, again for both platforms.

## Promotion

Once we had some documents to work with, we started promoting the idea of using these as learning tools in the curriculum, using different approaches.

Dr. Demirjian started using the courseware in the classroom as part of his teaching material. The documents are displayed to the class with a Macintosh IIci, equipped with an overhead projector and a color LCD projection panel.

We also designed a kiosk, to be installed in the corridors of the Faculty of Dentistry, so that students and faculty members could be exposed to our work and access it whenever they felt like it. That kiosk is also used to demonstrate its possibilities in special events taking place in Montreal, like conventions and expositions (Figure 23).

A few computers (Macs & PC's), were also made available to students to use these documents in a more comfortable setting, with technical support for the unfamiliar users.

Having access to the Internet network gave us the opportunity to make our work known all over the world. Demonstration packages were made available to anybody using the network, for them to download and evaluate. The network even turned out to be one of the best ways to promote our documents right here in Montreal!

Finally we produced a brochure, or catalogue, of our documents with all the necessary descriptive and technical information. This catalogue is distrib-
<figure><img src="image_23.png" /><figcaption>FIGURE 23.
*Interactive Kiosk*</figcaption></figure>

uted to individuals, students and professionals, universities, and libraries, all over the world.

Conclusion

This experience has proven to be very positive from a research and development point of view. The expectations were largely surpassed, due mainly to the fact that new possibilities were constantly uncovered as we progressed through each new project, allowing us to experiment further.

Though vary well received by undergraduate students, it took a long time to get some basic recognition from other faculty members, administrators, and practicing health professionals. And it will surely take more time to see some kind of real implementation in the curriculum of undergraduate studies as well as for continuing education. That time frame will depend mostly on how many experts will come forward and invest themselves to produce new subject-matter for development of more titles.

Once a critical mass of titles is available, both on fundamental and clinical aspects, we expect interest in using multi-media instructional materials will increase. At the moment, production of such computer-based materials in the field of dentistry owes much to the commitment and vision of a relatively small number of developers. It is not yet market driven.
Even with the involvement of well-established specialized people in the marketing field, we found that commercial distribution of such materials is very difficult. Several factors are probably responsible, such as the product's novelty, its specialized aspect, the relatively small number of potential buyers, and the difficulty of integrating the material with other faculty's curricula. We look forward to a point in the not so distant future when usage of interactive multi-media courseware will be considered a standard teaching and learning strategy. Development of such material will then be more easily justified.

AUTHORS

Benoit David is an Industrial Designer and Multimedia Specialist. He is currently a Graduate Student in Educational Technology at Concordia University, Montreal, Quebec.

Arto Demirjian is Professor of Anatomy at the Faculty of Dentistry, Universite de Montreal, Quebec.

Steven Shaw is Assistant Professor of Educational Technology at Concordia University, Montreal, Quebec.
Educational Technology Adoption and
Implementation: Learning from
Information Systems Research

Karen Lee Jost

Scott L. Schneberger

**Abstract:** Many educational academics believed that educational technology would play a large part in educational reform. A survey of the literature, however, illustrates the low impact that educational technology has had on established education with few implementation successes. These remarks are similar to those heard in the information systems field during the 1980s. A considerable amount of research was done on information systems adoption and implementation. It is proposed that the root problems and issues in Educational Technology adoption are not unique to education and educational technology-but problems faced by proponents of any new technology and, in particular, by those in Information Systems. If the theoretical and practical parallels between Educational Technology and Information Systems adoption hold. Educational Technology can benefit from what Information Systems has learned about adoption. Conceptual and historical similarites between educational technology and information systems are explored IS adoption theory and research is explained. and recommendations for ET adop: tion and implementation are discussed.

Résumé: Un grand nombre d'universitaires experts en éducation ont cru que la technologie éducative jouerait un grand rôle dans la réforme pédagogique. Une étude de la documentation écrite sur le sujet a toutefois révélé qu'elle a eu peu d'impact sur l'éducation traditionnelle et peu de succès dans l'application de ses principes. Ces remarques rappellent d'ailleurs cellesqui ont été faites au cours des années '80 sur les systèmes d'information.

La recherche effectuée sur l'adoption et l'application des systèmes d'information est considérable. Nous pensons que les problèmes fondamentaux et les questions portant sur l'adoption de la technologie éducative ne touchent pas seulement l'éducation et la technologie éducative mais sont des problèmes qui confrontent les tenants de toute nouvelle technologie et, en particulier, les tenants des systèmes d'information. Si les parallèles théoriques et pratiques établis entre l'adoption de la technique éducative et des systèmes d'information tiennent, la technique Educative peut bénéficier de ce que les systèmes d'information ont appris à propos de cette adoption,

Les similarités conceptuelles et historiques entre la technologie éducative et les systèmes d'information sont étudiées ici: la théorie de l'adoption et la recherche en systèmes d'information sont expliquees; et, les recommandations pour l'adoption et l'application de la technologie éducative font l'objet de discussion.
INTRODUCTION

Educational reform is a leading topic in socio-political debate, and was one of the key issues in the latest U.S. presidential election. Many educators envisioned educational technology (ET) becoming a large part of that reform. Heinich (1970), for example, foresaw a major role for educational technology as a tool to support teachers, as a replacement for teachers and a conduit for directly educating students, and/or as a means to forming a partnership with teachers whereby technology delivers routine instruction and teachers focus on planning and educational management. Indeed, interest in educational technology and some practical successes during the last 50 years led educational technology to become a unique academic and professional field. The field has academic departments and courses, professional organizations, journals and conferences, academic professionals who identify themselves as educational technologists, and of particular significance, a considerable amount of scientific research. One could easily assume that with so much interest in educational improvements, with so much potential for educational technology as part of the expanding information age of technology, and with all of the research within the field of educational technology, deep and broad improvements in established education would have resulted.

Many educational technologists, however, lament what they perceive to be few implementation successes and a decidedly low impact of educational technology on established education (Reigeluth, 1989; Winn, 1989; Gentry and Csete, 1991; Heinich, 1991). Educational technology is said to be an applied field, yet its knowledge, based on empirical research, is not applied by practitioners to the degree expected.

Many authors, publishing within the field of educational technology, have analyzed the problem and blamed a wide range of factors. Some view teachers themselves as the culprits; citing the idea that teachers are threatened by perceived professional irrelevance that would cause them to naturally resist educational technology (e.g. Heinich, 1991). Other authors blame simple bureaucratic inertia and lack of educational funding (Gentry and Csete, 1991). While there have been many accusations concerning weak adoption of educational technology in general education, some educational technologists criticize research as a malefactor, it is either too descriptive and not prescriptive enough (Clark, 1989), it is based on too many confusing or conflicting theories (Ross and Morrison, 1989), the research simply lacks external validity to everyday situations (Reigeluth, 1989), or that it fails to take advantage of related research in other fields (Clark, 1989).

The problem of innovation and adoption (and ruminating self-examination) is not, however, unlike what occurred in the information systems (IS) field during the early 1980s before the widespread proliferation of personal computers and readily available commercial software packages. A considerable amount of information systems research and writing has been done on who, what, when, where and why (or why not) information systems are
adopted, including research on why some information systems are adopted but then not used. It is unlikely that the problems with the adoption of educational technology innovations are entirely unique to education and educational technology. Instead they include problems commonly faced by proponents of any new technology. As stated by Fullan (1993), "change of all kinds has certain generic properties in complex societies" (p. vii).

This paper concentrates on the *how* problems of educational technology adoption. It presumes the validity of educational technology research on effective and efficient innovations and focuses instead on the adoption and implementation process and its factors. This paper outlines and compares the conceptual definitions of educational technology and information systems and relates the histories of ET and IS adoption. It outlines and explains information systems adoption paradigms, models, and frameworks and suggests similarities and differences between IS adoption/implementation and educational change. Finally, this paper discusses educational change and what can be learned from information systems adoption models.

## DEFINITIONS

One of the immediate issues in discussing the educational technology adoption problem is the varying definitions of educational technology. The Association for Educational Communications and Technology (AECT) defines *educational technology* as:

> a complex, integrated process involving people, procedures, ideas, devices and organization, for analyzing problems, and devising, implementing, evaluating and managing solutions to those problems, involved in all aspects of human learning.
> 
> (AECT, 1977, p.59)

Others define educational technology as a methodology or set of techniques (Cleary et al., as cited by Gentry, 1991), a "body of knowledge" (Dieuzeide, 1971, p.1) and as procedures and devices (Silverman, as cited by Gentry, 1991).

Instructional technology (IT), a phrase frequently used interchangeably with educational technology, often carries two connotations, The definition stated by the Presidential Commission on Instructional Technology (1970) includes both the view of instructional technology as:

> the media born of the communications revolution which can be used for instructional purposes along side the teacher, textbook, and blackboard. (p.19)
and as:

a systematic way of designing, carrying out, and evaluating the total process of learning and teaching in terms of specific objectives, based on research in human learning and communications, and employing a combination of human and non-human resources, to bring about more effective instruction (p.19).

Engler (1972) similarly defines instructional technology within two categories, the first as "hardware-television, motion pictures, audio tapes and discs, textbooks, blackboards, and so on" and secondly as:

a process by means of which we apply the research findings of the behavioral sciences to the problems of instruction. (p. 59)

One should not view these definitions of educational technology (and instructional technology) as nebulous, contradictory or exclusive definitions, but rather as inclusive definitions to bound the area of interest. By combining the essence of the definitions above (and others cited by Gentry, 1991), this paper defines *educational technology* as:

the application of people, devices, knowledge, and procedures for efficient and effective education.

This definition, resulting from combining the varying ET and IT definitions has a clear correlation with a common textbook definition of *computer-based information systems* (e.g. Davis and Olson, 1985; Laudon and Laudon, 1993):

devices (usually computer hardware and software-based), people, and procedures for organizing, storing, accessing, and maintaining information.

The definitions of educational technology and information systems identically focused on devices, people, knowledge, and process suggest a theoretical linkage between ET and IS application and adoption. The problems and issues associated with adopting information systems appear to have direct bearing on the problems and issues with adopting educational technology. Information systems research on adoption, therefore, would seem to offer rich insight and direction for fruitful educational technology adoption research. If the theoretical and practical parallels between ET and IS adoption hold, what information systems has learned about IS adoption may be what educational technology can benefit from in the future.
EDUCATIONAL CHANGE AND TECHNOLOGY ADOPTION:
A SUMMARIZED HISTORY

Fullan (1991) identifies four distinct historical phases in educational
change: adoption (1960s), implementation failure (1970-77), implementation
success (1978-82), and intensification versus restructuring (1983-90).

Fullan's first phase (adoption) came largely as a result of Soviet success
in launching a satellite in 1957 (years before the U.S.). The subsequent
"Sputnik crisis" led to large-scale curriculum innovations, technologically-
oriented instructional systems, and the advocacy of inquiry-oriented and
student-centered instruction. In the rush to meet the crisis, according to
Fullan, the emphasis was on how many innovations could be adopted, the
more the better as a mark of progress. During this period instructional
systems were researched and developed. Significant federal funding for R &
D laboratories, mandated evaluation of federally funded educational projects,
and the redefinition of audiovisual instruction to include instructional
development and technology gave the field of educational technology
increased visibility and credibility with educators.

During the 1970s however, innovation got a bad name. According to
Fullan, the 1960s' innovations had been adopted haphazardly with little
follow-through, leading to pronounced implementation failures. By the end
of the 1970s nevertheless, there were some significant, well-documented
successes that provided important frameworks and theories for
comprehensive educational reforms. The comprehensive reform movements
that began in 1983 (as a result of the watershed document *A Nation at Risk*
by the National Commission on Excellence in Education) took many
approaches, including the use of educational technology.

The advent of microcomputers in the 1980s appeared to offer the dawn of
a new era with computer-based instructional systems. The wide availability
of relatively inexpensive desktop computers, the capabilities of computer-
driven media, and the inherent ease of developing, using, and improving
software, provided a ready vehicle for applying educational technology. By
1989, 76,395 of the 79,693 U.S. public schools had two or more
microcomputers, averaging about 20 per school (Quality Education Data,
1989). Their use, however, was primarily for administrative and clerical
applications and not for the process of teaching and learning. The most
common educational use of microcomputers was limited to teaching computer
literacy (Ely, 1991). Higher education was not reported as any better; the
average U.S. university, in terms of its use of information technology in
teaching, was substantially behind the typical elementary and secondary
school (Newman, 1989 as cited by Ely 1991).
THE INFORMATION SYSTEMS ADOPTION PROBLEM A BRIEF HISTORY AND COMPARISON TO THE ADOPTION OF EDUCATIONAL TECHNOLOGY WITHIN EDUCATION

Although electronic computers were used for military purposes in the 1940s the public application of computers for information processing began in 1954 when one of the first computers was installed to process payroll at a large U.S. corporation (Davis and Olson, 1985). There have been three generally recognized eras in Information Systems adoption.

The first era was from 1954 to about 1964 when computers were used for accounting and clerical applications in major organizations. Information systems were expensive and very difficult to use. Few people understood how they worked, even fewer knew what to do with them. There was a wealth of research and theory that predicted enormous benefits from computers in everyday business and personal life, but potential users and those in management positions could only wonder at the futuristic predictions while continuing traditional work habits. With considerable simplification, this era roughly equates to the adoption of educational technology innovations in education prior to 1983.

During the second era, from around 1965-1980, the breadth of applications expanded due to improved general purpose programming languages. Major businesses saw computers as a strategic weapon, or at least an image maker, and management began to see the potential *efficiency* benefits from computers. There were large investments in computers and one-of-a-kind application software. Computers were ensconced within glass "throne rooms" tended to by computer specialists who were intermediaries to users of computed data. Users still did not understand computers or their potential, but they began to be exposed to the effects of computing. People were mostly forced to adapt to computers and, increasingly, to depend on them for record keeping as well as finance and accounting. These systems were designed by computer specialists who tended to oversell capabilities, had little understanding of user needs, and increasingly built systems that either did not work, went way over budget, or users would not use. Management perceived the importance of computers, but not how to apply them. As the chief strategist for a major U.S. bank said:

> [Computer] technology is our top strategic concern, not because it outweighs everything else, but because we are unsure what to do with it. Although we have a strategy for the marketplace, the technology issues seem to be eluding us. We can't seem to grasp the bigger picture (Parsons, 1983).

Information systems academics and professionals bemoaned the death of effective IS applications taking advantage of empirical research, while man-
agement complained that IS research was not practical enough or relevant.

This second IS era seems to correlate with the present state of educational technology research, development, and adoption. During the 1980s educational technologists also foresaw the importance of the use of technology in education. Educational software increased in availability and became more "user-friendly." Innovators have, however, made mistakes similar to IS designers. Educational programs and products have often been designed by specialists who do not understand the user (teacher) or the classroom learning environment. These innovations, therefore, are not implemented as the designers intended. Thus, while the use of technology has grown, it has not dramatically impacted on educational practice in general. Most educators like their IS counterparts in the second IS era, still do not seem quite sure of what to do with the technology.

The third era began with the advent of microcomputers around 1980. The entire mindset of users adapting to computers was reversed as powerful applications that *adapted to users* were mass-produced and were made commercially available. Moreover, non-procedural programming languages allowed non-programmers to write software specifically tailored to their needs, conditions, and location. Simultaneous communication innovations that digitally tied computers together allowed the full potential (widely predicted by researchers in the 1940s) of computers and IS to overcome time and distance. For most industries, information systems was no longer a service, or simply a medium for information; it had become the core impetus for an entire *r-e-engineering* of organizational processes. The second era issues about *what* can be done with information systems became third era *how* issues as new, practical applications spread. Information systems researchers began to struggle just to keep up with IS practice, let alone perform research that was not obsolete before it was published.

This third IS era parallels current trends in educational technology in the 90s. The third ET era exists in the literature and in isolated settings. The full potential of computer and communications technologies has yet to be utilized within the educational system as a whole. True organizational change has occurred in a limited number of specific school settings.

Given the theoretical and practical parallels between educational technology and information systems, educational technology should explore information systems research on adoption and implementation for insights and guidance.

## INFORMATION SYSTEMS ADOPTION THEORY AND RESEARCH

Information systems research on IS adoption and implementation has been ongoing since the 1950s with the earliest computer system applications. By the 1980s implementation was one of the four most heavily researched areas within the discipline of information systems (Culnan and Swanson,
1986). Two basic adoption paradigms were used for research: factors and process.

Factor Paradigm. The factor paradigm, the dominant paradigm in information systems implementation research, sought to identify and relate the many factors involved in IS implementation success, the what behind successful adoption. Six key variables have been identified from scores of empirical research and analysis efforts:

1) organizational need and support

2) user personal stake in success

3) user assessment of system and organizational support for it

4) user acceptance of system

5) use of system

6) satisfaction (Lucas, Ginsberg and Schultz, 1990).

These factors are linked into a generic model for IS implementation as shown in Figure 1.

<figure><img src="image_24.png" /><figcaption>Figure 1.
Information Systems Implementation Factor Model</figcaption></figure>
In this model, management support for a system, organizational changes required as a result of the system, and the urgency of the problem the system is supposed to address combine to affect the user's perceived stake in the system's adoption. User stake, in turn, influences user perception about the system (how efficiently and effectively it works toward the user's goals) as well as the organizational support behind the system (e.g., corrective maintenance, improvement, supplies). The user's perception of the system and its organizational support in turn directly affects the user's acceptance of the system, in addition to the technical characteristics of the system and the characteristics of the user. User acceptance, overt organizational support, and the user's personal stake in the system then determine how (or whether) the system is used. Experience using the system then directly determines satisfaction with the system from a user and organizational standpoint. Also generally believed to be important factors (but not empirically confirmed with strong data or consistently among researchers) are: user knowledge of the system purpose, user decision making style, user job characteristics, user/designer joint system development, and user knowledge of the system (Lucas, Ginzberg and Schultz, 1990). Underlying the entire model is the assumption that user acceptance and use are voluntary; the model changes considerably when system use is mandatory.

Under the IS factor implementation model, adoption and successful implementation largely depend on:

1) gaining support and commitment from the user's management (e.g., funding, job re-design, organizational changes, rewards and incentives, operational support and training);

2) seeking out potential users as early adopters who have a significant personal stake in the problem the system is designed to address, directly involving them in the design process, designing the system to target their technical needs as well as personal characteristics, and focusing attention on their adoption and early use; and

3) ensuring that the system addresses user personal stakes in system use.

*Process Paradigm*. This paradigm for information systems adoption and implementation research addresses the process of organizational change and management support behind system adoption. This paradigm takes the standpoint that systems simply address organizational and user change needs and provide a vehicle to implement those change needs. Therefore, *how* one implements a technological change is the key in this paradigm to successful adoption and use. Three models are prevalent in the IS adoption and implementation research under the process paradigm: *technological imperative, organizational imperative, and emergentperspective* (Keil, 1991).
The technological imperative model is based on the sociological assumption that external forces (the environment) cause internal changes, namely technological changes, to user behavior. In consonance with innovation theories, this model revolves around two change process factors: the technological advantage the system provides a user in performing his or her functions, and the system's ease of use. Together, these process factors determine system use. To promote adoption, management ensures that the system provides technological advantages (or at least that the benefits outweigh the detriments) and that the system is technologically easy to use. Management's agents to this end are IS specialists who are trained in systems, the organization, how to elicit requirements, and how to appropriately design systems for the users. This model is consistent under voluntary or mandatory use situations.

The organizational imperative model assumes that people are causative decision makers in anticipation or in response to environmental changes. Successful adoption and implementation therefore depend on successfully managing the decision making and implementation processes. This model, primarily based on the change and innovation work of Lewin (1947), consists of three phases. According to this model, successful change depends on **unfreezing** a situation by creating a climate or motivation for change. The second phase consists of the actual **change** based on analysis, design, development, implementation and training for a system and the organizational changes that must accompany the system. The final phase requires **refreezing** by institutionalizing the new system (with resulting organizational stability). This model (as shown in Figure 2) emphasizes that an organization with stable political, personal, and social coalitions must first be disturbed before change can be accepted. Although there are many roles (e.g., the user, management, IS developers), management plays the key organizational role in directing the change process.

<caption>Figure 2.</caption>

Organizational Imperative Model for IS Adoption and Implementation

<figure><img src="image_25.png" /></figure>
The key to adoption according to this model, therefore, is management awareness of the need for change, awareness and support for a change vehicle (the system with attendant personnel, data, process and organizational structure changes), determination and follow-through on changes, and institutionalization of the changes. While this model is associated more with mandatory than voluntary IS adoption, it can apply equally to both situations. Based on the managerial approach to implementing the change, management can serve as a catalyst to user change as well as an orchestrator.

The final model under the process paradigm is the *emergent perspective model* that assumes people and technologies interact in unpredictable ways. What is important is perpetually adjusting that interaction in response to uncovered barriers to success, as shown in Figure 3 (Leonard-Barton, 1988). The important point of this model is that there must be mutual adaptation of technological systems and the organization (including the organizational structure, its management, support, and the users). Change is assumed to be the norm, whether from internal or external environmental forces. No technological system, the model presumes, can ever satisfy all organizational needs forever and will therefore require continual, incremental changes. Likewise, no organization can remain static in light of technological changes or opportunities provided by systems.

<figure><img src="image_26.png" /><figcaption>Figure 3. *Emergent Perspective Model for IS Adoption and Implementation*</figcaption></figure>
The key to adoption in this model is the initial deployment of a new technology, followed by orchestrated monitoring and adaptation. Management and users must be willing to innovate and to take risks on the initial adoption and implementation with the understanding that problems will occur. Management and users must also be willing to invest resources (e.g., time, personnel, budgets) to identify and analyze implementation problems. Most importantly, they must be willing to continually implement technological and organizational changes in a perpetual cycle of change, analysis, and correction. Leonard-Barton (1988) views the initial implementation of a new technology as an extension of the invention process and the mutual adaptation process as occurring at multiple levels within the organization. She argues that "the successful management of technology transfer from developers to users requires that managers recognize and assume responsibility for both technical and organizational change" (p. 253). In organizational terms, this is conflict management, an essential feature of organizational management that entails managerial processes, structure, and content.

These information systems adoption and implementation paradigms, models, factors and processes provide ample suggestions for how to increase and improve successful educational technology implementation in education, as well as provide plentiful opportunities for research.

## EDUCATIONAL CHANGE THEORY AND RECOMMENDATIONS TO EDUCATIONAL TECHNOLOGY ADOPTION AND IMPLEMENTATION

Research on educational change has also produced knowledge of factors associated with adoption and affecting implementation. Fullan (1982) synthesized existing information and reported the factors contained in Tables 1 and 2 below.

<table><caption>**TABLE 1**
*Factors Associated with Adoption* (Fullan, 1982, p. 42)</caption><thead><tr><th>FACTORS ASSOCIATED WITH ADOPTION</th></tr></thead><tbody><tr><td>1. Existence and quality of innovations</td></tr><tr><td>2. Access to information</td></tr><tr><td>3. Advocacy from central administrators</td></tr><tr><td>4. Teacher pressure/support</td></tr><tr><td>5. Consultants and change agents</td></tr><tr><td>6. Community pressure/support/apathy/opposition</td></tr><tr><td>7. Availability of federal or other funds</td></tr><tr><td>8. New central legislation or policy (federal/state/provincial)</td></tr><tr><td>9. Problem-solving incentives for adoption</td></tr><tr><td>10. Bureaucratic incentives for adoption</td></tr></tbody></table>
<caption>TABLE 2</caption>

Factors Affecting implementation (Fullan, 1982, p. 56)

FACTORS AFFECTING IMPLEMENTATION

A. Characteristics of the Change

1. Need and relevance of the change

2. Clarity

3. Complexity

4. Quality and practicality of program (materials, etc.)

B. Characteristics at the School District Level

5. The history of innovative attempts

6. The adoption process

7. Central administrative support and involvement

8. Staff development (in-service) and participation

9. Time-line and information system (evaluation)

10. Board and community characteristics

C. Characteristics at the School Level

11. The principal

12. Teacher-teacher relations

13. Teacher characteristics and orientations

D. Characteristics External to the Local System

14. Bole of government

15. External assistance

When comparing the factors considered in educational change (above) to the IS Implementation Factor Model, a number of apparent consistencies can be noted. An important difference is that the characteristics described as affecting the adoption / implementation process in education are stated and treated in a static manner. What is omitted is any consideration of *organizational change*. Even the mutual-adaptation perspective which considers implementation as a process in which both the user and the innovation adapts or changes, defines the user narrowly, and does not consider changes which may be necessary at the organization or the larger systems level.

The major factors identified by educational change research are associated with introducing single innovations, rather than comprehensive reform (Fullan, 1993). The nature of systems suggests that introducing isolated changes does not produce lasting change, but that the system will revert back to the status quo. The complexity of comprehensive educational reform requires a dynamic systems perspective. Educational systems have remained
as relatively closed systems. Too much isolation exists both within schools and between schools and their environment. This isolation is an additional barrier to educational change.

Successful corporations, such as the Hanover Insurance Company (Hampden-Turner, 1992), and schools that have been successful in achieving major reform (Louis and Miles, 1990; Rosenholtz, 1989) are systems which learn from their environments. Successful organizations are continually changing and adapting, The vast majority of schools, however, have attempted to adopt isolated innovations into a conservative system that remains the same.

Schools that will be successful at serving a diverse and changing student body, must interact with and learn from their environment through collaborations and partnerships (Fullan, 1993). From past research we have learned that comprehensive change is a complex process that cannot be totally pre-planned. Each change causes numerous consequences, including many which can not be anticipated. We can again learn from studies of other types of organizations. In a study of business computing, Attewell (1992) reconceptualizes the diffusion of technology in terms of organizational learning, skill development, and knowledge barriers. He notes important differences when dealing with complex technologies which can inform the diffusion of technology in education.

Despite similarities to information systems, educational technology adoption research and practice should also bear in mind that educational systems have a characteristic rarely seen in general organizations used in information systems research. Educational systems are professional bureaucracies with a unique organizational structure, unique coordinating and controlling apparati, user roles and culture, communication channels, flow of decision making and authority, and situational factors. For example, information systems factor research consistently reveals management support as the most important, overriding factor in IS adoption and implementation success, but the role of management in a professional bureaucracy is small, existing mainly to provide resources to the professionals (i.e., educators), resolve conflicts among the professionals, and liaise with the external environment. In a professional bureaucracy, a successful decision to adopt an innovation will not be made by the administration alone, it will be made and carried out by individual professional educators. This characteristic, however, does not negate ET application of IS adoption models, it only suggests that the factors and processes for successful educational technology adoption will likely have different relative weights than the factors and processes in successful information systems adoption.

Educational systems are also social systems. As such there exists a variety of stakeholders insisting on having a voice in organizational change and/or instructional changes. The challenge is in learning from the different perspectives and working together rather than polarizing and working separately on opposing goals.
REFERENCES

Association for Educational Communications and Technology (1977). *The definition of educational technology*. Washington, DC: AECT.

Attewell, P. (1992). Technology diffusion and organizational learning: The case of business computing. *Organizational Science*, 3 (1), 1-19.

Clark, R.E. (1989). Current progress and future directions for research in instructional technology. *Educational Technology Research & Development*, 37(1), 57-66.

Cleary, A et al. (1976). *Educational technology: Implications for early and special education*. New York: John Wiley and Sons.

Commission on Instructional Technology (1970). *To improve learning. A report to the President and the Congress of the United States*. Washington, DC: Commission on Instructional Technology.

Culnan, M.J., & Swanson, E.B. (1986). Research in management information systems, 1980-84. *MIS Quarterly*, 10(3), 289-301.

Davis, G.B., & Olson, M.H. (1985). *Management information systems* (2nd ed.). New York McGraw-Hill.

Dieuzeide, H. (1971). *Educational technology: Sophisticated, adapted and rational technology. Series B: Opinions* (No. 30). Paris: International Commission on the Development of Education, UNESCO.

Ely, D.P. (1990). Trends and issues in educational technology, 1989. In G. Anglin (Ed.), *Instructional technology*. (pp. 34-58). Englewood, CO: Libraries Unlimited.

Engler, D. (1972). Instructional technology and the curriculum. In F.J. Pula and R.J. Goff (Eds.), *Technology in education: Challenge and change*. Worthington, OH: Charles A Jones.

Fullan, M. (1993). *Change forces: Probing the depths of educational reform*. New York: The Falmer Press.

Fullan, Michael (1991). *The new meaning of educational change* (2nd ed.). New York: Teachers College Press.

Fullan, Michael (1982). *The meaning of educational change*. New York: Teachers College Press.

Gentry, C.G. (1991). Educational technology: A question of meaning. In G. Anglin (Ed.), *Instructional technology*. (pp. 1-10). Englewood, CO: Libraries Unlimited.

Gentry, C.G., & Csete, J. (1991). Educational technology in the 1990s. In G. Anglin (Ed.), *Instructional technology*. (pp. 20-33). Englewood, CO: Libraries Unlimited.

Hampden-Turner, C. (1992). Charting the dilemmas of Hanover Insurance. *Planning Review*, January-February, 22-28.

Heinich, R. (1970). *Technology and the management of instruction*. Washington, DC: Association for Educational Communications and Technology.
Heinich, R. (1991). The proper study of instructional technology. In G. Angl (Ed.), *Instructional technology*. (pp. 59-81). Englewood, CO: Libraries Unlimited.

Jost, KL. (1992). The implementation of technology in the calculus classroom: An examination of teacher beliefs, practice and curriculum change. (Doctoral dissertation, Syracuse University). Dissertation Abstracts International, Order no. 9229677.

Keil, M.J. (1991). Managing MIS implementation. Unpublished doctoral dissertation, Harvard University Graduate School of Business Administration.

Laudon, KC. and Laudon, J.P. (1993). Business information systems (2nd ed.). New York: Dryden Press.

Leonard-Barton, D. (1988). Implementation as mutual adaptation of technology and organization. *Research Policy*, 17,251-267.

Lewin, K. (1947). Frontiers in group dynamics. *Human Relations*, 1,541.

Louis, K, & Miles, M.B. (1990). *Improving the urban high school: What works and why*. New York: Teachers College Press.

Lucas, H.C., Ginzberg, M.J., and Schultz, R.L. (1990). *Information systems implementation*. Nor-wood, NJ.: Ablex Publishing.

Mintzberg, H. (1993). *Structure in fives: Designing effective organizations*. Englewood Cliffs, NJ: Prentice Hall.

National Commission on Excellence in Education (1983). *A nation at risk: The imperative for educational reform*. Washington, D.C.: U.S. Dept. of Education.

Newman, F. (1989). *Uneven marriage: The university and the management of information technology*. Keynote address presented at the CAUSE National Conference on Managing Information Technologies: Facing Issues, San Diego, CA.

Parsons, G.L. (1983). Information technology: A new competitive weapon. In P. Gray, et al. (Eds.) *The management of information systems*. (pp. 473-502). Chicago: The Dryden Press.

Quality Education Data (1989). *1989-90 Catalog of educational mailing lists and marketing services*. Denver, CO: Quality Education Data.

Reigeluth, C.M. (1989). Educational technology at the crossroads: New mindsets and new directions. *Educational Technology Research & Development*, 37(1), 67-80.

Rosenholtz, S. (1989). *Teuchers' workplace: The social organization of schools*. New York: Longman.

Ross, S.M., & Morrison, G.R. (1989). In search of a happy medium in instructional technology research. *Educational Technology Research & Development*, 37(1), 19-33.

Silverman, R.E. (1969). Two kinds of technology. *Educational Technology*, January, 3.

Winn, W. (1989). Toward a rationale and theoretical basis for educational technology. *Educational Technology Research & Development*, 37(1), 35-46.
AUTHORS

Karen Lee Jost is Assistant Professor, Instructional Technology at Georgia State University, Atlanta, GA 30303.

Scott L. Schneberger is a Doctoral Candidate in Computer Information Systems, Georgia State University, Atlanta, GA 30303.
# Authentic Activity as a Model for Appropriate Learning Activity: Implications for Emerging Instructional Technologies

David G. Lebow
Walter W. Wager

**Abstract:** In this paper, we discuss the implications of authentic activity as a model of learning, particularly in the design of computer-based simulations and project-based learning activities. We first address the characteristics of real-life problem-solving situations, educational simulations, and authentic learning activities. We then identify three key guidelines in the design and development of authentic learning activities: (a) support the learner in establishing a learning enterprise. (b) insure that the learner practices what is essential for the transfer situation, and (c) base design decisions on values consistent with constructivist principles of teaching and learning. In our view, authentic activity represents a holistic and generative process of learning and motivation. Authentic learning activities place emphases on self-directed learning and on development of metacognitive abilities necessary to support it.

**Résumé:** Dans cet article nous étudions les implications de l'activité authentique comme modèle d'apprentissage, notamment dans la conception de simulations informatisees et dans les activités d'apprentissage a l'intérieur d'un projet. Nous nous penchons d'abord sur ce qui caracterise les situations de solution de problemes reelles, les simulations educatives, et les activités authentiques d'apprentissage. Nous identifions ensuite trois lignes directrices clés pour la conception et le développement d'activites authentiques d'apprentissage: a) le support de l'apprenant dans l'établissement d'un projet d'apprentissage, b) l'assurance que l'apprenant met les éléments essentiels a la situation de transfert en pratique, et c) des décisions sur les valeurs de conception de base compatibles avec les principes constructivistes de l'enseignement et de l'apprentissage. Nous pensons que l'activité authentique représente un processus holistique et generatif de l'apprentissage et de la motivation. Les activités authentiques d'apprentissage mettent l'accent sur l'apprentissage auto-dirige et sur le développement d'aptitudes métacognitives nécessaires a son support.

Today, interest in learning through authentic use is widespread as the theory base for situated learning matures and as innovations in computer-based multi-media systems outstrip the development of theory-based instructional strategies (Dick, 1991). Beginning with Dewey (1972) and the progressive educators of the 1920s and 1930s theorists have argued that learning
should take place in meaningful contexts where students work cooperatively to solve everyday problems. More recently, this idea has provided the basis for a number of educational approaches including cognitive apprenticeship (Brown, Collins, & Duguid, 1989), project-based learning (Blumenfeld, Soloway, Marx, Krajcik, Guzdial, & Palincsar, 1991), theme-based learning (Wager, 1994), guided microworlds (Rieber, 1992), computer-supported intentional learning environments (Scardamalia, Bereiter, McLean, Swallow, & Woodruff, 1989), and reciprocal teaching (Brown & Campione, 1990).

At the same time, the availability of powerful low-cost computers has stimulated interest in the design and development of simulations. Simulations have long been used to deliver instruction in schools, military, industrial, and other educational settings on the basis that they increase the ability of participants to apply what they have learned in the classroom to the real-world or transfer situation.

It is an often-stated conviction that producing transfer is the main job of education. Yet, an increasing body of research shows that the way students learn something in school often results in students knowing something but failing to use it when relevant. Brown et al. (1989) have concluded that this condition, originally identified by Whitehead (1929) as the problem of inert knowledge and also referred to as a transfer problem, occurs because classroom learning environments generally lack the contextual features of real-life problem-solving situations. With this problem in mind, Brown et al. have proposed that "understanding is developed through continued and situated use" (p. 33). They have further suggested that cognitive apprenticeships should be designed that immerse students in the culture of traditional academic domains by engaging students in authentic activity.

In this paper, we discuss the implications of adopting authentic activity as a model for appropriate learning activity, particularly in the design of computer-based simulations and project-based learning activities. The term "model" as used in this context is meant in the sense of something worthy of imitation and not in the scientific sense of a theoretical model of learning or instruction. We suggest, however, that there is much more to transforming the conventional classroom into an authentic learning environment than simply incorporating features of real-life situations into school work. Use of computer-based simulations and reality-centered projects does not insure that students will assume a positive orientation to learning or derive the benefits of in-context learning. Much additional support is required to strengthen the learners' tendencies to engage in intentional learning processes and to help them progressively assume responsibility for learning.

The following discussion is divided into three sections, organized around the following three questions that also serve as headings for the sections:

a) What characteristics of real-life problem-solving situations are different from problem solving in school?

b) What characteristics of educational simulations are most important?
c) What characteristics of authentic learning activity are most important?

**Question 1. What characteristics of real-life problem-solving situations are different from problem solving in school?**

A primary principle of the cognitive apprenticeship framework is that understanding develops through application and manipulation of knowledge within the context of the ordinary practices of the target culture-in other words, through authentic activity. This principle represents the primary rationale for using authentic activity as the model for appropriate learning activities. It extends various cognitive theories of transfer that emerged from information processing theories of human learning and memory in the 1970s (e.g., Bransford & McCarrell, 1974). These theories stimulated thinking about developing educational practices to enhance far transfer and, more particularly, transfer from school learning to real-world situations (Royer, 1979). As Brown et al. (1989) suggested, conventional classroom tasks frequently lack the contextual features that support transfer from the school setting to the outside world. For advocates of situated approaches to learning, the provision of authentic activity in schools is a way to increase cognitive engagement, support meaningful learning, and facilitate transfer.

Table 1 summarizes differences between real-life problems and problem-solving activities typical of actual practitioners versus problem solving typical of students in school that are particularly relevant to the design of authentic activities. The differences identified in the table are further elaborated below.

<caption>TABLE 1
Real-Life Versus In-School Problem Solving</caption>

**Real-Life**

1. Involves ill-formulated problems and ill-structured conditions.

2. Problems are embedded in a specific and meaningful context.

3. Problems have depth, complexity, and duration.

4. Involves cooperative relations and shared consequences.

5. Problems are perceived as real and worth solving.

**In-School**

1. Involves "textbook examples" and well-structured conditions.

2. Problems are largely abstract and decontextualized.

3. Problems lack depth, complexity, and duration.

4. Involves competitive relations and individual assessment.

5. Problems typically seem artificial with low relevance for students.
1) Real-life problems are frequently ill-formulated and conditions are ill-structured. Understanding develops through experience in multiple case contexts and from multiple perspectives within the same context. In school (and more generally in the design of instruction), performance requirements are simplified and the learning situation is well-structured (Spiro, Feltovich, Jacobson, & Coulson, 1991).

2) In real life, skill practice is embedded in performing some larger task at hand that justifies developing the skills in the first place (Brophy & Alleman, 1991). The larger task serves as an integrating structure or enterprise that helps organize new information and lends meaning and purpose for learning. Individuals assume responsibility for establishing and monitoring their goals and strategies when the reasons for performing procedures, even tedious ones, are understood within the context of a broad, global task environment (Honebein, Duffy, & Fishman, 1994). In school, teachers often assign students to low-level work involving recognition and reproduction of memorized information or practice of isolated skills, without providing links to a larger functional context (Doyle, 1986). All too often, the primary enterprise for students is to pass tests rather than apply new knowledge and skills in meaningful ways.

3) Real-life projects frequently have depth, complexity, and duration (Berliner, 1992). When people engage in active and generative problem-solving activities that involve personal values and beliefs, they experience a feeling of ownership over the activity and its goals and thus, the tendency to engage in intentional and self-regulated learning processes is enhanced. "School activities" generally lack depth, complexity, duration, and relevance to the real world. Teachers and students are constrained by requirements to cover "essential" content.

4) In real life, the intelligence to solve a problem or perform an activity is often distributed across a group of peers, a learner-mentor system, and/or an electronic performance support tool (EPSS), or other form of cognitive technology (Pea, 1993). The quality of interactions among participants is frequently of primary importance in undertaking a project or accomplishing a goal. In school, teachers serve as authoritative sources of all information rather than as guides or information managers. Relationships between students are predominantly competitive rather than cooperative, as individualistic modes of learning and assessment are generally the norm. When group activities do occur, students are usually judged exclusively on what they can do on their own.

5) In real-life, all problems do not have previously known solutions or just one possible explanation. When people work collaboratively on solving real-life problems, they share in substantive conversation, which has a different quality from conventional school talk (Newmann, 1991). An individual's orientation toward learning is qualitatively different when learning is embedded in the context of achieving personally relevant and valued goals versus working for a grade or some distant, future goal. In school, problems are often
artificial and of no particular interest to the students. Students who see no relevance in the problems, also see no relevance in the knowledge and skills required to solve them.

**Question 2. What characteristics of educational simulations are most important?**

The theoretical assumptions underlying the designs of simulations are as varied as the purposes for which they are used and the contexts in which they appear. According to Cunningham (1984), a simulation duplicates some essential aspect of reality for purposes of experimentation, prediction, evaluation, or learning. An educational simulation is designed to increase one's ability to respond appropriately in a real-world or transfer setting. It allows the learner to practice decision-making, problem solving, and/or role playing in the context of a controlled representation (model) of a real situation (Smith, 1986).

From an instructional-design perspective, educational simulations support predetermined learning outcomes by providing users with opportunities to deal with the consequences of their actions and to respond to feedback Within Pea's (1985) framework of distributed intelligence, computer-assisted simulations have the potential to reorganize mental processes by "closing of the temporal gaps between thought and action [and] between hypothesis and experiment" (p. 85). Pea envisions a partnership between computer and student that extends and redefines thinking capabilities and transforms how problem solving occurs. Such effects are made possible by allowing the user to engage in "what-if thinking" where the consequences of different approaches to a problem may be displayed immediately.

In the literature on simulation, the definition offidelity varies depending on the context to which it is applied and the theoretical orientation of the author. For example, in proposing a model for assessing the fidelity of task simulators used in industry, Bruce (1987) identified three criteria: a) physical similarity; b) functional similarity; and c) task commonality. After assigning a value to each category involved, the values are combined to produce a fidelity index for a particular training device. In contrast, Smith (1986) believes that the essential reality factor in a simulation is not its form but the information-processing demands it imposes on the learner. He has called this its "cognitive realism" the degree to which the simulation engages participants in a decision-making or problem-solving process that parallels the mental activities required in the transfer situation.

Oddly enough, research does not support the idea that maximizing realism, or fidelity of a simulation maximizes learning outcomes (Alessi 1987). With this in mind, Reigeluth and Schwartz (1989) recommended that when designing for a novice learner, it is best to start with low fidelity and to add fidelity and complexity progressively. Similarly, Blumenfeld et al. (1991) have proposed that a great strength of simulation for instructional purposes is its potential to allow students active exploration in simplified environ-
ments. They believe that when extraneous details are minimized, interac-
tions among variables are easier to notice than in a highly realistic simulation
or in the transfer environment itself.

Reigeluth and Schwartz (1989) have described three major elements in the design of computer-based simulations that they believe determine their effectiveness: the scenario, the underlying model, and the instructional overlay. They have suggested that the scenario (the situation and the learner interface with the simulation) and the model (usually a mathematical formula for establishing causal relationships) should duplicate to some degree the essential characteristics of the transfer situation. In other words, the characteristics of the scenario and the model determine whether essential aspects of the transfer situation are represented, although how to identify such essential characteristics is not addressed. Reigeluth and Schwartz have concluded, on the basis of their own analysis of simulations, that the instruc- tional overlay (the features that function to optimize learning and motivation) are generally the weakest aspect in educational simulations.

One element of the instructional overlay that Reigeluth and Schwartz (1989) feel should receive more attention from designers is artificial feedback. Alessi and Trollip (1985) have distinguished between natural feedback that the real-life situation provides, and artificial feedback that the designer builds into the simulation. One of the strengths of simulation for instructional purposes is its potential to shelter learners from costly forms of natural feedback (skidding into a snow bank) and to provide real-time artificial feedback (aural, verbal, or visual instructions to turn in the direction of the skid.)

The simplifying-conditions method proposed by Reigeluth (1993) appears
to take advantage of strengths inherent in simulation without sacrificing
authenticity of the learning activity. In this method, experts identify (a) a
simple case that closely represents a real-world task and (b) the ways in which
this "epitome" version of the task differs from more complex versions. Over
time, complexity and variations are added systematically to the learning
activity with the expectation that the method preserves the potential benefits
of in-context learning. Reigeluth has claimed that this is a more holistic way
to sequence instruction than the traditional parts-to-whole approach and is
compatible with context-based design models.

The ideas that sequencing within a simulation should progress from
simple-to-complex and that different levels of fidelity and complexity are
appropriate for different levels of learners appear to be inconsistent with
constructivist principles held by a number of theorists. For example,
Honebein et al. (1994) believe that the learning situation should parallel the
transfer environment with all its complexity and messiness. They have
argued that the complexity of the learning environment in the early stages of
learning should reflect the complexity of the authentic context to the extent
practical. Otherwise, when instructional designers simplify the learning
environment, they may unwittingly alter the metacognitive and affective demands of the authentic task complex.

Carroll (1990) has suggested that in order to facilitate transfer, promote metacognitive and affective learning, support an adaptive motivational pattern to learning, and encourage a high degree of ownership and personal relevance, educators should provide training on real tasks. Similarly, Spiro Vispoel, Schmitz, Samarapungavan, and Boeger (1987) believe that "case; and examples must be studied as they really occur, in their natural contexts, not as stripped down 'textbook examples' that conveniently illustrate some principle" (p. 181). From this perspective, the role of instruction changes from controlling student learning through imposing a simplifying structure on the environment to developing appropriate "scaffolding," including new strategies, tools, and resources that support the student in functioning within the authentic learning context.

Of note, Spiro et al. (1991) have identified two factors that determine when it is appropriate to maintain the complexity of the transfer environment within the learning situation, and when it is more effective to simplify the learning situation. They distinguish between a) well-structured domains versus ill-structured domains; and b) introductory learning versus advanced learning. Whereas instruction that focuses on general principles with application across cases is effective in well-structured domains, such an approach may impede attainment of more ambitious goals in ill-structured domains. To attain high-level thinking skills in ill-structured domains, the learning environment should provide experience in multiple case contexts and from multiple perspectives within the same context.

Theorists hold a variety of viewpoints on when to maintain the complexity of the transfer situation within the learning situation and when to simplify the learning situation. Vygotsky's (1978) concept of the zone of proximal development (ZPD) and Pea's (1985) ideas concerning distributed intelligence appear particularly relevant to this concern. The ZPD represents the limits of an individual's development defined as the distance between independent problem solving and what a person can accomplish under adult guidance or in collaboration with more capable peers (Vygotsky, 1978). Pea has suggested that technology, in Vygotskian terms, can expand zones of proximal development to enable novices to engage in problem-solving activities that would otherwise remain beyond their reach. From this perspective, intelligence may be distributed across a system not only through collaborative efforts but also as a result of the partnership between learner and computer (Pea, 1993). This suggests the following principle: The complexity of the learning situation should not exceed the capacity of the environment to adequately expand the ZPD through both social and technological scaffolding.

**Question 3. What characteristics of authentic learning activity are most important?**

Examined from a variety of viewpoints, the assumption underlying most traditional educational practices that knowledge is context-independent,
appears increasingly difficult to justify. A renewed educational awareness is
growing that recognizes the priority of supporting higher mental processes
through situated learning and the collaborative construction of meaning.
Although educators have long recognized the need to foster higher order
thinking skills and positive disposition toward learning, they have disagreed
about how to achieve these ends (Resnick, 1989).

The idea that students need to acquire facts and theories as conceptual
tools for solving problems in meaningful contexts, rather than for maximizing
success on "school" tests, has led to an interest in apprenticeship learning and
learning through authentic use (Brown et al., 1989; Pea & Gomez, 1992).
Results from cognitive skill research that examine differences between
experts and novices support this view. Experts and novices differ in many
ways other than the amount of subject matter mastered, including "the
organization of knowledge, the ability to process problems in depth, and the
appropriateness of the mental model possessed by the learner" (Royer, Cisero,
& Carlo, 1993, p. 235).

The notion of authentic activity is based on the idea that the learning
situation should help learners develop ways of thinking and acting that
characterize the target culture or professional community. This does not
mean, however, that design for authentic learning activity is merely a matter
of maximizing the similarity of the learning situation to the transfer environ-
ment. As is explained through Tripp's (1994) example below, design for
authentic learning is also concerned with the values that the learning
environment supports and models, and whether the learner practices what is
essential for the transfer situation.

Tripp (1994) believes that one way to learn something about instructional
design is to analyze proven educational approaches by reverse engineering
them and trying to extract some principles. For example, he believes that the
instructional model applied in the education of Pacific navigators for over a
thousand years represents such a proven approach. Although Pacific Island-
ers educated their navigators in part through informal situated learning at
sea, much of the students' education occurred in a formal instructional
context. For example, instruction in wave pattern recognition and star
relationships was done on land with substitute media-stick charts, stone
canoes, and the interior of a building that served as a kind of low-fidelity
planetarium — even though real waves and stars were readily available.

One principle that Tripp (1994) derived from his analysis is as follows:
"Reality is not necessarily superior to artificiality. When reality is subtle or
complex, substitute media that increase saliency by simplification are more
efficient" 3). The neophyte navigator knows in a meaningful way that what
he is learning is necessary for his survival. The idea that he is engaged in a
learning enterprise, in Gagne and Merrill's (1991) terms, is well established
in his mind before he enters the formal phase of training. Thus, in this case,
authenticity is primarily a quality of the larger task environment as under-
stood by the learner.
Tripp's (1994) analysis illustrates a fundamental principle of instruc-
tional design that emerges from an open-systems view of the learner: The
orientation of the individual to learning is part of the learning context
Learning orientation encompasses an individual's beliefs about the nature of
knowledge and how it is acquired as well as personal goals, expectations, and
attitudes. Considered as a whole, these factors influence students' cognitive
engagement and the learning activities they employ.

From an open systems conception, human learning is essentially a matter
of self-regulation. It is a process of making sense, where external
perturbations, anomalies, and errors trigger internal transformations toward
reorganization and new understanding (Doll, 1989). With an open systems
model of the learner, the role of instruction is to support the process of
meaning-making by influencing the thinking processes students use to learn
and sustain motivation, rather than controlling external conditions to achieve
pre-set ends.

Buchanan (1992) has suggested that since conditions are ill-structured
and problems are ill-formulated in many areas of human endeavor, all but the
most clearly linear design problems assume a fundamental indeterminacy.
Von Bertalanffy's (1967) distinction between open and closed systems is
relevant to this view. Closed systems such as cybernetic or feedback systems
are open to information but do not exchange matter with the environment.
Open systems, on the other hand, such as organisms and other living systems
are maintained in a continuous exchange of components. Instructional design
within an open-systems framework recognizes the constructive nature of
reality and further requires a shift in preferred metaphor for education from
transmission of information to building representations of meaning.' In a
sense, an open system is open to possibilities.

Based on our own design experience and on the literature related to
constructivist philosophy, higher order thinking, achievement motivation
and computer-supported collaborative learning environments we have tenta-
tively identified a set of interrelated values that may guide educators in the
design of authentic learning environments. In the face of what Doll (1989)
describes as an emerging post-modern agenda for curriculum the traditional
ID values of replicability, reliability, communication, and control (Heinich
1984) appear increasingly restrictive. An alternative (and not necessarily
mutually exclusive) set of values has emerged including mutual inquiry
collaboration, multiple perspectives, pluralism, personal autonomy activity'
reflectivity, generativity, authenticity, complexity, personal relevance, self-
regulation, ownership, and transformation (Lebow, 1993). These values are
supported by a growing body of educational research and theory that advo-
cates holistic and generative approaches to education and the use of technol-
ogy to assist students in developing higher order thinking skills and impor-
tant long term dispositions toward learning.

When the design of instruction is guided by such values means and ends
become integrated and the desired results and preferred instructional tech-
niques and strategies appear as reflections of the same whole. To put this in other terms, when the transfer task is the model for the learning activity, means support values inherent in ends. For example, to develop interpersonal skills for sustaining cooperative group work is both a goal of instruction and a means to achieve the goal. Students practice group process skills in the context of achieving personally relevant goals. Another goal, to develop the ability to reflect on one's own learning processes, is also a means to self-correction and self-regulation of the learning process. Within an authentic learning activity, instruction is a model for the values that instruction is designed to support, and, in line with the concept of continuous progress, a good test is also a good learning activity. As a result, students experience the relevance of new information, not by being told about its relevance, but by experiencing changes in perceptions, understandings, beliefs, feelings, and capabilities as a function of new information (Bransford, Franks, Vye, & Sherwood, 1989).

Gustafson (1993) believes that the fundamentals of instructional design must change in response to mounting evidence supporting the claims of cognitivists and constructivists. He has suggested that "what we now await is a much greater amount of specific guidance on how to apply cognitivism and constructivism, such as we have for behaviorism" (p. 30). Our current research efforts are pointed in this direction as we develop a set of guidelines that will serve to make the application of constructivist values to the design of instruction more concrete, in two ways. The guidelines will (a) provide information on how to design for higher order thinking skills and positive disposition toward learning and (b) suggest solutions to a variety of classroom management issues that may arise when authentic activities serve as the model for learning activities.

The guidelines initially emerged from a review of literature about how cognitivist and constructivist principles may contribute to the design of complex learning environments, and to an understanding of how to use emerging technologies in education. We are now in the process of refining the guidelines based on results of our own research efforts in examining a prototype computer-supported learning environment for graduate level education.

At this point, we have organized the guidelines into four main categories including collaboration, ownership, meaning, and practice as a framework for communicating our ideas and as a basis for future research. Taking the first letter of each category provides the convenient mnemonic of COMP. Under each category, we have listed a number of tentative goal statements that reflect instructional design principles consistent with constructivist and cognitivist thinking and in line with the values mentioned previously. Although it is beyond the scope of this paper to report on our progress in this area, an example of a goal statement from each of the COMP categories is offered below:
1) **Collaboration:** Provide students with opportunities to engage in activities traditionally reserved for teachers by a) shifting from all students learning the same things to different students learning different things; b) creating group problem-solving situations that give students responsibility for contributing to each other's learning; and c) helping students see the value of what they are learning and choose to share that value.

2) **Ownership:** Support development of reflective self-awareness and other self-regulated learning skills as the basis for assuming personal responsibility for learning by a) providing new structure, process-relevant feedback and sufficient time to support reflection on the learning process and to help learners experience the value of self-monitoring; b) modeling and offering coached practice at self-questioning and other metacognitive skills for developing both executive control of learning activities and critical thinking skills for evaluating intellectual products including one's own; and c) promoting meta-affective skills for increasing ability to concentrate and persevere.

3) **Meaning:** Support development of a learning versus grade orientation to the academic enterprise and help learners build commitment to their goals by a) embedding the reasons for learning something into the learning situation and helping students learn something in a way that includes experiencing its significance or function; and b) providing opportunities for students to experience how increasing cognitive engagement is tied to achievement of personally relevant goals.

4) **Practice:** Support development of cognitive flexibility by providing a) problem-solving experience in multiple case contexts and from multiple perspectives within the same context; and b) repeated practice in environments similar to those in which learners will use their problem-solving skills (Spiro et al., 1991).

In summary, authentic activity is consistent with a holistic and generative view of learning and motivation that places emphasis on self-directed learning and on development of metacognitive abilities necessary to support it. From this perspective, the traditional split between cognition and affect is seen as an often unproductive application of reductionist thinking. When authentic activity is the model for appropriate learning activity, the perceptions of the learner and the affordances of the environment represent an integral and inseparable context of learner/environment. The implications for instruction are primarily threefold: (a) design must support the learner in establishing a learning enterprise within the larger global task environment (b) the learning situation must afford the kinds of activities essential success in the transfer environment, and (c) the instructional designer should base design decisions on values consistent with constructivist principles of teaching and learning.
REFERENCES

Alessi, S. (1987, April). *Fidelity in the design of instructional simulations*. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, D.C.

Alessi, S., & Trollip, S. (1985). *Computer based instruction: Methods and development*. New York: Prentice Hall.

Berliner, D. C. (1992). Redesigning classroom activities for the future. *Educational Technology, 10*, 7-13.

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. *Educational Psychologist, 26* (3 & 4), 369-398.

Bransford, J. D., Franks, J. J., Vye, N. J., & Sherwood, R. D. (1989). New approaches to instruction: Because wisdom can't be told. In S. Vosniadou, & A. Ortony (Eds.), *Similarity and analogical reasoning* (pp. 470-497). Cambridge, England: Cambridge University Press.

Bransford, J. D., & McCarrell, N. S. (1974). A sketch of a cognitive approach to comprehension: Some thoughts about understanding what it means to comprehend. In W. B. Weimer and D. S. Palermo (Eds.), *Cognition and the symbolic process*, Hillsdale, NJ: Erlbaum.

Brophy, J., & Alleman, J. (1991). Activities as instructional tools: A framework for analysis and evaluation. *Educational Researcher, 20*(4), 9-23.

Brown, A L., & Campione, J. C. (1990). Communities of learning and thinking, or a context by any other name. In D. Kuhn (Ed.), *Developmental perspectives on teaching and learning thinkingskills. Contributions to human development*, 21, 108-126.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. *Educational Researcher*, 18(1), 32-42.

Bruce, R. C. (1987). *Simulation fidelity: A rational process for its identification and implementation*. Unpublished manuscript.

Buchanan, R. (1992). Wicked problems in design thinking. *Design Issues*, 8(2), 5-21.

Carroll, J. M. (1990). *The Nurnbergfunnel: Designing minimalist instruction for practical computer skill*. Cambridge, MA: MIT Press.

Clinchy, E. (1989). Education in and about the real world. *Equity and Choice*, 3,19-29.

Cunningham, J. B. (1984). Assumptions underlying the use of different types of simulations. *Simulations and Games, 15*(2), 213-234.

Dick, W. (1991). An instructional designer's view of constructivism. *Educational Technology, 5*,41-44.

Doll, W. E., (1989). Foundations for a post-modern curriculum. *Journal of Curriculum Studies, 21*(3), 243-253.

Doyle, W. (1986). Content representation in teachers' definitions of academic work. *Journal of Curriculum Studies, 18*,365-379.
*Dewey, J.* (1972). *Experience and Education*. New York: Collier Books.
Originally published in 1938 by Kappa Delta Pi.)

Gagne, G. M., & Merrill, D. (1991). Integrative goals for instructional design.
*Educational Technology Research & Development, 38*(1), 23-30.

Gustafson, K. L. (1993). Instructional design fundamentals: Clouds on the horizon.
*Educational Technology, 2,27-32*.

Heinich, R. (1984). The proper study of instructional technology. *Educa-
tional Communications and Technology Journal, 32*, (2), 67-87.

Honebein, P. C., Duffy, T. M., & Fishman, B. J. (1994). Constructivism and the design of learning environments: Context and authentic activities for learning. In T. M. Duffy, J. Lowyck, & D. Jonassen (Eds.), *Designing environments for constructivist learning*. Hillsdale, NJ: Lawrence Erlbaum.

Lebow, D. G. (1993). Constructivist values for instructional systems design: Five principles toward a new mindset. *Educational Technology Research & Development, 41*(3), 4-16.

Newmann F. M. (1991). Linking restructuring to authentic student achievement. *Phi Delta Kappan, 2,458-463*.

Pea, R. D. (1985). Beyond amplification: Using computers to reorganize mental functioning. *Educational Psychologist, 20*(4), 167-182.

Pea, R. D. (1993). Practices for distributed intelligence and designs for education. In G. Salomon (Ed.), *Distributed cognitions: Psychological and educational considerations* (47-87).

Pea, R. D., & Gomez, L. M. (1992). Distributed multi-media learning environments: *Why and how? Interactive Learning Environments, 2*(2), 73-109.

Reigeluth, C. M. (1992). Elaborating the elaboration *theory*. *Educational Technology Research & Development, 40*(3), 80-86.

Reigeluth, C. M., & Schwartz, E. (1989). An instructional theory for the design of computer-based simulations. *Journal of Computer-Based Instruction*, 16(1), 1-10.

Resnick, L. B. (1989). Introduction. In L. B. Resnick (Ed.), *Knowing, learning and instruction* (1-24). Hillsdale, NJ: Lawrence Erlbaum.

Rieber, L. P. (1992). Computer-based microworlds: A bridge between constructivism and direct instruction. *Educational Technology Research & Development, 40*(4), 107-109.

Royer, J. M. (1979). Theories of transfer of learning. *Educational Psychologist, 14*, 53-69.

Royer, J. M., Cisero, C. A., & Carlo, M. S. (1993). Techniques and procedures for assessing cognitive skills. *Review of Educational Research, 63*(2), 201-243.

Scardamalia, M., Bereiter, C., McLean, R. S., Swallow, J., & Woodruff E. (1989). Computer-supported intentional learning environments. *Journal of Educational Computing Research, 5*(1), 51-68.
Smith, P. (1986). *Instructionalsimulation: Research, theoryanda casestudy*. (ERIC Document Reproduction Service No. ED 267 793)

Spiro, J. R., Vispoel, W., Schmitz, J., Samarapungavan, A., & Boerger, A. (1987). Knowledge acquisition for application: Cognitive flexibility and transfer in complex content domains. In B.C. Britton (Ed.), *Executive control processes* (177-199). Hillsdale, NJ: Erlbaum.

Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advancedknowledge acquisition in ill-structured domains. *Educational Technology, 31(5), 24-33.

Tripp, S. (1994, February). *Reverse engineering the "Stone Canoe- (Learning from designs). Paper presented at the annual convention of the Association for Educational Communication and Technology, Nashville, TN.*

von Bertalanffy, L. (1967). *Robots, men, and minds*. New York: George Braziller.

Vygotsky, L. S. (1978). *Mind in society*. Cambridge, MA: Harvard University Press

Wager, W. W. (1994, February). *Designing thematic units and integrated learning activities*. Paper presented at the annual convention of the AECT, Nashville, TN.

Whitehead, A. N. (1929). *The aims of education*. New York: Harper & Row.

## AUTHORS

David G. Lebow is a doctoral Candidate in Instructional Technology at Florida State University, Tallahassee, FL 32301.

Walter W. Wager is Professor of Instructional Technology at Florida State University, Tallahassee, FL 32301.
# Mediaware Review

## *The Apple Media Kit Version 7.7*

L. F. (Len) Proctor, Editor

### System Information and Requirements

* Apple Programmers Development Association (APDA)

* Apple Computer, Inc.

* Order: 1-800-637-651 1

* Fax: 716-871-6511

* Applelink: APDA

* Price Range: $1,000- $2,000

### System Requirements:

* Macintosh IIci or later

* 8 MB of RAM memory

* At least 40 MB hard disk drive (although larger drives are recommended)

* System 7.1 or later

* QuickTimeTM, version 1.6.1 or later

* CD-ROM optional

### Program Description

The Apple Media Kit is a cross-platform, multimedia authoring tool. The Media Tool component facilitates the assembly of sound, picture, computer graphics, text and movie files. The AMT Programming Environment is an object-oriented programming language that enables a programmer to add interactivity to a project. The Kit contains Macintosh installation disks, Windows installation disks, documentation, a demo CD containing an electronic version of the Programming Environment documentation, sample Mac and Windows projects and a demonstration version of the Media Tool.

### Basic Functions

Media Tool projects use an icon map to structure the user path and the order in which screens are displayed. A library folder is used to hold all the media components that are used to construct the screen. When a completed Media Tool project is saved, the description of objects and their actions can be
output to the AMT Programming Environment as a text file. In text form, the code can be edited and enhanced with the Apple Media Language or other standard programming language such as Pascal or C. Conditional branching based on user input can only be achieved within the programming environment.

## Documentation

Documentation for the Media Tool has two forms. Getting Started is a tutorial guide that explains the basic operation of the program and the User's Guide provides a reference type, feature by feature explanation. The Programming Environment has both a User's Guide and a Reference Guide. An electronically searchable copy of the programming documentation is also available on the companion CD-ROM. For the Media Tool, the Getting Started Guide presumes that the user has an extensive working knowledge of Macintosh operating conventions. The Programming Environment presumes a basic knowledge of the Macintosh Programmer's Workshop (MPW) shell, programming in C, object-oriented programming, Macintosh programming fundamentals and the Media Tool.

## Critique and Recommendations

The Media Tool functions as a media assembly tool because it does not contain any resident media editors. Editing sound, text, picture or movie files has to be done by jumping out of the Media Tool to an appropriate outside editor. In order to use external media editors while the Media Tool is being used, there must be enough RAM memory available to handle each task. Second, if cross platform development is anticipated then MS-DOS file naming conventions must be used when the final editing is done, otherwise, the project may not run properly in a DOS or WINDOWS environment. The necessity of having to buy external editors adds to the cost of the program. Depending on the quality of subsidiary media editors selected, the cost will vary from a few dollars for shareware products to several hundred dollars for full-featured commercial editing programs.

The Media Tool requires a minimum of 3 MB on a hard drive. The documentation suggests that a minimum of 4 MB of RAM memory be exclusively dedicated to AMT, but recommends that the minimum be increased to 8 MB. Given that system files can easily consume another 2 to 3 MB, it is easy to see that if any multi-tasking is required, a minimum of 16 MB of RAM memory will be required for even a modest sized project. Second, because sound, still picture and video files are notorious for their ability to fill up hard disk space, the larger the hard drive available the better. Memory requirements for the Programming Environment are similar. The important point to note here is that while the program will run on smaller machines like the Macintosh IIci, using equipment with limited RAM memory and disk storage capabilities will very quickly limit the size and complexity of projects that can be developed.