U.S. Army Research Institute 

for the Behavioral and Social Sciences 

Research Report 1802

 

The Effectiveness of Web-based Training

Robert A. Wisher 

U.S. Army Research Institute 

Tatana M. Olson 

Purdue University

Consortium Research Fellows Program

January 2003

Approved for public release; distribution is unlimited. 
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the U.S. Total Army Personnel Command 

ZITA M. SIMUTIS 

 Director 

Technical review by 

 

Brooke Schaab, U.S. Army Research Institute 

Joseph Psotka, U.S. Army Research Institute 

Curtis J. Bonk, Indiana University 

NOTICES 

DISTRIBUTION: Primary distribution of this Research Report has been made by ARI. Please 

address correspondence concerning distribution of reports to: U.S. Army Research Institute for 

the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001 Eisenhower Ave., Alexandria, 

VA 22333-5600. 

FINAL DISPOSITION: This Research Report may be destroyed when it is no longer needed. 

Please do not return it to the U.S. Army Research Institute for the Behavioral and Social 

Sciences. 

NOTE: The findings in this Research Report are not to be construed as an official Department 

of the Army position, unless so designated by other authorized documents. 
REPORT DOCUMENTATION PAGE 

1. REPORT DATE (dd-mm-yy)

January 2003 

2. REPORT TYPE

Interim

3. DATES COVERED (from. . . to)

 June 2001 – July 2002 

4. TITLE AND SUBTITLE

The Effectiveness of Web-based Instruction 

5a. CONTRACT OR GRANT NUMBER

5b. PROGRAM ELEMENT NUMBER

63007A

5c. PROJECT NUMBER

A792 

6. AUTHOR(S) 

Robert A. Wisher (U.S. Army Research Institute) & Tatana M. Olson

(Purdue University) 5d. TASK NUMBER

208 

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Institute for the Behavioral and Social

Sciences

Attn: TAPC-ARI-II

5001 Eisenhower Avenue 

Alexandria, Virginia 22333-5600

8. PERFORMING ORGANIZATION REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

ARI 

 11. MONITOR REPORT NUMBER

Research Report 1802

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words): 

As the Army focuses more on Web-based instruction as a delivery option for training, professional development, and 

education, it is important to understand its instructional effectiveness. A search of the literature between 1996 and 2002 

identified more than 500 reports concerning Web-based courses, that is courses delivered over the Internet or an

intranet and conveyed through a browser. Most concerned recommendations for design or technology issues rather

than empirical measures of learning outcomes. From this larger set, 47 studies reported empirical measures of

outcomes; of these, 15 studies reported data with sufficient detail to compute an effect size. The effect size, comparing 

Web-based instruction to conventional classroom instruction, was .24, which means the “average” student moved from

the 50th to the 59th percentile. In earlier analyses of the effectiveness of other forms of computer-based instruction, 

effects sizes of between .32 and .41 have been reported (corresponding to the 63rd and 66th percentiles). In terms of

instructional effectiveness, it appears that current practices in Web-based instruction improve learning when compared 

to the classroom, but many more complete studies are needed before the full effect size is understood.

15. SUBJECT TERMS

distance learning, distributed learning, E-Learning, online instruction, computer-based training, training

 SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

16. REPORT

Unclassified

17. ABSTRACT

Unclassified

18. THIS PAGE

Unclassified

ABSTRACT

Unlimited 

OF PAGES

 29 

(Name and Telephone Number)

Dr. Franklin Moses 

703/617-5948 

i 
 
Research Report 1802 

The Effectiveness of Web-Based Instruction 

 Robert A. Wisher 

U.S. Army Research Institute 

 

Tatana M. Olson 

Purdue University

Consortium Research Fellows Program 

Advanced Training Methods Research Unit 

Franklin L. Moses, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences

 5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

January 2003 

Army Project Number Personnel, Performance 

2O633007A792 and Training

 

Approved for public release; distribution is unlimited.

iii 
 
FORWORD 

The U.S. Army Research Institute is examining the use of distributed learning

technologies for use by soldiers in an “on demand” environment, where training becomes more 

soldier centered rather than classroom based. Part of the work involves a thorough assessment of 

the distance learning research literature as it pertains to training. The present report under ARI’s 

WEBTRAIN project provides an examination of empirical evidence concerning the effectiveness 

of Web-based instruction. It forms a foundation for guidelines to help training developers, 

training managers, and policy makers. 

The results of this research were presented to the Army Training and Doctrine Command, 

Deputy Chief of Training, on 17 October 2002. 

 

 Franklin L. Moses

 Acting Technical Director

v 
vi 
THE EFFECTIVENESS OF WEB-BASED INSTRUCTION 

EXECUTIVE SUMMARY 

Research Requirement:

The Army is embarking on a major change to deliver standardized individual, self-development, 

and small group training to soldiers through the application of networked communication 

systems. Planning calls for Army training to become more learner-centric, in which soldiers

assume increased responsibility for the acquisition of knowledge and the development of skills, 

delivered over the Internet or military intranets. Reports in the literature on the effectiveness of

instruction delivered over the Web have begun to appear, but there has not been an integrated 

analysis of these early findings. This report offers an initial analysis of the reported findings. 

Procedure: 

Searches on relevant research databases yielded 40 reports with learning outcome data from the 

more than 500 reports on the topic of Web-based instruction. When including multiple studies 

documented in some reports, the number of cases that provided empirical evidence increased to

47. Of these, 15 provided sufficient data to compute an effect size, a standard way of

quantifying differences between groups. Here, the groups refer to a comparison group that 

received instruction in a conventional classroom and an experimental group that received 

comparable instruction over the Web. 

Findings: 

The average effect size, comparing Web-based instruction to conventional classroom instruction, 

was .24, which means the “average” student moved from the 50th to the 59th percentile. The

relatively small number of studies reporting outcome data demonstrated wide variability; the

effect sizes ranged from -.4 to 1.6. In earlier analyses of the effectiveness of non Web-based 

forms of computer-based instruction that used hundreds of studies, effects sizes of between .32 

and .41 have been reported (corresponding to the 63rd and 66th percentiles). In terms of

instructional effectiveness, it appears that current practices in Web-based instruction lead to an 

improvement in learning when compared to the classroom, but the central tendency falls short of 

that for traditional computer-based instruction. In view of the shortage of empirical findings and 

the relative novelty of using the Web for instructional purposes, dozens of more controlled 

studies are needed before a more accurate effect size can be resolved. There is no practical 

reason why the effect size should be less than that demonstrated for computer-based instruction. 

Utilization of Findings: 

These findings can be used as a basis for (1) selecting best practices in an objective manner, (2) 

establishing objective performance criteria for future learning content, and (3) creating an initial 

data set to which future evidence on the effectiveness of Web-based instruction can be added. 

vii 
viii 
THE EFFECTIVENESS OF WEB-BASED INSTRUCTION 

CONTENTS 

 Page

ADVANTAGES OF TRAINING TECHNOLOGY...................................................................................1 

THE USE OF THE WORLD WIDE WEB IN COURSE INSTRUCTION....................................................2 

Roles of the web in instructional settings............................................................................... 3 

THE EFFECTIVENESS OF CBI.........................................................................................................4 

THE EFFECTIVENESS OF WEB-BASED INSTRUCTION......................................................................7 

METHODS AND PROCEDURE................................................................................................... 8 

STUDY CHARACTERISTICS ............................................................................................................9 

Content Area ............................................................................................................................9 

Educational Level ....................................................................................................................9 

METHODOLOGICAL CHARACTERISTICS.......................................................................................10 

Sample Size........................................................................................................................... 10 

Use of A Comparison Group ................................................................................................ 10 

COURSE CHARACTERISTICS ........................................................................................................10 

Level of Web-Use ..................................................................................................................10 

Variables Assessed.................................................................................................................11 

Attrition..................................................................................................................................12 

COMPARISON TO CBI..................................................................................................................12 

CONCLUSION............................................................................................................................. 13 

REFERENCES ............................................................................................................................. 15 

Appendix A Characteristics of Studies on Web-Based Instructional Effectiveness..............A-1 

LIST OF TABLES 

Table 1 Average effect sizes for four types of knowledge outcomes for CBI.........................5 

Table 2 Average effect sizes for three instructional groups using CBI................................5 

Table 3 Average effect sizes for six categories of CBI..................................................6

ix
Table 4 Descriptive statistics for studies included in the review.......................................9 

Table 5 Comparison of effect sizes across analyses....................................................12 

x 
THE EFFECTIVENESS OF WEB-BASED INSTRUCTION 

The U.S. Army plans to increase the use of distributed learning to meet the future training

needs of a responsive and versatile force. The goals of the Army Training and Doctrine 

Command are to provide digital training facilities within 50 miles of 95 percent of Army duty

stations and develop more than 525 distributed learning courses. The courses will encompass 

online instruction distributed over communication networks, blended to varying degrees with 

CD-ROM, print material, email and chats, and conventional hands-on instruction and testing as

needed. For the versatile force of the future, learning will be centered around the soldier instead

of the classroom. A central question, then, is just how effective is online instruction? In

particular, how does it compare to both the conventional classroom and established forms of 

stand-alone, computer-based instruction (CBI) prior to the World Wide Web?

For the purposes of this review, online instruction is considered to be any educational or 

training program distributed over the Internet or an intranet and conveyed through a browser, 

such as Internet Explorer or Netscape Navigator. Hereafter, it is referred to as Web-based 

instruction. The use of browsers and the Internet is a relatively new combination in instructional 

technology. While the effectiveness of traditional CBI has been reviewed quite thoroughly

(Kulik, 1994; Lou, Abrami, & d’Apollonia, 2001), the effectiveness of online instruction has 

received little analysis. Part of the reason is because few cases have been detailed in the 

literature. This report serves, then, as an initial look at the empirical evidence for its 

instructional effectiveness. 

Advantages of Training Technology

The Army seeks to take advantage of the benefits offered by distributed learning, such as 

reduced travel costs, increased accessibility, and improvements in learning. The cost advantage

of even basic forms of technology-based instruction has been documented (Fletcher, 1990; 

Wisher, Priest, & Glover, 1997). Learning advantages have consistently been found whenever 

well-designed instruction is coupled with computer delivery. Fletcher (2001), for example, has 

established the “Rule of Thirds” based on an extensive review of the empirical findings in 

educational and training technology. This rule advises that the use of CBI reduces the cost of 

instruction by about one-third, and additionally either reduces the time of instruction by about 

one-third or increases the effectiveness of instruction by one-third. The analysis for this rule was 

based primarily on stand-alone CBI, not the contemporary use of online technologies. 

There is no practical reason to expect Web-based instruction to be any less effective than

traditional CBI. Both are capable of interactivity, individual feedback, and multi-media

presentation. However, technical limitations with current Web-based configurations may dilute 

some of these advantages. Inherent limitations such as a small viewing area for video, video 

with a slow frame speed, or delays in responsiveness as a result of high traffic load on the 

Internet may restrict its current effectiveness. On the other hand, Web-based instruction offers 

new advantages to the learner, such as interactivity with instructors and students and quick 

access to supplementary online resources. As the technology improves, Web-based instruction 

may have an ultimate advantage.

1 
This report first reviews the numerous ways in which the World Wide Web is influencing

instructional practices. Then, the parameters for and results of a search on the evaluation 

literature are described. The results of a meta-analysis of the evaluation data are reported and a

central tendency of instructional effectiveness is identified. How well this central tendency,

represented as a standardized effect size, compares to both the conventional classroom and to 

traditional uses of the computer as a stand-alone delivery device is discussed. Finally, some 

conclusions are drawn from this initial inquiry into the effectiveness of Web-based instruction. 

The Use of the World Wide Web In Course Instruction 

The World Wide Web can be used in various ways for instructional and instructional 

support purposes. For example, Web-based instruction offers the learner unparalleled access to 

instructional resources, far surpassing the reach of the conventional classroom. Web-based 

instruction also makes possible learning experiences that are open, flexible, and distributed, 

providing opportunities for engaging, interactive, and efficient instruction (Kahn, 2001). Phrases 

such as “flexible navigation,” “richer context,” “learner centered,” and “social context of 

learning,” are populating the literature on Web-based instruction. Furthermore, the rendering of 

cognitive-based theories of learning into the terminology of Web-based instruction extends the

technical nomenclature to actual instructional practices (Bonk & Dennen, 1999). Indeed, Dills 

and Romiszowksi (1997) have identified more than 40 instructional paradigms seeking to 

advance and improve the online learning experience beyond the conventional classroom. 

Some researchers have argued, however, that the tried-and-true principles of instructional 

design, namely interaction and timely feedback, are often absent from Web-based instruction, 

particularly from individual websites devised to teach (Eli-Tigi & Branch, 1997). The absence 

of a sturdy pedagogical underpinning for a Web-based program can diminish an otherwise

worthy opportunity to improve learning. Well-designed CBI developed in the 1970s and 1980s, 

for example, has demonstrated an enhancement to learning outcomes when compared to

classroom instruction (Kulik & Kulik, 1991). How does Web-based instruction compare to the 

classroom at this relatively early stage of its implementation? In view of the extraordinary

potential for Web-based instruction, it is somewhat surprising that there are few empirical 

assessments of its effectiveness. 

Unlike the fixed resources in conventional CBI, Web-based instruction can be 

conveniently modified and redistributed, readily accessed, and quickly linked to related sources 

of knowledge. This establishes a backbone for “anytime, anywhere” learning (Fletcher &

Dodds, 2001). Compare these features to, say, a pre-Internet CD-ROM in which instructional 

messages were encoded in final form, availability was limited to specific computers, and 

immediate access to a vast array of related materials was not possible. To be sure, many key

instructional features, such as learner control and feedback, are shared between Web-based and 

conventional CBI. 

A reasonable assumption concerning the effectiveness of Web-based instruction, then, is 

that it should be at least “as good as” conventional forms of CBI. That is, the effect size of 

evaluations that compare the effectiveness of either to the classroom should be comparable. The 

2 
qualities shared by the two delivery media include multi-media formats, self-pacing, tailored 

feedback, and course management functions. Additionally, the unique features of Web-based 

instruction – flexible courseware modification, broad accessibility, and online links to related 

materials, instructors, and fellow students – should make possible improvements in learning

outcomes beyond what CBI has demonstrated. Within the CBI literature as described in detail 

shortly, the learning outcomes (when compared to conventional classroom instruction) have 

demonstrated effect sizes of between 0.3 and 0.4 standard units (Fletcher, 1990; Kulik, 1994). 

The measure of effect size is simply a way of quantifying the difference between two groups. 

For example, if one group had an experimental treatment (Web-based instruction) and the other 

did not (the conventional classroom), then the effect size is a measure of the effectiveness of the 

Web-based treatment compared to that of the classroom. In statistical terms, the effect size

describes the difference between two group means divided by either the pooled standard 

deviation or the standard deviation of the treatment group (Glass, McGaw, & Smith, 1991). An 

advantage of using effect size is that numerous studies can be combined to determine an overall 

best estimate, or central tendency, of the effect. Generally, values of 0.2, 0.5, and 0.8 standard 

units are considered to correspond to small, medium, and large effect sizes, respectively (Cohen, 

1988). (N.B. Effect sizes can be quickly translated to percentiles from a table of standard 

distributions.) Our premise in the present analysis is that if Web-based instruction is employed 

properly, and in a similar manner to conventional CBI, it should lead to an effect size that is 

comparable to that demonstrated for CBI. The assumption of being “as good as” CBI is 

reasonable, of course, when the Web is applied for purposes similar to conventional CBI, namely

for achieving specific learning objectives. 

Roles of the Web In Instructional Settings

Web-based instruction presents multiple dimensions of use in education and training

environments. As with CBI, it is capable of providing direct instruction to meet individual 

learning objectives. Due to its networking capability, the Web can assume additional roles. 

These include promoting and facilitating enrollment into courses, availing the syllabus or

program of instruction, posting and submitting assignments, email between instructors and 

fellow students, collaboration on assignments, and building learning communities. 

The Web has become a powerful tool for learning and teaching at a distance. Its inherent

flexibility allows application in a variety of ways within an educational context, ranging from 

simple course administration and student management to teaching entire courses online. Each of

these “levels of use” works towards a different goal. These goals should be recognized when 

evaluating the use of the Web. For example, an instructor may hold face-to-face lectures in a 

classroom but post the class syllabus, assignments, and grades on the Web. In this case, it would 

not be appropriate to evaluate the use of the Web with respect to learning outcomes, since the 

Web was not used in a direct instructional role. 

There are a host of factors that contribute to a meaningful learning environment. In an 

attempt to gain a systematic understanding of these factors, Kahn (1997) developed a framework 

for Web-based learning, consisting of eight dimensions: pedagogical, technological, interface

design, evaluation, management, resource support, ethical, and institutional. Kahn (2001) later 

3 
offered a framework for the extent of Web-based instruction along a continuum ranging from 

“micro” uses to “macro” uses. The “micro” end of the continuum involves the use of the Web as 

a way to supplement or enhance conventional classroom instruction (e.g., providing students in a 

biology course with an interactive map of the brain to help them learn brain functions). Further 

along the continuum are courses that are partly delivered over the Web, such as elective modules 

that supplement classroom instruction. Clearly, factors beyond pedagogy such as technical 

reliability, interface design, and evaluation become more important at this level. Finally, at the

“macro” end of the continuum are complete distance learning programs and virtual universities. 

Other researchers have recognized the importance of determining the level of Web-use in 

a course. For example, Galloway (1998) identified three levels of Web-use. In Level 1, the Web 

is used to post course material with little or no online instruction. The instructor guides students 

to the relevant information rather than obliging the students to search for information. In Level 

2, the Web is used as the medium of instruction. Course assignments, group projects, and lecture 

notes are posted on the Web. The teacher becomes the facilitator of knowledge, guiding the 

student rather than telling them what to do. In addition, there is increased student-student 

interaction. Courses that are offered completely online fall into Level 3. Teachers and students 

interact only over the Internet, and knowledge of using the technology is extremely important at 

this level. 

Web-based instruction is still in an early stage of implementation. Nevertheless, 

educational institutions, private industry, the government, and the military anticipate immense 

growth in its use. In its 1999 distance education statistical analysis, for example, the National 

Center for Education Statistics found that 82% of higher education institutions plan to increase 

their use of Web-delivered courses (Johnson, 2001). In addition, it is estimated that by 2003, 

approximately $11.4 billion will be spent for online training and more than 30% of corporate 

education will be conducted online (Bonk & Cummings, 1998). 

Obstacles to realizing the Web’s full potential for learning clearly remain. These include

the appropriateness of pedagogical practices (Fisher, 2000) and the bandwidth bottleneck for 

certain learner requests, for example video on demand (Saba, 2000). From an evaluation 

perspective, there has been an inclination for media comparison, such as the effectiveness of a 

technology relative to the conventional classroom (Wisher & Champagne, 2000). However, the 

more appropriate benchmark might be a comparison to the historical findings on the 

effectiveness of conventional CBI. An assessment of current practices, then, must consider 

whether the capabilities of the Web are being tapped, how interpretable the findings are, and how 

strong those findings are in comparison to conventional CBI. 

 

The Effectiveness of CBI 

CBI has been a significant part of educational technology, beginning with the first 

reported use of the computer instructional purposes in 1957 (Saettler, 1990). Its emergence as a 

true multimedia delivery device occurred in the early 1980s with the coupling of a videodisc 

player with a computer. In recent years, the videodisc has been replaced by the CD-ROM. The 

combination of a computer controlling quality video and/or audio segments was a compelling

advancement in CBI, and the instructional effectiveness of this pairing has been documented. 

4 
Since our premise is that Web-based instruction should be at least “as good as” conventional 

CBI, a summary of the findings is needed in order to identify a comparative benchmark. 

Fletcher (1990) conducted a quantitative analysis of the education and training

effectiveness of interactive videodisc instruction. Specifically, empirical studies comparing

interactive videodisc instruction to conventional instruction were segmented into three groups: 

higher education, industrial training, and military training. The various learning outcomes 

investigated included: (1) Knowledge outcomes, which assessed a student’s knowledge of facts 

or concepts presented in the instructional program; (2) performance outcomes, which assessed a 

student’s skill in performing a task or procedure; (3) retention, which measured the durability of 

learning after an interval of no instruction; and (4) the time to complete the instruction. The 

effect sizes, or the difference between the mean scores of the treatment and comparison groups 

divided by the standard deviation of the control group, were computed for each of the 28 studies 

identified. 

The results of the Fletcher (1990) meta-analysis are presented in Table 1, broken down 

by learning outcome, and in Table 2, broken down by instructional group. The effect sizes are 

reported in standard units with a corresponding percentile from a normal distribution. 

Table 1 

Average Effect Sizes for Four Types of Knowledge Outcomes for CBI

Learning Outcome Effect Size Percentile 

Knowledge .36 64th

Performance .33 63rd

Retention .65 74th

Time to Complete 1.19 88th

Table 2 

Average Effect Sizes for Three Instructional Groups Using CBI

Instructional Group Effect Size Percentile 

Higher Education .66 74th

Industrial Training .17 57th

Military Training .39 65th

5 
The conclusion of the Fletcher (1990) analysis was that interactive video instruction was both 

effective and less costly than conventional instruction. 

In a later analysis of the effectiveness of CBI, Kulik (1994) took into account the 

conceptual and procedural differences in how the computer was used in the individual studies. 

In his analysis of 97 studies that compared classes that used CBI to classes that did not, Kulik 

(1994) computed the overall effect size as well the effect sizes corresponding to five categories 

of computer use relevant to the present report: 1) tutoring, 2) managing, 3) simulation, 4) 

enrichment, and 5) programming.

Kulik determined the overall effect size to be .32. This indicates that the average student 

receiving CBI performed better than the average student in a conventional class, moving from 

the 50th percentile to the 62nd percentile. However, the effect sizes when categorized by

computer-use yielded somewhat discrepant results. These results are presented in Table 3. 

Table 3 

Average Effect Sizes for Six Categories of CBI 

Application Effect Size Percentile

Tutoring .38 65th

Managing .14 56th

Simulation .10 54th

Enrichment .14 56th

Programming .09 53rd

The effect size for computer-based programs used for tutoring (.38) is significantly

higher than the rest, indicating that students who use computers for these purposes may achieve 

better outcomes than students who use CBI for management, simulation, enrichment, or 

programming purposes. In addition, it is clear from the table that basic programming and 

simulations had minimal effect on student performance. The conclusion of the Kulik (1994) 

analysis was that researchers must take into account all types of CBI when trying to assess their 

effects on student learning. 

Finally, Liao (1999) conducted a meta-analysis of 46 studies that compared the effects on 

learning of hypermedia instruction (e.g., networks of related text, graphics, audio, and video) to 

different types of non-hypermedia instruction (e.g., CBI, text, conventional, videotape). Results 

indicated that, overall, the use of hypermedia in instruction results in more positive effects on 

6 
students learning than non-hypermedia instruction with an average effect size equal to 0.41. 

However, the effect sizes varied greatly across studies and were influenced by a number of 

characteristics. Effect sizes were larger for those studies that used a one-group repeated measure 

design and simulation. In addition, effect sizes were larger for studies that compared hypermedia 

instruction to videotaped instruction than for studies that compared hypermedia instruction to

CBI. These results seem to suggest that the results of hypermedia instruction greatly depend on 

the type on instruction that it is compared to. 

While each of the studies reviewed above provides evidence for the positive effects of 

CBI on student learning, they also address the complexity of the issue. The relationship between 

CBI and learning is influenced by many variables including the type of media being used, what 

CBI is being compared to, and the type of research design employed, to name just a few. These 

issues increase in complexity when applied to Web-based instruction as instruction becomes less 

linear and more interactive and dynamic. 

The Effectiveness of Web-Based Instruction 

The review encompassed 40 articles published between 1996 and 2002. A tabulation of the 

documented findings into nine dimensions are offered in Appendix A, along with our 

assessments of the experimental designs, effect sizes, and the degree to which the evaluation 

incorporated features unique to Web-based instruction. The studies are first discussed in terms 

of four key features: 1) the degree of interaction in the course; 2) the measurement of learning

outcomes; 3) the experimental design used in evaluating the course; and 4) the extent of Web use 

throughout the course. Since the purpose of this review is to assess current practices in the use 

and evaluation of Web-based courses, the criteria that guided the choice of studies for this review

were broadly defined. The studies had to involve the use of the Web as an instructional tool 

either as a supplement to conventional classroom instruction or as the primary medium of 

instruction. In addition, the studies had to include an evaluation of the Web-based components 

of the course. The studies were directed at the undergraduate and graduate levels of education. 

The evaluations include in these studies fell into two broad categories. The first category

of evaluation consisted of comparisons of Web-based instructional approaches to conventional 

classroom instruction. These evaluations could involve the comparison of a control group with 

an experimental group derived from the same population of students or a simple comparison 

between a class taught at one time without the use of the Web and the same class taught at a 

different time using the Web. The second category of evaluations involved the assessment of 

student performance and reactions relative to a single course. 

Methods and Procedure 

The Educational Resources Information Center (ERIC) and Psychological Abstracts 

databases were searched using the following combinations of key words: “web-based courses,” 

“web-based instruction,” “web-based courses and evaluation,” “course evaluation and web,” 

“course evaluation and Internet,” “web and distance education,” and “online course and 

evaluation.” Because we were trying to assess current practices in evaluating Web-based 

instruction, the search was restricted to the years between 1996 and 2002. This search, 

7 
conducted between August 2000 and July 2002, identified more than 500 qualifying studies. 

However, most of these studies concerned recommendations for the design of online courses or

technology concerns rather than an evaluation of a specific course, so they were not included in 

the review. In addition, we found several relevant studies from the Journal of Asynchronous 

Learning, Education at a Distance, and the previous four years of the Proceedings of the 

Distance Teaching and Learning Conference. Finally, we received useful references and 

citations from researchers in the field of distance education that had not been identified by our 

search of the databases. 

We narrowed the larger set of studies to 47 based on reports of empirical evidence for 

instructional effectiveness. These are summarized in Appendix A, which is organized 

alphabetically by the first author’s last name. Although most of the studies involved courses in 

the physical sciences, the instructional content of the courses concerned a variety of subject 

matters, including philosophy, nutrition, economics, and sports science. 

Our review of the literature on Web-based instruction is organized according to three

categories: 1) study characteristics, 2) methodological characteristics, and 3) course 

characteristics. The analysis of these characteristics provides some insight into what questions 

people are asking about Web-based instruction and how well they are being answered. 

Descriptive statistics for these characteristics are presented in Table 4. 

8 
Table 4 

Descriptive Statistics for Studies Included in Review

Variables N % Variables N % 

 Content Area Course Characteristics 

 Math/Engineering/Computers 11 23 Level of Web Use 

 Science/Medicine 12 26 All-Online 29 62 

 Distance Education Programs 7 15 Blend 17 36 

 Social Sciences/Education 11 23 Mixed between courses 1 2 

 Business 4 9 

 Language 2 4 Attrition Data 

 Yes 18 38 

 Educational Level No 29 62 

 Undergraduate 38 81 

 Graduate 8 17 Variables Assessed 

 Both 1 2 Course design 21 45 

 Demographics 22 47 

Methodological Characteristics Computer Experience 13 28 

 Sample Size Effectiveness of instructor 18 38 

 9-50 13 28 Technical issues 12 26 

 51-100 10 21 Interaction/Participation 12 26 

 Over 100 13 28 Desire to take additional online 6 13

 Not Reported 11 23 Recommendation of course to 

 others

3 6 

 

 Comparison Group 

 Yes 29 62 

 No 18 38 

Study Characteristics 

Content Area

Content areas represented in the studies were wide-ranging. Approximately 23%

examined the effects of Web-based instruction for teaching math, engineering, and computer 

courses whereas 26% focused on the teaching of science and medical courses. Another 23% 

focused on the social sciences and education. In addition, about 15% of the studies evaluated 

entire programs of distance learning, which most likely were comprised of many types of

courses. The wide variety of content areas discovered in this review demonstrates the flexibility

of Web-based instruction to be adapted to the requirements of students and teachers in many

different subject areas. 

Educational Level

Both undergraduate and graduate students were represented in this review. Of the 47 

studies, 81% evaluated Web-based instruction for undergraduate students, 17% evaluated 

graduate students, and 2% evaluated Web-based instruction combined for both graduate and 

9 
undergraduate students. Given the differences in both course content and teaching styles 

between undergraduate classes and graduate classes, it would have been interesting to assess the

differential impact of Web-based instruction on student learning between the two. However, 

insufficient information was provided by the studies to be able to draw any firm conclusions on 

this issue. For example, many of the studies failed to provide means or standard deviations for 

learning outcome measures. In addition, may of the studies with graduate students did not 

describe the methods used to assess student learning and performance or provide adequate

descriptive information as to assess course content and level of Web use. 

Methodological Characteristics 

Sample Size

The sample size of a study can significantly affect the statistical power of the underlying

tests for differences. Of the 47 studies, 36 reported information about the sample sizes of 

participants. These ranged from 9 to 1,406 (M = 153, SD = 245). Of these 36 studies that 

provided sample sizes, most (64%) had sample sizes of fewer than 100 participants. Effect sizes 

were available for 10 of these studies, and the mean effect size was approximately 0.09. For 

studies in which the sample size exceeded 100, the mean effect size increased to 0.55. In

general, the larger the sample-size, the stronger the statistical power. However, since effect sizes 

were available for only 15 of the studies, these results must be interpreted with caution. 

Use of A Comparison Group

The majority of the studies identified for this review used a comparison group in which 

students took the same course face-to-face or the same course with no Web-based components. 

However, 41% of the studies simply evaluated the Web-based course with no comparison group. 

A similar pattern was found in evaluations of distance learning technology in training

environments in which 55% of evaluations did not use a comparison group (Wisher &

Champagne, 2000). While a comparison group is not a requirement for course evaluation, the 

absence of one can threaten the internal validity of the study and restrict the interpretation of the

data. It is difficult to draw conclusions about the impact of instruction using the Web on student 

learning, satisfaction, and other outcomes without an equivalent comparison group. 

Furthermore, of the 29 studies that had a comparison group, only one (Schutte, 1996) randomly

assigned students to conditions. Thus, even most of the studies with comparison groups were 

subject to many possible confounding variables that may have influenced the relationships 

between Web-based instruction and learning outcomes. 

Course Characteristics 

Level of Web-Use

As demonstrated by the results of both Kulik (1994) and Liao (1999), different forms of 

CBI can differentially affect student outcomes. Thus, it is important to take into account how a 

particular medium of instruction is applied when evaluating a course. This is especially true of 

Web-based instruction in view of the medium’s tremendous ability to distribute seemingly

unlimited resources and information to anyone at anytime. The flexibility of the Web allows it 

to be used for a variety of purposes, from course administration and management to complete

course delivery, and each of these “levels of use” works towards a different goal. 

10 
All of the studies surpassed the use of the Web for purely management purposes. Of the 

47 studies, 17 evaluated “blend courses” or courses that are a mix of both face-to-face instruction 

and Web-based components (e.g., posting of course syllabus and lecture notes, online tutorials 

and graphics, etc.). While the use of the Web in these “blend courses” fulfilled many

management functions, students needed to access the Web regularly to be a productive member 

of the class. This access, in turn, caused higher levels of Web-use as defined by the frameworks 

described above. The remainder of the studies included courses that were completely online. At 

this level of use, there is little or no face-to-face interaction, so evaluating the degree of online 

interaction between students and instructors or other students, the availability of feedback, and 

technological issues becomes critical. Of the 30 courses that were completely online, 10 

evaluated student and teacher interactions, the availability of instructor feedback, and 

technological issues. Since these variables are the cornerstones of good instructional design, 

regardless of delivery medium, it is important that they be included in any assessment of Webbased instruction. 

In terms of learning outcomes, effect sizes were available for six of the “blend” courses 

and nine of the all-online courses. The mean effect size for the “blend” courses was 0.48 while 

the mean effect size for the all-online courses was 0.08. Although this is not a statistically

significant difference (Mann-Whitney U Test, p = .14), the direction of the difference is the same

as Liaos’s (1999) analysis, which showed that the mean effect size for courses that used 

hypermedia as a supplement to conventional instruction was 0.18 standard units higher than 

courses that replaced conventional instruction with hypermedia. Taken together, the results 

appear to suggest that Web-based instruction may be more beneficial for student learning when 

used in conjunction with conventional classroom instruction but a larger sample is needed. 

Variables Assessed 

Much of the research that evaluates Web-based instruction, and distance education as a 

whole, lacks a guiding theoretical framework (Saba, 2000; Berge & Mrozowski, 2001). 

Furthermore, there is no consensus as to what variables are important to examine when 

evaluating Web-based courses. Determining the evaluation elements becomes more complicated

for online courses than for face-to-face instruction as online courses incorporate unique elements 

such as flexibility, a wide range of corresponding resources and tools, and technological 

considerations among others. As a result, researchers have assessed a wide range of variables. 

For the studies included in this analysis, the variables assessed can be grouped into eight major

categories:

 Demographics (age, gender, race,)  Technical issues 

 Previous computer/Internet 

experience 

 Level of participation/collaboration 

 Course design

 Recommendation of course to 

others 

 Effectiveness of the instructor 

 Desire to take additional online

courses 

11 
Although many of the studies assessed the design of the Web-based course and the demographic 

characteristics of the participants, fewer evaluated the quality of interaction or collaboration in 

the course, the effectiveness of the instructor, and the technology itself. 

Attrition

It has been widely recognized that the attrition of students is a greater problem for online 

courses than classroom courses (Phipps & Merisotis, 1999; Terry, 2001). In addition, some 

research has shown that blended courses should be considered separately from completely online 

courses when assessing student attrition as blended courses have lower attrition rates (Bonk 

2001). However, only 14 (34%) of the studies reported information about attrition. This is 

unfortunate because of the lost opportunity to better understand the characteristics and motives 

underlying students’ decisions to drop online courses. 

Comparison to CBI

When compared to conventional classroom instruction, the learning outcomes from 

conventional CBI have demonstrated effect sizes significantly above the “no significant 

difference” threshold (Fletcher, 1990; Kulik, 1994). The original premise stated that if Webbased instruction was employed properly, and used to achieve specific learning objectives, it 

should lead to effect sizes that are at least comparable to CBI. Of the 15 studies in this analysis 

that provided sufficient information to calculate effect sizes, eight (53%) of the effect sizes were

positive and favored the group that used Web-based instruction, while seven (47%), were 

negative and favored the group that did not use Web-based instruction. The effect sizes ranged 

from -.40 to 1.60. The grand mean for all 15 effect sizes was 0.24, and the grand median was 

0.095. The standard deviation of 0.58 indicates the great variability of effect sizes across studies. 

Because these data did not meet the assumptions of normality and homoscedasticity, nonparametric tests were performed on the mean effect sizes of the current analysis as well as the 

analyses conducted by Kulik (1994) and Liao (1999) to compare Web-based instruction to CBI. 

Results of a Kruskal-Wallis Test indicated that there was no significant difference (p = .47) in 

mean effect size across the three sets of analyses. The average effect sizes and their 

corresponding standard deviations for the studies included in this article and the previous 

analyses conducted by Kulik and Liao are listed in Table 5. 

Table 5 

Comparison of Effect Sizes Across Analyses 

Web-Based 

Instruction 

CBI

Kulik (1994) 

CBI

Liao 

(1999)

Average Effect Size .24 .32 .41 

Percentiles 59th 63rd 66th

Average S.D. of Effect

Sizes .58 .39 .87

Number of Studies 15 97 46 

12 
These results would seem to suggest that Web-based instruction is “as good” as CBI but, as will 

be discussed in the following section, this may be a premature conclusion. 

Conclusion 

On the basis of a limited number of empirical studies, Web-based instruction shows an

improvement over conventional classroom instruction. However, it is debatable whether Webbased instruction compares favorably to CBI. On the surface, the overall effect size is smaller,

but not statistically significant. This, of course, cannot be interpreted to say that they are 

equivalent, but rather that there is no detectable difference. This is apparently due to inconsistent 

and widespread variability in the findings. As the number of studies that report comparative data

increase, leading possibly to a more stable central tendency in the effect size, a more reliable

assessment of how well Web-based instruction compares to CBI will be possible.

There are numerous reasons why the effectiveness of Web-based instruction may not yet 

be fully realized. For example, many of the early adapters were apparently faculty from a 

diversity of fields who were not necessarily trained in the art and science of instructional design. 

Their comparisons were, in some cases, a first attempt at Web-based instruction compared to a

highly practiced classroom rendition of the course. Another restriction may have been response 

delays not uncommon during peak usage periods on the Internet, in contrast to the immediate

responses possible with a stand-alone computer. With packet-based networks, variable delays

cause latency problems in the receipt of learning material, particularly with graphic images. 

Previous research has demonstrated a slight decrement in learning due to inherent delays of

transmitting complex graphics over the Internet (Wisher & Curnow, 1999). 

One objective of this article was to discuss the various roles that the Web plays in 

educational courses and the importance of identifying this factor when evaluating courses. Here, 

we have limited our analysis to those applications involving direct instruction through the Web. 

As described earlier, there are many other advantages that the Web offers (e.g. access, flexibility,

enrollment, and management) that must be factored in when determining the overall value that

the Web offers to a learning enterprise. 

How large a learning effect, in terms of an effect size, can we expect from the Web? One 

possibility comes from research on intelligent tutoring systems. These are knowledge-based 

tutors that generate customized problems, hints, and aids for the individual learner as opposed to 

ad-hoc, frame-oriented instruction. When compared to classroom instruction, evaluations 

indicate an effect size of 1.0 and higher (Woolf & Regian, 2000; Wisher, Macpherson. 

Abramson, Thornton, & Dees, 2001). If these individual learning systems are further 

complemented by collaborative learning tools and online mentoring, effect sizes on the order of 

2 standard deviation units, as suggested by (Bloom, 1984), may someday be possible. 

The use of the Web for instruction is at an early stage of development. Until now, there

has been a lack of tools for instructional developers to use, but this is beginning to change with 

the emergence of common international standards (Fletcher & Dodds, 2001). Such standards 

and specifications, such as the Sharable Content Object Reference Model, allow vendors to 

13 
develop educational and training content, authoring tools, and repositories that are interoperable. 

The potential of Web-based instruction for the Army will be furthered as pedagogical practices

improve, advances in standards for structured learning content progress, and improvements in 

bandwidth are made. For now, it appears that Web- based instruction is moving towards the 

level of effectiveness previously achieved by CBI. As suggested by Fletcher (2001), the Rule of

Thirds, reduce the cost of instruction by about one-third, and additionally either reduce the time 

of instruction by about one-third or increase the effectiveness of instruction by one-third, is 

likely to apply to Web-based instruction. 

14 
References 

Angulo, A. J., & Bruce, M. (1999). Student perceptions of supplemental Web-based instruction. 

Innovative Higher Education, 24, 105-125. 

Arvan, L., Ory, J. C., Bullock, C. D., Burnaska, K. K., & Hanson, M. (1998). The SCALE 

Efficiency Projects. Journal of Asynchronous Learning Networks, 2, 33-60. 

Bee, R. H., & Usip, E. E. (1998). Differing attitudes of economics students about web-based 

instruction. College Student Journal, 32, 258-269.

Berge, Z. L., & Mrozowski, S. (2001). Review of research in distance education, 1990 to 1999. 

The American Journal of Distance Education, 15, 5-19. 

Bloom, B. S. (1984). The 2-Sigma problem: The search for methods of group instruction as 

effective as one-on-one tutoring. Educational Researcher, 13, 4-16. 

Bonk, C. J., & Cummings, J. A. (1998). A dozen recommendations for placing the student at the 

centre of web-based learning. Educational Media International, 35, 82-89. 

Bonk, C. J., & Dennen, V. P. (1999). Teaching on the Web: With a little help from my 

pedagogical friends. Journal of Computing in Higher Education, 11, 3-28. 

Bonk, C. J. (2001). Online teaching in an online world. Bloomington, IN: Courseshare.com 

Available at: http://www.courseshare.com/reports.php. 

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Ed.). Hillsdale, NJ: 

Lawrence Erlbaum Associates. 

Cooper, L. W. (2001). A comparison of online and traditional computer applications classes. 

T.H.E. Journal, 28, 52-58. 

Davies, R. S., & Mendenhall, R. (1998). Evaluation comparison of online and classroom 

instruction for HEPE 129 – Fitness and lifestyle management course. Report to Brigham 

Young University. 

Dills, C., & Romiszowski, A. J. (1997). Instructional development paradigms. Englewood 

Cliffs, NJ: Educational Technology Publications. 

Eli-Tigi, M., & Branch, R. M. (1997). Designing for interaction, learner control, and feedback 

during Web-based learning. Educational Technology, 37, 23-29. 

Fisher, S. G. (2000). Web-based training: One size does not fit all. In K. Mantyla (Ed.), The

2000/2001 Distance learning yearbook. New York: McGraw-Hill 

15 
Fletcher, J. D. (1990). Effectiveness and cost of interactive videodisc instruction in defense 

training and education. Institute for Defense Analyses Paper P-2372. 

Fletcher, J. D. (2001). Technology, the Columbus effect, and the third revolution in learning. 

Institute for Defense Analyses Paper D-2562. 

Fletcher, J. D., & Dodds, P. (2001). All about ADL. In K. Mantyla & J.A. Woods (Eds.), The 

2001/2002 ASTD distance learning yearbook (pp. 229-235). New York: McGraw Hill. 

Fredericksen, E., Pickett, A., Shea, P., Pelz, W., & Swan, K. (2000). Student satisfaction and 

perceived learning with on-line courses: Principles and examples from the SUNY Learning 

Network. Journal of Asynchronous Learning Networks, 4, 1-28. (Includes separate

evaluations of two courses) 

Gagne, M., & Shepherd, M. (2001). Distance learning in accounting: A comparison between a 

distance and traditional graduate accounting class. T.H.E. Journal, 28, 58-65. 

Galloway, G. M. (March, 1998). A model of Internet usage for course delivery. Paper 

presented at the 9th Annual International Conference of the Society for Information Technology

and Teacher Education (SITE), Washington D.C.

Glass, G. V., McGaw, B, & Smith, M. L. (1981). Meta-analysis in social research. Beverly

Hills, CA: Sage Publications 

Green, R., & Gentemann, K. (2001). Technology in the curriculum: An assessment of the 

impact of on-line courses. Available at: http://assessment.gmu.edu/reports/Eng302. 

Johnson, S. D., Aragon, S. R., Shaik, N., & Palma-Rivas, N. (2000). Comparative analysis of 

online vs. face-to-face instruction. (ERIC Document No. ED 448722). 

Johnson, S. M. (2001). Teaching introductory international relations in an entirely web-based 

environment. Education at a Distance, 15, 5-14. 

Jones, E. R. (February, 1999). A comparison of an all web-based class to a traditional class. 

Paper presented at the 10th Annual International Conference of the Society for Information 

Technology and Teacher Education, San Antonio, Texas. 

Kahn, B. H. (1997). A framework for Web-based learning. Paper presented at the Instructional 

Technology Department, Utah State University, Logan, Utah. 

Kahn, B. H. (2001). Web-based training: An introduction. In B.H. Kahn (Ed.) Web-based 

Training. Englewood Cliffs, NJ: Educational Technology Publications. 

Kulik, C. L., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated 

analysis. Computers in Human Behavior, 7, 75-94. 

16 
Kulik, J. A. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker, 

E.L. & O’Neil, H.F. (Eds.), Technology assessment in education and training (pp. 9-33). 

Hillsdale, NJ: Lawrence Erlbaum Associates. 

LaRose, R., Gregg, J., & Eastin, M. (July, 1998). Audiographic telecourses for the web: An 

experiment. Paper presented at the International Communication Association, Jerusalem, 

Israel. 

Leasure, A. R., Davis, L., & Thievon, S. L. (2000). Comparison of student outcomes and 

preferences in a traditional vs. World Wide Web-based baccalaureate nursing research course. 

Journal of Nursing Education, 39, 149-154. 

Liao, Y. K. C. (1999). Effects of hypermedia on students’ achievement: A meta-analysis. 

Journal of Educational Multimedia and Hypermedia, 8, 255-277. 

Lou, Y., Abrami, P. C., & d’Apollonia, S. (2001). Small group learning and individual learning

with technology: A meta-analysis. Review of Educational Research, 71, 449-521. 

Magalhaes, M. G. M., & Schiel, D. (1997). A method for evaluation of a course delivered via 

the World Wide Web in Brazil. In Moore, M. G. & Cozine, G. T. (Eds.) Web-Based 

Communications, the Internet, and Distance Education. University Park, PA: The 

Pennsylvania State University. 

Maki, W. S., & Maki, R. H. (2002). Multimedia comprehension skill predicts differential 

outcomes of web-based and lecture courses. Journal of Experimental Psychology: Applied, 8, 

85-98. 

Maki, R. H., Maki, W. S., Patterson, M., & Whittaker, P. D. (2000). Evaluation of a web-based 

introductory psychology course: I. Learning and satisfaction in on-line versus lecture courses. 

Behavior Research Methods, Instruments, and Computers, 32, 230-239. 

McNulty, J. A., Halama, J., Dauzvardis, M. F., & Espiritu, B. (2000). Evaluation of web-based 

computer-aided instruction in a basic science course. Academic Medicine, 59-65. 

Murphy, T. H. (2000). An evaluation of a distance education course design for general soils. 

Journal of Agricultural Education, 41, 103-113. 

Navarro, P., & Shoemaker, J. (2000). Performance and perceptions of distance learners in 

cyberspace. In Moore, M.G. & Cozine, G.T. (Eds.) Web-Based Communications, the Internet, 

and Distance Education. University Park, PA: The Pennsylvania State University. 

Phelps, J., & Reynolds, R. (1999). Formative evaluation of a web-based course in meteorology. 

Computers and Education, 32, 181-193. 

17 
Phipps, R., & Merisotis, J. (1999). What’s the difference?: A review of contemporary research 

on the effectiveness of distance learning in higher education. Washington D.C.: The Institute

for Higher Education Policy.

Powers, S. M., Davis, M., & Torrence, E. (October, 1998). Assessing the classroom 

environment of the virtual classroom. Paper presented at the Annual Meeting of the MidWestern Educational Research Association, Chicago, Illinois. 

Ryan, M., Carlton, K. H., & Ali, N. S. (1999). Evaluation of traditional classroom teaching

methods versus course delivery via the World Wide Web. Journal of Nursing Education, 38, 

272-277. 

Saba, F. (2000). Research in distance education: A status report. International Review of 

Research in Open and Distance Learning, 1, 1-9. 

Saettler, P. (1990). The evolution of American educational technology. Englewood, CO: 

Libraries Unlimited. 

Sandercock, G. R. H., & Shaw, G. (1999). Learners’ performance and evaluation of attitudes 

towards web course tools in the delivery of an applied sports science module. Asynchronous

Learning Networks Magazine, 3, 1-10. 

Schlough, S., & Bhuripanyo, S. (March, 1998). The development and evaluation of the internet 

delivery of the course “Task Analysis”. Paper presented at the 9th International Conference for 

Information Technology and Teacher Education (SITE), Washington D.C.

Schulman, A. H., & Sims, R. L. (1999). Learning in an online format versus an in-class format: 

An experimental study. T.H.E. Journal, 26, 54-56. 

Schutte, J. G. (1996). Virtual teaching in higher education: The new intellectual superhighway 

or just another traffic jam? California State University, Northridge. Available at: 

http://www.csun.edu/sociology/virexp.htm. 

Serban, A. M. (2000). Evaluation of fall 1999 online courses. Education at a Distance, 14, 45-

49. 

Shapley, P. (2000). On-line education to develop complex reasoning skills in organic chemistry. 

Journal of Asynchronous Learning Networks, 4. 

Shaw, G. P., & Pieter, W. (2000). The use of asynchronous learning networks in nutrition 

education: Student attitude, experiences, and performance. Journal of Asynchronous Learning 

Networks, 4, 40-51. 

Shuell, T. J. (April, 2000). Teaching and learning in an online environment. Paper presented at 

the meeting of the American Educational Research Association, New Orleans, Louisiana. 

18 
Stadtlander, L. M. (1998). Virtual instruction: Teaching an online graduate seminar. Teaching 

of Psychology, 25, 146-148. 

Summary, R., & Summary, L. (April, 1998). The effectiveness of the World Wide Web as an 

instructional tool. Paper presented at the 3rd Annual Mid-South Instructional Technology

Conference, Murfreesboro, Tennessee.

Taylor, C. D., & Burnkrant, S. R. (1999, Virginia Tech). Spring 1999: Online Courses. Assessment 

Report to the Institute for Distance and Distributed Learning. 

Terry, N. (2001). Assessing the enrollment and attrition rates for the online MBA. T.H.E. 

Journal, 28, 64-68. 

Thomson, J. S., & Stringer, S. B. (August, 1998). Evaluating for distance learning: Feedback 

from students and faculty. Paper presented at the 14th Annual Conference on Distance 

Teaching and Learning, Madison, Wisconsin. 

Trier, K. K. (1999). Comparison of the effectiveness of Web-based instruction to campus 

instruction. Available at: http://www.as1.ipfw.edu/99tohe/presentations/trier.htm. 

Verbrugge, W. (June, 1997). Distance education via the Internet: Methodology and results. 

Paper presented at the 30th Annual Summer Conference Proceedings of Small Computer Users 

in Education, North Myrtle Beach, South Carolina.

Wang, X. C., Kanfer, A., Hinn, D. M., & Arvan, L. (2001). Stretching the boundaries: Using

ALN to reach on-campus students during an off-campus summer session. Journal of 

Asynchronous Learning Networks, 5, 1-20. 

Waschull, S. B. (2001). The online delivery of psychology courses: Attrition, performance, and 

evaluation. Teaching of Psychology, 28, 143-147.

Wegner, S. B., Holloway, K. C., & Garton, E. M. (1999). The effects of internet-based 

instruction on student learning. Journal of Asynchronous Learning Networks, 3, 98-106. 

White, S. E. (April, 1999). The effectiveness of web-based instruction: A case study. Paper 

presented at the Annual Meeting of the Central States Communication Association, St. Louis, 

Missouri 

Wisher, R.A., & Champagne, M. (2000). Distance learning and training: An evaluation 

perspective. In S. Tobias & J. Fletcher (Eds.), Training and retraining: A handbook for 

business, industry, government, and military. New York: Macmillan Reference USA. 

Wisher, R.A, & Curnow, C.K. (1999). Perceptions and effects of image transmissions during

Internet-based training. The American Journal of Distance Education, 13, 37-51. 

19 
Wisher, R. A, Macpherson, D. H., Abramson, L. J., Thornton, D. M., & Dees, J. J. (2001). The

virtual sand table: Intelligent tutoring for field artillery training. Research Report # 1768, 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Wisher, R.A, Priest, A., & Glover, E. (1997). Audio teletraining for unit clerks: a costeffectiveness analysis. (Research Report 1712) Alexandria, VA: U.S. Army Research Institute 

for the Behavioral and Social Sciences. 

Woolf, B. P., & Regian, J. W. (2000). Knowledge-based training systems and the engineering of 

instruction. In S. Tobias and J. Fletcher (Eds.), Training & retraining: A handbook for 

business, industry, government, and the military (pp. 339-356). New York: Macmillan 

Reference USA. 

20 
Appendix A 

Characteristics of Studies On Web-Based Instructional Effectiveness

Source Course Evaluated Educational

Level

Extent of

Web Use Sample Size Comparison 

Group

Attrition

Data

Learning

Outcome Effect Size

Angulo & Bruce (1999)

Psychology, Drama,

Philosophy, English 

Literature, Chemistry

Undergraduate 

and Graduate Blend 290 No No N/A N/A

Arvan et al. (1998a) Spanish Undergraduate Blend 118 (78 Web,

40 No Web) Yes No ExamScores -0.19

Arvan et al. (1998b) Statistics Undergraduate Blend 304 (203 Web,

101 No Web) Yes No Exam

Scores 0.36

Arvan et al. (1998c) Electrical and Computer

Engineering Undergraduate Blend Not Reported No No N/A N/A

Arvan et al. (1998d) Chemistry Undergraduate All Online Not Reported No No N/A N/A

Arvan et al. (1998e) Calculus Undergraduate Blend Not Reported Yes No N/A N/A

Arvan et al. (1998f) Biology Undergraduate Blend Not Reported No No N/A N/A

Bee & Usip (1998) Statistics Undergraduate Blend 153 (78 Web,

75 No Web) Yes No N/A N/A

Cooper (2001) Fundamentals of Computer 

Applications Undergraduate All Online 131 (37 Web,94 No Web) Yes Yes N/A N/A

Davies & Mendenhall 

(1998)

Fitness and Lifestyle 

Management Undergraduate All Online 96 (57 Web, 39 

No Web) Yes No Exam

Scores -0.13

Fredericksen et al.

(2000a)

Program-wide evaluation of

SUNY Learning Network

(online instructional 

program)

Undergraduate All Online 1,406 No No N/A N/A

Fredericksen et al.

(2000b)

Program-wide evaluation of

the Internet Academy at

Herkimer County Community

College

Undergraduate All Online Not Reported Yes Yes N/A N/A

Fredericksen et al.

(2000c)

Master's Degree program in 

Instructional Technology Graduate All Online Not Reported Yes Yes CourseGrades

Insufficient

Data

Gagne & Shepherd 

(2001) Accounting Graduate All Online Not Reported Yes No

Exam

Scores;

Project

Grades

Insufficient

Data

Green & Gentemann 

(2001) English Undergraduate All Online 57 No Yes Course

Grades

Insufficient

Data

Johnson (2001) Political Science Undergraduate All Online 89 (40 Web, 49 

No Web ) Yes Yes Exam

Scores

Insufficient

Data

Johnson, Aragon, Shaik,

& Palma-Rivas (2000) Human Resources Graduate All Online 38 (19 Web, 19 

No Web) Yes No CourseProjects -0.12

Jones (1999) Statistics Undergraduate All Online 89 (33 Web, 56 

No Web) Yes Yes Exam

Scores

Insufficient

Data

LaRose, Gregg, & Eastin

(1998) Telecommunications Undergraduate Blend 49 (25 Web, 24 

No Web) Yes No ExamScores

Insufficient

Data

Leasure, Davis, & 

Thievon (2000) Nursing Research Graduate All Online 66 (18 Web, 48 No Web) Yes No

Exam

Scores; 

Course

Grades

0.14; 0.28

Magalhaes & Schiel 

(1997) Graphic Mechanics Undergraduate All Online Not Reported No No N/A N/A

Maki et al. (2000) Psychology Undergraduate Blend 333 (151 Web,

182 No Web) Yes Yes Exam

Scores 1.60

Maki & Maki (2002) Psychology Undergraduate Blend 189 (95 Web,

94 No Web) Yes Yes ExamScores 0.37

McNulty et al. (2000) Anatomy Undergraduate Blend 124 No No Course

Grades

Insufficient

Data

Murphy (2000) General Soils Undergraduate Blend 80 (10 Web, 70 

No Web) Yes Yes Exam

Scores 0.66

Navarro & Shoemaker 

(2000) Economics Undergraduate All Online 200 (49 Web,

151 No Web) Yes No ExamScores 0.59

Phelps & Reynolds

(1999) Meteorology Undergraduate All Online Not Reported No No N/A N/A

Powers, Davis, &

Torrence (1998)

Instructional Technology,

Information Technology and 

Media Literacy,

Technologies of Distance

Learning

Graduate All Online 13 No No N/A N/A

Ryan, Carlton, & Ali 

(1999) 7 Nursing Courses Graduate Blend 96 No No N/A N/A

Sandercock & Shaw

(1999) Sports Science Undergraduate Blend 80 Yes No

Assignment

Scores; 

Exam

Scores

1.11; 0.095

Schlough & Bhuripanyo

(1998) Task Analysis Undergraduate All Online 22 No No N/A N/A

Schulman & Sims (1999)

Organizational Behavior,

Personal Finance,

Managerial Accounting,

Sociological Foundations of

Education, Environmental 

Studies

Undergraduate All Online 99 (40 Web, 59 

No Web) Yes No ExamScores 0.012

Schutte (1996) Social Statistics Undergraduate All Online 33 (16 Web, 17 

No Web) Yes Yes Exam

Scores

Insufficient

Data

A-1 
Source Course Evaluated Educational 

Level

Extent of

Web Use Sample Size Comparison 

Group

Attrition

Data

Learning

Outcome Effect Size

Serban (2000) Various Courses Undergraduate

All Online 

and Blend 

courses

Not Reported Yes Yes Course

Grades

Insufficient

Data

Shapley (2000) Organic Chemistry Undergraduate All Online Not Reported Yes No Exam

Scores

Insufficient

Data

Shaw & Pieter (2000) Nutrition Undergraduate Blend 46 No Yes Course

Grades

Insufficient

Data

Shuell (2000) Educational Psychology Undergraduate All Online 12 No Yes N/A N/A

Stadtlander (1998) Motivation Graduate All Online 9 No Yes Course

Grades

Insufficient

Data

Summary & Summary

(1998) Economics Undergraduate Blend 408 Yes No N/A Insufficient

Data

Taylor & Burnkrant

(1999)

 Computer Science, English,

Entomology, Geography,

Math, Political Science,

Women's Studies,

Philosophy

Undergraduate All Online 520 No No N/A N/A

Thomson & Stringer 

(1998) Agricultural Science Undergraduate Blend 170 No No N/A N/A

Trier (1999) Sociology Undergraduate All Online 54 (21 Web, 33 

No Web) Yes Yes

Paper 

Scores; 

Exam

Scores

0.34; 1.21

Verbrugge (1997) Computer Operating 

Systems Undergraduate All Online 13 Yes Yes Course

Grades

Insufficient

Data

Wang et al. (2000) Economics Undergraduate All Online 24 No Yes Course

Grades

Insufficient

Data

Waschull (2001) Psychology Undergraduate All Online 41 (18 Web, 23 

No Web) Yes Yes Exam

Scores -0.33

Wegner, Holloway, & 

Garton (1999)

Curriculum Design and 

Evaluation Graduate All Online 31 (14 Web, 17 No Web) Yes No ExamScores -0.27

White (1999) Communication Technology

and Change Undergraduate All Online 40 (16 Web, 24 

No Web) Yes No Course

Grades -0.40

A-2