U.S. DE PA RTM E N T OF ED U C AT ION

E d u c at ional technology EX PE RT PA N E L

Exemplary&Promising

Educational

TEchnology

2 0 0 0 p r o g r a m s

Educational

TEchnology

EX E M P LA RY A N D PROM I S I N G

ED U C A T I O N A L

TE C H N O L O G YPR O G R A M S

2 0 0 0

U.S. Department of Education

Educational Technology Expert Panel
U.S. Department of Education

Rod Paige

S e c re t a r y

O ffice of Educational Research and Impro v e m e n t

G rover J. Whitehurst

Assistant Secre t a r y

O ffice of Reform Assistance and Dissemination

P e i rce Hammond

D i re c t o r

January 2002

This publication is in the public domain. Authorization to re p roduce it in whole or in part for

educational purposes is granted. The full text of this publication is available at the

Department’s home page at http://www.ed.gov

For online ordering via the Intern e t :

w w w . e d . g o v / p u b s / e d p u b s . h t m l

For ordering information on this report, (publication number ORAD 2001–1015), write:

U.S. Department of Education

Ed Pubs

P.O. Box 1398

Jessup, MD 20794–1398

or call toll free 1–877–4ED–Pubs.

ii
U.S. DEPA RTMENT OF EDUCAT I O N

ED U C A T I O N A L TE C H N O L O G Y EX P E R T PA N E L I S T S

Elsie Brumback

Educational Technology Consultant

(Retired State Technology Coordinator)

North Carolina

Milton Chen

The George Lucas Educational Foundation

California

Jere Confrey

The University of Texas at Austin

Texas

Betty Cronin

Covington High School

Louisiana

Christopher Dede

Harvard University

Massachusetts

Allen Glenn

University of Washington

Washington

Louis Gomez

Northwestern University

Illinois

Margaret Honey

Center for Children and Technology

New York

Caroline McCullen

SAS inSchool

North Carolina

Peter Mich

The Joyce Foundation

Illinois

Jacqueline Miller

Independent Consultant

(Retired Principal,Southeastern High School of

Technology)

Michigan

Lawrence Mixon, Sr.

Columbus Public Schools

Ohio

David Moursund

International Society for Technology in Education

Oregon

James Parry

Technology and Innovations in Education 

South Dakota

Nora Sabelli

National Science Foundation

Virginia

Steven Sanchez

New Mexico State Department of Education

New Mexico

Karen Sheingold

Independent Consultant

(Former Director, Center for Performance 

Assessment,Educational Testing Service)

New Jersey

Robin Willner

IBM Corporation

New York

iii
U.S. DEPA RTMENT OF EDUCAT I O N

E D U C ATIONAL TECHNOLOGY EXPERT PA N E L

Ex e m p la ry and Prom i s i n g

ED U C A T I O N A L TE C H N O L O G Y PR O G R A M S

2 0 0 0

Panel Co-Chairs

Elsie Bru m b a ck

Milton Chen

Wo r k g roup Chairs

C h ristopher Dede,

Chair of the Cri t e ria Wo rk gro u p

Louis Gomez,

Chair of the Vi s i o n ,O ve rall Philosophy and Use of the Pa n e l ’s Wo rk Wo rk gro u p

M a rg a ret Honey,

Chair of the Dissemination Wo rk gro u p

D avid Moursund and Steven Sanch e z ,

C o - C h a i rs of the Rev i ew Process and Rev i ewe rs ’ Training Wo rk gro u p

K a ren Sheingo l d ,

Chair of the Rubrics Wo rk gro u p

Federal Panel Coord i n a t o r s

C h e ryl P.G a rn e t t e ,

C o - C o o rdinator of the Educational Te ch n o l o gy Expert Panel and Director of the Learning Te ch n o l o gies 

D i v i s i o n ,O ffice of Refo rm Assistance and Dissemination,O ffice of Educational Research and Improve m e n t ,

U. S .D e p a rtment of Education,Wa s h i n g t o n ,D C

Diane T.A l e e m ,

C o - C o o rdinator of the Educational Te ch n o l o gy Expert Pa n e l ,L e a rning Te ch n o l o gies Division,

O ffice of Refo rm Assistance and Dissemination,O ffice of Educational Research and Improve m e n t ,

U. S .D e p a rtment of Education,Wa s h i n g t o n ,D C

iv
TA B L E OF CON T E N T S

v

PREFACE............................................................................................................................vi

REPORT ON THE EDUCATIONAL TECHNOLOGY EXPERT PANEL .................................1

EVALUATION CRITERIA ....................................................................................................7

EXEMPLARY PROGRAMS

Challenge 2000 Multimedia Project ..............................................................................11

Generation www.Y.......................................................................................................15

PROMISING PROGRAMS

Maryland Virtual High School CoreModels Project......................................................19

Middle-school Mathematics through Applications Program (MMAP)...........................23 

Modeling Instruction in High School Physics...............................................................27

One Sky, Many Voices...................................................................................................31

The WEB Project...........................................................................................................35 

FURTHER REFLECTIONS BY PANEL...............................................................................39
vi

PR E FA C E

This publication is the result of 20 months of work by the members of the Educational Technology Expert

Panel. From September 1998 to April 2000, the Panel met a total of five times to discuss its mandate, create

review criteria, establish a review process, and review programs. This report includes a look at the Panel

process, the criteria by which submissions were judged, and the descriptions of programs recommended by

the Panel and designated exemplary or promising by the Secretary of Education in 2000. Based on additional

insights and knowledge gained from this major undertaking, the report concludes with a section on further

reflections by the Panel. 
RE P ORT ON T H E ED U CAT ION A L TE C H NOLO G Y

EX P E RT PA N E L

Educational technology: A variety of electronic tools, media, and environments that can be used to

enhance learning, foster creativity, stimulate communication, encourage collaboration, and engage in the 

continuous development and application of knowledge and skills.

IN T ROD U CT ION

Educational technology has now evolved to a point where its power to transform schools is evident. This

report, focused on the potential of technology to affect preK–12 student learning outcomes, is expected to

advance the dynamic field of educational technology by helping 

● practitioners make better informed curriculum and budget decisions; 

● researchers design studies, assessments, and evaluations that are immediately relevant and meaningful 

to practitioners; and

● policymakers shape and support initiatives that are well thought out and that include evaluation as an 

essential element.

By 2001, public schools had invested almost $5.5 billion dollars in technology (Market Data Retrieval’s 2000-

2001 Public School Technology Survey), and 98 percent of public schools had access to the I n t e r n e t

(Internet Access in U.S. Public Schools and Classrooms: 1994-2000, NCES). Currently, the critical questions about educational technology for practitioners, researchers, policymakers as well as parents are: How

can digital technology best be put to use in the service of precollege education? How can this technology be

used to accelerate conventional learning as well as transform what students are learning and understanding?

How can technology contribute to the acquisition of information as well as to the ability to assess reliability of

findings and to synthesize incomplete but relevant data across several disciplines? How can technology enable

students to demonstrate high standards of performance as well as to become contributing citizens of our

increasingly global society and economy?

The field of technology is in a state of constant change. Development has been uneven. Leaders in the field

see pockets of excellence but also areas of genuine need (such as more universal individual classroom access

to the Internet and home computers for more students), particularly on the part of underserved populations.

Many in the education community are aware of the need for more guidance and discourse on how to maximize any investment in technology. Many educators want to know how technology will make a difference to

their students in the classroom. Trained to evaluate curricula and develop methods for teaching traditional

disciplines, many educators feel unprepared to evaluate or use technology, yet they have an obligation to

explain their technology expenditures to the tax-paying public by demonstrating how technology improves

educational outcomes. The use of technology cuts across all academic areas, and it is changing rapidly, gaining greater capabilities seemingly every day. Most institutions—from businesses supported by resource-rich

technology departments dedicated to mastering these powerful, expensive, and evolving electronic tools, to

schools and school districts—are hard-pressed to keep current. Few general, informative, and stable guide1
lines exist anywhere. Thus, concrete examples of successful technology use and thoughtful criteria can be an

important source of practical expertise and information for others.

In the field of education, the basic issues are: What is the appropriate role of the new technologies in education? How can the evolution of technology be directed efficiently and effectively toward improving student

learning for all? How can what is known be shared and developed further? The work of the Educational

Technology Expert Panel is a beginning attempt to address some of these questions. Much more work

remains to be done. 

TH E EX P E RT PA N E L

In 1994, Congress directed the Office of Educational Research and Improvement (OERI) to establish “panels

of appropriate qualified experts and practitioners” to evaluate educational programs and recommend to the

Secretary of Education those programs that should be designated as exemplary or promising. The purpose of

these panels is to provide teachers, administrators, policymakers, and parents with reliable information about

the quality and effectiveness of programs and materials so that they can make better-informed decisions in

their efforts to improve the quality of student learning. Other expert panels have been convened for

Mathematics and Science Education; Gender Equity; and Safe, Disciplined, and Drug-Free Schools.

In 1998, OERI convened an Expert Panel on Educational Te c h n o l o g y. The experts—drawn from universities,

school-district administrations, preK–12 classrooms, research institutions, industry, government, philanthropic organizations—encompassed a wide variety of backgrounds in using technology for education. 

After several months of in-depth discussions, and after an interactive process that solicited and used feedback

from the field, the Expert Panel released its application and criteria for program submissions. The application and criteria are the result of a rare coming together of disparate, occasionally adversarial, points of view.

Teachers insisted on the need for usable ideas from academe and practical support with technology in the

classroom. Researchers insisted on the need for sound data, not anecdotes or hunches. Everyone agreed on

the need for clarity—on the part of the Panel and on the part of programs for the purpose of dissemination. 

The Educational Technology Expert Panel recognized the importance of precise definitions of terms and

objectives very early in the process. The Panel began by defining information technology as a tool with multiple potential applications for education, and agreed that learning to use this tool is necessary but is only one

component of sound educational practice. In the past, technology discussions tended to focus on the infrastructure and hardware, such as how to wire schools or whether to put the computers in the classroom or a

separate laboratory. Today the field has matured sufficiently to focus on how to use the technology to improve

teaching and learning. In fact, the objective of using educational technology should be to create an instructional environment that fosters significant learning for preK–12 students. Technology affords new ways of

teaching established disciplines that are likely to be more effective for more students; equally important, technology can inspire teachers and students alike to move beyond conventional content and deepen the process

of learning in various disciplines. Technology can be used to advance administrative and business functions

in a school or district (e.g., to improve purchasing power, record-keeping, data-driven decision-making,

communications with parents or between principal and faculty), which are important precursors to advancing

2
student achievement. The Panel, however, decided that programs designated by the Department of Education

should focus primarily on student learning. 

The Panel viewed significant learning as being multifaceted and active. It is a combination of skills, knowledge, and the ability to apply and communicate understanding as well as to learn further.

What examples of technology programs working toward this kind of learning would be most useful as models

to the field? The Expert Panel observed that with any endeavor involving technology, what is innovative today

will be standard practice tomorrow. If the Panel had been assembled 2 years previous, the picture would

have been very different. Then, the Internet was still a promising but exotic novelty; a technology plan meant

dealing with equipment and was rarely linked to a plan for education.

One lasting indicator of technological innovation, however, might be student learning that cannot be achieved

without technology. To d a y, for example, schools are using the Internet to connect disparate communities that

otherwise would have no opportunity to meet or interact. Another powerful use of technology is modeling, an

aid to experimentation that seeks to understand the physical world.

The Expert Panel further recognized that an exhaustive review of the many programs currently active would be

impossible. Compared to math and science, disciplines in which the content is fairly stable but approaches

change over time—particularly in the early grades—educational technology is a fast-changing, dynamic tool

that responds to emerging innovations and allows for many different approaches to many disciplines, often several at the same time. As the capacity of technology grows, so does the power of new programs. Thus, the

Panel concluded that the most lasting, far-reaching guidance to the field would come from establishing useful,

thoughtful criteria and dissemination strategies rather than from any particular programs it designated.

The Expert Panel developed the six evaluation criteria and program indicators based on the four categories of

criteria—(1) quality of program, (2) educational significance, (3) evidence of effectiveness, and (4) usefulness to others—established by federal regulations for all panels. The Panel’s six evaluation criteria were

carefully field-tested and resulted in the final review criteria presented on pages 7-10. 

The Panel determined that an exemplary program

● addresses significant educational issues and identifies goals and a design supported by research; 

● improves preK–12 learning;

● contributes to educational excellence for all;

● promotes organizational change;

● makes possible educational gains that cannot be achieved without the use of technology; and

● serves as a model for other educational institutions because it is sustainable, adaptable, and scalable. 

The Panel also determined that a promising program also possesses these qualities, but may not yet have rigorous and sufficient evidence for the program’s achievements. 

3
4

The Expert Panel was not seeking programs that teach basic technology skills (how to use e-mail or the

Internet) or that automate current classroom activities (“drill and practice”). Nor was the Panel interested in

exciting programs developed by teachers in single classrooms because historically, successful scale-up of

approaches idiosyncratic to an individual instructor have proved frustratingly elusive. Rather, the Panel

focused on significant learning for preK–12 students and sought innovative, thoughtful programs that reflect

systemic improvement within and across disciplinary boundaries with various student populations.

From the Panel’s research review and familiarity with educational outcomes in the classroom, it was well

aware that schools are complex organizations. Among the essential concomitants of increased student learning are organizational change (e.g., scheduling of common planning time, new job titles such as technology

c o o r d i n a t o r, and prompt on-site technical support) and structural alignment (e.g., coordinating teacher professional development, curricula for inquiry-based learning, and assessment and curriculum standards) as

well as an active commitment to equity. A truly powerful program benefits from contributions from different

points of view and is successful with many kinds of learners, not just a narrow segment of students.

In addition, the Panel had observed that the most effective programs are the most coherent; that is, they have

a clear, purposeful linkage between actions and outcomes. The Panel attributed coherence to a clear articulation of the program’s objectives and conditions for success, combined with a constant, critical monitoring of

evidence that supports or contravenes the program’s assumptions. The Panel further noted that useful evidence might consist of simple pre- and posttests, or the tracking of changes in student behavior (e.g., attendance and graduation rates) and is not limited to formal quantitative studies. 

The Panel believes that successful outcomes are the cumulative result of effort in multiple areas and has structured its criteria to reflect that belief. 

The Panel’s definition of p r o g r a m goes well beyond that of a textbook or curriculum for a single discipline: 

P r o g r a m : The implementation of a set of interrelated strategies and activities directed toward increased 

learning. A program includes assessment that demonstrates its impact on participants and on a wider 

organizational structure (i.e., a school or cluster of schools, a district, community organization, partnership, or other distributed system). Eligible programs depend on technology to achieve their goals.

The Educational Technology Expert Panel believes that no one electronic device can fulfill its definition of p r og r a m without a sustaining and complex structure of support. A program cannot be dissected and only certain

components extracted and used—a program functions as an integral, interlocking whole. 

TH E DE S IG N AT E D PRO G R AM S

The selection of exemplary and promising programs reflects the potential of educational technology for 

significant learning. Equally important, the programs serve to affirm and encourage the continued efforts 

of many others across the country who share a commitment to maximizing educational technology for the

benefit of improved learning.
Given that new projects are constantly being developed and that the Panel was dependent on the set of selfselected applications received, there is no assurance that the list of selected programs is fully representative of

the variety of programs being implemented across the country. Rather, the list is illustrative; the programs

exemplify excellence regarding certain features the Panel considered critical, including their effectiveness in

exploiting the unique capabilities of technology and their contribution to professional development, equity,

and assessment. The designated programs stand as models of the kind of thinking that leads to increased and

sustained student learning. 

The process of identifying excellence is ongoing. The exemplary programs and many of the promising programs can serve as models to all schools. The learning goals, methods, and design have universal application.

Virtually any school with leadership and imagination can adapt these programs to their own populations,

strengths, and weaknesses. The winners will be the students, teachers, administrators, and the community.

HO W TO U S E T H E CR I T E R I A

The Panel’s criteria reflect how experts in the field are currently thinking about technology and learning. The

criteria can be used as guidelines when creating and improving programs that use educational technology.

They provide a rigorous, analytical model for districts and schools undergoing planning and development.

They are useful for policymakers deciding on how to benefit fully from connecting their buildings to the

Internet, what objectives they want to achieve and with whom, what sorts of technology programs they want to

create or adapt, what sorts of implementation issues must be considered, and what sort of assessment will be

effective and persuasive. 

The criteria can also be used to explain to school boards, parents, and others why educators should bring

technology into schools. They can also be used as an assessment guide to help schools scrutinize their current use of technology: How is technology currently being used for student learning? Is equity addressed

consciously and sufficiently? Are systemic supports in place, and are more supports needed? 

The criteria reflect the experience of experts in the field, and, as such, can stimulate creativity and help 

avoid pitfalls.

TH E REV I E W PRO C E S S

The submission process was open, widely publicized, and designed to encourage applications from a variety

of organizations. One hundred thirty-four programs were submitted for review. The review encompassed

three stages. First, every submission was examined by a team drawn from 30 reviewers, each of whom had

expertise in technology in addition to a content specialty such as math or an area of expertise such as school

reform, research, or evaluation. This was the Quality Review Panel (QRP). Reviewer training included a

series of online discussions with Expert Panel members over a period of 3 weeks, and then an onsite orientation conducted by a team of 4 Panel members. These QRP teams determined eligibility and verified that the

application was complete; scored the quality of the submission in accord with the Panel’s published criteria;

and determined whether the submission merited being forwarded to the next group of experts—the Impact

5
6

Review Panel (IRP). In the second stage, the IRP, consisting of national experts in evaluation design and analysis,

assessed the persuasiveness of the evaluation data and the claims made by the program. Third, members from the

Expert Panel reviewed all of the programs, along with the ratings and comments of the QRP and IRP review teams,

to determine which programs to recommend to the Secretary of Education as exemplary or promising. 

The criteria used to judge the programs and the program descriptions for the two exemplary and five promising

programs designated by the Secretary of Education follow. The descriptions are based on information provided by

the developers. Contact information for each program is included.

NEXT ST E PS

The main goal of the Educational Technology Expert Panel was to develop and promulgate a shared vision, a common framework, and specific definitions and criteria to raise the level of conversation and agreement within the

field about educational technology in order to improve teaching and learning. Educational technology is still

viewed by some as a matter of installing hardware and software, with insufficient attention being devoted to quality

teacher training and evaluation. We hope that the Panel’s work will help policymakers, school administrators, educators, and parents gain a more comprehensive view of the integral role that technology can play in curriculum

development, organizational change, and school reform.
EVA LUAT ION CR I T E R I A

The diagram below presents the interrelatedness of the six criteria upon which programs were reviewed.

Recognized programs have significant educational goals that result in complex learning supported by technolo g y. These programs also promote organizational change as well as greater equity and educational excellence

for all students. The programs demonstrate persuasively their effectiveness regarding these outcomes and are

useful and adaptable in other school settings. 

Significance 

of Goals

Learning Equity Organizational

Change

Evidence of

Effectiveness

Usefulness 

to Others

7
8

A. QUA L I TY OF PRO G R AM

Criterion 1. The program addresses an important educational issue or issues and 

articulates its goals and design clearly.

● The educational goals are significant.

● The program design is thoughtful and supported by research.

● The program description is clear and complete.

Include the following items:

a . Need or problem the program addresses and how it relates to teaching and learning 

in preK–12 schools.

b . Program goals.

c . Technology used and how it helps to achieve the program’s goals.

d . Subject population(s): ethnic, racial, socioeconomic, and gender percentages. 

The size of any special populations served (e.g., ESL, AP biology students, 

students with disabilities).

e . Content and learning goals.

f . Program design (structure and components).

g . Professional development provided as part of the program.

h . Overall size and maintenance costs (funding and staff requirements, number of people 

in target population).

i . Key learning activities for participants.

j . Assessment(s) used to determine the program’s efficacy and achievements.

k . Keys to the program’s success.

l . A specific, concrete example that best captures the changes achieved by this program.

B. ED U CAT ION A L SIG N I F ICA NC E

The Expert Panel considers the following three areas—learning, equity, and organizational

c h a n g e—essential to fulfilling the promise of educational technology. A sound program must address all

three, and all three must be shown to have impact on or linkage to preK–12 student learning.

Criterion 2. The program develops complex l e a r n i n g and thinking skills for its target 

audience. 

If the target audience is other than preK–12 students, the applicant should 

articulate the program’s goals and their connection to student learning. 

If the target audience is preK–12 students, the indicators might include one 

or more of the following objectives:
● The program increases students’ in-depth understanding and competence in at least 

one content discipline. 

● The program develops the habits of lifelong learning (e.g., the ability to collaborate, 

direct one’s own learning, solve problems, communicate ideas clearly, and think 

flexibly and critically).

● The program helps students become proficient and critical consumers and producers 

of educational technology.

● The program includes preparation for entering a technology-infused workplace.

Criterion 3. The program contributes to educational excellence for all.

● The program conveys high expectations for all learners.

● The program responds to the diverse needs of varied populations of learners.

● The program includes active outreach and partnerships to encourage broad 

p a r t i c i p a t i o n .

● The program increases the participation or achievement of underserved learners so 

that the gap between this group and other categories of students diminishes.

Criterion 4. The program promotes coherent organizational change.

● The program reflects a vision of educational reform consistent with disciplinary content 

standards, recommendations from national commissions, findings from educational 

research, and documented best practices.

● Policies, funding, and practice are aligned to support sustainable change.

● Through partnerships and professional development, the program builds human 

capacity to accomplish its goals (e.g., allocates time for teachers’ and administrators’

collaboration and planning).

● The program increases the educational involvement of parents, professional groups, 

and communities.

C. EV I DE NC E OF EF F E CT I V E N E S S

Criterion 5. The program has rigorous, measurable evidence for its achievements for at 

least one criterion among Criteria 2, 3, and 4 (learning, equity, and 

organizational change).

Valid evidence will meet generally accepted standards in the field and may include 

● one or more comparison groups;

9
10

● a formal evaluation; 

● a quantified demonstration of positive change among participants as a result of the 

program (e.g., increased parental involvement in school governance; diminished gaps 

in achievement between groups; increased enrollment in rigorous mathematics courses 

or graduation rates among subject populations; changes in the base funding and 

requirements for professional development); and 

● an in-depth, qualitative analysis of change among participants as a result of the 

program (e.g., case studies, ethnographies, and principled analyses of observations and

i n t e r v i e w s ) .

D. US E F U L N E S S TO OT H E RS

Criterion 6. The program is adaptable for use in multiple contexts.

● The program’s technology requirements are easily available to potential users.

● The program is cost-effective relative to its benefits.

● After its initial implementation, the program is sustainable with existing resources 

(i.e., does not require extraordinary/unreasonable time, effort, or funding), and 

scalable (i.e., can naturally expand its scope to several teachers, multiple grade 

levels/subjects/sites, different disciplines). 

● The program is adaptable to a range of educational settings with learners similar to the 

intended population.

● The program provides clear and detailed guidelines about the conditions required for 

its successful implementation.

Note: The term “learners” can refer to preK–12 students, educators, or parents.
E x e m p l a r y P r o g r a m

C h a l l e n ge 2000 

Multimedia Pro j e c t
Exemplary program

CH A L L E NG E 2000 MU LT I M E DI A PROJ E CT

PRO G R AM DE S C R I P T ION

The Challenge 2000 Multimedia Project, created in 1995 and designed for students in grades K-12,

establishes a learning community in which students use appropriate technology and acquire skills

needed for the high-tech workplace—content knowledge, problem-solving acumen, communication and collaboration skills, as well as self-assessment. On-site mentor-teachers support the

design of classroom experiences in which students analyze and propose solutions for real-world

problems by planning and producing interdisciplinary, multimedia projects. Students then present

the final projects to the community at a multimedia fair. 

Support for teachers involves four components: (a) teaching and professional development

intended to transform practice by helping teachers become effective designers and implementers

of student-centered, technology-supported, project-based learning using multimedia; (b) computers in the classrooms (rather than a separate computer lab) for all teachers and students; (c)

classroom connections to the “information superhighway”; and (d) effective, engaging software

and online learning resources as an integral part of school curriculum.

Student multimedia projects might be a hypermedia stack, computer presentation, Web site, or

video. Classes have studied such topics as the rain forest, water pollution, designing a research

facility and living quarters for scientists in Antarctica, the physics of amusement park rides, and

colonizing Mars. 

QUALITY AND EDUCATIONAL SIGNIFICANCE

LE A R N I NG

Students develop in-depth understanding and competence in at least one content discipline. They

also develop habits of lifelong learning (e.g., ability to collaborate, direct one’s own learning, solve

problems, communicate clearly, and think flexibly and critically).

For example, at one school, 12th-grade physics students are paired and work to develop videos

combining footage of amusement park rides and their associated forces and acceleration changes.

The goal is to teach middle school and 9th-grade students the physics of the rides. The teacher

observes that preparing a presentation for a younger audience really pushes the students to a 

PRO G R AM CO S T S

For cost information, please contact program designee.

12
FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

Michael Simkins

Challenge 2000 Multimedia Pro j ec t

P O Box 6361

Los Osos, California 93412

Te l e p h o n e : 8 3 1 – 4 7 7 – 5 5 0 1

E-mail: msimkins@smcoe.k12.ca.us

Home Page: http://pblmm.k12.ca.us

deeper level of understanding than would “spouting back calculus formulas.” Videos are judged on

correct science content, the suitability of the material for the intended audience, and the effective

use of multimedia to explore the planned content (incorporation of video, pictures, and data). 

The school holds an annual “Physics Day” when students collect data and images with video and

digital cameras at the local amusement park. Students plan accordingly, working out what data they

want to collect, how they will collect it, and how to share equipment. Sometimes they also share

materials and calculations. To capture force in two of the three possible planes, each student pair is

given two accelerometers and a graphing calculator with special software, which they program to

collect the data they want before going on the rides. 

Back in the classroom, students upload their data and print out a graph of the data points. They

can thus see patterns of acceleration associated with, for example, the spin of a car. One group of

students chose to explain how roller coaster loops keep people in their seats, showing how the tearshaped loops at the park worked and what would happen if the loops were circular. They calculated that the force of a circular loop would make people six times heavier at the bottom than the

force of a tear-shaped loop, so that circular loops were impractical. 

At the end of the spring semester, the videos are presented in a film festival and viewed on a full

screen with high-quality amplification, accompanied by much celebration and cheering. 

EXC E L L E NC E F OR AL L

Teachers report that where the implementation of project-based learning using multimedia has been

greatest, there have been significant increases in teacher expectations of student work. Further, they

report that low-achieving students benefit most from the project-based multimedia approach, followed by academically gifted and talented students. Other groups who particularly benefit are students with ADHD, English-language learners, and special education students. When adding teams of

schools, the project has sought to increase participation by adding teachers from schools with the

greatest number of disadvantaged students.

ORG A N I ZAT ION A L CH A NG E

Challenge 2000 advances organizational change by providing effective models of change and supporting the teacher learning community as they implement project-based learning in their classrooms. It promotes the sharing of expertise by establishing partnerships focused around the impleUS E F U L N E S S TO OT H E RS

F rom 1995 to 1999, Challenge

2000 increased its teacher cohort

f rom 40 to 116. The pro j e c t ’s

technology re q u i rements are

a ff o rdable and realistic for most

schools; the software is not pro p r ie t a ry and can vary according to the

re s o u rces of the school. Costs per

unit are high during start-up but

d e c rease as the program expands

to a greater number of students

and teachers. Participants include

a wide range of ethnic, racial, and

socioeconomic gro u p s .

13
mentation of student-centered projects in classrooms. The partnerships are composed of a more experienced

mentor teacher, who has previously implemented multimedia projects, and one to three teachers new to the

project-based multimedia approach. Based on their ideas for projects, the partnerships have developed proposals to secure hardware, software, and network tools needed to implement their projects. During implementation, teachers often discuss their projects within their school and have the opportunity to meet with

teachers from other schools who have formed their own partnerships at program-wide meetings throughout

the year. Their work culminates each year in a presentation of their students’ work at a Multimedia Fair, to

which parents and community members are invited to witness the accomplishments of the teachers and their

students. 

EV I DE NC E OF EF F E CT I V E N E S S

Teacher reports of classroom practice and observations conducted in Multimedia Project classrooms and

comparison classrooms show the impact of the program on teaching. Most teachers report implementing

multiple projects with their students, and the more experience teachers have with Challenge 2000, the more

likely they are to engage students in sustained inquiry—asking questions, investigating them, and making predictions about phenomena. Students in these classrooms were more likely than those in comparison classrooms to be engaged in long-term projects and to work collaboratively with their peers. They were also more

likely to be engaged in cognitive activities of design—interpreting and analyzing information, deciding on the

structure of a presentation, revising a presentation, and thinking about how their audience learns. 

In addition, the project conducted a performance assessment designed to measure students’ skill in constructing a presentation aimed at a particular audience, using content provided to them. Students from Multimedia

Project classrooms outperformed comparison classrooms in all three areas scored by researchers and teachers who analyzed the student products: content, attention to audience, and design.

E x e m p l a r y P r o g r a m

14
e x e m p l a r y P r o g r a m

G e n e ration www. Y
GE N E R AT ION W W W. Y

PRO G R AM DE S C R I P T ION

Generation www.Y (GenY), started in 1996, focuses on today’s new generation of youth as partners,

and often leaders, in bringing technology into the classroom. Instead of teaching technology skills

to teachers, GenY trains students to partner with teachers to improve teaching and learning. Along

with educators, students become agents of change, not mere recipients of change. The class has

been implemented in grades 4–12, with the majority of participating students in grades 6–8.

The core of the program is the GenY course, taught by a coordinating teacher and offered as an 18-week

class in secondary schools or a 30-week class in elementary schools. GenY students are trained in computing and telecommunications skills for the purpose of helping rebuild the curricular units in their

school so that teachers, administrators, and all students can make better use of modern technology.

GenY students also receive training and coaching on communication and collaboration with teachers,

project planning and management, standards-based curriculum development, and effective presentation

of information to learners. Coordinating teachers participate in a one-day training workshop.

One coordinating teacher and one class of students per semester are the kernel of the program in

each school. Each GenY student in these classes forms a partnership with a teacher, known as the

p a r t n e r- t e a c h e r, and together the team designs and completes curriculum-building projects. Many

teachers schoolwide participate as partners. The partner-teacher chooses a lesson plan or curriculum unit to upgrade or build from scratch, then teacher and student plan a project that enriches

the unit or lesson plan, taking advantage of the available technology. The student focuses on the

technical aspects of the project, meeting regularly with the partner-teacher to ensure that the project fits with the teacher’s curriculum plan and classroom needs. The partner-teacher uses this

technology-enhanced lesson in classroom teaching, often continuing afterwards to develop or

extend the project with continued technical assistance from GenY-trained students. The students

provide Internet expertise (e.g., search out Web sites for information and help with electronic presentation of materials), while the teachers model skills such as communicating effectively, mentoring, solving problems, and project management. 

The GenY course is supported by an extensive curriculum guide, student workbook, CD-ROM, and

videotape, all developed during the first 2 years of the program. Participants also have access to an

online database of previous project descriptions, a planning database with facilities for aligning

projects with state standards, and an online system staffed by content-area experts who provide

feedback to students on their developing projects. 

PRO G R AM CO S T S

For cost information, please contact program designee.

16

Exemplary program
QUALITY AND EDUCATIONAL SIGNIFICANCE

LE A R N I NG

GenY students receive training in network and Internet use, in a variety of computing skills, as well as

in communication, collaboration, teamwork, project planning, project management, and project presentation. The training is designed to help students support teachers in rebuilding their lesson plans

and curricular materials to enhance learning by using technology.

The GenY experience involves a wide range of technical, academic, cognitive, and social skills.

Students are expected to become proficient and critical consumers and producers of educational

technology as well as to use these skills to help teachers use and produce educational technology.

This requires understanding, then communicating and collaborating effectively with someone who may

have a different point of view. When building collaborative projects, students must understand the

needs of the partner- t e a c h e r, frame problems productively, seek out appropriate tools and information, then plan and manage their time and work in order to accomplish a discrete project within a

specified timeline. Delivery of the projects entails good communication and presentation skills, using

a style appropriate to the audience and the pedagogical purpose. The projects are typically unique

and require students to think flexibly and creatively, direct their own learning, get help when needed,

be responsive to feedback from the partner- t e a c h e r, and take responsibility for an academic and technical product. One student, for example, developed a HyperStudio unit on prefixes, suffixes, and root

words that was easy for the class to understand and included a quiz that was fun to take.

Benefits for the partner-teacher include an understanding of how to teach lessons enhanced by technology and aimed at national and state learning goals and educational standards. In collaboration

with the GenY student, the teacher develops a more collaborative relationship with students as

colearners and partners in the educational process, in which both student and teacher make important contributions. The GenY class is divided into 10 units that provide a suggested content and format

for schools to implement the model. The format provides a model of project-based, authentic, student-centered, multidisciplinary teaching and learning enhanced by technology.

EXC E L L E NC E F OR AL L

The GenY model has been adopted in rural, suburban, and urban settings in several statesand the Vi r g i n

Islands. As a result of outreach efforts during 1999–2000, the GenY program will be implemented in several innercity schools, western schools with a high Native American population, and in high-diversity

schools in Hawaii. Because of aslight gender imbalance in GenY classes in general as ofSeptember 1,

1999 (56–60 percent males vs. 40–44 percent females), one school has implemented special versions of

the class aimed at appealing to girls. Program developers are studyingthiseffort to determine how to

improvethe gender imbalance in other schools. All GenY students are held to the same s t a n d a r d s :

FOR FU RT H E R IN F OR MAT ION, CON TA CT:

Dennis Harper

Generation YES

711 Capitol Wa y, South, Suite 702

Olympia, Washington 98501

Te l e p h o n e : 888–941–4369, ext. 111 or

360–528–2345, ext. 111

Fax: 360–528–2350

E-mail: dennis@genyes.org

Home Page: http://www. g e n y e s . o rg / g e n w w w y/

US E F U L N E S S TO OT H E RS

GenY began in the Olympia School

District in the State of Wa s h i n g t o n .

As of September 1, 1999, GenY

has been adapted for use in 140

schools in 11 states and the Vi rg i n

Islands. Several hundred more

schools across the nation are

beginning to use the model. After

two years of intensive development eff o rts, the course is supported with an array of re s o u rc e s

(some online) for coord i n a t i n g

teachers that can be adapted for

d i ff e rent school settings serving all

populations using virtually any

h a rd w a re / s o f t w a re infrastru c t u re .

The course materials have been

used successfully from upper elem e n t a ry through high school levels

and include materials for administrators, part n e r-teachers, coord inating teachers, and parents. The

model is flexible, allowing classes

to be conducted before school,

after school, as a regular school

elective during the day, or on

w e e k e n d s .

17
satisfactory completion of the curriculum units, including exercises, classroom activities, and assessments; delivery of a fully

articulated, planned project proposal during the middle of the semester; and completion and documentation of the project

by the end of the semester.

In addition to the core model, the program includes an opportunity for GenY graduates and preservice teachers to staff

school-based or community computer labs. The labs provide increased computer and network access as well as

coaching for underserved students, their families, and other community members. Students who use these labs often

have no computer, no Internet access, or inadequate software at home, so the labs serve an important function in

increasing the equitable distribution of access to technology.

ORG A N I ZAT ION A L CH A NG E

The premise of GenY is that students are a largely untapped, creative, enthusiastic source of partnership and leadership

in an effort to use technology to support school reform. Thus, the program supports increased cooperation and collaboration among students, teachers, administrators, family and community members, as well as business, educational and

community organizations. The program requires coordinating teachers, partner-teachers, and students to think carefully about the alignment of curriculum units, lesson plans, and collaborative projects with state standards. In the state of

Washington, GenY materials and online support are linked to the Washington Essential Academic Learning

Requirements. Further, the online resources for project planning and reporting have been expanded to allow participants in each state to work with their own system of learning goals and state standards.

EV I DE NC E OF EF F E CT I V E N E S S

GenY students, partner-teachers, and coordinating teachers reported that 90 percent of the students completed their projects. These students generally feel proud of their projects, and their partner-teachers consider the projects to be of high

quality (93 percent). The majority of the partner teachers (92 percent) reported using the projects in their regular classes

and planning to update or extend the projects in future class activities (94 percent). Almost half of the projects completed

in spring 1999 were multimedia presentations and 25 percent were Web pages. The remainder involved teaching technical

skills to teachers and other students, providing assistance in Web searching, and other activities.

In spring 1999, 807 partner-teachers completed surveys before beginning the program, and 631 completed follow-up

surveys at the end of the semester. About 90 percent of the responding teachers intend to use the new lessons repeatedly or will continue to develop and expand them; this same percentage also reported that as a result of the GenY, they

learned about technology and their students learned about technology and a content discipline. Nearly all (about 95

percent) consider GenY a good method for providing support and assistance to teachers as they integrate technology

into their classes, reported a desire to continue participating in the program, and said they would continue to rebuild

lesson plans to benefit from using educational technology. Eighty-two percent of the teachers reported that the experience would change the way that they teach in the future.

When asked their opinion about using technology in education after participating in GenY, virtually everyone (98 percent) said

they felt technology facilitates positive changes in classroom teaching and learning practices; 52 percent said they wanted to

learn more about using new technologies; and 62 percent attributed these results to their participation in GenY.

An external evaluation documents substantial learning gains on the part of participating students. Reviewers were

impressed by the creativity of the project, creating a role reversal in which students help support the school's technology infrastructure and partner with teachers in curriculum development. The latter is crucial to the success of the project and to fostering learning gains for all students in participating districts. While a few projects have taken similar

approaches, this particular implementation is better conceived, more thoroughly implemented, and more carefully documented than other comparable programs. 

E x e m p l a r y P r o g r a m

18
p r o m i s i n g P r o g r a m

The Mary l a n d

Vi rtual High School (MVHS)

C o reModels Pro j e c t

The Mary l a n d

Vi rtual High School (MVHS)

C o reModels Pro j e c t
Promising program

TH E MA RY LA N D VI RT UA L HIG H SC HO OL

(MVHS) COR EMODE L S PROJ E CT

PRO G R AM DE S C R I P T ION

The Maryland Virtual High School (MVHS) CoreModels Project has two primary goals: (1) to use

computer modeling to help all high school students achieve state and national science standards,

and (2) by developing and refining a process of network-based peer leadership and collaboration,

to give teachers in their classrooms the support they need to integrate modeling activities into their

instruction. Started in 1997, CoreModels is based on, and is intended to institutionalize, the gains

achieved by the Maryland Virtual High School program that connected rural and underserved

Maryland schools to a magnet high school via the Internet. Through this connection, MVHS supported teachers in developing and implementing computational science projects with their students. Crucial components of the MVHS program were: (1) its focus on using computational

modeling to teach complex science content, and (2) its mentoring and support for remote teachers and students provided by magnet teachers and their students. The mentors won national

recognition for their computational science projects and gained experience through a student-run

laboratory at Montgomery Blair High School Magnet Program. 

A model of distributed support that creates the conditions for sustainability of the changes and

gains achieved by schools, the program is active in 6 to 15 schools in each of 3 CoreModels

regional centers in Maryland.

Students construct their own models, utilizing the software STELLA, in areas of science such as

wildlife populations, the carbon cycle, hurricane prediction, projectile motion, chemical reactions,

and rock formation cycles. Students hypothesize about the results of their model based on the

parameters used in the graphical model definition. They compare their results with other predictions, such as those from scientists or developed by other students, or with actual results as measured in practice, thus gaining an understanding of important recurring scientific concepts involving

equilibrium processes, feedback, and multiple causal relationships, among others. 

This process of learning brings students closer to science as it is practiced in the world outside of

school. As an example of the conceptual core of the use of modeling, students not only learn

about the laws of conservation of mass and conservation of energy, but they learn to apply them in

many different situations; to identify control and independent variables; to collect data and analyze

the results; and to write clear step-by-step instructions for investigating their hypotheses. In the 

PRO G R AM CO S T S

For cost information, please contact program designee.

20
FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

M a ry Ellen Ve rona or Susan Ragan

M a ryland Vi rtual High School

C o reModels Pro j e c t

M o n t g o m e ry County Public Schools

51 East University

Silver Spring, Maryland 20901

Telephone: 301–649–2880

Fax: 301–649–8245

E - m a i l : m v e rona@mvhs.mbhs.edu or 

s r a g a n @ m v h s . m b h s . e d u

Home Page: http://mvhs.mbhs.edu

process, students apply higher-level mathematics skills, such as ratio and proportion and setting up

equations. For example, in the glucose regulation model students simulate the glucose-insulin feedback process that the body uses in attempting to maintain homeostasis (equilibrium) and predict

the effect on the body of eating a candy bar. They practice graph analysis by determining how the

model output differs from their prediction and describe what the graphs mean in terms of the body’s

response. They also use the graphs to describe how the body’s blood glucose and insulin concentrations differ when pasta is eaten instead of a candy bar.

The MVHS CoreModels project works through regional centers to integrate curriculum reform, staff

development, and teacher professionalism in a sustainable manner. Technology enhances and facilitates this initiative by allowing educators and students to become better at teaching and learning.

Teachers are recruited and trained to

● implement computer modeling to help students understand and learn more complex 

scientific concepts central to biology and physics; 

● help students learn in a collaborative, problem-solving, interactive manner that brings 

them closer to the world of science; and 

● implement innovative pedagogical practices by being provided with scientific and technology

support. 

I d e a l l y, as more teachers are recruited and trained, the pool of trainers for new teachers becomes

l a r g e r, and more and more teachers become trained in the use of CoreModels. The project had

been in existence for 2 full years as of September 1, 1999. 

QUA L I TY A N D ED U CAT ION A L SIG N I F ICA NC E

LE A R N I NG

MVHS CoreModels increases students’ in-depth understanding and competence in their current science subject area by focusing on an analysis of change over time and on the transfer of understanding from one activity to the next. One of the hallmarks of the project is the alignment of CoreModels

activities with Maryland Science Core Learning Goals (CLGs). CoreModels materials also focus on

the American Association for the Advancement of Science (AAAS) Benchmark common themes

(similar to National Science Education Standards themes), which emphasize connections between

seemingly disparate science content. 

In using and creating computer models, student attention can be focused on similar structures and

behavior across areas of study. For example, a predator-prey interaction model and a physical 

US E F U L N E S S TO OT H E RS

The very stru c t u re of MVHS

C o reModels lends credence to a claim

of usefulness to others. The pro j e c t

includes regional centers, 15 local

educational authorities, and 31

schools. Growth and dissemination

have been extremely successful in

regions with very diff e rent demographic and political characteristics,

and the program has been adapted

d i ff e rently on the school, district, and

regional levels. 

The pro g r a m ’s technology re q u i rements are relatively simple, understandable, and aff o rdable. Schools

can purchase the site license for STELLA software for $500. Unlike subjectspecific simulation software, STELLA is

technology problem-solving software

(National Educational Te c h n o l o g y

S t a n d a rds). In addition to computers

running STELLA, Internet access

should be available for teachers to collaborate with peers and to access

models and materials. In adapting the

MVHS CoreModels program, leaders

could access materials available

t h rough the MVHS site, as well as

other system dynamic modeling

a rchives such as the Creative Learn i n g

E x c h a n g e .

21
spring model share the same mathematical oscillatory structure. Disruption and resumption of equilibrium

can be found in both biological and Earth systems. Assimilating an understanding of such core structure and

behavior concepts leads to acquisition of the “schemas” of science content, which have been shown to distinguish experts from novices (P. Chandler & J. Sweller [1991], “Cognitive Load Theory and the Format of

Instruction,” Cognition and Instruction 8, 293–332).

EXC E L L E NC E F OR AL L

Modeling activities have been used with mainstreamed special education students as well as with advanced

placement students. MVHS CoreModels developed Earth science and environmental science activities because

these classes often target students considered less successful in previous science courses. In addition, center

directors, who observe all teachers within their region, help teachers develop higher expectations for students

in nonacademic classes by suggesting visits to teachers already successfully using activities with similar groups

of students.

One reason exemplary science activities are not available to all students is that only a fraction of eligible teachers take the initiative to investigate and apply to summer training programs. Facilitating organizational change

through the CoreModels Centers is needed to provide training for additional teachers and learning opportunities for all their students. The group of teachers thus targeted typically teach the non-AP and nonhonors classes

so that affecting the practice of these teachers is one step toward contributing to educational excellence for all.

ORG A N I ZAT ION A L CH A NG E

One vital component of the MVHS CoreModels project is the role of the teacher in developing activities and

evaluating their effectiveness in the classroom. Extending the initial success of MVHS in developing a remote,

network-based collaborative learning environment, the CoreModels program supports teachers’ ability to

change and integrate curriculum and technology through the use of peer mentors.

Capacity building is another important aspect of MVHS CoreModels. The project is deeply committed to the

development of leadership among participating teachers; its predecessor, MVHS, was a teacher- d e v e l o p e d ,

t e a c h e r-led program. Although the CoreModel program’s institutionalization goals call for a leadership team

(the project director, three regional center directors, and eight supporting teachers), the participating teachers

collaborate with leaders in presenting outreach workshops and in developing materials. This expanding leadership has allowed the project to respond to opportunities to create districtwide and school-based training

opportunities as well as to reach out to all teachers, meeting them where they are. Teachers are immersed in a

profound form of professional development when they work together to adapt materials for the specific needs

of a particular class; when they prepare a presentation for a group of colleagues; or when they think through

how modeling can be used to facilitate and assess student learning in a new content or skill area. 

EV I DE NC E OF EF F E CT I V E N E S S

Preliminary results show that learning appeared to have been enhanced by the use of the CoreModels project.

Students in one science class using the CoreModels project improved their scores 46 percent on the Force

Concept Inventory test, while students in another school, regarded as the best in the county but not using the

program, improved by only 27 percent. In addition, a school with all its biology teachers involved in the

CoreModels project led its district in the initial results of the state biology exam.

22

P r o m i s i n g P r o g r a m
p r o m i s i n g P r o g r a m

Middle-school Mathematics

t h rough Ap p l i c a t i o n s

P ro gram (MMAP)

Middle-school Mathematics

t h rough Ap p l i c a t i o n s

P ro gram (MMAP)
MI DDL E-S C HO OL MAT H E MAT IC S T H RO U G H

AP P L ICAT ION S PRO G R AM ( M MA P )

PRO G R AM DE S C R I P T ION

The Middle-school Mathematics through Applications Program (MMAP) is a comprehensive set

of middle school materials based on projects. The projects, which utilize software tools,

engage students with real-world problem solving and require them to learn mathematics and to

apply what they have learned. MMAP, started in 1992, addresses the needs of students in

grades 6 through 8 with a series of units that emphasize proportional reasoning, algebraic

expressions, and functions. It also covers statistics, probability, measurement, and geometry.

The materials are designed to increase students’ conceptual understanding of mathematics,

their competence with standard symbolic notations for mathematical concepts, and their ability

to communicate their mathematical ideas to others. They also learn mathematical practices

such as making conjectures, coming up with counter examples, and writing proofs.

For example, with the design unit Antarctica and the software ArchiTech, sixth- and seventhgrade students play the roles of architects who design an ideal living space for themselves or a

client; seventh- and eighth-graders design a scientific research station in Antarctica. Thus, students learn about scale and standard proportional notation; learn to represent the real world

symbolically; research Antarctica and the needs of their clients; consider the building and heating costs of their design; and present their floor plans, budget, and a written report to their

class. These same skills are approached from a different perspective with the design unit

Guppies and the software Habitech, for which students play the roles of population biologists

who research, analyze, model, and make predictions about guppy behavior. They research

b e h a v i o r, reproductive habits, and life cycles of guppies; organize their data; model guppy

breeding and feeding behavior; design a sustainable tank environment for guppies; and share

their findings and suggestions with each other.

QUA L I TY A N D ED U CAT ION A L SIG N I F ICA NC E

LE A R N I NG

Central to MMAP is the way in which the program engages students by placing them in software

environments to solve open-ended design problems requiring the use of mathematics concepts.

For example, Coding Toolbox allows students to use shift cipher, function, matrix, analysis, and

PRO G R AM CO S T S

For cost information, please contact program designee.

24

Promisingprogram
compression tools to develop and test privacy and efficiency codes. The software complements, rather

than dominates, the classroom activities. As students strive to improve their designs, they find themselves using sophisticated combinations of math tools, representations, and techniques.

By applying concepts to find solutions to relevant problems, MMAP students appreciate the power of

mathematics. Their responses can be mathematically complex as well as tied to real concerns. Reallife contexts for learning and work-world technology tools help prepare students for the workplace.

M M A P ’s use of design processes, ongoing embedded performance assessments, and technology-based

tools help students make explicit links between informal and formal mathematics. In the course of

the Antarctica unit, for example, students are asked to compute the average yearly temperature in

coastal Antarctica (computational skills, developing equations); furnish their floor plan design using

resizable furniture icons in the ArchiTech software (scale and proportion); as well as graph and compare the relationship of building costs, heating costs, and insulation costs (patterns and functions).

Other activities include building biological models and building digital maps with data layers. 

EXC E L L E NC E F OR AL L

The overall design of MMAP is intended to engage more students successfully with mathematics, particularly students traditionally underserved in school math or those who find themselves alienated

from it. The software and topics were designed to be equally unfamiliar but engaging to most middle

schoolers (i.e., the simulation context of Antarctica, the professions of population biologist or architect). In addition, the technology is tool-oriented and relies on multiple representations; prior computer or math savvy is not necessary to achieve results. 

The design process itself encourages the participation of many students as well as multiple solutions to

a problem. Design begins with brainstorming, welcoming the different skills of different students and

encouraging students to bring their personal experiences, skills, and identities to the task. For example, while working with ArchiTech on house design, one student compared the group's proposed

design to the size of his own house and realized it was huge; the group revised its floor plan. Students

work with others to learn what they need to know, rather than being singled out for not knowing

s o m e t h i n g .

ORG A N I ZAT ION A L CH A NG E

Among MMAP's greatest accomplishments is that teachers—both those who are codevelopers as well

as those who use the materials at a distance but participate in the ongoing online discussions and support activities—have rejuvenated their careers. Teachers report varied experiences that have helped

FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

Shelley Goldman

Middle-school Mathematics thro u g h

Applications Program (MMAP)

School of Education 

S t a n f o rd University

Cubberley Building, 485 Lasuen Mall

S t a n f o rd, California 94305–3096

Telephone: 650–723–8422

Fax: 650–725–7412

E-mail: sgoldman@stanford . e d u

Home Page: http://mmap.wested.org 

o r ➼

US E F U L N E S S TO OT H E RS

MMAP has been successfully integrated in many settings and is currently used in racially, economically,

and linguistically diverse communities in California, Alaska, New

York, Florida, Michigan, Ore g o n ,

and Washington. It also serv e s

students with learning disabilities.

The program is most successfully

implemented in schools where students have constant access to a set

of mathematical tools, including

calculators and measuring devices,

and where each MMAP class has

access to at least four to six comp u t e r s .

25
them change and make changes. For some, it is seeing extraordinary things happen for students as they participate in new ways. Some are empowered when they start to use technology and find that it enables them and their

students to improve. Teaching to standards and teaching with technology open teachers to new possibilities for

student learning.

EV I DE NC E OF EF F E CT I V E N E S S

MMAP encourages flexibility in the ongoing assessment process, allowing teachers to gear their assessments to

meet the individual needs of their students. The assessment process stresses the importance of a variety of means

of communicating student understanding. Options include pre- and posttests, journals, log books, oral presentations, writing, peer reviews, teacher-student conferences, group meetings, and more. Progress reports can be

given almost daily, affording the student immediate feedback.

A 1994 external evaluation surveyed 24 teachers and 42 students on program perception. Students, especially

from traditionally underserved populations, indicated excitement and high motivation. Teachers also reported

notable positive effects of MMAP on students’ various skills in both cognitive and affective domains and on students’ understanding of some mathematical topics. Results indicated a positive perception towards the program

that was similar across gender and special education status. A 1995 teacher survey of 10 teachers provides some

evidence that teachers have positive perceptions of the program and that they believe the program is beneficial to

s t u d e n t s .

An internal evaluation utilized classroom observation and videotape analyses, reviewed school test data from existing accountability assessments, and reviewed report card data to demonstrate student concept development and

mathematical power (single group design). In 1997, program students in one classroom showed higher scores

on algebra readiness tests in comparison to the same teacher’s students in previous years with less usage of MMAP,

and a 100 percent passing rate at mid-year in eighth-grade algebra, in comparison to the previous school average

of 50 percent passing at mid-year. Anecdotal evidence also supports the claim of students’ growth in mathematical

understanding and improvement in student attitudes towards learning mathematics.

P r o m i s i n g P r o g r a m

26

FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

Jennifer Knudsen

Middle-school Mathematics thro u g h

Applications Program (MMAP)

We s t E d

300 Lakeside Drive, 18th Floor

Oakland, California 94612

Telephone: 510–302–4273

Fax: 510–302–4242

E-mail: jknudsen@wested.org
p r o m i s i n g P r o g r a m

Modeling Instruction in 

High School Phy s i c s
Promising program

MODE L I NG IN S T RU CT ION I N HIG H

SC HO OL PH Y S IC S

PRO G R AM DE S C R I P T ION

Modeling Instruction in High School Physics, started in 1990, uses computers to teach models and

modeling, central components of modern science. These components are focal points to develop

the content and pedagogical knowledge of physics teachers, who then serve as local experts on the

use of technology in teaching and learning science. Science, and physics in particular, is a content

area for which students need to learn how to use computers as a scientific tool for observation,

data acquisition, analysis, and problem solving. Teachers are trained to support technology-based

learning in up to 8 weeks of intensive Modeling Workshops conducted over two summers, and

with ongoing year-round electronic network support. Teachers are thus engaged in a complete

revamping of high school physics to incorporate both technology and the insights of educational

research in full accord with the National Science Education Standards. The training provides them

with a robust new teaching methodology that greatly increases students’ understanding of basic

physics. 

In the Modeling Workshops, participants are introduced to modeling as a systematic approach to

the design of curriculum and instruction. Teachers identify a small number of models around

which to base their physics course and learn strategies to help students develop those models.

They collaborate on the redesign of the high school physics course to enhance learning and

employ technology to achieve their goals. They learn how to use computers as an integral part of

their teaching practice. They implement a student-centered instructional strategy which engages

students in active scientific inquiry, discourse, and evaluation of evidence. Further, they examine

the implications of educational research for physics teaching. They do all this while immersed in

studying the content of the entire year, which also provides extensive remediation for underprepared teachers.

The participating teachers then become leaders at their schools, modeling the best use of technology in the science classroom, training and assisting other teachers in the scientific use of technology in instruction, and advising schools on cost-effective infusion of technology into all science

courses. These leaders must remain practicing science teachers, not simply technology specialists,

because effective instruction in the use of computers as scientific tools requires special competence in pedagogy and science as well as in technology.

PRO G R AM CO S T S

For cost information, please contact program designee.

28

Note: Modeling Instruction in High School Physics was designated exemplary by the 

U.S. Department of Education’s Mathematics and Science Education Expert Panel, 2000. 
FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

Jane C. Jackson

Modeling Instruction in High School

P h y s i c s

Box 871504

D e p a rtment of Physics and Astro n o m y

Arizona State University

Tempe, Arizona 85287–1504

Telephone: 480–965–8438

Fax: 480–965–7331

E-mail: jane.jackson@asu.edu

Home Page: http://modeling.asu.edu

QUA L I TY A N D ED U CAT ION A L SIG N I F ICA NC E

LE A R N I NG

Teachers learn to 

●organize course content around a small set of basic models as the content core of physics;

●engage students collaboratively in making and using models to describe, explain, predict, 

design, and control physical phenomena; 

●involve students in using computers as scientific tools for collecting, organizing, analyzing, 

visualizing, and modeling real data; 

●provide students with basic conceptual tools for modeling physical objects and processes, 

especially mathematical, graphical, and diagrammatic representations;

●show how scientific knowledge is validated by engaging students in evaluating scientific 

models through comparison with empirical data;

●assess student understanding in more meaningful ways and experiment with more authentic 

means of assessment;

●improve continuously and update instruction with new software, curriculum materials and 

insights from educational research; and

●work collaboratively in action research teams to mutually improve their teaching practice.

For example, in one experiment, students are asked to develop principles for the motion of a pendulum. With the teacher as recorder, students brainstorm about properties of the pendulum which

might affect its period. After compiling the list, teacher and students decide which properties should

be investigated. In this example, they determine to investigate how changes in mass of bob, length of

string, and amplitude of motion affect the period. Students then work in teams and determine their

own procedure for collecting data. After collecting data, they plot variables appropriately and then

elicit the equations of motion and relationships among the variables. Then, using a technique called

“whiteboarding,” groups present the results of their experiments to the class. At the end of this

process, the class can agree on an appropriate model to describe the behavior of the pendulum.

They do this without being given the answer by a text or a teacher.

Students learn to understand scientific claims and to make sense of their experiments themselves.

They must articulate coherent opinions of their own, defending their findings in a variety of formats,

using concise English sentences, graphs and/or diagrams, and through algebraic expressions of the

relationship. Students are forced to reflect on why they choose an answer and how they evaluate

e v i d e n c e .

US E F U L N E S S TO OT H E RS

The Modeling Instruction Project has

demonstrated its adaptability for use

in multiple contexts. It began with

pilot workshops in Arizona, where it

has flourished for the past decade in

two dozen urban, suburban, and

rural high schools. It is now used in

44 states in diverse schools, including impoverished urban public and

exclusive private pre p a r a t o ry high

schools. Its practitioners include

experienced high school physics

teachers with exceptional qualifications and motivation as well as

u n d e r p re p a red teachers who were

drafted into teaching physics. More

than half of the physics teachers in

Arizona have learned Modeling

I n s t ruction, and the pro j e c t ’s

a p p roach is making inroads in several other states.

29
EXC E L L E NC E F OR AL L

The project developed three new evaluation instruments and found that the program increases the achievement

of underserved learners. For example, 36 Phase 1 teachers (teachers who had participated in one summer

workshop) were found to fall into one of two groups: those implementing all aspects of Modeling Instruction

consistently (17 teachers), and those implementing some aspects of modeling consistently (19 teachers). In

disadvantaged/lower income schools, the mean FCI gains of the 93 students of the 2 teachers who implemented

the Modeling Method consistently were 25 percent higher than for the 335 students of the 6 teachers who were

implementing less modeling. Students of both groups did better than those in traditional courses; the FCI gains

of the 93 students were double that of students under traditional instruction, and the FCI gains of the 335 students were 50 percent higher than those under traditional instruction. 

Many teachers, including inner-city teachers, report that their enrollments have increased since they started

using the Modeling Method.

ORG A N I ZAT ION A L CH A NG E

Graduates of Modeling Workshops are in great demand in their schools to assist in incorporating technology in

other science and math courses. In consequence of its well-documented success, the project has stimulated

formation of the Arizona Science and Technology Education Partnership (AzSTEP), which is institutionalizing

the reforms and methods of the Modeling Workshops in a statewide program for professional development of

in-service physics teachers as local leaders of science teaching reform. AzSTEP is an exemplar of a universityhigh school partnership to drive sustained reform of science teaching with technology.

EV I DE NC E OF EF F E CT I V E N E S S

An internal evaluation of data from 1995–1999 utilized a pre/posttest comparison design with matched students on the Force Concept Inventory (FCI) test, a test used to determine the effectiveness of mechanics courses in teaching students to reliably discriminate between the applicability of scientific concepts and naïve alternatives in common physical situations. The nationwide sample contained approximately 10,000 high school

physics students involved in the program. The comparison groups consisted of approximately 8,000 high

school physics students of the same teachers in the year before the teachers began the Modeling Wo r k s h o p

series and 700 high school students of teachers in traditional courses. Results indicate that Modeling

Instruction students demonstrate greater gains on the FCI than comparison students. In the nationwide sample

of students, the average FCI pretest score was approximately 26 percent, slightly above the random guessing

level of 20 percent. Students of traditional high school instruction (lecture, demonstration, and standard laboratory activities) achieved an average posttest score of about 42 percent. In contrast, Modeling Instruction students recorded posttest FCI scores of approximately 53 percent when the teacher was in her first year of the

program and an average of 69 percent when the teacher was an expert teacher with 2 years of modeling experience. 

An external evaluation utilizing classroom observation of teacher training workshops determined that there was

“overwhelming and consistent support” for this teaching approach. In interviews, participants reported that the

workshops were well run and that the facilitators were extremely knowledgeable about how best to teach

physics. They also commented that the group itself was an excellent resource as they could all help each other.

30

P r o m i s i n g P r o g r a m
p r o m i s i n g P r o g r a m

One Sky,

M a ny Vo i c e s
Promising program

ON E SK Y, MA N Y VOIC E S

PRO G R AM DE S C R I P T ION

One Sky, Many Voices (OSMV) is a research-focused learning program based on the student's application of knowledge to solve science problems using real-time weather data. Visualization software

allows students to track real-life events such as hurricanes, blizzards, floods, and tornadoes. The

program, started in 1992, covers 4 or 8 weeks of coordinated study in middle school science,

teacher support in the form of local study groups and focused networked discussions by experienced

teachers, daily scientists’ updates, and a suite of state-of-the art technological tools including current,

customizable weather imagery and message board systems. OSMV has been implemented, with varying levels of support, with a large number of students in impoverished urban schools as well as suburban and isolated rural schools. Science- and data-based interactions among students from these

diverse populations is a strength of the program.

OSMV promotes important attributes associated with significant learning experiences. Middle school

students use an Internet browser for enhancing their investigations of current weather phenomena.

Guided by their teachers and linked with experts in the atmospheric and environmental sciences, students view real-time images and converse with their peers on Web-based message boards as an integral part of the their studies. Thus, students might experience tracking and predicting current hurricanes, collaboratively studying and discussing current weather fronts in their region, or developing

content-rich explanations of weather phenomena in their area to be shared with students who do not

live with the same extreme weather patterns.

Studying real-time weather as part of One Sky, Many Voices has afforded students the opportunity to

become involved in real-life occurrences. For example, after tracking Hurricane Mitch, many students helped organize relief efforts for victims of the weather phenomenon that they had come to

know so intimately as part of their studies.

One Sky, Many Voices engages and empowers students as scientists and provides support for teachers

using the associated technology and pedagogy. Students take an active role in the teaching and learning experience. The exchange of roles between students and teachers in this program reflects a

sound, progressive school reform effort.

PRO G R AM CO S T S

For cost information, please contact program designee.

32
FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

Nancy Songer

One Sky, Many Vo i c e s

The University of Michigan

1323 SEB/610 East University

Ann Arbor, Michigan 48109–1259

Telephone: 734–647–7369

Fax: 734–763–1368

E-mail: songer@umich.edu

Home Page: http://www. o n e s k y. u m i c h . e d u

QUA L I TY A N D ED U CAT ION A L SIG N I F ICA NC E

LE A R N I NG

Teachers learn to 

●provide students with basic conceptual tools for understanding the use of modeling 

processes, especially in visual, mathematical, and graphical, representations; 

●assess student understanding in more meaningful ways and experiment with more authentic 

means of assessment; 

●improve continuously and update instruction with new software, curriculum materials, and 

insights from educational research; and 

●work collaboratively in action research teams to mutually improve their teaching practice.

Students learn to

●engage collaboratively in making and using models to describe, explain, and predict physical 

p h e n o m e n a ;

●understand how scientific knowledge is validated by evaluating scientific models through 

comparison with empirical data; and 

●use computers as scientific tools for collecting, organizing, analyzing, visualizing, and 

modeling real data.

Students increase the abilities necessary to conduct and understand scientific inquiry as well as to

make sense of their predictions themselves. They hone their communication skills while conducting

online interactive discussions with content mentors and peers distributed nationwide on their focus

topics. Peer explanations and predictions are critiqued and discussed in a group forum. After participation in OSMV, students display knowledge of significantly more weather terms, more scientifically

valid claims, and more sophisticated measures of scientific thinking.

This integrated program was specifically designed to help learners of all backgrounds, whether students,

teachers, researchers, or scientists, to use each other as resources and to collaboratively study interdisciplinary science and its impact on humanity. This is in contrast to the use of curriculum materials in isolation.

OSMV is also intended to help learners of all backgrounds better understand the tremendous power of the

Internet, when utilized well, to afford profoundly rich and meaningful learning experiences.

US E F U L N E S S TO OT H E RS

Between 1992 and 1999, OSMV

g rew from 6 teachers and 125 students to over 240 teachers and

11,000 students. Its designers started small in order to re s e a rch what

models of teacher support, studentl e a rning support, and emerg i n g

technologies really worked for a

wide range of classrooms and audiences. They believe that the key to

the pro g r a m ’s success is its re s p o nsiveness to the needs of teachers

and students. Although OSMV is

concentrating its eff o rts on the particular challenge of urban classrooms, the program, with its

focused use of the Internet and curriculum combined with full support

for teachers, fulfills a need

e x p ressed by many teachers everyw h e re .

33
EXC E L L E NC E F OR AL L

OSMV achieves its goals by fully supporting teachers in bringing to their community and their colleagues the

kind of different learning that Internet-rich programs can offer to schools. OSMV targets schools with low connectivity and that face similar challenges as a way to best meet the needs of all schools. The program currently

focuses its local activities on schools in Detroit, many of which have very low socioeconomic status, high crime

rates, and challenging teaching environments.

ORG A N I ZAT ION A L CH A NG E

L o c a l l y, teachers form study groups before, during, and after curricular implementation as a means of scaffolding the learning of new teachers. Because many of the teachers are distributed nationwide, OSMV also organized similar scaffolded study groups online and compensates returning OSMV teachers to act as discussion

moderators, to seed discussions, and to respond to the questions and issues of other teachers. After several

years of OSMV implementation, a strong cohort of returning teachers serves as an extremely valuable resource

for new teachers and sees their professional and in-school roles change. Content support is facilitated through

a “message of the day” posted daily by OSMV scientists to assist teachers in guiding student learning to an interesting current weather event of the day. This allows teachers to be content-savvy about scientific phenomena

not normally covered in traditional textbooks and makes the content more interesting because of its real-life

r a m i f i c a t i o n s .

EV I DE NC E OF EF F E CT I V E N E S S

OSMV makes a measurable difference in learning. In 1997, one study found that Detroit seventh-graders in the

program (95 percent African-American) not only outperformed eighth-grade African-American students nationwide on content addressed in the curriculum but outperformed all eighth-graders nationwide on these items.

These same Detroit students performed significantly lower statistically than all eighth-graders, including

African-American eighth-graders, on items not addressed in the program.

The research results also demonstrate that both students and teachers become significantly more computer- l i terate after work with our programs, and that their attitudes about both science and technology are improved

after participation.

OSMV programs have been implemented with urban, rural, suburban, and homeschooled populations. In one

important measure of effectiveness, girls’ confidence in science increased significantly to match that of boys at

the end of the program. In addition, girls indicated an increase in expectations of achievement at the end of

the program.

Teacher comments speak eloquently to the program’s effectiveness: “Very valuable. My students even gave up

their recess to read messages.” “I can’t say enough about Hurricanes... The people who put it together really

understood what teachers need, and need at their fingertips. The kids were on fire with how much they were

learning and they hated to quit.” 

34

P r o m i s i n g P r o g r a m
p r o m i s i n g P r o g r a m

The WEB Pro j e c t
Promising program

TH E W E B PROJ E CT

PRO G R AM DE S C R I P T ION

The WEB Project created a consortium of community organizations, small businesses, and educational institutions, which collaboratively learned how to employ new technologies to effect systemic

reform in school systems throughout Vermont. The Project, based in Putney, Vermont, is designed

for students in grades 2–12. It utilizes multimedia production and telecommunications as: (a) an

educational environment for student inquiry and expression; (b) a medium for presenting and

assessing student work; and (c) a virtual faculty room for professional discussions about work.

Begun in 1995, the WEB Project has developed systems for the dynamic use of multimedia and

telecommunications to improve student learning in the Vermont Vital Results—communication,

problem solving, civic responsibility, and personal development—as they apply to arts, language, literature, history, and social studies. Partnership initiatives include sharing music compositions, art

projects, literature, and historical research online within the context of specified learning goals.

Each initiative specifies standards that students will reach as a result of participation. 

QUA L I TY A N D ED U CAT ION A L SIG N I F ICA NC E

LE A R N I NG

The WEB Project uses current multimedia technology as a tool to provide opportunities for students

to gather rich source materials; solve problems focused on content; critique their own work as a natural part of the production process; determine trade-offs between time, quality, and technological

limitations; create new knowledge; and express themselves artistically. Several studies cited by the

Web Project indicate that the multimedia and music composition activities lead to complex learning

and provide evidence that student learning is improving in important and complex areas.

Multimedia production requires teamwork. Collaborative problem solving and design adds a 

dynamic, real-world connection to student experiences. 

Through the WEB Project process, students learn how to realize their intentions more fully by learning, for example, how to differentiate foreground from background more clearly; how to use shading

to give a drawing more dimension; and how to focus the viewer’s attention on the important elements

of a drawing or sculpture. In the virtual faculty room, for example, teachers discussed a disturbing

drawing (“Cop Killer”) posted by a student. They covered the issue of censorship, suggesting that 

PRO G R AM CO S T S

For cost information, please contact program designee.

36
FOR FU RT H E R IN F OR MAT ION, CON TA CT: 

F e rn Ta v a l i n

The WEB Pro j e c t

270 Putney Mountain Road

P u t n e y, Ve rmont 05346

Telephone: 802–387–2624

E-mail: tavalin@sover. n e t

Home Page: http://www. w e b p ro j e c t . o rg

students practice voluntary restraint, and asking the “Cop Killer” artist what his intention was. Upon asking, his teacher was surprised to learn that he had seen a photograph of a teenager killed by police while

throwing grenades and wanted to warn other kids to think about the consequences of their actions. This

led to further discussions among students and mentors about censorship, the need to inquire before

drawing conclusions, and the power of images, presentation, and labeling, among other issues.

Student interviews held across Vermont in 1995 by the coordinator of the Vermont Assessment Project

revealed that students want two types of input about their work: where they stand (yardstick) and

how to make what they are doing better (reflection and critique of work in progress). The WEB

Project translated these ideas into a Web-based system that gathers and exhibits multimedia forms of

student performance while simultaneously offering support for reflection and critique. Students

report feeling more motivated because of their access to online mentors who take their work seriousl y. The emphasis on multiple forms of evidence of learning impels students to show what they know

in many ways. The following quote, from a parent whose son worked on a multimedia team investigating community memories of World War II, is one illustration of the depth of engagement, complexi t y, and relevance of the resultant learning: 

I was thinking that if you knew Matt by whether he wrote his essays or not...you would think

of him as a totally different kind of student than you get from watching him do a technology

project...For me, it was a chance to see that this young man is really engaging in positive ways

in the world which I don’t see in terms of his responses to homework and things like that. He

came home and talked about this a lot. He related it to contemporary events. It wasn’t just

that he learned about World War II. He talked about issues of race and gender that were

going on at the time. He was saying that the sociocultural climate was really different in the

thirties and forties than it is now.

EXC E L L E NC E F OR AL L

By design, the WEB Project serves all students. It contributes to excellence for all students, from talented and gifted to students in danger of dropping out or failing.

ORG A N I ZAT ION A L CH A NG E

The WEB Project design focuses on reform at all levels. It has engaged preservice teachers and university faculty in online mentoring. An action research course has been developed for teachers and

potential trainers to improve their programs through the use of technology. The challenge for organizational change even reached the evaluators, who generated a new model—based on theories of

change, systems approaches, and participatory evaluation—of how they believe student learning is

best mediated through technology. 

US E F U L N E S S TO OT H E RS

The WEB Project is highly adaptable

to any subject area and is part i c u l a rly effective at overcoming traditional

b a rriers of access, re p re s e n t a t i o n ,

socioeconomic conditions, as well as

ethnic and gender bias. The online

faculty room offers immediately re levant professional development in

c o re content and includes mentoring in the use of technology in

teaching and learning. The pro j e c t

also supports core discipline practices that had been absent in the

lives of many teachers. For example, prior to the WEB Pro j e c t ,

Ve rmont music teachers neither

composed music nor taught composition, yet composition is the raison

d ’ ê t re of music. Now, Ve rm o n t

music teachers are acquiring and

transmitting the core skills and activities practiced by their re a l - w o r l d

c o u n t e r p a rt s .

To gain the benefits of the WEB

P roject, schools, community org a n izations, and state departments need

low-cost hard w a re and software. A

new project implementation without existing infrastru c t u re would

need to establish or secure a network connection for each part i c ipant and mentor, establish a We b

site with the online tools developed

by or adapted from WEB Pro j e c t

examples, then find, train, and supp o rt personnel to be online content

mentors. (Small districts such as

those found in rural states like

Ve rmont should band together at a

statewide level to increase the number of effective professionals and

teachers in the work; large districts

a re likely to have ample human

re s o u rces at hand.) Four or five

mentors may be needed in a content area to effectively serve up to

2,000 students and 100 teachers at

a time, assuming that students work

in groups and use mentors only

when responses to work or inquiries

about content are not available

l o c a l l y.

37
The Project has spent the last 5 years building capacity so that current participants would have the skills needed to continue in the absence of federal funding. People who use the multimedia and telecommunications

tools well can now be found in all locations around Vermont, providing people-to-people contact when needed

for new users.

Perhaps the primary contribution of the WEB Project has been the creation of a permanent online program

enhancement for all schools in Vermont, which consists of a virtual space where student and teacher work can

be critiqued by the professionals in the arts and humanities community. 

EV I DE NC E OF EF F E CT I V E N E S S

A variety of measures were used to evaluate the effectiveness of the WEB Project. These included an online surv e y, student survey, classroom observations, document and artifact analyses, teacher focus groups and interviews, student focus groups, and interviews with curriculum coordinators, initiative coordinators, and program

coordinators. In addition, student results were measured by teacher-created student product assessments and

selected process rubrics. 

The research associated with the WEB Project demonstrates that students and collaborating organizations have

engaged in rich and substantive online discussion. The project has helped school districts in the state to align

instruction with the Vermont State Frameworks. As a result of their participation in the WEB Project, students

improve their technology skills as well as their performance in the arts by engaging in discussion with mentor

experts. There is greater student engagement in learning tasks when students are using the technology to

design and deliver products and performances than when working on traditional classroom projects or assignments. In addition, students exhibited increased time on task.

More than half of the students used the technology to ask experts for feedback; 65 percent revised their products at least once based on critiques and feedback from experts and mentors. Measures taken at one school

where control and experimental groups for arts and technology were observed revealed test score gains in

problem solving for low-performing math students. 

Student work was recognized in a host of arenas. One student’s videotape of the town’s activity to clean up the

river was broadcast on a local television program. Another student created portfolio pieces that were used by

the local Chamber of Commerce. A talented senior received a full 4-year university scholarship as a result of

his multimedia skills. Two high school students obtained part-time positions as multimedia designers with

local corporations after school and during the summer. Two teachers at different schools in one district

observed improvements in social behavior, attendance, morale, and attention span for several troubled students, which transferred to other core courses. And finally, one failing student who was repeating his senior

year obtained an internship with a local artist and has had several offers to buy his work.

38

P r o m i s i n g P r o g r a m
FU RT H E R RE F L E CT ION S BY T H E PA N E L

39

The following section reflects the work and thinking of the Educational Technology Expert Panel after the completion of

the review process and the recommendation of programs to the Secretary. The material in this section should not be

construed or interpreted as an endorsement by the U.S. Department of Education. After much discussion, the Panel

leaves with you a definition of significant learning for your study and consideration. The Panel also offers for your consideration and use the following framework and guidance as you plan or review your educational technology program. 

❋ ❋ ❋ ❋ ❋ ❋ ❋ ❋ ❋ ❋ ❋

You may have noted that “learning” is cited many times throughout the Educational Technology Expert Panel’s

“Guidelines and Materials for Submitting Educational Technology Programs for Review” (June 1999). The Panel applied

the criterion of complex learning and thinking skills as outlined on page 8 of this Report in making its program recommendations. After the proceedings, however, the Panel concluded that it needed to propose to the field a much more

encompassing term than complex learning; thus, it offers to the field the term significant learning. As stated in the

Report (page 3), “The objective of using educational technology should be to create an instructional environment that

fosters significant learning for preK–12 students.” The Panel’s working definition of the term follows: 

Significant learning is learning that stimulates the student to further inquiry and reflection, a process

that teaches the student how to master ever-deeper levels of understanding. It entails a combination of

mastery of basic skills (such as number facts and literacy) and acquisition of foundational information.

Fundamental skills and knowledge include facility with and understanding of quantitative relationships;

the ability to recognize and construct a persuasive argument using evidence based on observation and the

vocabulary of a specific discipline; and sufficient familiarity with the natural and historical world to recognize patterns of behavior, causality, and meaning. Significant learning enables the student to know

what questions to ask when confronting new information or circumstances and how to marshal appropriate processes and resources to solve problems.

Significant learning is ultimately what the teaching and learning process is aiming to accomplish with all learners in our

n a t i o n ’s schools.
40

R EV I S I T I NG THE CRITERIA FOR EVA LUAT I NG 

T E C H NOLOGY PRO G R AM S

The preceding criteria and indicators (pages 7–10) were used to evaluate the educational technology programs submitted to the Expert Panel. Based on the knowledge gained by the Panel and as a result of a careful examination of the

review process, the Panel proposes the following refinements in the criteria and indicators for greater clarity. You will

also see below a set of rubrics developed and offered by the Panel to (l) clarify the scoring system; and (2) help you

evaluate the soundness and effectiveness of your educational technology program. The Panel hopes that this refinement

in the indicators and its proposed set of rubrics will help to stimulate meaningful conversations and thinking around

what constitutes effective teaching and student learning in our nation’s schools when technology is used as part of the

educational process. 

A. QUA L I TY OF PRO G R AM

Criterion 1. The program addresses an important educational issue or issues and 

articulates its goals and design clearly.

Please describe the program in detail. Readers should be able to understand the overall program, as well as 

what participants actually do, sufficiently well to explain the program to others. Include the items following each

bullet point.

❑ The educational goals are significant. Include in description the 

● need or problem the program addresses and how it relates to teaching and learning 

in preK–12 schools; 

● program goals; and 

● content and learning goals.

❑ The program design is thoughtful and coherent. Include in description the

● program design—how the parts relate to the whole; 

● technology used and how it is essential for achieving the program’s goals; 

● professional development provided as part of the program; and 

● assessments used to determine the program’s efficacy and achievements.

❑ The program goals and design are supported by research. Include in description the

● relationship of goals and design to relevant research on educational reform, the science 

of teaching and learning, or technology.

❑ The program description is clear and complete. It should include

● key learning activities for participants; 

● subject population(s), including ethnic, racial, socioeconomic, and gender percentages and

the size of any special populations served (e.g., ESL, AP biology students, and students with 

disabilities); 

● overall size and maintenance costs (e.g., funding and staff requirements, and 

number of people in target population); 

● keys to the program’s success; and 

● a specific, concrete example that best captures the changes achieved by this program.
41

Criterion 1 Rubrics

Level 1 There is little or no demonstration that the program addresses important educational issues or articulates its goals and design clearly. The program’s goals, design, and description are vague and incomplete. The

design for the use of technology may be an example of already-existing widespread practice and therefore will

not significantly advance or inform the field. The design may be vague, incoherent, unclear, or unsupported by

research; the program description may be unclear. In most cases no indicator is adequately addressed.

Level 2 There is insufficient or incomplete demonstration that the program addresses important educational

issues or articulates its goals and design clearly. The goals, program design, and program description are often

too general to be considered adequate. In some cases, however, the program is designed or described adeq u a t e l y, but the goals are not significant. The goals may be limited or local—not set in a broader educational or

research context. Generally, only one indicator is adequately addressed.

Level 3 The goals are adequately significant and the design adequately thoughtful and coherent. The description, however, may be insufficiently clear or incomplete. The goals and design are often not supported by

research. At least two indicators, including the significance of the educational goals, are adequately addressed.

Level 4 The goals, design, and description may be convincingly significant, thoughtful, coherent, clear, and

complete. Alternatively, the goals and design may be compelling in their significance and thoughtfulness, but the

description only adequate in its clarity and completeness. Program goals and designs are at least adequately

supported by research.

Level 5 The goals, design, and description are convincing or compelling. The program goals and designs are

convincingly supported by research. 
42

B. ED U CAT ION A L SIG N I F ICA NC E

The Expert Panel considers the following three areas—learning, equity, and organizational change—

essential to fulfilling the promise of educational technology. A sound program must address all three, and 

all three must be shown to have impact on or linkage to preK–12 student learning.

Criterion 2. The program develops complex l e a r n i n g and thinking skills. 

If the participants are other than preK–12 students, the applicant should articulate the pro g r a m ’s goals 

and their connection to student learn i n g .

❑ Students increase their in-depth understanding and competence in at least one content discipline. 

❑ Students develop the habits of self-directed, lifelong learning (e.g., the ability to collaborate, to 

direct one’s own learning, to solve problems, to communicate ideas clearly, and to think flexibly and

c r i t i c a l l y ) .

❑ Students become proficient and critical consumers and producers of educational technology.

❑ Students are prepared to enter a technology-infused workplace.
43

Criterion 2 Rubrics 

Level 1 There is little or no demonstration that the program develops complex learning and thinking skills. 

It may be unclear just what activities learners engage in, and how and what they may be learning from these

activities. The contribution of technology to the learning may not be well specified. At this level, there is insufficient demonstration of any of the indicators. 

Level 2 There is insufficient demonstration that the program develops complex learning and thinking skills.

The program may adequately demonstrate that it is addressing one indicator, or may provide more vague

demonstration of more than one indicator. Programs may describe clearly the activities learners engage in, without making the case for how these activities lead to the development of complex learning and thinking skills. If

the program is for teachers, it may be unclear how the program can lead to their students’ developing complex

learning and thinking skills. Programs that focus on the transition from school to work may not clearly demonstrate how they connect to content learning or to self-directed, lifelong learning.

Level 3 There is adequate demonstration that the program develops complex learning and thinking skills. The

program adequately addresses at least two indicators. These programs describe program activities sufficiently,

and make a case for how these activities contribute to learning, for two of the indicators. Programs that serve

teachers may not fully clarify how the program will result in complex learning for students. Programs that focus

on the transition from school to work may fail to demonstrate how they connect to content learning or to selfdirected, lifelong learning.

Level 4 There is convincing demonstration that the program develops complex learning and thinking skills.

The program addresses a minimum of three indicators, at least one convincingly (the other two at least adequately). The program provides a clear description of program activities, and makes a case, through both argument and examples, for how these activities contribute to learning. Programs that serve teachers clarify the links

to student learning—that is, how teacher participation in the program is intended to result in complex learning

for students. Programs that focus on the transition from school to work demonstrate their intended connection

to content learning or to self-directed, lifelong learning.

Level 5 There is compelling demonstration that the program develops complex learning and thinking skills.

The program addresses at least three indicators, at least two convincingly (the third at least adequately). The

program provides a clear description of program activities and makes a case, through both argument and examples, for how these activities contribute to learning. Programs that serve teachers demonstrate the links to student learning. Specifically, they show, through examples of student activities and student work, how teacher participation in the program leads to complex learning for students. Programs that focus on the transition from

school to work demonstrate their actual connection to content learning or to self-directed, lifelong learning.
44

Criterion 3. The program contributes to educational excellence for all.

❑ The program conveys high expectations for all learners.

❑ The program responds to the diverse needs of varied populations of learners. Learners may vary, for 

example, by achievement level, ethnicity, socioeconomic level, English language proficiency, gender, 

learning style, and handicapping conditions.

❑ The program includes active outreach and partnerships with the community or relevant organizations to 

encourage broad participation of diverse groups of learners.

❑ The program increases the participation or achievement of underserved learners so that the gaps between 

these learners and other categories of students diminishes.
45

Criterion 3 Rubrics

Level 1 There is little or no demonstration that the program contributes to educational excellence for all. It

may not be clear who has access to or is served by the program. There is no evidence of outreach or collaboration. While the program may, for example, set high expectations for its learners, it may serve only students who

are already well served without attempting to include and serve a broader range of learners .

Level 2 There is limited and insufficient demonstration that the program contributes to educational excellence

for all. A program may address the needs of an underserved group of learners, but without conveying expectations that are high or clear. A program in teacher development may serve a diverse group of teachers without

making any connection to the diverse needs of the students they teach. A software-based program may claim that

the software is designed to serve diverse learners but not demonstrate how or that it does. There is no evidence

of active outreach or collaborative partnership.

Level 3 There is adequate demonstration that the program contributes to educational excellence for all.

Programs generally set high expectations and serve diverse groups of learners. Many do not have strong outreach programs. Programs may have partners, but be unable to show how these partnerships contribute to

broader participation. Some programs are able to demonstrate that they are closing gaps in the participation of

underserved learners.

Level 4 There is clear and convincing demonstration that the program contributes to educational excellence for

all. Programs set high expectations for all learners, meet the needs of diverse and underserved learners, and

have active outreach and collaborative partnerships. Programs show that they narrow the gaps in participation

for underserved learners, although at this level they do not yet demonstrate an ability to narrow the gaps in student performance.

Level 5 There is complete and compelling demonstration that the program contributes to educational excellence for all. The program fully addresses all four indicators. Programs are able to demonstrate that they have

increased both the participation and the performance of underserved groups of learners.
46

Criterion 4. The program promotes coherent organizational change.

❑ The program reflects a vision of educational renewal consistent with disciplinary content standards, 

recommendations from national commissions, findings from educational research, and documented best 

p r a c t i c e s .

❑ The program has identified a set of goals that will lead to educational renewal within the organization.

❑ The program has identified and involved key constituents related to its set of goals—parents, professional 

groups, and community members.

❑ Through partnerships and professional development, the program enhances the human capacity necessary 

to accomplish its goals (e.g., allocates time for teachers’ and administrators’ collaboration and planning).

❑ As a result of the program, policies, procedures, funding, and practice have been changed to increase their 

alignment in support of sustainable change.
47

Criterion 4 Rubrics

Level 1 There is little or no demonstration that the program promotes coherent organizational change. Either

the response to this criterion is too vague, or the program has not demonstrated the vision, goals, involvement of

constituencies, enhancement of human capacity, or changes in policy that can promote organizational change.

Level 2 There is limited and insufficient demonstration that the program promotes coherent organizational

change. For example, a program may claim to be guided by a vision without making clear what that vision is, or

may recognize the need for capacity building to support change without actually building that capacity.

A l t e r n a t e l y, it may indicate that it has established partnerships without showing how they contribute to organizational change. The program may adequately demonstrate one indicator.

Level 3 There is adequate demonstration that the program promotes some organizational change. The program adequately addresses at least three of the indicators in ways that can result in organizational change.

Change is not yet comprehensive or fully coherent.

Level 4 There is clear and convincing demonstration that the program promotes coherent organizational

change. The program addresses four indicators, at least three convincingly. 

Level 5 There is complete and compelling demonstration that the program promotes coherent organizational

change. All five indicators are addressed, at least four convincingly.
48

C. EV I DE NC E OF EF F E CT I V E N E S S

Criterion 5. The program has rigorous, measurable evidence of its achievements for one or more

among Criteria 2, 3, and 4 (learning, equity, and organizational change).

❑ The evidence is clearly related to the claims made for program effectiveness. For example: 

● framework for gathering evidence is closely tied to program goals; and 

● evidence bears directly on claims made for program effectiveness.

❑ The design for collecting evidence meets generally accepted standards of quality and may include:

● one or more comparison groups;

● a quantified validation by an external authority; 

● a formal evaluation; 

● a quantified demonstration of positive change among participants as a result of the program 

(e.g., increased parental involvement in school governance; diminished gaps in achievement between 

groups; increased enrollment in rigorous mathematics courses or graduation rates among subject 

populations; changes in the base funding and requirements for professional development); and 

● an in-depth, qualitative analysis of change among participants as a result of the program 

(e.g., case studies, ethnographies, principled analysis of observations and interviews).

❑ The evidence is complete and convincing. This includes the following: 

● methods are fully described and explained;

● sample sizes are appropriate;

● results are clearly and completely documented;

● data are disaggregated for relevant groups; 

● multiple methods are used to support key claims; and

● results are generally positive.
49

Criterion 5 Rubrics

Level 1 There is no rigorous, measurable evidence for the program’s achievements. If evidence is presented,

it is not clearly related to program goals and claims, does not meet standards of quality, and is often anecdotal.

Programs at this level are often in too early a stage to have collected valid evidence, although they may have

plans for doing so. 

Level 2 There is incomplete, insufficient evidence for the program’s achievements. Although considerable

amounts of data may be presented, the data do not constitute credible evidence of effectiveness. Often the program lacks a framework for gathering, analyzing, and interpreting evidence. A program may make numerous

claims, for example, but present data that bear on only one or two, or none, of them. A program may be very

limited in the kind of data it collects; it may, for example, rely solely on attitudinal evidence. A program may not

sample adequately given the size and type of groups it serves. A program may not describe its methods with sufficient clarity. Generally, programs use only one method to support any claim. Often, programs are in too early

a stage of development to present credible evidence of impact.

Level 3 There is credible, adequate evidence for the program’s achievements. The evidence is clearly related

to program goals and to claims of effectiveness. The evaluation design meets adequate standards of quality.

The design includes most of the elements required to make valid inferences about the effectiveness of the program. The evidence is sufficiently well documented, analyzed, and complete to present an adequate case for the

p r o g r a m ’s effectiveness with respect to at least one criterion.

Level 4 There is convincing evidence for the program’s achievements. The evidence is clearly related to program goals and to claims for effectiveness. The evaluation design meets high standards of quality. The design

includes most of the elements required to make valid inferences about the effectiveness of the program. The

evidence is well documented, carefully analyzed, and complete. It presents a convincing case for the program’s

effectiveness with respect to at least one criterion. It further presents an adequate case for the effectiveness of

one other criteria.

Level 5 There is compelling evidence for the program’s achievements. The evidence is clearly related to program goals and to claims for effectiveness. The evaluation design is driven by these goals and claims. For all

claims, clear and appropriate evidence is provided. The design meets high standards of quality. The design

includes all the elements required to make valid inferences about the effectiveness of the program. The evidence

is completely documented, carefully analyzed, and presents a compelling case for the program’s effectiveness

with respect to at least one criterion. A program at this level shows evidence of at least adequate effectiveness for

the other two criteria.
50

D. US E F U L N E S S TO OT H E RS

Criterion 6. The program is adaptable for use in multiple contexts.

❑ The program’s technology requirements are easily available to potential users.

❑ The program is cost-effective relative to its benefits.

❑ After its initial implementation, the program is sustainable with existing resources (i.e., does not require 

extraordinary/unreasonable time, effort, or funding), and scalable (i.e., can naturally expand its scope to 

several teachers, multiple grade levels/subjects/sites, and different disciplines). 

❑ The program is adaptable to a range of educational settings.

❑ The program provides clear and detailed guidelines about the conditions required for its successful 

implementation (e.g., philosophical assumptions and level of personnel training prior to start-up).

❑ The program provides clear and detailed guidelines about the start-up and annual ongoing costs required 

for its successful implementation (e.g., equipment and infrastructure; personnel; training costs; technical 

support; replacement or upgraded hardware or software; materials and supplies).
51

Criterion 6 Rubrics

Level 1 There is little or no demonstration that the program is adaptable for use in multiple contexts. In some

cases, the program is too new to be able to show adaptability. In some cases, the program description is so vague

or unfocused that what would be replicated or sustained is not clear.

Level 2 There is limited or insufficient demonstration that the program is adaptable for use in multiple contexts.

Programs may demonstrate that their technology and cost requirements are reasonable and that they are locally sustainable, but not that they are scalable or adaptable to a range of settings; or they may provide no guidelines for

implementation. Programs at this level may have made no effort to expand beyond a set of limited goals for a particular target population. Alternatively, they may be prohibitively costly, although their model is, in principle, adaptable. 

Level 3 There is adequate demonstration that the program is adaptable for use in multiple contexts. Programs at

this level have adequately available technology, are cost effective, and demonstrate adequate scalability or adaptabilit y. Some may have clear guidelines for implementation, including costs and other requirements.

Level 4 There is convincing demonstration that the program is adaptable for use in multiple contexts. Programs

at this level are at least adequate with respect to five of the six indicators.

Level 5 There is compelling demonstration that the program is adaptable for use in multiple contexts. Programs

address all of the indicators and are able to show that they can be been widely used in multiple settings.
The purpose of the Expert Panel is to help practitioners make better-informed decisions by setting forth the

experience and insights of others. In the last decade, technology has achieved a critical mass of availability

and power. We stand at the opening of a revolutionary pathway built on interactive capabilities. No one

knows where this pathway will ultimately lead, but many have contributed to the development of a reliable

compass to guide us all forward.
ORAD 2001-1015

Office of Educational Research and Improvement

U.S. Department of Education

Postage and Fees Paid

U.S. Department of Education

Permit No. G-17

Standard 

United States

Department of Education

E D - P u b s

8242-B Sandy Court

Jessup, MD 20794-1398

Official Business

Penalty for Privacy Use, $300

ORAD 2001-1015

Office of Educational Research and Improvement

U.S. Department of Education