USING RESEARCH AND REASON IN EDUCATION

WHAT IS

SCIENTIFICALLY BASED

RESEARCH?

A GUIDE FOR TEACHERS
Early in the 17th century, two astronomers competed 

to describe the nature of our solar system.

Galileo built a telescope and found new planets and

moons. Francesco Sizi ridiculed Galileo’s findings. 

There must be only seven planets, Sizi said. After all, 

there are seven windows in the head—two nostrils, two

ears, two eyes, and a mouth. There are seven known

metals. There are seven days in a week, and they are

already named after the seven known planets. If we

increase the number of planets, he said, the whole

system falls apart. Finally, Sizi claimed, these so-called

satellites being discovered by Galileo were invisible to

the eye. He concluded they must have no influence on

the Earth and, therefore, do not exist.

Sizi’s most valuable contribution to history may have

been to remind us that true understandings of the

world, and how it works, cannot be based on pure

thought alone, no matter how logical, creative, or

contemporary such thought may seem.

True understandings require some measure of science 

and the willingness to seek information when making

decisions.
●

1

BECOMING A WISE CONSUMER

OF EDUCATION RESEARCH

More than ever, educators are expected to make decisions

that guarantee quality instruction. As knowledge emerges, so do

philosophies, opinions, and rhetoric about definitions of instructional

excellence. From policy makers to classroom teachers, educators need ways to

separate misinformation from genuine knowledge and to distinguish scientific

research from poorly supported claims.

Effective teachers use scientific thinking in their classrooms all the time. They

assess and evaluate student performance, develop Individual Education Plans,

reflect on their practice, and engage in action research. Teachers use experimental

logic when they plan for instruction: they evaluate their students’ previous

knowledge, construct hypotheses about the best methods for teaching, develop

teaching plans based on those hypotheses, observe the results, and base further

instruction on the evidence collected.

In short, teachers use the concepts of rigorous research and evaluation in

profoundly practical ways.

Teachers can further strengthen their instruction and protect their students’

valuable time in school by scientifically evaluating claims about teaching

methods and recognizing quality research when they see it. This booklet,

distilled from the monograph Using Research and Reason in Education: How

Teachers Can Use Scientifically Based Research to Make Curricular and

Instructional Decisions, provides a brief introduction to understanding and

using scientifically based research.
2

RECOGNIZING EFFECTIVE RESEARCH

Teachers can use a simple set of questions to distinguish between research that

confirms the effectiveness of an instructional practice and research that does not:

● Has the study been published in a peer-reviewed journal or approved

by a panel of independent experts?

● Have the results of the study been replicated by other scientists?

● Is there consensus in the research community that the study’s findings

are supported by a critical mass of additional studies?

● ●

The federal perspective on scientifically 

based research

The No Child Left Behind (NCLB) Act of 2001 encourages and, in

some cases such as Reading First, requires the use of instruction based

on scientific research. The emphasis on scientifically based research

supports the consistent use of instructional methods that have been

proven effective.

To meet the NCLB definition of “scientifically based,” research must:

● employ systematic, empirical methods that draw on observation or

experiment;

● involve rigorous data analyses that are adequate to test the stated

hypotheses and justify the general conclusions;

● rely on measurements or observational methods that provide valid

data across evaluators and observers, and across multiple

measurements and observations; and

● be accepted by a peer-reviewed journal or approved by a panel of

independent experts through a comparatively rigorous, objective, 

and scientific review.
3

Independent peer review

Peer review subjects a paper to scrutiny by scientists in the relevant field of

specialization. This happens in two ways. In one method, a paper submitted 

for publication in a peer-reviewed journal is examined by other scientists in the

field before an editor (usually an expert in the field) passes judgment on it. The

second method is review by an independent panel of experts who, using rigorous

criteria, determine whether the findings of the paper are credible.

Peer review provides a baseline of quality control because it exposes ideas and

experimentation to examination and criticism by other researchers. Its absence

should raise doubt about the quality of the research. Presentations at education

conferences that make claims about specific educational practices should also

be held to this standard.

It is relatively easy for teachers to determine if a paper has been published in a

peer-reviewed journal; it can be more difficult to determine whether a panel

review (without publication) has occurred unless it is specified in the paper.

Not all education journals are peer-reviewed

Education journals have different purposes that contribute to our

understanding of teaching. The American Educational Research Journal,

the Journal of Educational Psychology, and Reading Research

Quarterly are examples of journals that conduct peer reviews and

contain empirical evidence about teaching techniques. Phi Delta

Kappan and Educational Leadership, by contrast, contain original

thought, but neither publishes peer-reviewed original research.

Peer-reviewed journals on other topics such as cognitive psychology

and other social sciences can also make useful contributions to

educational practice.
4

Replication of results by other scientists

Teachers should look for evidence that an instructional technique has been

proven effective by more than one study. Knowledge generated by one study

without scrutiny and criticism by others cannot be fully scientific. To be

considered scientifically based, a research finding must be presented in a way

that enables other researchers to reach the same results when they repeat the

experiment.

True scientific knowledge is public and open to challenge. It is held tentatively,

subject to change based on contrary evidence.

Consensus within a research community

A single experiment rarely decides an issue, supporting one theory and ruling

out all others. Issues are most often decided when the community of scientists

in a field comes to agreement over time that sufficient evidence has converged

to support one theory over another. Scientists do not evaluate data from a single,

perfectly designed experiment. They evaluate data from many experiments, each

containing some flaws but providing part of the answer.

SCIENTIFIC INVESTIGATION

PROCEEDS BY STAGES

Becoming more aware of how the scientific process manifests itself every day 

in both research and teaching can enhance a teacher’s effectiveness, depth of

expertise, and ability to justify the choice of instructional methods to parents,

peers, and administrators. As in formal evaluations of educational programs, 

the tenets and themes of scientific research have relevance and application in 

the classroom. But because there are different stages of scientific investigation,

teachers should take care to use data generated at each stage in appropriate ways.

● ●
For example, some teachers rely on their own observations to make judgments

about the success of educational strategies. A collection of observations leads to

some understanding of the world, but observations have limited value. Scientific

observations must be structured in order to support or reject theories about the

causes that underlie events. Scientists—and teachers—make predictions about

causes based on their structured observations and then use other techniques to

test specific outcomes.

In the early stages of investigation, case studies—highly detailed descriptions of

individuals or small groups and the context surrounding them—can be useful.

Case studies provide descriptive information about how an educational

program operates in a classroom, for example, descriptions of instructional

strategies, amount of time, and types of materials used in a new vocabulary

program. This qualitative design uses a variety of data collection methods from

multiple sources to study a single entity in depth, over a period of time, and in

its context. Case studies lack the comparative information needed to determine

cause-effect relationships, but they can point researchers to variables that

deserve further study and help generate hypotheses. They can be helpful in

developing theories about what is or is not working instructionally. However,

case studies cannot provide the measurable results that are necessary to

understand and confirm outcomes.

5
6

Correlational studies take things a step further by testing whether there are

links between variables and outcomes. They are useful in early and middle

stages of an investigation once hypotheses have been developed. For example, if

a researcher hypothesizes that vocabulary instruction leads to improved reading

comprehension, he or she could conduct a correlational study, using statistical

techniques, to determine if there is a link between vocabulary instruction and

reading comprehension. If the study finds a link, the researcher could design 

a randomized controlled trial, or true experiment, to confirm whether the

vocabulary instruction causes the improvement in comprehension.

In order to draw conclusions about outcomes and their causes, data must come

from true experiments. True experiments, or randomized field or controlled

trials, test specific predictions and rule out alternative explanations. In an

experiment, an investigator assigns subjects randomly to experimental and

control groups, varies the apparent cause (the independent variable) and looks

at the apparent effect (the dependent variable) while holding all other variables

constant. Only true experiments can provide evidence of whether an

instructional practice works or not.

The experimental method controls for the many other variables that could 

have an impact on an outcome. Unlike case studies and correlational methods,

experiments use techniques such as random assignment of subjects to treatment

and control conditions and the matching of subjects in the treatment groups on

background and ability variables.

For example, imagine an experimental study that investigates whether

vocabulary instruction has a positive effect on reading comprehension. A

sample of third-grade students is selected and half of the students are then

randomly assigned to the treatment group and half are assigned to the

control/comparison group. Random assignment ensures (if the sample 

size is sufficient) that the two groups will be relatively matched on various

demographic characteristics and on overall ability level. This is why random

assignment is so important—it ensures the equivalence/comparability of the
● ●

7

students in the treatment and control groups. Students in the treatment group

then receive instruction in learning 100 new vocabulary words, while students

in the control group do not receive instruction in learning the new vocabulary

words (they engage in an alternative activity). At the end of the instructional

period, students are given a standardized comprehension test.

Results of the study will demonstrate whether students in the treatment

condition do better than comparable students in the control group on 

the test of reading comprehension. If the treatment group shows reading

comprehension scores that are higher than those of the control group to a

statistically significant degree, then the experiment provides evidence that helps

to establish a causal relationship between vocabulary instruction and reading

comprehension. As mentioned earlier, multiple studies that replicate these

methods and find similar results would need to be conducted for further

confidence in the results.

While teachers certainly would not be expected to carry out true experimental

research in the classroom, understanding the role of experimental research as

well as the other stages of scientific investigation and the data they generate—

from observations to standardized assessments—can prepare teachers to

interpret research better, decide what and how to teach, and make legitimate,

defendable statements about the impact of their instructional choices.

USING THE RESEARCH LITERATURE

AS A GUIDE

In many cases, science has not yet provided the answers teachers and others

need to make fully informed decisions about adopting, or dropping, particular

educational strategies. What if an area of education lacks a research base, has

not been evaluated according to the principles of scientific evaluation described

above, and no consensus exists? In those cases, teachers have to rely on

scientific reasoning to find their way. An important first step is to look at 

the findings and principles from the established research base.
8

Imagine, for example, that two untested treatments for children with extreme

reading difficulties have emerged. The first treatment suggests a new strategy

for teaching phonemic awareness by using only songs and clapping to teach

children how language can be broken down into sound segments, or phonemes.

The second treatment trains children in vestibular sensitivity by having them

walk on balance beams while blindfolded. In this hypothetical case, neither of

these new treatment ideas has been tested.* Neither has produced empirical

evidence to prove that it is effective.

The lack of such evidence need not automatically lead to the conclusion that

the methods do not work. Even without empirical evidence, one might find

support for one or both methods from other studies conducted on similar

strategies. In this case, the strategy featuring awareness of sound segments merits

consideration first, because it makes contact with a broad consensus in the

research literature that children with severe reading difficulties are hampered 

by an insufficiently developed awareness of language’s segmented structure. 

The second does not have a comparable link to existing research. A teacher

thinking scientifically can make a reasonable conclusion that the first method 

is preferable by knowing that there is a link to the broader research base.

Teachers supporting teachers

It can be difficult for a teacher to sort through claims of educational

impact. Teachers may want to form reading/discussion groups to talk

about research studies and to challenge each other in a collegial way

about what works, or does not work, in the classroom. 

The Institute of Education Sciences’ What Works Clearinghouse

(www.whatworks.ed.gov) can be a resource for summaries of

scientifically based studies. By talking and learning more about how

to apply the findings of scientific research in their teaching, educators

can practice and refine their skills—and follow the example of

Galileo in bringing new knowledge to the world.

*Some treatments related to phonemic instruction have, in fact, been scientifically tested and

proven effective. This hypothetical example imagines a new strategy—using only songs and

clapping—that has not been proven effective.
9

IN SUMMARY

Teachers play a variety of roles in their work—instructor, coach,

advocate, and learner—but they also act as scientists in several

ways. As they make the important decisions about what and how

to teach, they must evaluate the claims associated with educational

strategies and programs. And in the classroom, they must

constantly assess and reassess the value of programs and their

impact on students.

The basic principles of the scientific method

● Science progresses by investigating testable problems.

● A testable theory yields predictions that could possibly be

proven wrong.

● Scientific knowledge has passed some minimal tests.

● Data and theories are considered in the public domain, or

included in the research base, only after a peer review,

either by a journal or a panel.

● Published data and theories allow for replication and

criticism by other scientists.

● Theories are tested by systematic observation bound by

the logic of true experiments.

●Correlational studies, useful when experiments can not be

carried out, only help rule out hypotheses.

● Researchers use many different methods to reach

conclusions. Most often, they draw conclusions only after

a slow accumulation of data from many studies.
This brochure is based on Using Research and Reason in Education: How Teachers Can Use

Scientifically Based Research to Make Curricular and Instructional Decisions, written by 

Paula J. Stanovich and Keith E. Stanovich of the University of Toronto, and published by The

Partnership for Reading.

For copies of the original monograph, or for more copies of this brochure, download PDF 

or HTML versions at www.nifl.gov/partnershipforreading. To order print copies, contact the

National Institute for Literacy at ED Pubs, PO Box 1398, Jessup, Maryland 20794-1398. 

Call 1-800-228-8813, or email edpubs@inet.ed.gov.

The Partnership for Reading, a project administered by the National Institute for Literacy, is a

collaborative effort of the National Institute for Literacy, the National Institute of Child Health

and Human Development, the U. S. Department of Education, and the U. S. Department of

Health and Human Services to make scientifically based reading research available to educators,

parents, policy makers, and others with an interest in helping all people learn to read well.

The National Institute for Literacy, an independent federal organization, supports the

development of high quality state, regional, and national literacy services so that all Americans

can develop the skills they need to succeed at work, at home, and in the community.

This brochure was written, edited, and designed by RMC Research Corporation under National

Institute for Literacy Contract No. ED-00CO-0093. The views expressed herein do not

necessarily represent the policies of the National Institute for Literacy. No official endorsement

by the National Institute for Literacy of any product, commodity, service, entity, or enterprise is

intended or should be inferred.

The National Institute for Literacy

Sandra Baxter, Director

Lynn Reddy, Deputy Director

2005