What is Scientifically 

Based Research?

USING RESEARCH AND 

REASON IN EDUCATION

A Guide for Teachers
Early in the 17th century, two astronomers competed to describe the

nature of our solar system.

Galileo built a telescope and found new planets and moons.

Francesco Sizi ridiculed Galileo’s findings. There must be only seven

planets, Sizi said. After all, there are seven windows in the head—two

nostrils, two ears, two eyes, and a mouth. There are seven known

metals. There are seven days in a week, and they are already named

after the seven known planets. If we increase the number of planets,

he said, the whole system falls apart. Finally, Sizi claimed, these socalled satellites being discovered by Galileo were invisible to the eye.

He concluded they must have no influence on the Earth and,

therefore, do not exist.

Sizi’s most valuable contribution to history may have been to remind

us that true understandings of the world, and how it works, cannot

be based on pure thought alone, no matter how logical, creative, or

contemporary such thought may seem.

True understandings require some measure of science and the

willingness to seek information when making decisions.
BECOMING A WISE CONSUMER OF EDUCATION RESEARCH

More than ever, educators are expected to make decisions that guarantee quality

instruction. As knowledge emerges, so do philosophies, opinions, and rhetoric about

definitions of instructional excellence. From policy makers to classroom teachers,

educators need ways to separate misinformation from genuine knowledge and to

distinguish scientific research from poorly supported claims.

Effective teachers use scientific thinking in their classrooms all the time. They assess

and evaluate student performance, develop Individual Education Plans, reflect on their

practice, and engage in action research. Teachers use experimental logic when they plan

for instruction: they evaluate their students’ previous knowledge, construct hypotheses

about the best methods for teaching, develop teaching plans based on those hypotheses,

observe the results, and base further instruction on the evidence collected.

In short, teachers use the concepts of rigorous research and evaluation in profoundly

practical ways.

Teachers can further strengthen their instruction and protect their students’ valuable

time in school by scientifically evaluating claims about teaching methods and recognizing

quality research when they see it. This booklet, distilled from the monograph Using

Research and Reason in Education: How Teachers Can Use Scientifically Based Research

to Make Curricular and Instructional Decisions, provides a brief introduction to

understanding and using scientifically based research.

1

The federal perspective on scientifically based research

The No Child Left Behind (NCLB) Act of 2001 encourages and, in some cases

such as Reading First, requires the use of instruction based on scientific

research. The emphasis on scientifically based research supports the consistent

use of instructional methods that have been proven effective. To meet the NCLB

definition of “scientifically based,” research must:

• employ systematic, empirical methods that draw on observation or

experiment;

• involve rigorous data analyses that are adequate to test the stated hypotheses

and justify the general conclusions;

• rely on measurements or observational methods that provide valid data across

evaluators and observers, and across multiple measurements and

observations; and

• be accepted by a peer-reviewed journal or approved by a panel of independent

experts through a comparatively rigorous, objective, and scientific review.
RECOGNIZING EFFECTIVE RESEARCH

Teachers can use a simple set of questions to distinguish between research that confirms

the effectiveness of an instructional practice and research that does not:

• Has the study been published in a peer-reviewed journal or approved by a panel of

independent experts?

• Have the results of the study been replicated by other scientists?

• Is there consensus in the research community that the study’s findings are supported by

a critical mass of additional studies?

Independent peer review

Peer review subjects a paper to scrutiny by scientists in the relevant field of

specialization. This happens in two ways. In one method, a paper submitted for

publication in a peer-reviewed journal is examined by other scientists in the field before

an editor (usually an expert in the field) passes judgment on it. The second method is

review by an independent panel of experts who, using rigorous criteria, determine

whether the findings of the paper are credible.

Peer review provides a baseline of quality control because it exposes ideas and

experimentation to examination and criticism by other researchers. Its absence should

raise doubt about the quality of the research. Presentations at education conferences that

make claims about specific educational practices should also be held to this standard.

It is relatively easy for teachers to determine if a paper has been published in a peerreviewed journal; it can be more difficult to determine whether a panel review (without

publication) has occurred unless it is specified in the paper.

2

Not all education journals are peer-reviewed

Education journals have different purposes that contribute to our understanding

of teaching. The American Educational Research Journal, the Journal of

Educational Psychology, and Reading Research Quarterly are examples of journals

that conduct peer reviews and contain empirical evidence about teaching

techniques. Phi Delta Kappan and Educational Leadership, by contrast, contain

original thought, but neither publishes peer-reviewed original research.

Peer-reviewed journals on other topics such as cognitive psychology and

other social sciences can also make useful contributions to educational practice.
Replication of results by other scientists

Teachers should look for evidence that an instructional technique has been proven

effective by more than one study. Knowledge generated by one study without scrutiny

and criticism by others cannot be fully scientific. To be considered scientifically based, a

research finding must be presented in a way that enables other researchers to reach the

same results when they repeat the experiment.

True scientific knowledge is public and open to challenge. It is held tentatively,

subject to change based on contrary evidence.

Consensus within a research community

A single experiment rarely decides an issue, supporting one theory and ruling out all

others. Issues are most often decided when the community of scientists in a field comes

to agreement over time that sufficient evidence has converged to support one theory over

another. Scientists do not evaluate data from a single, perfectly designed experiment.

They evaluate data from many experiments, each containing some flaws but providing

part of the answer.

SCIENTIFIC INVESTIGATION PROCEEDS BY STAGES

Becoming more aware of how the scientific process manifests itself every day in both

research and teaching can enhance a teacher’s effectiveness, depth of expertise, and

ability to justify the choice of instructional methods to parents, peers, and administrators.

As in formal evaluations of educational programs, the tenets and themes of scientific

research have relevance and application in the classroom. But because there are different

stages of scientific investigation, teachers should take care to use data generated at

each stage in appropriate ways.

For example, some teachers rely on their own observations to make judgments about

the success of educational strategies. A collection of observations leads to some

understanding of the world, but observations have limited value. Scientific observations

must be structured in order to support or reject theories about the causes that underlie

events. Scientists—and teachers—make predictions about causes based on their

structured observations and then use other techniques to test specific outcomes.

In the early stages of investigation, case studies—highly detailed descriptions of

individuals or small groups and the context surrounding them—can be useful. Case

3
studies provide descriptive information about how an educational program operates in a

classroom, for example, descriptions of instructional strategies, amount of time, and

types of materials used in a new vocabulary program. This qualitative design uses a

variety of data collection methods from multiple sources to study a single entity in depth,

over a period of time, and in its context. Case studies lack the comparative information

needed to determine cause-effect relationships, but they can point researchers to

variables that deserve further study and help generate hypotheses. They can be helpful in

developing theories about what is or is not working instructionally. However, case studies

cannot provide the measurable results that are necessary to understand and confirm

outcomes.

Correlational studies take things a step further by testing whether there are links

between variables and outcomes. They are useful in early and middle stages of an

investigation once hypotheses have been developed. For example, if a researcher

hypothesizes that vocabulary instruction leads to improved reading comprehension, he or

she could conduct a correlational study, using statistical techniques, to determine if there

is a link between vocabulary instruction and reading comprehension. If the study finds a

link, the researcher could design a randomized controlled trial, or true experiment, to

confirm whether the vocabulary instruction causes the improvement in comprehension.

In order to draw conclusions about outcomes and their causes, data must come from

true experiments. True experiments, or randomized field or controlled trials, test specific

predictions and rule out alternative explanations. In an experiment, an investigator

assigns subjects randomly to experimental and control groups, varies the apparent cause

(the independent variable) and looks at the apparent effect (the dependent variable) while

holding all other variables constant. Only true experiments can provide evidence of

whether an instructional practice works or not.

The experimental method controls for the many other variables that could have an

impact on an outcome. Unlike case studies and correlational methods, experiments use

techniques such as random assignment of subjects to treatment and control conditions

and the matching of subjects in the treatment groups on background and ability variables.

For example, imagine an experimental study that investigates whether vocabulary

instruction has a positive effect on reading comprehension. A sample of third-grade

students is selected and half of the students are then randomly assigned to the treatment

4
group and half are assigned to the control/comparison group. Random assignment

ensures (if the sample size is sufficient) that the two groups will be relatively matched 

on various demographic characteristics and on overall ability level. This is why random

assignment is so important—it ensures the equivalence and comparability of the

students in the treatment and control groups. Students in the treatment group then

receive instruction in learning 100 new vocabulary words, while students in the control

group do not receive instruction in learning the new vocabulary words (they engage in 

an alternative activity). At the end of the instructional period, students are given a

standardized comprehension test.

Results of the study will demonstrate whether students in the treatment condition 

do better than comparable students in the control group on the test of reading

comprehension. If the treatment group shows reading comprehension scores that are

higher than those of the control group to a statistically significant degree, then the

experiment provides evidence that helps to establish a causal relationship between

vocabulary instruction and reading comprehension. As mentioned earlier, multiple studies

that replicate these methods and find similar results would need to be conducted for

further confidence in the results.

While teachers certainly would not be expected to carry out true experimental

research in the classroom, understanding the role of experimental research as well as the

other stages of scientific investigation and the data they generate—from observations 

to standardized assessments—can prepare teachers to interpret research better, decide

what and how to teach, and make legitimate, defendable statements about the impact of

their instructional choices.

USING THE RESEARCH LITERATURE AS A GUIDE

In many cases, science has not yet provided the answers teachers and others need 

to make fully informed decisions about adopting, or dropping, particular educational

strategies. What if an area of education lacks a research base, has not been evaluated

according to the principles of scientific evaluation described above, and no consensus

exists? In those cases, teachers have to rely on scientific reasoning to find their way. 

An important first step is to look at the findings and principles from the established

research base.

5
Imagine, for example, that two untested treatments for children with extreme

reading difficulties have emerged. The first treatment suggests a new strategy for

teaching phonemic awareness by using only songs and clapping to teach children how

language can be broken down into sound segments, or phonemes. The second treatment

trains children in vestibular sensitivity by having them walk on balance beams while

blindfolded. In this hypothetical case, neither of these new treatment ideas has been

tested.* Neither has produced empirical evidence to prove that it is effective.

The lack of such evidence need not automatically lead to the conclusion that the

methods do not work. Even without empirical evidence, one might find support for one or

both methods from other studies conducted on similar strategies. In this case, the

strategy featuring awareness of sound segments merits consideration first, because it

makes contact with a broad consensus in the research literature that children with severe

reading difficulties are hampered by an insufficiently developed awareness of language’s

segmented structure. The second does not have a comparable link to existing research. A

teacher thinking scientifically can make a reasonable conclusion that the first method is

preferable by knowing that there is a link to the broader research base.

6

Teachers supporting teachers

It can be difficult for a teacher to sort through claims of educational impact.

Teachers may want to form reading/discussion groups to talk about research

studies and to challenge each other in a collegial way about what works, or does

not work, in the classroom. 

The Institute of Education Sciences’ What Works Clearinghouse

(www.whatworks.ed.gov) can be a resource for summaries of scientifically based

studies. By talking and learning more about how to apply the findings of scientific

research in their teaching, educators can practice and refine their skills—and

follow the example of Galileo in bringing new knowledge to the world.

*Some treatments related to phonemic instruction have, in fact, been scientifically tested and proven effective. This hypothetical example

imagines a new strategy—using only songs and clapping—that has not been proven effective.
Teachers play a variety of roles in their work—instructor, coach,

advocate, and learner—but they also act as scientists in several ways.

As they make the important decisions about what and how to teach,

they must evaluate the claims associated with educational strategies

and programs. And in the classroom, they must constantly assess and

reassess the value of programs and their impact on students.

The basic principles of the scientific method

• Science progresses by investigating testable problems.

• A testable theory yields predictions that could possibly be proven

wrong.

• Scientific knowledge has passed some minimal tests.

• Data and theories are considered in the public domain, or included in

the research base, only after a peer review, either by a journal or a

panel.

• Published data and theories allow for replication and criticism by

other scientists.

• Theories are tested by systematic observation bound by the logic of

true experiments.

• Correlational studies, useful when experiments can not be carried

out, only help rule out hypotheses.

• Researchers use many different methods to reach conclusions. Most

often, they draw conclusions only after a slow accumulation of data

from many studies.

7

IN SUMMARY
8

This brochure is based on Using Research and Reason in Education: How Teachers Can Use

Scientifically Based Research to Make Curricular and Instructional Decisions, written by Paula J.

Stanovich and Keith E. Stanovich of the University of Toronto, and published by the Partnership 

for Reading.

For copies of the original monograph, or for more copies of this brochure, download PDF 

or HTML versions at www.nifl.gov. To order print copies, contact the National Institute for 

Literacy at ED Pubs, PO Box 1398, Jessup, Maryland 20794-1398. Call 1-800-228-8813, or 

email edpubs@inet.ed.gov.

The Partnership for Reading, a project administered by the National Institute for Literacy, is a

collaborative effort of the National Institute for Literacy, the National Institute of Child Health and

Human Development, the U. S. Department of Education, and the U. S. Department of Health and

Human Services to make scientifically based reading research available to educators, parents,

policy makers, and others with an interest in helping all people learn to read well.

The National Institute for Literacy, an agency in the Federal government, is authorized to help

strengthen literacy across the lifespan. The Institute works to provide national leadership on

literacy issues, including the improvement of reading instruction for children, youth, and adults by

sharing information on scientifically based research. 

This brochure was written, edited, and designed by RMC Research Corporation under National

Institute for Literacy Contract No. ED-00CO-0093. The views expressed herein do not necessarily

represent the policies of the National Institute for Literacy. No official endorsement by the

National Institute for Literacy of any product, commodity, service, entity, or enterprise is intended

or should be inferred.

The National Institute for Literacy

Sandra Baxter, Director

Lynn Reddy, Deputy Director

2006