Scientifically based research and the Comprehensive School Reform (CSR)
Program

August 2002

> Comprehensive School Reform Program Office
>
> Office of Elementary and Secondary Education
>
> U.S. Department of Education

This document may be duplicated for dissemination.

[Introduction]{.underline}

The primary responsibility of schools undertaking comprehensive school
reform is creating programs that result in improved student achievement.
One of the most important tasks in this process is choosing highly
effective reform strategies, methods, and programs, those that are
grounded in scientifically based research. This guide is designed to
help school staffs and those with whom they work to increase their
understanding of what scientifically based research is, and to use that
understanding to assess the quality, relevance and usefulness of the
research they examine.

**[How has the reauthorization of the ESEA raised the standard for
evidence of effectiveness?]{.underline}**

Determining that reform strategies are effective before implementing
them is not only common sense, but is also one of the requirements of
the Comprehensive School Reform program. The 2001 reauthorization of the
Elementary and Secondary Education Act (ESEA) requires that only those
strategies and methods proven effective by the standard of
scientifically based research should be included in school reform
programs. As defined in the ESEA*,* "scientifically based research"
emphasizes experimental and quasi-experimental studies that are
systematic, empirical, well-designed, replicable, and have been accepted
by independent reviewers. (See Appendix 2 for complete definition.)

Prior to the 2001 reauthorization, CSR-supported programs were required
to use "*innovative* strategies and proven methods for student learning,
teaching, and school management . . . based on *reliable research* and
effective practices, and . . . replicated successfully in schools with
diverse characteristics." The new law requires comprehensive reform
programs that "employ *proven* strategies and *proven* methods for
student learning, teaching, and school management that are based on
*scientifically based research* and effective practices and have been
replicated successfully in schools."

In addition, the law now addresses the school's comprehensive program as
a whole. It requires of schools a "program that---has been found,
through scientifically based research, to significantly improve the
academic achievement of students participating in such program as
compared to students in schools who have not participated in such
program *or* has been found to have strong evidence that such program
will significantly improve the academic achievement of participating
children."

Clearly, the new law sets a higher bar for evaluating the research base
of both the comprehensive program and its individual strategies and
methods. Both must reflect "scientifically based research" when such
research exists. However, in some cases, research that meets these
criteria is minimal or non-existent. For instance, little or no
scientifically based research has been conducted that examines the
programmatic impact of the interaction of the eleven required components
of the CSR program. In this case, school leaders will need to rely on
the best available empirical evidence and some degree of professional
judgment in creating their programs. As the quantity, quality and
availability of empirical, randomized studies increases, schools will be
able to make a stronger connection between their design decisions and
the evidence of "what works."

[CSR components]{.underline}

To receive CSR funding, schools must implement a comprehensive school
reform program that:

1.  **Uses proven strategies and methods for learning, teaching, and
    school management based on scientifically based research and
    effective practices, and used successfully in multiple schools**

2.  Integrates a comprehensive design with aligned components focused on
    helping students meet standards and addressing needs identified in a
    school needs assessment

3.  Provides high quality, ongoing professional development

4.  Includes measurable goals and benchmarks for student academic
    achievement

5.  Has the support of staff within the school

6.  Provides support for all faculty and staff

7.  Provides for parental and community support and involvement

8.  Uses high quality, external technical support and assistance from an
    experienced provider

9.  Includes a plan for the annual evaluation of the implementation of
    the reform program and the outcomes achieved

10. Identifies other resources to support the reform effort

11. **Has been found through scientifically based research to
    significantly improve student academic achievement, or has shown
    strong evidence that it will.**

Components one and eleven identify a legislative (and common sense)
standard of scientifically based research. The first component requires
that schools apply that standard to the instructional[^1] strategies
contained within their comprehensive school reform programs. Component
eleven, on the other hand, requires either scientifically based research
or "strong evidence" of effectiveness; this standard applies to the
remaining components, those that make up the rest of the school's
comprehensive program. For these components, little or no scientifically
based research exists, so practitioners must rely on a less rigorous
standard. Findings from research studies that fulfill most, but not all
of the requirements of scientifically based research as defined in the
law demonstrate "strong evidence."

**[Standards for identifying effective approaches]{.underline}**

The reauthorized ESEA stresses that schools should review scientifically
based research to determine that the reform approaches they are
considering are likely to have **a positive impact on student
achievement**. The questions on the following pages and the accompanying
*Guidelines for Judging the Quality of a Study* can guide that review
process. Although information about "what works" ranges from folklore to
case studies to studies with randomized trials, this publication focuses
on identifying the "silver" and "gold" research standards, the highest
levels of scientifically based research as it is defined in the law.

**[Research consumer questions:]{.underline}**

Finding evidence of effects on student achievement is important, but in
order to gain a broad understanding of the potential usefulness of a
reform, consumers are advised to act as discriminating "research
consumers," examining the research from three perspectives:

1.  the **theoretical base** of the reform practice or program

2.  **implementation and replicability** information

3.  **evidence of effects** **on student achievement**

Some questions designed to help school staffs think about these issues
are provided on the next page. Additional resources are identified in
Appendix 1.

Question 1:

Is there a theoretical base for the practice or program being
considered?

+---------------------------+------------------------------------------+
| ### Questions a           | ###                                      |
| bout the theoretical base |  Judging quality of the theoretical base |
+---------------------------+------------------------------------------+
| -   What are the ideas    | 1\) Is there a clear, non-technical      |
|     behind this practice  | description of the central idea and      |
|     or program?           | goals of the practice or program?        |
|                           |                                          |
| -   What are its guiding  | 2\) Is there a clear description of the  |
|     principles?           | instructional activities that are        |
|                           | central to this program or practice?     |
| -   How does it work?     |                                          |
|                           | 3\) Is the practice clearly tied to an   |
| -   Why does it work?     | established learning theory, e.g. child  |
|                           | development or language acquisition?     |
+---------------------------+------------------------------------------+

# 

Question 2:

Is there evidence that this practice or program has been successfully
implemented and has produced positive outcomes in a variety of
situations? Has it been successful in a context similar to that of the
school considering this practice?

+---------------------------+------------------------------------------+
| ####                      | ##### Judging qua                        |
| # Questions about impleme | lity of implementation and replicability |
| ntation and replicability |                                          |
+---------------------------+------------------------------------------+
| -   Has this program or   | 1\) How many schools have used this      |
|     practice been widely  | practice or program?                     |
|     used?                 |                                          |
|                           | 2\) Did the schools using it fully       |
| ```{=html}                | implement the practice or program?       |
| <!-- -->                  |                                          |
| ```                       | 3\) In what settings has it been         |
| -   Where is this reform  | implemented?                             |
|     likely to work?       |                                          |
|                           | 4\) Has improved student achievement     |
| -   Under what            | been convincingly demonstrated in a      |
|     circumstances is it   | variety of settings?                     |
|     most effective?       |                                          |
+---------------------------+------------------------------------------+

#  

# Question 3: Is there evidence that this practice or program has a significant positive effect on student achievement?

+---------------------+-----------------------+------------------------+
|                     | ### Judging the qu    |                        |
|                     | ality of the evidence |                        |
+---------------------+-----------------------+------------------------+
| **Question about    | # Scientif            | **Developing toward    |
| individual          | ically based research | scientifically based   |
| practices or        |                       | research**             |
| programs**          |                       |                        |
+---------------------+-----------------------+------------------------+
| Is there evidence   | For each practice or  | For each practice or   |
| based on rigorous   | program identified:   | program identified:    |
| research showing    |                       |                        |
| that this practice  | 1\) Are there studies | 1\) Are there studies  |
| and/or program      | looking at the impact | looking at the impact  |
| improves student    | on students of that   | on students of that    |
| achievement?        | practice or program?  | practice or program?   |
|                     |                       |                        |
|                     | 2\) Are those studies | 2\) Are those studies  |
|                     | of high quality? (See | of reasonable quality? |
|                     | Guidelines for        | (See Guidelines for    |
|                     | Judging Quality of    | Judging Quality of     |
|                     | Study, pp. 7-10).     | Study, pp. 7-10).      |
|                     |                       |                        |
|                     | 3\) Are there at      | 3\) Are there at least |
|                     | least 5 high quality  | 5 high or reasonable   |
|                     | studies?              | quality studies?       |
|                     |                       |                        |
|                     | 4\) Do 4 of the 5     | 4\) Do 4 of the 5      |
|                     | high quality studies  | reasonable quality     |
|                     | show that the         | studies show that the  |
|                     | practice improves     | practice improves      |
|                     | student achievement?  | student achievement?   |
|                     |                       |                        |
|                     | 5\) If yes, are the   | 5\) If yes, are the    |
|                     | findings significant  | findings significant   |
|                     | in 3 of those 4       | in 3 of those 4        |
|                     | studies? (See p. 11). | studies? (See p. 11).  |
+---------------------+-----------------------+------------------------+
|                     | If the answer to all  | If the answer to all   |
|                     | of these questions is | of these questions is  |
|                     | YES, there is         | YES, there is "strong  |
|                     | scientifically based  | evidence" regarding    |
|                     | research regarding    | this practice or       |
|                     | this practice or      | program, even though   |
|                     | program.              | the research on which  |
|                     |                       | it is based did not    |
|                     | **The "Gold"          | meet all the           |
|                     | Standard**            | requirements of        |
|                     |                       | scientifically based   |
|                     |                       | research as defined in |
|                     |                       | the law.               |
|                     |                       |                        |
|                     |                       | #####                  |
|                     |                       |  The "Silver" Standard |
+---------------------+-----------------------+------------------------+

[Guidelines for judging the quality of a study]{.underline}

The criteria for judging the quality of research studies are contained
in the definition of scientifically based research in section 9101(37)
of the reauthorized Elementary and Secondary Education Act (ESEA).
Although there is no universally accepted standard, for the purposes of
this publication, a **high quality study** meets *all* of the criteria
described below. A **reasonable quality study** meets *all but one* of
the criteria. For example, a reasonable quality study might be
*systematic, empirical,* and use *rigorous data analysis* on *reliable
and valid data*, but might use a longitudinal study *design* that does
not involve random assignment to study groups or statistical controls on
background characteristics.

+-----+----------------------------------------------------------------+
| #   |                                                                |
| ### |                                                                |
| ##  |                                                                |
+-----+----------------------------------------------------------------+
| #   |                                                                |
| ### |                                                                |
| ##  |                                                                |
| Cri |                                                                |
| ter |                                                                |
| ia  |                                                                |
| 1:  |                                                                |
| Sys |                                                                |
| tem |                                                                |
| ati |                                                                |
| c a |                                                                |
| nd  |                                                                |
| emp |                                                                |
| iri |                                                                |
| cal |                                                                |
+-----+----------------------------------------------------------------+
| ### |                                                                |
| ### |                                                                |
|  Hi |                                                                |
| gh  |                                                                |
| qua |                                                                |
| lit |                                                                |
| y r |                                                                |
| ese |                                                                |
| arc |                                                                |
| h i |                                                                |
| s c |                                                                |
| arr |                                                                |
| ied |                                                                |
|  ou |                                                                |
| t i |                                                                |
| n a |                                                                |
|  ma |                                                                |
| nne |                                                                |
| r t |                                                                |
| hat |                                                                |
|  is |                                                                |
|  co |                                                                |
| nsi |                                                                |
| ste |                                                                |
| nt, |                                                                |
|  di |                                                                |
| sci |                                                                |
| pli |                                                                |
| ned |                                                                |
| , a |                                                                |
| nd  |                                                                |
| met |                                                                |
| hod |                                                                |
| ica |                                                                |
| l-- |                                                                |
| -no |                                                                |
| t s |                                                                |
| lop |                                                                |
| py  |                                                                |
| or  |                                                                |
| hap |                                                                |
| haz |                                                                |
| ard |                                                                |
| . S |                                                                |
| uch |                                                                |
|  re |                                                                |
| sea |                                                                |
| rch |                                                                |
|  sh |                                                                |
| ows |                                                                |
|  ev |                                                                |
| ide |                                                                |
| nce |                                                                |
|  of |                                                                |
|  ca |                                                                |
| ref |                                                                |
| ul  |                                                                |
| pla |                                                                |
| nni |                                                                |
| ng  |                                                                |
| and |                                                                |
|  ke |                                                                |
| en  |                                                                |
| att |                                                                |
| ent |                                                                |
| ion |                                                                |
|  to |                                                                |
|  de |                                                                |
| tai |                                                                |
| l.  |                                                                |
| Emp |                                                                |
| iri |                                                                |
| cal |                                                                |
|  re |                                                                |
| sea |                                                                |
| rch |                                                                |
|  is |                                                                |
|  gr |                                                                |
| oun |                                                                |
| ded |                                                                |
|  in |                                                                |
|  da |                                                                |
| ta  |                                                                |
| dra |                                                                |
| wn  |                                                                |
| fro |                                                                |
| m o |                                                                |
| bse |                                                                |
| rva |                                                                |
| tio |                                                                |
| n o |                                                                |
| r e |                                                                |
| xpe |                                                                |
| rim |                                                                |
| ent |                                                                |
| ; t |                                                                |
| he  |                                                                |
| cla |                                                                |
| ims |                                                                |
|  be |                                                                |
| ing |                                                                |
|  ma |                                                                |
| de  |                                                                |
| are |                                                                |
|  su |                                                                |
| ppo |                                                                |
| rte |                                                                |
| d b |                                                                |
| y m |                                                                |
| eas |                                                                |
| ura |                                                                |
| ble |                                                                |
|  ev |                                                                |
| ide |                                                                |
| nce |                                                                |
| , n |                                                                |
| ot  |                                                                |
| opi |                                                                |
| nio |                                                                |
| n o |                                                                |
| r s |                                                                |
| pec |                                                                |
| ula |                                                                |
| tio |                                                                |
| n.  |                                                                |
+-----+----------------------------------------------------------------+
| ##  |                                                                |
| ### |                                                                |
| # W |                                                                |
| hen |                                                                |
|  ev |                                                                |
| alu |                                                                |
| ati |                                                                |
| ng  |                                                                |
| res |                                                                |
| ear |                                                                |
| ch, |                                                                |
|  co |                                                                |
| nsi |                                                                |
| der |                                                                |
|  th |                                                                |
| e f |                                                                |
| oll |                                                                |
| owi |                                                                |
| ng: |                                                                |
+-----+----------------------------------------------------------------+
|     | ###### Does the research have a sound theoretical foundation?  |
|     |                                                                |
|     | > (See Research Consumer Questions, pp. 4 and 5.)              |
+-----+----------------------------------------------------------------+
|     | #####                                                          |
|     | # Were the data obtained using observation or experimentation? |
+-----+----------------------------------------------------------------+
|     | -   Were the data collected from *all* appropriate groups of   |
|     |     respondents and not just from certain groups? For example, |
|     |     does a school reform program that claims to benefit all    |
|     |     students include special education students in its         |
|     |     research? If the research uses test results for a given    |
|     |     school, did all of the students in the school take the     |
|     |     test?                                                      |
+-----+----------------------------------------------------------------+
|     | ###### Were the data observed or collecte                      |
|     | d from multiple subjects (teachers, students, schools, etc.)?  |
+-----+----------------------------------------------------------------+
|     | ####                                                           |
|     | ## Are the research findings supported by measurable evidence? |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| ##  |                                                                |
| # C |                                                                |
| rit |                                                                |
| eri |                                                                |
| a 2 |                                                                |
| : R |                                                                |
| igo |                                                                |
| rou |                                                                |
| s d |                                                                |
| ata |                                                                |
|  an |                                                                |
| aly |                                                                |
| sis |                                                                |
+-----+----------------------------------------------------------------+
| E   |                                                                |
| ven |                                                                |
| the |                                                                |
| h   |                                                                |
| igh |                                                                |
| est |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| d   |                                                                |
| ata |                                                                |
| are |                                                                |
| of  |                                                                |
| lit |                                                                |
| tle |                                                                |
| va  |                                                                |
| lue |                                                                |
| unl |                                                                |
| ess |                                                                |
| an  |                                                                |
| aly |                                                                |
| zed |                                                                |
| tho |                                                                |
| ugh |                                                                |
| tfu |                                                                |
| lly |                                                                |
| and |                                                                |
| c   |                                                                |
| are |                                                                |
| ful |                                                                |
| ly. |                                                                |
| The |                                                                |
| d   |                                                                |
| efi |                                                                |
| nit |                                                                |
| ion |                                                                |
| of  |                                                                |
| sc  |                                                                |
| ien |                                                                |
| tif |                                                                |
| ica |                                                                |
| lly |                                                                |
| ba  |                                                                |
| sed |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| re  |                                                                |
| qui |                                                                |
| res |                                                                |
| t   |                                                                |
| hat |                                                                |
| d   |                                                                |
| ata |                                                                |
| col |                                                                |
| lec |                                                                |
| ted |                                                                |
| m   |                                                                |
| ust |                                                                |
| be  |                                                                |
| an  |                                                                |
| aly |                                                                |
| zed |                                                                |
| us  |                                                                |
| ing |                                                                |
| m   |                                                                |
| eth |                                                                |
| ods |                                                                |
| t   |                                                                |
| hat |                                                                |
| are |                                                                |
| ap  |                                                                |
| pro |                                                                |
| pri |                                                                |
| ate |                                                                |
| for |                                                                |
| the |                                                                |
| ta  |                                                                |
| sk, |                                                                |
| and |                                                                |
| ad  |                                                                |
| equ |                                                                |
| ate |                                                                |
| to  |                                                                |
| t   |                                                                |
| est |                                                                |
| the |                                                                |
| sta |                                                                |
| ted |                                                                |
| h   |                                                                |
| ypo |                                                                |
| the |                                                                |
| ses |                                                                |
| and |                                                                |
| j   |                                                                |
| ust |                                                                |
| ify |                                                                |
| the |                                                                |
| g   |                                                                |
| ene |                                                                |
| ral |                                                                |
| co  |                                                                |
| ncl |                                                                |
| usi |                                                                |
| ons |                                                                |
| dra |                                                                |
| wn. |                                                                |
| F   |                                                                |
| ail |                                                                |
| ure |                                                                |
| to  |                                                                |
| ap  |                                                                |
| ply |                                                                |
| ap  |                                                                |
| pro |                                                                |
| pri |                                                                |
| ate |                                                                |
| m   |                                                                |
| eth |                                                                |
| ods |                                                                |
| co  |                                                                |
| uld |                                                                |
| p   |                                                                |
| rod |                                                                |
| uce |                                                                |
| i   |                                                                |
| nac |                                                                |
| cur |                                                                |
| ate |                                                                |
| or  |                                                                |
| m   |                                                                |
| isl |                                                                |
| ead |                                                                |
| ing |                                                                |
| fin |                                                                |
| din |                                                                |
| gs. |                                                                |
+-----+----------------------------------------------------------------+
| S   |                                                                |
| ome |                                                                |
| key |                                                                |
| que |                                                                |
| sti |                                                                |
| ons |                                                                |
| to  |                                                                |
| co  |                                                                |
| nsi |                                                                |
| der |                                                                |
| ab  |                                                                |
| out |                                                                |
| the |                                                                |
| d   |                                                                |
| ata |                                                                |
| an  |                                                                |
| aly |                                                                |
| sis |                                                                |
| i   |                                                                |
| ncl |                                                                |
| ude |                                                                |
| the |                                                                |
| f   |                                                                |
| oll |                                                                |
| owi |                                                                |
| ng: |                                                                |
+-----+----------------------------------------------------------------+
|     | -   Does the research test the stated hypothesis, and do the   |
|     |     findings justify the general conclusions drawn?            |
+-----+----------------------------------------------------------------+
|     | -   Does the research report the sample size and the           |
|     |     statistical procedures used?                               |
+-----+----------------------------------------------------------------+
|     | -   Do the researchers analyze the data in a manner            |
|     |     appropriate to the research question of interest? Are the  |
|     |     statistical procedures used adequate for answering the     |
|     |     research question?                                         |
+-----+----------------------------------------------------------------+
|     | -   Do the analysis methods correspond to the structure of the |
|     |     data? Does the analysis account for the complexities of    |
|     |     the data? for missing data? for unique groupings? for      |
|     |     changes in the data over time?                             |
|     |                                                                |
|     | > For example, in school research studies that unfold over     |
|     | > time, subjects may drop out of the study (for example, by    |
|     | > moving out of a study school). Adequate data analyses        |
|     | > address these issues.                                        |
+-----+----------------------------------------------------------------+

###### 

+-----+----------------------------------------------------------------+
| ##  |                                                                |
| # C |                                                                |
| rit |                                                                |
| eri |                                                                |
| a 3 |                                                                |
| : R |                                                                |
| eli |                                                                |
| abl |                                                                |
| e a |                                                                |
| nd  |                                                                |
| val |                                                                |
| id  |                                                                |
| dat |                                                                |
| a c |                                                                |
| oll |                                                                |
| ect |                                                                |
| ion |                                                                |
+-----+----------------------------------------------------------------+
| H   |                                                                |
| igh |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| d   |                                                                |
| ata |                                                                |
| p   |                                                                |
| rod |                                                                |
| uce |                                                                |
| ac  |                                                                |
| cur |                                                                |
| ate |                                                                |
| and |                                                                |
| cr  |                                                                |
| edi |                                                                |
| ble |                                                                |
| fin |                                                                |
| din |                                                                |
| gs. |                                                                |
| Sc  |                                                                |
| ien |                                                                |
| tif |                                                                |
| ica |                                                                |
| lly |                                                                |
| ba  |                                                                |
| sed |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| rel |                                                                |
| ies |                                                                |
| on  |                                                                |
| mea |                                                                |
| sur |                                                                |
| eme |                                                                |
| nts |                                                                |
| or  |                                                                |
| o   |                                                                |
| bse |                                                                |
| rva |                                                                |
| tio |                                                                |
| nal |                                                                |
| m   |                                                                |
| eth |                                                                |
| ods |                                                                |
| t   |                                                                |
| hat |                                                                |
| p   |                                                                |
| rov |                                                                |
| ide |                                                                |
| re  |                                                                |
| lia |                                                                |
| ble |                                                                |
| and |                                                                |
| va  |                                                                |
| lid |                                                                |
| d   |                                                                |
| ata |                                                                |
| acr |                                                                |
| oss |                                                                |
| e   |                                                                |
| val |                                                                |
| uat |                                                                |
| ors |                                                                |
| and |                                                                |
| obs |                                                                |
| erv |                                                                |
| ers |                                                                |
| and |                                                                |
| acr |                                                                |
| oss |                                                                |
| mu  |                                                                |
| lti |                                                                |
| ple |                                                                |
| mea |                                                                |
| sur |                                                                |
| eme |                                                                |
| nts |                                                                |
| and |                                                                |
| o   |                                                                |
| bse |                                                                |
| rva |                                                                |
| tio |                                                                |
| ns. |                                                                |
| Re  |                                                                |
| lia |                                                                |
| bil |                                                                |
| ity |                                                                |
| i   |                                                                |
| mpl |                                                                |
| ies |                                                                |
| t   |                                                                |
| hat |                                                                |
| re  |                                                                |
| pea |                                                                |
| ted |                                                                |
| mea |                                                                |
| sur |                                                                |
| eme |                                                                |
| nts |                                                                |
| on  |                                                                |
| su  |                                                                |
| bje |                                                                |
| cts |                                                                |
| ta  |                                                                |
| ken |                                                                |
| un  |                                                                |
| der |                                                                |
| s   |                                                                |
| imi |                                                                |
| lar |                                                                |
| c   |                                                                |
| irc |                                                                |
| ums |                                                                |
| tan |                                                                |
| ces |                                                                |
| or  |                                                                |
| o   |                                                                |
| ver |                                                                |
| t   |                                                                |
| ime |                                                                |
| w   |                                                                |
| ill |                                                                |
| p   |                                                                |
| rod |                                                                |
| uce |                                                                |
| s   |                                                                |
| imi |                                                                |
| lar |                                                                |
| re  |                                                                |
| sul |                                                                |
| ts. |                                                                |
| If  |                                                                |
| un  |                                                                |
| rel |                                                                |
| iab |                                                                |
| le, |                                                                |
| the |                                                                |
| d   |                                                                |
| ata |                                                                |
| may |                                                                |
| hin |                                                                |
| der |                                                                |
| the |                                                                |
| r   |                                                                |
| ese |                                                                |
| arc |                                                                |
| her |                                                                |
| \'s |                                                                |
| a   |                                                                |
| bil |                                                                |
| ity |                                                                |
| to  |                                                                |
| d   |                                                                |
| isc |                                                                |
| ern |                                                                |
| r   |                                                                |
| eal |                                                                |
| di  |                                                                |
| ffe |                                                                |
| ren |                                                                |
| ces |                                                                |
| am  |                                                                |
| ong |                                                                |
| su  |                                                                |
| bje |                                                                |
| cts |                                                                |
| or  |                                                                |
| pro |                                                                |
| gra |                                                                |
| ms. |                                                                |
| To  |                                                                |
| be  |                                                                |
| c   |                                                                |
| ons |                                                                |
| ide |                                                                |
| red |                                                                |
| val |                                                                |
| id, |                                                                |
| the |                                                                |
| d   |                                                                |
| ata |                                                                |
| col |                                                                |
| lec |                                                                |
| ted |                                                                |
| m   |                                                                |
| ust |                                                                |
| m   |                                                                |
| eas |                                                                |
| ure |                                                                |
| the |                                                                |
| ou  |                                                                |
| tco |                                                                |
| mes |                                                                |
| t   |                                                                |
| hey |                                                                |
| w   |                                                                |
| ere |                                                                |
| de  |                                                                |
| sig |                                                                |
| ned |                                                                |
| to  |                                                                |
| me  |                                                                |
| asu |                                                                |
| re, |                                                                |
| (e  |                                                                |
| .g. |                                                                |
| t   |                                                                |
| hat |                                                                |
| s   |                                                                |
| tud |                                                                |
| ent |                                                                |
| m   |                                                                |
| ath |                                                                |
| kno |                                                                |
| wle |                                                                |
| dge |                                                                |
| is  |                                                                |
| w   |                                                                |
| hat |                                                                |
| is  |                                                                |
| be  |                                                                |
| ing |                                                                |
| mea |                                                                |
| sur |                                                                |
| ed, |                                                                |
| not |                                                                |
| stu |                                                                |
| den |                                                                |
| ts' |                                                                |
| a   |                                                                |
| bil |                                                                |
| ity |                                                                |
| to  |                                                                |
| gu  |                                                                |
| ess |                                                                |
| t   |                                                                |
| est |                                                                |
| ans |                                                                |
| wer |                                                                |
| s). |                                                                |
| Th  |                                                                |
| ere |                                                                |
| m   |                                                                |
| ust |                                                                |
| be  |                                                                |
| a   |                                                                |
| ma  |                                                                |
| tch |                                                                |
| b   |                                                                |
| etw |                                                                |
| een |                                                                |
| the |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| qu  |                                                                |
| est |                                                                |
| ion |                                                                |
| and |                                                                |
| the |                                                                |
| ob  |                                                                |
| ser |                                                                |
| ved |                                                                |
| be  |                                                                |
| hav |                                                                |
| ior |                                                                |
| on  |                                                                |
| wh  |                                                                |
| ich |                                                                |
| the |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| fi  |                                                                |
| ndi |                                                                |
| ngs |                                                                |
| are |                                                                |
| bas |                                                                |
| ed. |                                                                |
+-----+----------------------------------------------------------------+
| Que |                                                                |
| sti |                                                                |
| ons |                                                                |
| ab  |                                                                |
| out |                                                                |
| the |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| of  |                                                                |
| d   |                                                                |
| ata |                                                                |
| c   |                                                                |
| oll |                                                                |
| ect |                                                                |
| ion |                                                                |
| i   |                                                                |
| ncl |                                                                |
| ude |                                                                |
| the |                                                                |
| f   |                                                                |
| oll |                                                                |
| owi |                                                                |
| ng: |                                                                |
+-----+----------------------------------------------------------------+
|     | -   Was data collection conducted professionally and           |
|     |     consistently? For example, was there some system to ensure |
|     |     that different data collectors had the same focus and      |
|     |     attention to detail ?                                      |
|     |                                                                |
|     | > (E.g. training before data collection or interrater          |
|     | > reliability tests)                                           |
+-----+----------------------------------------------------------------+
|     | -   Were research biases minimized? Developers of reform       |
|     |     models supply a natural example: was the evaluation of the |
|     |     reform model conducted by the model developers or by a     |
|     |     third-party, independent evaluator?                        |
+-----+----------------------------------------------------------------+
|     | -   Does the study look at the appropriate information to      |
|     |     address its questions? Are the measures valid? That is, do |
|     |     the measures discussed and analyzed correspond to the      |
|     |     concepts being studied?                                    |
+-----+----------------------------------------------------------------+
|     | -   Are the data reliable? Did repeated measurements on        |
|     |     subjects taken under similar circumstances produce similar |
|     |     results? Do the data represent counts of actions, records, |
|     |     responses, etc., that directly reflect what the practice   |
|     |     or program is supposed to be doing and affecting?          |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| ##  |                                                                |
| # C |                                                                |
| rit |                                                                |
| eri |                                                                |
| a 4 |                                                                |
| : S |                                                                |
| tro |                                                                |
| ng  |                                                                |
| res |                                                                |
| ear |                                                                |
| ch  |                                                                |
| des |                                                                |
| ign |                                                                |
+-----+----------------------------------------------------------------+
| S   |                                                                |
| tud |                                                                |
| ies |                                                                |
| m   |                                                                |
| ust |                                                                |
| be  |                                                                |
| de  |                                                                |
| sig |                                                                |
| ned |                                                                |
| to  |                                                                |
| op  |                                                                |
| tim |                                                                |
| ize |                                                                |
| the |                                                                |
| inv |                                                                |
| est |                                                                |
| iga |                                                                |
| tor |                                                                |
| \'s |                                                                |
| a   |                                                                |
| bil |                                                                |
| ity |                                                                |
| to  |                                                                |
| ans |                                                                |
| wer |                                                                |
| the |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| qu  |                                                                |
| est |                                                                |
| ion |                                                                |
| or  |                                                                |
| hy  |                                                                |
| pot |                                                                |
| hes |                                                                |
| is. |                                                                |
+-----+----------------------------------------------------------------+
| The |                                                                |
| fol |                                                                |
| low |                                                                |
| ing |                                                                |
| que |                                                                |
| sti |                                                                |
| ons |                                                                |
| are |                                                                |
| re  |                                                                |
| lev |                                                                |
| ant |                                                                |
| to  |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| d   |                                                                |
| esi |                                                                |
| gn: |                                                                |
+-----+----------------------------------------------------------------+
|     | -   Does the study follow an experimental or                   |
|     |     quasi-experimental design? That is, are the subjects in    |
|     |     the study divided randomly into at least two groups, with  |
|     |     at least one group using the practice or program of        |
|     |     interest and one group not using it?                       |
+-----+----------------------------------------------------------------+
|     | -   Does the study design contain appropriate controls in      |
|     |     order to be able to evaluate the effects of the condition  |
|     |     of interest? Were the subjects of the research randomly    |
|     |     assigned, or were there other within-condition or          |
|     |     across-condition controls as part of the design? (Random   |
|     |     assignment of students is a way to ensure that it is the   |
|     |     practice or program and not particular student             |
|     |     characteristics that are producing the measured results.)  |
+-----+----------------------------------------------------------------+
|     | -   If subjects are not divided into the groups randomly, are  |
|     |     the groups selected to ensure that subjects share similar  |
|     |     background characteristics such as economic well-being or  |
|     |     previous academic achievement? If not, does the study      |
|     |     explain how statistical controls were used to account for  |
|     |     these differences in background characteristics of the     |
|     |     students in the study? (See criteria 2.)                   |
+-----+----------------------------------------------------------------+
|     | -   Did the research minimize alternative explanations for     |
|     |     observed effects?                                          |
+-----+----------------------------------------------------------------+
|     | -   Does the study make a determination that the practice or   |
|     |     program was used appropriately and fully as intended?      |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| ##  |                                                                |
| # C |                                                                |
| rit |                                                                |
| eri |                                                                |
| a 5 |                                                                |
| : D |                                                                |
| eta |                                                                |
| ile |                                                                |
| d r |                                                                |
| esu |                                                                |
| lts |                                                                |
|  th |                                                                |
| at  |                                                                |
| all |                                                                |
| ow  |                                                                |
| for |                                                                |
|  re |                                                                |
| pli |                                                                |
| cat |                                                                |
| ion |                                                                |
+-----+----------------------------------------------------------------+
| The |                                                                |
| r   |                                                                |
| esu |                                                                |
| lts |                                                                |
| of  |                                                                |
| h   |                                                                |
| igh |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| s   |                                                                |
| tud |                                                                |
| ies |                                                                |
| are |                                                                |
| pre |                                                                |
| sen |                                                                |
| ted |                                                                |
| in  |                                                                |
| s   |                                                                |
| uff |                                                                |
| ici |                                                                |
| ent |                                                                |
| det |                                                                |
| ail |                                                                |
| to  |                                                                |
| al  |                                                                |
| low |                                                                |
| for |                                                                |
| th  |                                                                |
| eir |                                                                |
| rep |                                                                |
| lic |                                                                |
| ati |                                                                |
| on, |                                                                |
| or  |                                                                |
| to  |                                                                |
| at  |                                                                |
| le  |                                                                |
| ast |                                                                |
| p   |                                                                |
| rov |                                                                |
| ide |                                                                |
| o   |                                                                |
| ppo |                                                                |
| rtu |                                                                |
| nit |                                                                |
| ies |                                                                |
| to  |                                                                |
| bu  |                                                                |
| ild |                                                                |
| sy  |                                                                |
| ste |                                                                |
| mat |                                                                |
| ica |                                                                |
| lly |                                                                |
| on  |                                                                |
| th  |                                                                |
| eir |                                                                |
| fin |                                                                |
| din |                                                                |
| gs. |                                                                |
| To  |                                                                |
| in  |                                                                |
| cre |                                                                |
| ase |                                                                |
| th  |                                                                |
| eir |                                                                |
| u   |                                                                |
| sef |                                                                |
| uln |                                                                |
| ess |                                                                |
| to  |                                                                |
| pr  |                                                                |
| act |                                                                |
| iti |                                                                |
| one |                                                                |
| rs, |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| fi  |                                                                |
| ndi |                                                                |
| ngs |                                                                |
| m   |                                                                |
| ust |                                                                |
| be  |                                                                |
| re  |                                                                |
| por |                                                                |
| ted |                                                                |
| in  |                                                                |
| a   |                                                                |
| way |                                                                |
| t   |                                                                |
| hat |                                                                |
| ma  |                                                                |
| kes |                                                                |
| t   |                                                                |
| hem |                                                                |
| eas |                                                                |
| ily |                                                                |
| a   |                                                                |
| cce |                                                                |
| ssi |                                                                |
| ble |                                                                |
| and |                                                                |
| un  |                                                                |
| der |                                                                |
| sto |                                                                |
| od. |                                                                |
| The |                                                                |
| in  |                                                                |
| for |                                                                |
| med |                                                                |
| lay |                                                                |
| rea |                                                                |
| der |                                                                |
| sho |                                                                |
| uld |                                                                |
| be  |                                                                |
| a   |                                                                |
| ble |                                                                |
| to  |                                                                |
| u   |                                                                |
| nde |                                                                |
| rst |                                                                |
| and |                                                                |
| the |                                                                |
| s   |                                                                |
| tud |                                                                |
| y's |                                                                |
| d   |                                                                |
| esi |                                                                |
| gn, |                                                                |
| me  |                                                                |
| tho |                                                                |
| ds, |                                                                |
| and |                                                                |
| fin |                                                                |
| din |                                                                |
| gs. |                                                                |
+-----+----------------------------------------------------------------+
| W   |                                                                |
| hen |                                                                |
| e   |                                                                |
| val |                                                                |
| uat |                                                                |
| ing |                                                                |
| the |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| of  |                                                                |
| re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| r   |                                                                |
| epo |                                                                |
| rti |                                                                |
| ng, |                                                                |
| co  |                                                                |
| nsi |                                                                |
| der |                                                                |
| the |                                                                |
| f   |                                                                |
| oll |                                                                |
| owi |                                                                |
| ng: |                                                                |
+-----+----------------------------------------------------------------+
|     | -   Are the findings clearly described and reported, free from |
|     |     technical terms and jargon?                                |
+-----+----------------------------------------------------------------+
|     | -   Are the description of the design and the results of the   |
|     |     research sufficiently detailed so that replication of the  |
|     |     design is possible? For example, do researchers report the |
|     |     sample size (number of people or schools involved) and the |
|     |     statistical procedures used?                               |
+-----+----------------------------------------------------------------+
|     | -   Are the findings presented fairly and objectively?         |
+-----+----------------------------------------------------------------+
|     | -   Are technical aspects of the study, such as statistical    |
|     |     significance or confidence intervals made available and    |
|     |     explained? Do the reports supply any supporting technical  |
|     |     materials, perhaps in appendices?                          |
+-----+----------------------------------------------------------------+
|     | -   Is the presentation balanced? That is, are shortcomings    |
|     |     reported as well as strengths? Were possible explanations  |
|     |     provided for findings that ran counter to the researcher's |
|     |     expectations?                                              |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| #   |                                                                |
| ### |                                                                |
| ##  |                                                                |
| Cri |                                                                |
| ter |                                                                |
| ia  |                                                                |
| 6:  |                                                                |
| Exp |                                                                |
| ert |                                                                |
|  Sc |                                                                |
| rut |                                                                |
| iny |                                                                |
+-----+----------------------------------------------------------------+
| A   |                                                                |
| str |                                                                |
| ong |                                                                |
| st  |                                                                |
| udy |                                                                |
| sho |                                                                |
| uld |                                                                |
| be  |                                                                |
| a   |                                                                |
| ble |                                                                |
| to  |                                                                |
| m   |                                                                |
| eet |                                                                |
| cri |                                                                |
| tic |                                                                |
| ism |                                                                |
| by  |                                                                |
| ind |                                                                |
| epe |                                                                |
| nde |                                                                |
| nt, |                                                                |
| exp |                                                                |
| ert |                                                                |
| r   |                                                                |
| evi |                                                                |
| ewe |                                                                |
| rs. |                                                                |
| P   |                                                                |
| eer |                                                                |
| r   |                                                                |
| evi |                                                                |
| ewe |                                                                |
| rs, |                                                                |
| eit |                                                                |
| her |                                                                |
| f   |                                                                |
| rom |                                                                |
| s   |                                                                |
| cie |                                                                |
| nti |                                                                |
| fic |                                                                |
| jo  |                                                                |
| urn |                                                                |
| als |                                                                |
| or  |                                                                |
| f   |                                                                |
| rom |                                                                |
| an  |                                                                |
| in  |                                                                |
| dep |                                                                |
| end |                                                                |
| ent |                                                                |
| pa  |                                                                |
| nel |                                                                |
| of  |                                                                |
| e   |                                                                |
| xpe |                                                                |
| rts |                                                                |
| in  |                                                                |
| a   |                                                                |
| gi  |                                                                |
| ven |                                                                |
| fie |                                                                |
| ld, |                                                                |
| p   |                                                                |
| rov |                                                                |
| ide |                                                                |
| q   |                                                                |
| ual |                                                                |
| ity |                                                                |
| c   |                                                                |
| ont |                                                                |
| rol |                                                                |
| in  |                                                                |
| the |                                                                |
| f   |                                                                |
| orm |                                                                |
| of  |                                                                |
| a   |                                                                |
| rig |                                                                |
| oro |                                                                |
| us, |                                                                |
| o   |                                                                |
| bje |                                                                |
| cti |                                                                |
| ve, |                                                                |
| and |                                                                |
| s   |                                                                |
| cie |                                                                |
| nti |                                                                |
| fic |                                                                |
| rev |                                                                |
| iew |                                                                |
| of  |                                                                |
| res |                                                                |
| ear |                                                                |
| ch. |                                                                |
| Re  |                                                                |
| sea |                                                                |
| rch |                                                                |
| con |                                                                |
| sum |                                                                |
| ers |                                                                |
| can |                                                                |
| pl  |                                                                |
| ace |                                                                |
| m   |                                                                |
| ore |                                                                |
| c   |                                                                |
| onf |                                                                |
| ide |                                                                |
| nce |                                                                |
| in  |                                                                |
| fi  |                                                                |
| ndi |                                                                |
| ngs |                                                                |
| t   |                                                                |
| hat |                                                                |
| h   |                                                                |
| ave |                                                                |
| b   |                                                                |
| een |                                                                |
| sub |                                                                |
| jec |                                                                |
| ted |                                                                |
| to  |                                                                |
| exp |                                                                |
| ert |                                                                |
| r   |                                                                |
| evi |                                                                |
| ew. |                                                                |
+-----+----------------------------------------------------------------+
| W   |                                                                |
| hen |                                                                |
| e   |                                                                |
| val |                                                                |
| uat |                                                                |
| ing |                                                                |
| res |                                                                |
| ear |                                                                |
| ch, |                                                                |
| co  |                                                                |
| nsi |                                                                |
| der |                                                                |
| the |                                                                |
| f   |                                                                |
| oll |                                                                |
| owi |                                                                |
| ng: |                                                                |
+-----+----------------------------------------------------------------+
|     | -   Has the research been accepted and published by a          |
|     |     competitive, peer-reviewed scientific journal, or was it   |
|     |     reported only in media such as newspapers, magazines, or   |
|     |     trade journals?                                            |
+-----+----------------------------------------------------------------+
|     | -   If the work was not published, is there evidence that it   |
|     |     was reviewed by independent experts and subjected to       |
|     |     external verification? If so, did the reviewers approve    |
|     |     the study methodology and interpretation of the findings?  |
+-----+----------------------------------------------------------------+

#  

# [Significance of effects:]{.underline}

Before reformers make a final decision about the usefulness of available
research findings, they must determine their significance. Even
high-quality research studies can produce findings that are not
statistically or practically significant. **Significance** is a
statistical term that helps readers to understand the likelihood that
the findings of a study were the result of the designed intervention and
would not be observed independent of that intervention. For
practitioners, two standards of significance apply, *statistical*
significance and *educational,* or *practical*, significance.

-   For findings to be considered *statistically significant*,
    researchers should indicate in the study that the findings are
    "significant" at the .01 level (American Institutes for Research,
    1999; Slavin & Fashola, 1998). In the social science research
    community, a .01 level of significance is highly regarded because it
    means that the positive results of the study would be the result of
    some unrelated cause only 1 time in 100 repetitions of the study's
    procedures. Thus, a .01 level of significance indicates that the
    program's activities have led to the achievement gains reported.

-   For the findings to be considered *educationally, or practically,
    significant*, the effect on student achievement should be large
    enough to be of practical value. For example, gains on a
    standardized test should be 10 percentile points to be considered
    educationally significant (American Institutes for Research, 1999).

A study may have a significant statistical result in that the
achievement gains reported in the study are very clearly the result of
the reform program. However, the actual gains in achievement may be
modest in size; when this is the case, the study may be said to have
limited practical significance. In this context, school community
members need to decide if the cost of implementing the program is
outweighed by the size of the student gains that can be achieved.

**[Putting scientifically based research in the school reform
context:]{.underline}**

The comprehensive school reform program usually begins with school staff
gathering and analyzing data about their school. Once this process is
completed, they identify and prioritize their needs, establish goals,
and then begin to research strategies and methods that will help them
achieve those goals. The following step-by-step process illustrates the
place that scientifically based research occupies in the context of
school reform.

  ----------- -----------------------------------------------------------
  **Step      Schools collect data that gives a complete, accurate
  One:**      picture of the current state of the school.

  **Step      School community analyzes data to determine and prioritize
  Two:**      needs.

  **Step      After prioritizing needs, schools write goal statements to
  Three:**    address prioritized needs.

  **Step      Schools develop potential strategies to achieve stated
  Four:**     goals.

  **Step      Schools begin topical literature review, identifying
  Five:**     research base of potential strategies. (See Appendix 1 for
              sources of research abstracts.)

  **Step      Schools use abstracts to determine if the research
  Six:**      addresses (1) the theoretical basis of a program or
              practice, and/or (2) issues of implementation,
              replicability, and relevance, or (3) evidence of the
              effects of that program or practice on student achievement.
              Not all research addresses all three topics, but more than
              one may be addressed in an individual research study.

  **Step      If an examination of the abstract indicates that a study is
  Seven:**    relevant to the identified needs of the school, the study
              should be examined in more detail. Schools can use the
              *Standards for Identifying Effective Approaches* (pp. 4-6)
              to make a determination about the practice or program of
              interest. *Guidelines for Judging the Quality of a Study*
              (pp.7-10), and *Understanding Significance*, (p.11) will
              provide additional help.

  **Step      Schools determine if the examined research is of high
  Eight**:    quality, replicable, and relevant. The examined research
              might meet the criteria of scientifically based research,
              the "strong evidence" criteria, or be at some other point
              along the spectrum of "evidence based." In some cases there
              will be no, or little, high quality evidence that this
              practice, program or model will increase positive outcomes
              for students.
  ----------- -----------------------------------------------------------

# 

The Comprehensive School Reform program is built on the premise that
schools can improve when they create a unified, coherent approach to
reform, and when they implement practices and programs that are grounded
in scientifically based research. Assessing and evaluating the research
base of each method and strategy included in the school's comprehensive
design for improvement will improve the quality, implementation, and
outcomes of the reforms funded under this program.

**References**

American Federation of Teachers. (1997). *Raising student achievement: A
resource guide for redesigning low-performing schools*. (AFT Item Number
3780). Washington, DC: AFT.

American Institutes for Research. (1999). *An Educators' Guide to
Schoolwide Reform*. Arlington, VA: Educational Research Service.

Boruch, R.F., DeMoya, D., and Snyder, B. (in press). The importance of
randomized field trials in education and related areas. In F. Mosteller
and R. Boruch (Eds.), *Evidence matters: Randomized trials in education
research.* Washington, DC: Brookings Institution Press.

Cook, T.D.. A critical appraisal of the case against using experiments
to assess school (or community) effects.. Professor of Sociology,
Northwestern University.

Cook, T.D. (March 1999). Considering the major arguments against random
assignment: An analysis of the intellectual culture surrounding
evaluation in American schools of education. Northwestern University.
Paper presented at the Harvard faculty Seminar on Experiments in
Education.

Cook, T.D., Habib, F., Phillips, M., Settersten, R.A., Shagle, S.C., and
Degirmencioglu, S.M. (1999). Comer's School Development Program in
Prince George's County, Maryland: A theory-based evaluation. *American
Education Research Journal, 36 (3)*, pp. 543-597.

Cook, T.D., Hunt, H.D., and Murphy, R.F. Comer's School Development
Program in Chicago: A theory-based evaluation. Institute for Policy
Research, Northwestern University.

Cook, T.D., and Payne, M.R. Objecting to the Objections to Using Random
Assignment in Educational Research.

Cook, T.D. (Fall 2001). Why education researchers reject randomized
experiments. Sciencephobia, \[Online), Available:
http://www.educationnext.org/.

Elementary and Secondary Elementary Act of 1965 (20 U.S.C. 6301 et seq.)
as amended by the No Child Left Behind Act of 2001.

Grissmer, D. (Ed.). (1999). Class size: Issues and new findings (Special
issue). *Educational Evaluation and Policy Analysis*, *21 (2).*

Kaestle, C.F. (1993). The awful reputation of education research.
*Educational Researcher, 22 (1)*, pp. 26-31.

Miller, D.W. (2001, July 13). The problem with studying vouchers. *The
Chronicle of Higher Education,* pp. A14-A15.

National Clearinghouse of Comprehensive School Reform. (undated)
<http://www.csrclearinghouse.org/>

National Education Association. (2002). *Charting new frontiers:
Creating high performing schools*. Washington, DC: NEA.

National Research Council. (2001). *Scientific inquiry in education*.
Committee on Scientific Principles for Education Research. Shavelson,
R.J. and Towne, L. (eds.) Center for Education. Division of Behavioral
and Social Sciences and Education. Washington, DC: National Academy
Press.

Northwest Regional Educational Laboratory. (2001). *The catalogue of
school reform models*.. Portland, OR: NWREL.

Peterson, P.E., Howell, W.G., and Greene, J.P. (1999). An evaluation of
the Cleveland voucher program after two years. Program on Education
Policy and Governance Research Paper. Kennedy School of Government,
Harvard University. \[Online\]. Available:
<http://www.hks.harvard.edu/pepg/>. \[2001, August 21\]

Putnam, R.D. (1995). Bowling alone: America\'s declining social capital.
*Journal of Democracy, 6,* pp. 65-78.

Ritter, G.W., and Boruch, R.F. (1999). The political and institutional
origins of a randomized controlled trial on elementary school class
size\" Tennessee\'s Project STAR. EEPA, *21 (2),* pp. 111-125.

Slavin, R.E., and Fashola, O.S. (1998). *Show me the evidence! Proven
and promising programs for America's schools*. Thousand Oaks, CA: Corwin
Press.

Southwest Educational Development Laboratory (undated). *CSRD Awards
Database*. (<http://csrprogram.ed.gov/>)

Sroufe, G.E. (1997). Improving the \"awful reputation\" of educational
research. *Educational Researcher, 26 (7)*, pp. 26-28.

Willinsky, J. (2001). The strategic education research program and the
public value of research. *Educational Researcher, 30 (1)*, pp. 5-14.

Appendix 1

Resources for Schools

+-----------------------+----------------------------------------------+
| ####                  | ##### Where Can You Find this Information?   |
| # Type of Information |                                              |
+-----------------------+----------------------------------------------+
| Research abstracts    | -   ERIC Clearinghouse; ericir.syr.edu/Edu/  |
|                       |                                              |
|                       | -   National Clearinghouse for Comprehensive |
|                       |     School Reform;                           |
|                       |     <http://www.csrclearinghouse.org/> ,     |
|                       |     click on "CSR Library"                   |
+-----------------------+----------------------------------------------+
| Description of        | -   Materials from model or program          |
| theoretical base (the |     developers                               |
| idea behind the       |                                              |
| reform and why it     | -   *Catalog of School Reform Models*;       |
| should work).         |                                              |
|                       | > NWREL (2001);                              |
|                       | > [www.nwrel.org/scpd/c                      |
|                       | atalog/](http://www.nwrel.org/scpd/catalog/) |
|                       | > and                                        |
|                       | >                                            |
|                       | > National Clearinghouse for Comprehensive   |
|                       | > School Reform                              |
|                       | > <http://www.csrclearinghouse.org/>         |
|                       |                                              |
|                       | -   The Educators' Guide to Schoolwide       |
|                       |     Reform (1999). Herman, R., et.al.        |
|                       |     American Institutes for Research         |
|                       |                                              |
|                       | -   National Clearinghouse for Comprehensive |
|                       |     School Reform                            |
|                       |     <http://www.csrclearinghouse.org/>       |
+-----------------------+----------------------------------------------+
| Evidence of           | *Individual studies:*                        |
| effectiveness of an   |                                              |
| instructional         | -   Program or model developers              |
| practice              |                                              |
|                       | -   Education journals (e.g., American       |
|                       |     Educational Research Journal, American   |
|                       |     Journal of Education, Educational        |
|                       |     Leadership, Educational Research and     |
|                       |     Evaluation, Educational Researcher,      |
|                       |     Harvard Educational Review, Journal of   |
|                       |     Education for Students Placed at Risk,   |
|                       |     Review of Educational Research, School   |
|                       |     Effectiveness and School Improvement)    |
|                       |                                              |
|                       | *Research summaries*:                        |
|                       |                                              |
|                       | -   Slavin, R.E. and Fashola, O.S. (1998).   |
|                       |     *Show me the evidence! Proven and        |
|                       |     promising programs for America's         |
|                       |     Schools.* Thousand Oaks, CA: Corwin      |
|                       |     Press.                                   |
|                       |                                              |
|                       | -   Education journals (e.g., Review of      |
|                       |     Educational Research)                    |
+-----------------------+----------------------------------------------+
| Evidence of           | -   Program or model developers              |
| effectiveness of a    |                                              |
| set of practices or a | -   Education journals (e.g., American       |
| program               |     Educational Research Journal, American   |
|                       |     Journal of Education, Educational        |
|                       |     Leadership, Educational Research and     |
|                       |     Evaluation, Educational Researcher,      |
|                       |     Harvard Educational Review, Journal of   |
|                       |     Education for Students Placed at Risk,   |
|                       |     Review of Educational Research, School   |
|                       |     Effectiveness and School Improvement)    |
|                       |                                              |
|                       | -   National Clearinghouse for Comprehensive |
|                       |     School Reform                            |
|                       |     <http://www.csrclearinghouse.org/>       |
|                       |                                              |
|                       | -   *The Educators' Guide to Schoolwide      |
|                       |     Reform* (1999). Herman, R., et.al.       |
|                       |     American Institutes for Research         |
|                       |                                              |
|                       | -   Slavin and Fashola (1998) -- see above   |
+-----------------------+----------------------------------------------+
| Evidence of           | -   Program or model developers              |
| replicability         |                                              |
|                       | -   Education journals (e.g., American       |
|                       |     Educational Research Journal, American   |
|                       |     Journal of Education, Educational        |
|                       |     Leadership, Educational Research and     |
|                       |     Evaluation, Educational Researcher,      |
|                       |     Harvard Educational Review, Journal of   |
|                       |     Education for Students Placed at Risk,   |
|                       |     Review of Educational Research, School   |
|                       |     Effectiveness and School Improvement)    |
|                       |                                              |
|                       | -   The Educators' Guide to Schoolwide       |
|                       |     Reform (1999). Herman, R., et. al.       |
|                       |     American Institutes for Research         |
|                       |                                              |
|                       | -   Southwest Educational Development        |
|                       |     Laboratory (SEDL). CSRD Awards Database. |
|                       |     <http://csrprogram.ed.gov/>              |
+-----------------------+----------------------------------------------+

Appendix 2

Definition of Scientifically based research (ESEA; Title IX, Part A:

\(37\) Scientifically based research --"The term 'scientifically based
research'---

\(A\) means research that involves the application of rigorous,
systematic, and objective procedures to obtain reliable and valid
knowledge relevant to education activities and programs; and\
(B) includes research that---

\(i\) employs systematic, empirical methods that draw on observation or
experiment;

\(ii\) involves rigorous data analyses that are adequate to test the
stated hypotheses and justify the general conclusions drawn;

\(iii\) relies on measurements or observational methods that provide
reliable and valid data across evaluators and observers, across multiple
measurements and observations, and across studies by the same or
different investigators;

\(iv\) is evaluated using experimental or quasi-experimental designs in
which individuals, entities, programs, or activities are assigned to
different conditions and with appropriate controls to evaluate the
effects of the condition of interest, with a preference for random
assignment experiments, or other designs to the extent that those
designs contain within-condition or across-condition controls;

\(v\) ensures that experimental studies are presented in sufficient
detail and clarity to allow for replication or, at a minimum, offer the
opportunity to build systematically on their findings; and

\(vi\) has been accepted by a peer-reviewed journal or approved by a
panel of independent experts through a comparably rigorous, objective,
and scientific review."

######### 

######### 

######### 

######### Appendix 3

**No**

**Yes**

This booklet was designed to help school staffs and technical assistance
providers become critical consumers of educational research and
critically review the evidence of effects on student achievement. The
information in the text is provided graphically on page seventeen in the
form of a "decision tree". The following example illustrates how the
decision tree might be used.

Example:

Elementary School A is determined to take a comprehensive approach to
reforming their school. Using the eleven CSR components, they have
created a program that will affect all students and all teachers.

Their program includes an externally developed model that emphasizes
developing a family-style learning environment to help develop higher
order thinking skills for all students, with a special focus on
instruction. The CSR model has been used on 600 schools across the
country, and the model developers provide professional development and
materials.

The school will supplement that model with another change in instruction
(cross-age lessons) and with increased parent involvement (by having
parents act as tutors using the school's curriculum). Both of these
practices are consistent with the central idea of developing students'
higher order thinking skills in a family-style learning environment.

Before finalizing this plan, the staff wants to make sure that there is
research evidence supporting the practices and programs they are
considering. First, they look at the theory behind the externally
developed model, asking questions such as, "How does a family-style
learning environment work?" "Why does a family-style learning
environment improve higher order thinking skills?" and "What is involved
in creating this environment?" In asking these questions, they are
determining the theoretical basis of the model and ensuring that all of
the school's teachers have the same picture in their heads of what the
ideal family-style learning environment looks like, and why.

Next, the school staff critically reviews the evidence of effects. They
begin by identifying the central instructional practices of their
proposed program, such as cross-age instruction. They look for research
studies on that practice and any others they are considering. Let's say
the staff finds seven studies on cross-age instruction. They would then
determine the quality of each study, using the criteria summarized in
the decision tree and explained in the *Guidelines for Judging the
Quality of a Study*. Of the seven studies on cross-age instruction, they
might determine that four are of high quality, two of reasonable
quality, and one is not very good at all. Since there are at least four
high quality studies, the school staff would move to investigating the
findings. They might discover that three of the studies showed
significant results indicating that students' reasoning skills improved
after participating in cross-age instruction, with two showing that
students had gained more than 10 percentile points on state tests, an
educationally significant finding. Using this evidence, the staff would
determine that the instructional practice of cross-age instruction is a
proven practice. In the same manner, the school staff would review the
other instructional practices embedded in their comprehensive program,
such as small group project work.

If all of the instructional practices can be shown to be effective as in
the example above, then the CSR program would be "proven." If the
core---but not all---instructional practices can be shown to be
effective, then the CSR program would be "promising."

Finally, the school staff would want to consider implementation and
replicability. Although there are 600 schools using the CSR model that
is part of this school's program, no schools are known to be using their
exact CSR program, and hence there is no evidence that this combination
of components will work together. Therefore, school staff will rely
especially on the research evidence that indicates that each of these
components works well in isolation (evidence of effects) and that the
different components seem consistent with a common theme (theoretical
base).

[^1]: Management strategies also are named as part of Component 1 but,
    given the very limited nature of the research of the impact of
    management strategies on achievement, we recommend focusing on
    instructional practices here.