Archived Information

The most effective resource we have for improving American\
schools is scientifically valid knowledge about which\
strategies work and which don\'t work.

---Secretary Rod Paige

To improve education for all children, educators need to have an
understanding of which practices and policies are effective for
improving student achievement and which are not. Providing solutions to
the education problems in our nation can only be achieved with
trustworthy information on the effectiveness of teaching and learning
methods. In response to this need, the Department of Education has
committed to improving the quality and relevance of the research we fund
and conduct.

The No Child Left Behind Act grounds education improvement in the
application of scientifically based research, defined as "rigorous,
systematic, and objective procedures to obtain reliable and valid
knowledge relevant to education activities and programs."[^1] Implicit
in the legislation is the assumption that obtaining positive education
impacts requires research, and the type of research methods matters. The
history of other fields that have become grounded in science shows a
progression from decision-making based on precedent to decision-making
based on evidence derived from systematic protocols for collecting and
analyzing data. Further, in other fields, rapid technological and
functional advances have followed the shift to evidence-based
decision-making. The Department has made it a strategic priority to
bring about just such a change in education.

The Department of Education conducts research across a wide variety of
education research topics and funds specialized projects in special
education, rehabilitation, and disability research. The Institute of
Education Sciences, the Department's primary research arm, reflects our
commitment to advance the field of education research, supporting
evidence-based education through high standards for research methods and
the development and dissemination of research designed to ultimately
inform and improve teacher instruction and student achievement. In the
Department's Office of Special Education and Rehabilitative Services,
there are two research divisions: the Office of Special Education
Program's Research to Practice Division focuses on research,
demonstrations, and technical assistance and dissemination for students
with disabilities; the National Institute on Disability and
Rehabilitation Research maintains a comprehensive program of research
and development related to the rehabilitation of individuals with
disabilities. Together these three entities advance research in the
field to improve teaching and learning in schools and to increase access
for individuals with disabilities.

Department Expenditures

High Research Standards Result in Rigorous Studies

As we hold students, teachers, and schools accountable for their
performance, we are also committed to providing them with reliable
evidence about educational effectiveness. Education fads will come and
go, but the Department of Education encourages practitioners and
decision-makers to make decisions based on the best available evidence.
The first step in this process has been the impressive gains in the past
year that the Department has made to fund studies based upon rigorous
and scientifically based research standards.

Improving the quality of research begins with establishing more rigorous
standards for the quality of the projects supported by the Department.
The Department communicates these standards to researchers by releasing
funding announcements with detailed methodological requirements. To
ensure that these standards are met, the Department convenes scientific
peer review panels of experienced researchers to evaluate the technical
merit of research proposals and funds only applications that meet the
high standards for research quality.

**Performance Goals.** To determine whether newly funded education
research and evaluation efforts are of high quality, each year an
external panel of distinguished scientists appraises a randomly selected
sample of newly funded grant proposals. Based on preliminary FY 2004
data, the portion of education research deemed to be of high quality
increased by 20 percentage points since 2001, even though we did not
meet our target for fiscal year (FY) 2004. Final FY 2004 data will be
available in December 2004.

The Department also assesses the quality of our supported research by
annually tracking the proportion of funded proposals that employ
experimental methods to answer causal questions. As the "gold standard"
for research on the effectiveness of programs, randomized control trials
provide the most rigorous tests of what works in education. In FY 2004,
a large percentage of Department research projects met high
methodological standards: more than 90 percent of the education research
projects evaluated to date that address causal questions did so using
randomized experimental designs. FY 2004 data from special education
projects will be available in December 2004.

The Department's progress on our performance goals for this objective is
summarized in the table below. See p. 27 for methodology and appendix A,
pp. 202--03, for detailed data.

+------------------------------------------+-------------+------------+
| Quality of Research (Objective 4.1)      |             |            |
+------------------------------------------+-------------+------------+
| Performance Goals                        | Status      | Year       |
+------------------------------------------+-------------+------------+
| New IES and OSEP research and evaluation | Did not     | FY 2004    |
| efforts that are deemed to be of high    | meet        |            |
| quality                                  |             |            |
|                                          |             |            |
| -   Projects                             |             |            |
|                                          |             |            |
| -   Publications                         |             |            |
+------------------------------------------+-------------+------------+
| New IES and OSEP efforts addressing      | Exceeded    | FY 2004    |
| causal questions that employ randomized  |             |            |
| experimental designs                     |             |            |
|                                          |             |            |
| -   Projects                             |             |            |
|                                          |             |            |
| -   Publications                         |             |            |
+------------------------------------------+-------------+------------+

IES = Institute of Education Sciences

OSEP = Office of Special Education Programs

**Note.** There were no publications to report so the status reflects
only an assessment of projects.

In the long term, improving the quality of education research requires a
new generation of researchers who are trained to conduct rigorous
studies and evaluation. Currently the capacity of the education research
community to conduct rigorous research is limited. To create a
scientific workforce capable of high-quality education research, the
Department has established predoctoral and postdoctoral training
programs to develop a cadre of young investigators with the skills to
conduct the type of research needed to provide solutions to the
challenges in education facing our country.

Relevant Research Findings Inform Education Practice and Policy

According to Grover J. Whitehurst, Director of the Institute of
Education Sciences, "The current nationwide emphasis on ensuring that
all students and schools achieve at high levels has increased the demand
for sound evidence regarding 'what works' in education."[^2] The
Department's commitment to the production and dissemination of relevant
research findings provides the basis for improving education practices
for all students and improving access and rehabilitation techniques for
those with disabilities.

To increase the relevance of our education research and evaluation
activities, the Department emphasizes research designed to evaluate the
effectiveness of education programs and practices in the field. To this
end, the Department has established focused research programs in
reading, mathematics, and science education; teacher quality; education
finance, leadership, and management; and special education. Education
practitioners and decision-makers have indicated a need for research to
answer critical questions in these specific areas, and the Department
continues the development and evaluation of research to improve and
inform theory and practice.

**Review and Dissemination Activities.** Many evaluations purport to
show the effectiveness of an education intervention, but their design
and methods do not provide the basis for assessing impact. By reviewing
studies and evaluations for their scientific rigor, the Department's
What Works Clearinghouse[^3] analyzes the quality of education studies
and evaluations to determine whether they provide reliable evidence on
the impact of an intervention on student learning. To carry out its
work, the clearinghouse developed rigorous standards for reviewing
intervention studies, which are now widely regarded as scientifically
valid for assessing research on the impact of interventions in education
and other fields.

The clearinghouse prepares individual study reports for all studies
meeting its standards. These reports provide education decision-makers
with information on the quality of research on the impact of an
intervention; the type of intervention evaluated; characteristics of the
students, teachers, and schools involved in the evaluation; the outcome
measures that were assessed; and the results of the evaluation (i.e.,
was the intervention effective or not). In June, the clearinghouse
released initial study reports in two topic areas---peer-assisted
learning and middle school mathematics curricula. Work is underway to
evaluate the research in the following areas: beginning reading;
elementary school math curricula; character education; English language
learning; adult literacy; dropout prevention; and prevention of
delinquent, disorderly, and violent behavior.

Another improvement in making quality education research available to
the public came in FY 2004 when the Department awarded a major contract
to develop and operate the new database system for the Education
Resources Information Center (ERIC).[^4] ERIC will be linked to the
resources of the What Works Clearinghouse and other sources of
up-to-date information and research about education. The new ERIC uses
the latest technology to provide access to its documents and journal
articles. Users will find fast and effective search results in this
well-established directory of education research.

**Improvements in Statistical Reports.** The Department's National
Center for Education Statistics is responsible for informing the nation
on the condition of education in our country. *The Condition of
Education 2004*[^5] was submitted to the Congress and the public on June
1, 2004, to enhance understanding of the current status of education in
the United States. The Department also produced other key reports,
including *Projections of Education Statistics to 2013* and *Indicators
of School Crime and Safety: 2003*, and the results of over 35 studies.

A major accomplishment in statistical reporting this year was the
improvement in the timeliness of the release of the congressionally
mandated National Assessment of Educational Progress (NAEP) results.
During FY 2004, NAEP data for assessments conducted in spring 2003 were
released just eight months following data collection, the shortest data
lag in the history of the assessment. The 2003 NAEP national and state
assessments in reading and mathematics for fourth and eighth grades were
of particular importance this year because they provided baseline data
to support the assessment's new role in state accountability systems,
and, for the first time in the history of the program, they represented
all states.

+-----------------------------------------------------------------------+
| ![](media/image4.wmf)                                                 |
|                                                                       |
| NAEP = National Assessment of Educational Progress                    |
|                                                                       |
| **Source.** Data from the Department's Institute of Education         |
| Sciences.                                                             |
+-----------------------------------------------------------------------+

**Research on Disability and Rehabilitation.** Through the National
Institute on Disability and Rehabilitation Research, the Department
provides leadership and support for a comprehensive program related to
the rehabilitation of individuals with disabilities. The Department's
ongoing efforts maximize the full inclusion, social integration,
employment, and independent living of individuals of all ages with
disabilities. Accomplishments in technological advancements over this
past year include the following:

-   The development of 35 state and local\
    "visit-ability" programs that incorporate an affordable,
    sustainable, and inclusive design approach for integrating basic
    accessibility features into all newly built homes.[^6]

-   The publication of a book on universal design that has been
    disseminated nationwide and has been adopted by the New York City
    Department of Design and Construction as the official guide for all
    architects working for the city.[^7]

-   The implementation of accessible information kiosks at the new World
    War II Memorial in Washington, D.C., and by the U.S. Postal Service
    to allow individuals with all types of abilities to access needed
    public information with ease.[^8]

```{=html}
<!-- -->
```
-   The development and validation of a new more valid and reliable
    outcome measure of walking function in individuals with spinal cord
    injury, the Walking Index for Spinal Cord Injury (WISCI).[^9]

**Performance Goals.** To track the relevance of our research
investments, each year the Department submits a randomly selected sample
of newly funded projects to an external panel of experienced
practitioners, including superintendents, directors of special
education, directors of research and evaluation at the district and
state levels, and chief state school officers. The panel evaluates the
relevance of the proposed research to education practice. Since FY 2001,
the percentage of newly funded proposals that were of high relevance has
doubled; by FY 2003, over half of the projects sampled were rated as
highly relevant. While this increase did not meet the target set for
FY 2003, the Department is continuing efforts to improve the usefulness
of our research for education practitioners and decision-makers. We are
refining our Requests for Applications to specify the types of questions
and projects that are needed by people in the field and providing more
guidance to the scientific review panels so that they better understand
those needs. FY 2004 data will be available in January 2005.

The Department's progress on our performance goals for this objective is
summarized in the table below. See p. 27 for methodology and appendix A,
p. 203, for detailed data.

  -------------------------------------------- ------------- -------------
  Relevance of Research (Objective 4.2)                      

  Performance Goal                             Status        Year

  New research projects of high relevance to   Did not meet  FY 2003
  educational practice                                       
  -------------------------------------------- ------------- -------------

Programs Supporting Goal 4

Six of our grant programs most directly support Goal 4. These programs
are listed below. In the table we provide both FY 2004 appropriations
and FY 2004 expenditures for each of these programs. We also provide an
overview of the results of each program on its program performance
measures. Program performance reports are available on the Web at
<http://www.ed.gov/about/reports/annual/2004report/index.html>.

<table>
<colgroup>
<col style="width: 41%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 4%" />
</colgroup>
<thead>
<tr class="header">
<th>Program Name</th>
<th>Appro-pria-tions<sup>†</sup></th>
<th>Expend-itures<sup>‡</sup></th>
<th colspan="9"><p>Program Performance Results</p>
<p>Percent of Targets Met, Not Met, Without Data</p></th>
</tr>
<tr class="odd">
<th rowspan="2"></th>
<th rowspan="2"><p>FY 2004</p>
<p>$ in millions</p></th>
<th rowspan="2"><p>FY 2004</p>
<p>$ in millions</p></th>
<th colspan="3">FY 2004</th>
<th colspan="3">FY 2003</th>
<th colspan="3">FY 2002</th>
</tr>
<tr class="header">
<th><p>%</p>
<p>Met</p></th>
<th><p>%</p>
<p>Not Met</p></th>
<th><p>%</p>
<p>No Data</p></th>
<th><p>%</p>
<p>Met</p></th>
<th><p>%</p>
<p>Not Met</p></th>
<th><p>%</p>
<p>No Data</p></th>
<th><p>%</p>
<p>Met</p></th>
<th><p>%</p>
<p>Not Met</p></th>
<th><p>%</p>
<p>No Data</p></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>ESEA: Indian Education National Activities</td>
<td>5</td>
<td>5</td>
<td colspan="3"></td>
<td colspan="3"></td>
<td colspan="3"></td>
</tr>
<tr class="even">
<td>ESEA: Title I Evaluation</td>
<td>11</td>
<td>15</td>
<td colspan="3"></td>
<td colspan="3"></td>
<td colspan="3"></td>
</tr>
<tr class="odd">
<td>ESRA: Research, Development and Dissemination</td>
<td>181</td>
<td>126</td>
<td>60</td>
<td>20</td>
<td>20</td>
<td>100</td>
<td>0</td>
<td>0</td>
<td>100</td>
<td>0</td>
<td>0</td>
</tr>
<tr class="even">
<td>ESRA: Statistics</td>
<td>119</td>
<td>109</td>
<td>43</td>
<td>57</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>100</td>
<td colspan="3"></td>
</tr>
<tr class="odd">
<td>IDEA: Special Education Research and Innovation</td>
<td>83</td>
<td>86</td>
<td>0</td>
<td>0</td>
<td>100</td>
<td>0</td>
<td>0</td>
<td>100</td>
<td>0</td>
<td>13</td>
<td>88</td>
</tr>
<tr class="even">
<td>RA: National Institute on Disability and Rehabilitation
Research</td>
<td>119</td>
<td>126</td>
<td>0</td>
<td>0</td>
<td>100</td>
<td>43</td>
<td>29</td>
<td>29</td>
<td>50</td>
<td>50</td>
<td>0</td>
</tr>
<tr class="odd">
<td><strong>Total</strong></td>
<td><strong>518</strong></td>
<td><strong>467</strong></td>
<td colspan="9"></td>
</tr>
</tbody>
</table>

^†^ Budget for each program includes program budget authority and the
program's proportional share of salaries and expenses budget authority.

^‡^ Expenditures occur when recipients *draw down* funds to cover actual
outlays. FY 2004 expenditures may include funds from prior years'
appropriations. Expenditures for each program include the program's
proportional share of administrative expenditures.

A shaded cell denotes that the program did not have targets for the
specified year.

ESEA = Elementary and Secondary Education Act

ESRA = Education Sciences Reform Act

IDEA = Individuals with Disabilities Education Act

RA = Rehabilitation Act

PART Analysis for Goal 4 Programs

The Program Assessment Rating Tool (PART) was developed and implemented
by the Office of Management and Budget as a standardized process for
determining program effectiveness in a consistent way across agencies.
Over a five-year period, all programs will be evaluated under this
process. Results of PART reviews are used by agencies as one component
of justifying their budget requests. Following are summaries of PART
reviews that were conducted in conjunction with preparing the
Department's FY 2004 budget request and subsequent updated reviews of
those programs.[^10]

Program: Statistics

Year of Rating: *For FY 2004 Budget (Initial*)

*For FY 2005 Budget (Revised)*

Rating: Effective

Program Type: Research and Development

![](media/image5.wmf)

Recommendations:

1\. The 2002 PART assessment found a weakness in long-term performance
measurement for NCES.

2\. The Department of Education needs to improve the timeliness of NCES
products and services.

Response:

1\. The Department of Education has established long-term performance
measures for the program.

2\. The Department of Education is examining the timeliness of NCES
products and services, including National Assessment products and
services.

[^1]: Public Law 107-110, No Child Left Behind Act of 2001, section
    9101.

[^2]: Department of Education, July 2004 ED Results Agenda.

[^3]: Available at
    [http://]{.underline}[www.whatworks.ed.gov](http://www.whatworks.ed.gov/).

[^4]: Available at <http://www.eric.ed.gov/ERICWebPortal/Home.portal>

[^5]: Available at <http://nces.ed.gov/pubs2004/2004077.pdf>.

[^6]: Rehabilitation Engineering Research Center (RERC) on Universal
    Design at the State University of New York at Buffalo, School of
    Architecture and Planning, Edward Steinfeld, Arch. D., principal
    investigator.

[^7]: Ibid.

[^8]: Rehabilitation Engineering Research Center on Information
    Technology Access at the Trace Center, University of Wisconsin,
    Gregg Vanderheiden, Ph.D., principal investigator.

[^9]: ^8^ Spinal Cord Injury Model System Project, Thomas Jefferson
    University, John F. Ditunno Jr., M.D., principal investigator.

[^10]: Information about the PART process is available at:
    <http://www.whitehouse.gov/omb/part/> . Information on Department
    PARTs is available at: [http://www.whitehouse.gov/omb\
    /budget/fy2005/pdf/ap_cd_rom/part.pdf](http://www.whitehouse.gov/omb%0B/budget/fy2005/pdf/ap_cd_rom/part.pdf)
    and <http://www.whitehouse.gov/omb/budget/fy2005/pma/education.pdf>.