Education Research: Preliminary Observations on the Institute of Education Sciences' Research and Evaluation Efforts
Released: 2013-09-10T00:00:00

Letter

Background

ESRA authorizes IES to conduct and support many different types of research and evaluations. Specifically, ESRA contains several key provisions related to the management, core functions, and processes of IES: all research conducted by IES is to use scientifically based research standards that include, where appropriate, making claims of causal relationships only in random assignment experiments; education evaluations conducted by IES are to employ experimental designs using random assignment, when feasible; all research, statistics, and evaluation reports conducted by or supported through IES must be subjected to rigorous peer review before being published or otherwise made available to the public; and the establishment of an advisory board—the NBES—whose duties include (1) advising and consulting with the Director of IES regarding its policies and approving the Director’s overall research priorities, and (2) reviewing and approving procedures for peer review and reviewing the work of IES to ensure the consistency of scientifically valid research.

Education administers programs that support education research and technical assistance through grants and contracts involving several research groups—including the RELs and R & D Centers within IES and the Comprehensive Centers within the Office of Elementary and Secondary Education (see fig. 1).

The National Center for Education Evaluation and Regional Assistance administers the REL program, a network of 10 regional entities that conduct applied research, develop and disseminate research and products, and conduct technical assistance and other activities to support the needs of state and local education agencies in their region. The Comprehensive Centers Program is a network of 22 technical assistance grantees that help to increase the capacity of state educational agencies (SEA) to assist districts and schools in meeting student achievement goals. Selected Comprehensive Centers focus on specific areas of expertise and produce research-based information and products for use by SEAs. Lastly, ESRA includes specific requirements for the administration of R & D Centers, which are designed to address areas of national need and each of which IES assigns at least one of the broad research topics outlined in the law. R & D Centers are also responsible for the production and dissemination of rigorous evidence and products that provide practical solutions to important education problems in the United States. The National Center for Education Research and the National Center for Special Education Research administer 17 R & D Centers in total.

In addition to IES, other entities conduct education-related research and evaluations, and ESRA includes general requirements for the Director of IES to coordinate its research and evaluation work with these entities, both within Education and across the rest of the federal government. Within Education, the Office of Planning, Evaluation, and Policy Development (OPEPD) conducts analyses and program evaluations on behalf of the department. Other federal agencies, such as the National Science Foundation and the National Institute for Child Health and Human Development—part of the National Institutes of Health—also support education-related research and the directors of these agencies serve as nonvoting ex officio members on the NBES.

IES Has Supported High-Quality Research, but Lacks Key Processes and Performance Measures in Some Areas

In our ongoing work, we found that IES has substantially improved the education research field. In 2007, the Office of Management and Budget (OMB) assessed IES’s research and concluded that since its inception, IES had transformed the quality and rigor of research within Education and increased demand for scientifically based evidence of effectiveness in the education field as a whole. Likewise, many stakeholders we spoke with said IES’s research standards have improved the quality of Education’s research and had a positive influence on education research generally. More specifically, several stakeholders told us that IES products, such as its publications of education statistics reports, were useful for their work. While IES’s research grants and evaluations have resulted in many randomized controlled studies since the agency was established over 10 years ago, IES’s research standards also include guidelines for the implementation of other rigorous research methodologies, and it has recently funded studies using those methodologies, such as regression discontinuity or single-case designs. IES’s support of these multiple types of methodologies allows it to better meet its various stakeholders’ needs.

However, IES’s research is sometimes of limited usefulness to policymakers and practitioners. Some stakeholders told us that the research and evaluations supported by IES may not be completed soon enough to inform the decision making of policymakers and practitioners on important questions. For example, officials from one constituency based organization for policymakers said that IES’s evaluation of Education’s Race to the Top and School Improvement Grant Programs will not be released in time to give states the opportunity to implement lessons learned from these studies before these programs’ funding expires.

The peer review process may exacerbate timeliness concerns. In order to ensure the high quality of IES’s work, ESRA requires IES-supported research reports to be peer reviewed before being published. In the past year, however, the time it takes to complete this review process has substantially increased, from an average of 117 days in fiscal year 2011 to 175 days in fiscal year 2012. When asked for explanations for the increase in 2012, senior IES officials cited factors such as the complexity of the reports reviewed in that year and the time it took IES to work with its contractors on suitable responses to peer review comments. In accordance with GAO’s internal control standards, program managers should have access to and use operational data to determine whether they are meeting their agencies’ goals for effective and efficient use of resources. However, officials told us that while the peer review office within IES monitors the time taken for its review, IES does not monitor the time its Centers or contractors take to respond to peer review comments, which would allow it to take steps to mitigate delays, such as by holding contractors more accountable.

We previously reported on and made several recommendations related to the timeliness and dissemination activities of the What Works Clearinghouse. See GAO, Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse, GAO-10-644. (Washington, D.C.: July 23, 2010). experiment in Texas and the development of a system to monitor students’ social and emotional learning in a school district in Nevada.

Although IES has made recent efforts to increase the relevance of its research, IES does not have a structured process for incorporating feedback from policymakers and practitioners into its research agenda. Though there is no single way for government agencies to conduct research, in prior work we developed a framework to identify key elements that promote a sound federal research program using guidelines from several leading national research organizations. Within that framework, we found that agencies should establish a structured process for developing their research and evaluation priorities that considers key stakeholders’ input. found individual IES Center Commissioners and the Director of IES have at times gathered input from groups of policymakers and practitioners, but that IES did not have an ongoing, structured process for collecting this input. Inconsistent outreach by IES may have contributed to gaps in its research. For example, stakeholders said that there is a shortage of research using varied methodologies that could allow for shorter turnaround times among IES-funded research projects.

This framework includes five key elements: (1) agenda setting, (2) selecting research, (3) designing research, (4) conducting research, and (5) disseminating research results. See GAO-11-285. For more information on leading practices for research and evaluation planning, see also National Research Council, Rebuilding the Research Capacity at HUD (Washington, D.C.: National Academies Press, 2008) and American Evaluation Association, An Evaluation Roadmap for a More Effective Government, (September 2010), available at http://www.eval.org/d/do/107. accomplishments. In addition, according to our internal control standards and leading practices on performance management, agencies should establish performance measures for their activities and continually compare actual performance data against these goals. To measure the effectiveness of its research grants, IES counts the number of IES- supported interventions that have been determined to be effective in improving student outcomes in particular areas. However, IES officials told us that its current performance measures, which were developed after the agency was established in 2002, no longer capture the scope of IES’s current research and priorities. Furthermore, in some cases, senior IES officials told us they are not relevant to managing the agency’s operations. For example, one measure relies on the results of a survey of potential users of the What Works Clearinghouse, but IES officials told us they will not conduct this survey because they do not believe it would yield enough useful information to be worth the investment. Moreover, a new performance measure was reported in Education’s fiscal year 2014 budget request for IES that better reflects the results of its recent research. However this measure still does not include certain existing areas of IES research, such as research on the organization and management of schools and education systems. In addition, IES does not publicly report on the performance of the RELs, which constitute one of the agency’s largest investments. As we have reported, without performance measures, agencies may be at risk for failing to achieve their goals.

IES officials told us they have begun work on revising their performance measures. Officials told us that they plan to include revised performance measures in Education’s fiscal year 2015 budget request for IES, and that it has begun discussions with OMB to establish these new measures. As of August 2013, IES officials told us they intend for the new performance measures they are developing to include all programs, including the RELs and its new grant programs for researcher-practitioner partnerships.

Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts

Research and technical assistance groups have taken various steps to provide relevant research to the education field. More specifically, to identify topics of relevance, RELs, R & D Centers, and Comprehensive Centers have engaged policymakers and practitioners in planning research and technical assistance activities. For example, beginning in 2012, RELs were required to conduct their work through new or existing partnerships of practitioners, policymakers, and others—called research alliances—which would work together to use data and research to address specific concerns in education, such as improving low-performing schools or college readiness. All three groups use a range of methods to disseminate their research evidence and products, including publications, conferences, and in some cases, technical assistance resources specifically for teachers, such as professional development courses.

Despite these efforts, stakeholders—including practitioners and policymakers—have raised concerns about the relevance and dissemination of some of the research and products these groups have produced. For example, a stakeholder group that represents local school districts, as well as two superintendents we spoke with, said they did not find REL research as relevant or as timely as other sources of research information. At the same time, several stakeholders said that the RELs’ new strategy of creating and working through research alliances could help increase the RELs’ focus on research questions of concern to the field. We also heard from stakeholders we spoke with that R & D Center research may have limited relevance to practitioners, because these centers consider the research community the primary audience for their products. As a result, they do not always adapt their research findings in a format that is readily understandable by practitioners, such as by producing non-technical reports and shorter research summaries. In addition to concerns about relevance, REL and R & D Center dissemination efforts are not always reaching policymakers and practitioners. Officials from some intermediaries—such as industry associations—we spoke with said their organizations disseminate research information to policymakers and practitioners. However, some noted that further efforts are needed to leverage intermediary groups to better market REL and R & D Center work to reach IES’s target audiences.

IES does not have plans to evaluate the current group of RELs or R & D Centers, nor does it collect sufficient information to manage these groups’ efforts to disseminate relevant research. IES is still in the process of conducting a mandated evaluation of the prior group of RELs—whose contracts ended in 2011—but has no further plans to evaluate the current group of RELs. In addition, as of August 2013, IES had no plans to conduct a formal evaluation of the R & D Centers and there is no requirement to do so. Further, IES does not collect sufficient information about RELs and R & D Centers to manage these groups’ efforts to disseminate relevant research. For example, in 2012, IES, in collaboration with the RELs, developed nearly 50 performance indicators for these groups. IES officials told us that in August 2013 they had prioritized 24 of these indicators on which RELs must now report. However, IES has not established performance targets or goals for these selected measures. Were IES to establish targets or goals, there would be more incentive for the RELs to perform at the agency’s desired level, and IES would be better positioned to determine if RELs are meeting its expectations. Previous GAO work has indicated that agencies successful in measuring performance had performance measures including targets or goals that (1) demonstrate results; (2) are limited to the vital few; (3) cover multiple priorities; and (4) provide useful information for decision making. In addition, IES collects limited information to assess the R & D Centers’ dissemination efforts because IES does not require R & D Centers to report on their specific dissemination strategies or the strategies’ effectiveness. Despite its difficulty, we have previously reported on ways in which agencies can assess research programs with broad dissemination goals.

IES Coordinates with Other Federal Agencies, but Education Faces Challenges in Funding Program Evaluations

IES coordinates with other federal education research agencies on projects to increase federal agencies’ use of research evidence in guiding funding decisions. For example, IES co-led a joint Education-National Science Foundation working group to develop common evidence guidelines for federally-funded research in education. Several researchers we interviewed during our ongoing work, as well as two federal agencies we spoke with, said these guidelines will benefit the education field. In addition, IES helped develop and incorporate a tiered evidence framework for awarding grants for Education’s Investing in Innovation Fund (i3) grant program. According to an official from the Administration for Children and Families (ACF) at the Department of Health and Human Services, ACF has recently begun awarding its grants using Education’s i3 program model, which uses tiered-evidence standards to award larger grants to applicants with greater evidence of their programs’ effectiveness.

In addition to working with other federal agencies, IES coordinates within Education to facilitate collaboration between the REL and Comprehensive Center programs. In our ongoing work, directors of both groups told us coordination has improved with the current group of RELs and Comprehensive Centers, which began in 2012. In addition, some directors said that the RELs’ new structure of research alliances has improved coordination, as some Comprehensive Centers are now members of REL research alliances. This coordination, according to some directors with whom we spoke, has helped to address confusion among some SEA officials about the appropriate role and tasks performed by each. Some directors told us they have conducted joint visits with SEA officials, as well as made efforts to sequence their work for their state clients to better meet their needs. For example, some REL directors said the Comprehensive Center may conduct initial planning work for a project, and the REL would later assist with any aspects of the project requiring data analysis or original research.

Within Education, IES and OPEPD are jointly responsible for program and policy evaluation, and together, these offices lead a department-wide planning process to identify evaluation projects to conduct. However, according to Education officials, efforts to prioritize evaluation projects through this annual process are hindered in part because of statutory requirements related to funding and program evaluation, including the fact that amounts designated for evaluation purposes under a program must be used only for evaluating that specific program or its activities. Specifically, officials reported that Education does not have the authority to combine evaluation funds from programs across the Department and use them to evaluate any program. As a result, some evaluations may not occur, and high-priority evaluations may be delayed. In addition, OPEPD and IES officials said that smaller programs often have insufficient funds to conduct an evaluation. We have previously reported that many Education programs, especially smaller programs, have not been evaluated, which can limit the ability of Congress to make informed decisions about which programs to continue, expand, modify, consolidate, or eliminate. Further, the President’s Budget request contained a proposal to increase Education’s flexibility to conduct program evaluations, and bills pending in Congress to reauthorize the Elementary and Secondary Education Act, including that of this committee, would address some of these issues for certain education programs.

In conclusion, since its creation more than a decade ago, IES has made significant contributions to strengthening the rigor of the education research field and has considerably elevated the demand for conducting and has promoted the use of scientifically based research in our nation’s education system. However, IES could continue to build on these efforts by improving its ability to release relevant and timely information to policymakers and practitioners. With better management of its product review process to ensure the more timely release of reports and a more structured process for incorporating stakeholders’ needs into its research agenda, IES would position itself to more fully deliver on its mission to be responsive to the needs of its various stakeholders. Similarly, comprehensive performance measures and routine assessment of its grantees’ and contractors’ dissemination strategies would ensure its stewardship of federal investments in these areas. In addition, the ability to prioritize and conduct effective evaluations is critical to helping make the best use of limited resources and to support Congress in making informed decisions about which programs to continue, expand, modify, consolidate, or eliminate. As we complete our ongoing work, we will consider any recommendations needed to address these issues.

Chairman Kline, Ranking Member Miller, and Members of the Committee, this concludes my prepared statement. I will be pleased to answer any questions that you may have.

GAO Contact and Staff Acknowledgments

For future contact regarding this testimony, please contact George A. Scott at (202) 512-7215 or scottg@gao.gov. Key contributors to this testimony were Scott Spicer (Assistant Director), Lucas Alvarez, Deborah Bland, Nora Boretti, David Chrisinger, Elizabeth Curda, Rachel Frisk, Dana Hopings, Jean McSween and James Rebbe.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.