Education Research: Preliminary Observations on the Institute of Education Sciences' Research and Evaluation Efforts
Published: 2013-09-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government has a longstanding role in conducting education research and collecting related data. The Education Sciences Reform Act of 2002 established IES as Education's primary research and evaluation arm. With a budget of just under $600 million in fiscal year 2013, IES has a broad mission to provide its research, evaluations, and statistics to a wide variety of stakeholders, including researchers, parents, educators, and the general public.
This testimony reports on ongoing GAO work about IES. A full report will be issued later this year. Based on preliminary findings, this testimony will focus on: (1) the extent to which IES supports high-quality research and fulfills its mission, (2) the extent to which selected Education research and technical assistance groups disseminate relevant products to the education field, and (3) IES's coordination within Education and with other federal agencies.
For this work, GAO reviewed agency documents and relevant federal legislation, interviewed agency officials and stakeholders, and analyzed information from selected research and technical assistance groups. GAO also compared IES's practices to established guidelines for internal controls and effective program management and performance reporting.
section_title: What GAO Found
paragraphs: The Institute of Education Sciences (IES) supports high-quality research, according to stakeholders, but lacks certain key procedures needed to fulfill other aspects of its mission. Since its inception, IES has substantially improved the quality of education research. However, stakeholders expressed some concerns about IES's ability to produce timely and relevant research that meets their various needs. For example, IES's efforts to respond quickly to its stakeholders are slowed, in part, because the time IES's products have spent in peer review substantially increased this past year, and IES does not monitor some aspects of these timeframes. In addition, IES does not have a structured process for incorporating stakeholder input into its research agenda, which previous GAO work has shown to be key to sound federal research programs. Lastly, IES's performance measures do not fully reflect its current programs, which is not consistent with GAO's leading practices for performance management. IES officials said, however, that they have begun to develop new performance measures for all of their programs.
Although the Department of Education's (Education) research and technical assistance groups have taken steps to produce and disseminate relevant research to the field, IES does not always assess these efforts. Some stakeholders raised concerns about the relevance and dissemination of research and products from the Regional Educational Laboratories (REL) and Research and Development Centers (R & D Center). For example, they told us that these groups do not always adapt their products for use by both policymaker and practitioner audiences. Further, IES has not fully assessed REL and R & D Center relevance and dissemination efforts. As a result, IES does not know if these efforts are effective in meeting their mandated goal of providing usable research and information to policymakers and practitioners. GAO's prior work on information dissemination suggests that further assessment could help to inform IES's oversight of the RELs and R & D Centers to improve these groups' dissemination to key audiences.
IES works with federal education research agencies to increase the use of research evidence in federal decision-making, but according to officials, has limited ability to prioritize evaluations. IES and the National Science Foundation recently developed guidelines to help improve the quality of evidence resulting from federally-funded education research, which stakeholders said will benefit the education field. Within the department, IES plans evaluations of Education programs in concert with various other offices. However, Education officials said funding and program evaluation requirements prevent the agency from combining evaluation funds across programs, which limits their ability to conduct the evaluation projects they consider most important.

section_title: Letter
paragraphs: Thank you for inviting me here today to discuss the preliminary results of  our ongoing work for this committee on the Department of Education’s  (Education) Institute of Education Sciences (IES). For more than 100  years, the federal government has invested in conducting education  research and collecting education data. The Education Sciences Reform  Act of 2002 (ESRA) outlines a broad mission for IES to expand  fundamental knowledge and understanding of education and to provide  this information to a wide variety of stakeholders, including parents,  educators, researchers, policymakers, and the general public. As the  primary research and evaluation arm of Education, IES is charged with  providing information about educational policies, programs, and practices  that improve academic achievement and access to educational  opportunities for all students. In fiscal year 2013, IES had a budget of just  under $600 million, which it used to support a range of research, data  collection, and evaluation activities. In order to assist the Committee in  the reauthorization of ESRA, my testimony will focus on (1) the extent to  which IES supports high-quality research and fulfills its mission, (2) the  extent to which selected Education research and technical assistance  groups disseminate relevant products to the education field, and (3) IES’s  coordination within Education and with other federal agencies.
Our analysis is based on a review of relevant federal laws and regulations  as well as agency documents, including budget justifications, guidance  and procedures, grantee and contractor performance work statements,  and annual grantee and contractor performance reports. We also  conducted interviews with officials from IES, other offices within  Education, other federal agencies, and a range of stakeholder groups  representing researchers, policymakers, and practitioners, as well as  several members of IES’s advisory board—the National Board for  Education Sciences (NBES). In addition, we interviewed the directors of  the Regional Educational Laboratories (REL) and Comprehensive  Technical Assistance Centers (Comprehensive Center), and administered  a survey to the directors of the National Research & Development  Centers (R & D Center). We selected these three research and technical  assistance groups because they are all authorized under the public law  that included ESRA. Moreover, there is a statutory requirement for the  Comprehensive Centers to coordinate their activities and to collaborate  with the RELs, among others. We compared agency documents and  procedures to GAO’s standards for internal control and criteria on  performance management and reporting, as well as a framework GAO  developed to identify key elements of sound federal research and  evaluation programs. We intend to produce a report later this year that  will provide our final results. We discussed our preliminary results with  IES, and incorporated their comments as appropriate.
We are conducting our ongoing work in accordance with generally  accepted government auditing standards. Those standards require that  we plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions.
subsections: section_title: Background
paragraphs: ESRA authorizes IES to conduct and support many different types of  research and evaluations. Specifically, ESRA contains several key  provisions related to the management, core functions, and processes of  IES: all research conducted by IES is to use scientifically based research  standards that include, where appropriate, making claims of causal  relationships only in random assignment experiments; education evaluations conducted by IES are to employ experimental  designs using random assignment, when feasible; all research, statistics, and evaluation reports conducted by or  supported through IES must be subjected to rigorous peer review  before being published or otherwise made available to the public; and  the establishment of an advisory board—the NBES—whose duties  include (1) advising and consulting with the Director of IES regarding  its policies and approving the Director’s overall research priorities, and  (2) reviewing and approving procedures for peer review and reviewing  the work of IES to ensure the consistency of scientifically valid  research.
Education administers programs that support education research and  technical assistance through grants and contracts involving several  research groups—including the RELs and R & D Centers within IES and  the Comprehensive Centers within the Office of Elementary and  Secondary Education (see fig. 1).
The National Center for Education Evaluation and Regional Assistance  administers the REL program, a network of 10 regional entities that  conduct applied research, develop and disseminate research and  products, and conduct technical assistance and other activities to support  the needs of state and local education agencies in their region. The  Comprehensive Centers Program is a network of 22 technical assistance  grantees that help to increase the capacity of state educational agencies  (SEA) to assist districts and schools in meeting student achievement  goals. Selected Comprehensive Centers focus on specific areas of  expertise and produce research-based information and products for use  by SEAs. Lastly, ESRA includes specific requirements for the  administration of R & D Centers, which are designed to address areas of  national need and each of which IES assigns at least one of the broad  research topics outlined in the law. R & D Centers are also responsible  for the production and dissemination of rigorous evidence and products  that provide practical solutions to important education problems in the  United States. The National Center for Education Research and the  National Center for Special Education Research administer 17 R & D  Centers in total.
In addition to IES, other entities conduct education-related research and  evaluations, and ESRA includes general requirements for the Director of  IES to coordinate its research and evaluation work with these entities,  both within Education and across the rest of the federal government.  Within Education, the Office of Planning, Evaluation, and Policy  Development (OPEPD) conducts analyses and program evaluations on  behalf of the department. Other federal agencies, such as the National  Science Foundation and the National Institute for Child Health and  Human Development—part of the National Institutes of Health—also  support education-related research and the directors of these agencies  serve as nonvoting ex officio members on the NBES.
section_title: IES Has Supported High-Quality Research, but Lacks Key Processes and Performance Measures in Some Areas
paragraphs: In our ongoing work, we found that IES has substantially improved the  education research field. In 2007, the Office of Management and Budget  (OMB) assessed IES’s research and concluded that since its inception,  IES had transformed the quality and rigor of research within Education  and increased demand for scientifically based evidence of effectiveness  in the education field as a whole. Likewise, many stakeholders we spoke  with said IES’s research standards have improved the quality of  Education’s research and had a positive influence on education research  generally. More specifically, several stakeholders told us that IES  products, such as its publications of education statistics reports, were  useful for their work. While IES’s research grants and evaluations have  resulted in many randomized controlled studies since the agency was  established over 10 years ago, IES’s research standards also include  guidelines for the implementation of other rigorous research  methodologies, and it has recently funded studies using those  methodologies, such as regression discontinuity or single-case designs.  IES’s support of these multiple types of methodologies allows it to better  meet its various stakeholders’ needs.
However, IES’s research is sometimes of limited usefulness to  policymakers and practitioners. Some stakeholders told us that the  research and evaluations supported by IES may not be completed soon  enough to inform the decision making of policymakers and practitioners  on important questions. For example, officials from one constituency  based organization for policymakers said that IES’s evaluation of  Education’s Race to the Top and School Improvement Grant Programs  will not be released in time to give states the opportunity to implement  lessons learned from these studies before these programs’ funding  expires.
The peer review process may exacerbate timeliness concerns. In order to  ensure the high quality of IES’s work, ESRA requires IES-supported  research reports to be peer reviewed before being published. In the past  year, however, the time it takes to complete this review process has  substantially increased, from an average of 117 days in fiscal year 2011  to 175 days in fiscal year 2012. When asked for explanations for the  increase in 2012, senior IES officials cited factors such as the complexity  of the reports reviewed in that year and the time it took IES to work with  its contractors on suitable responses to peer review comments. In  accordance with GAO’s internal control standards, program managers  should have access to and use operational data to determine whether  they are meeting their agencies’ goals for effective and efficient use of  resources. However, officials told us that while the peer review office  within IES monitors the time taken for its review, IES does not monitor the  time its Centers or contractors take to respond to peer review comments,  which would allow it to take steps to mitigate delays, such as by holding  contractors more accountable.
We previously reported on and made several recommendations related to the timeliness  and dissemination activities of the What Works Clearinghouse. See GAO, Department of  Education: Improved Dissemination and Timely Product Release Would Enhance the  Usefulness of the What Works Clearinghouse, GAO-10-644. (Washington, D.C.: July 23,  2010). experiment in Texas and the development of a system to monitor  students’ social and emotional learning in a school district in Nevada.
Although IES has made recent efforts to increase the relevance of its  research, IES does not have a structured process for incorporating  feedback from policymakers and practitioners into its research agenda.  Though there is no single way for government agencies to conduct  research, in prior work we developed a framework to identify key  elements that promote a sound federal research program using  guidelines from several leading national research organizations. Within  that framework, we found that agencies should establish a structured  process for developing their research and evaluation priorities that  considers key stakeholders’ input. found individual IES Center Commissioners and the Director of IES have  at times gathered input from groups of policymakers and practitioners, but  that IES did not have an ongoing, structured process for collecting this  input. Inconsistent outreach by IES may have contributed to gaps in its  research. For example, stakeholders said that there is a shortage of  research using varied methodologies that could allow for shorter  turnaround times among IES-funded research projects.
This framework includes five key elements: (1) agenda setting, (2) selecting research,  (3) designing research, (4) conducting research, and (5) disseminating research results.  See GAO-11-285. For more information on leading practices for research and evaluation  planning, see also National Research Council, Rebuilding the Research Capacity at HUD  (Washington, D.C.: National Academies Press, 2008) and American Evaluation  Association, An Evaluation Roadmap for a More Effective Government, (September  2010), available at http://www.eval.org/d/do/107. accomplishments. In addition, according to our internal control  standards and leading practices on performance management, agencies  should establish performance measures for their activities and continually  compare actual performance data against these goals. To measure the  effectiveness of its research grants, IES counts the number of IES- supported interventions that have been determined to be effective in  improving student outcomes in particular areas. However, IES officials  told us that its current performance measures, which were developed  after the agency was established in 2002, no longer capture the scope of  IES’s current research and priorities. Furthermore, in some cases, senior  IES officials told us they are not relevant to managing the agency’s  operations. For example, one measure relies on the results of a survey of  potential users of the What Works Clearinghouse, but IES officials told us  they will not conduct this survey because they do not believe it would  yield enough useful information to be worth the investment. Moreover, a  new performance measure was reported in Education’s fiscal year 2014  budget request for IES that better reflects the results of its recent  research. However this measure still does not include certain existing  areas of IES research, such as research on the organization and  management of schools and education systems. In addition, IES does not  publicly report on the performance of the RELs, which constitute one of  the agency’s largest investments. As we have reported, without  performance measures, agencies may be at risk for failing to achieve  their goals.
IES officials told us they have begun work on revising their performance  measures. Officials told us that they plan to include revised performance  measures in Education’s fiscal year 2015 budget request for IES, and that  it has begun discussions with OMB to establish these new measures. As  of August 2013, IES officials told us they intend for the new performance  measures they are developing to include all programs, including the RELs  and its new grant programs for researcher-practitioner partnerships.
section_title: Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts
paragraphs: Research and technical assistance groups have taken various steps to  provide relevant research to the education field. More specifically, to  identify topics of relevance, RELs, R & D Centers, and Comprehensive  Centers have engaged policymakers and practitioners in planning  research and technical assistance activities. For example, beginning in  2012, RELs were required to conduct their work through new or existing  partnerships of practitioners, policymakers, and others—called research  alliances—which would work together to use data and research to  address specific concerns in education, such as improving low-performing  schools or college readiness. All three groups use a range of methods to  disseminate their research evidence and products, including publications,  conferences, and in some cases, technical assistance resources  specifically for teachers, such as professional development courses.
Despite these efforts, stakeholders—including practitioners and  policymakers—have raised concerns about the relevance and  dissemination of some of the research and products these groups have  produced. For example, a stakeholder group that represents local  school districts, as well as two superintendents we spoke with, said they  did not find REL research as relevant or as timely as other sources of  research information. At the same time, several stakeholders said that the  RELs’ new strategy of creating and working through research alliances  could help increase the RELs’ focus on research questions of concern to  the field. We also heard from stakeholders we spoke with that R & D  Center research may have limited relevance to practitioners, because  these centers consider the research community the primary audience for  their products. As a result, they do not always adapt their research  findings in a format that is readily understandable by practitioners, such  as by producing non-technical reports and shorter research summaries.  In addition to concerns about relevance, REL and R & D Center  dissemination efforts are not always reaching policymakers and  practitioners. Officials from some intermediaries—such as industry  associations—we spoke with said their organizations disseminate  research information to policymakers and practitioners. However, some  noted that further efforts are needed to leverage intermediary groups to  better market REL and R & D Center work to reach IES’s target  audiences.
IES does not have plans to evaluate the current group of RELs or R & D  Centers, nor does it collect sufficient information to manage these groups’  efforts to disseminate relevant research. IES is still in the process of  conducting a mandated evaluation of the prior group of RELs—whose  contracts ended in 2011—but has no further plans to evaluate the current  group of RELs. In addition, as of August 2013, IES had no plans to  conduct a formal evaluation of the R & D Centers and there is no  requirement to do so. Further, IES does not collect sufficient information  about RELs and R & D Centers to manage these groups’ efforts to  disseminate relevant research. For example, in 2012, IES, in  collaboration with the RELs, developed nearly 50 performance indicators  for these groups. IES officials told us that in August 2013 they had  prioritized 24 of these indicators on which RELs must now report.  However, IES has not established performance targets or goals for these  selected measures. Were IES to establish targets or goals, there would  be more incentive for the RELs to perform at the agency’s desired level,  and IES would be better positioned to determine if RELs are meeting its  expectations. Previous GAO work has indicated that agencies successful  in measuring performance had performance measures including targets  or goals that (1) demonstrate results; (2) are limited to the vital few; (3)  cover multiple priorities; and (4) provide useful information for decision  making. In addition, IES collects limited information to assess the R & D  Centers’ dissemination efforts because IES does not require R & D  Centers to report on their specific dissemination strategies or the  strategies’ effectiveness. Despite its difficulty, we have previously  reported on ways in which agencies can assess research programs with  broad dissemination goals.
section_title: IES Coordinates with Other Federal Agencies, but Education Faces Challenges in Funding Program Evaluations
paragraphs: IES coordinates with other federal education research agencies on  projects to increase federal agencies’ use of research evidence in guiding  funding decisions. For example, IES co-led a joint Education-National  Science Foundation working group to develop common evidence  guidelines for federally-funded research in education. Several  researchers we interviewed during our ongoing work, as well as two  federal agencies we spoke with, said these guidelines will benefit the  education field. In addition, IES helped develop and incorporate a tiered  evidence framework for awarding grants for Education’s Investing in  Innovation Fund (i3) grant program. According to an official from the  Administration for Children and Families (ACF) at the Department of  Health and Human Services, ACF has recently begun awarding its grants  using Education’s i3 program model, which uses tiered-evidence  standards to award larger grants to applicants with greater evidence of  their programs’ effectiveness.
In addition to working with other federal agencies, IES coordinates within  Education to facilitate collaboration between the REL and Comprehensive  Center programs. In our ongoing work, directors of both groups told us  coordination has improved with the current group of RELs and  Comprehensive Centers, which began in 2012. In addition, some  directors said that the RELs’ new structure of research alliances has  improved coordination, as some Comprehensive Centers are now  members of REL research alliances. This coordination, according to  some directors with whom we spoke, has helped to address confusion  among some SEA officials about the appropriate role and tasks  performed by each. Some directors told us they have conducted joint  visits with SEA officials, as well as made efforts to sequence their work  for their state clients to better meet their needs. For example, some REL  directors said the Comprehensive Center may conduct initial planning  work for a project, and the REL would later assist with any aspects of the  project requiring data analysis or original research.
Within Education, IES and OPEPD are jointly responsible for program  and policy evaluation, and together, these offices lead a department-wide  planning process to identify evaluation projects to conduct. However,  according to Education officials, efforts to prioritize evaluation projects  through this annual process are hindered in part because of statutory  requirements related to funding and program evaluation, including the fact  that amounts designated for evaluation purposes under a program must  be used only for evaluating that specific program or its activities.  Specifically, officials reported that Education does not have the authority  to combine evaluation funds from programs across the Department and  use them to evaluate any program. As a result, some evaluations may not  occur, and high-priority evaluations may be delayed. In addition, OPEPD  and IES officials said that smaller programs often have insufficient funds  to conduct an evaluation. We have previously reported that many  Education programs, especially smaller programs, have not been  evaluated, which can limit the ability of Congress to make informed  decisions about which programs to continue, expand, modify, consolidate,  or eliminate. Further, the President’s Budget request contained a  proposal to increase Education’s flexibility to conduct program  evaluations, and bills pending in Congress to reauthorize the  Elementary and Secondary Education Act, including that of this  committee, would address some of these issues for certain education  programs.
In conclusion, since its creation more than a decade ago, IES has made  significant contributions to strengthening the rigor of the education  research field and has considerably elevated the demand for conducting  and has promoted the use of scientifically based research in our nation’s  education system. However, IES could continue to build on these efforts  by improving its ability to release relevant and timely information to  policymakers and practitioners. With better management of its product  review process to ensure the more timely release of reports and a more  structured process for incorporating stakeholders’ needs into its research  agenda, IES would position itself to more fully deliver on its mission to be  responsive to the needs of its various stakeholders. Similarly,  comprehensive performance measures and routine assessment of its  grantees’ and contractors’ dissemination strategies would ensure its  stewardship of federal investments in these areas. In addition, the ability  to prioritize and conduct effective evaluations is critical to helping make  the best use of limited resources and to support Congress in making  informed decisions about which programs to continue, expand, modify,  consolidate, or eliminate. As we complete our ongoing work, we will  consider any recommendations needed to address these issues.
Chairman Kline, Ranking Member Miller, and Members of the Committee,  this concludes my prepared statement. I will be pleased to answer any  questions that you may have.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For future contact regarding this testimony, please contact George A.  Scott at (202) 512-7215 or scottg@gao.gov. Key contributors to this  testimony were Scott Spicer (Assistant Director), Lucas Alvarez, Deborah  Bland, Nora Boretti, David Chrisinger, Elizabeth Curda, Rachel Frisk,  Dana Hopings, Jean McSween and James Rebbe.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.