Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts
Published: 2013-12-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government has a longstanding role in conducting education research and collecting education data, and IES has a broad mission to provide this information to a wide variety of stakeholders. With a fiscal year 2013 budget of just under $600 million, IES is Education's primary research and evaluation arm. GAO was asked to review IES's performance. This report examines: (1) the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, (2) the extent to which selected Education research and technical assistance groups disseminate relevant products to the education field, and (3) how IES coordinates its activities with other relevant federal research agencies and within Education. GAO reviewed relevant federal laws and regulations, agency documents and data, interviewed agency officials and stakeholders, and analyzed information from selected research and technical assistance groups. GAO also compared IES's practices to federal internal control standards and leading practices for performance management and collaboration.
section_title: What GAO Found
paragraphs: The Department of Education's (Education) Institute of Education Sciences (IES) supports high-quality research, but lacks certain key procedures needed to fulfill other aspects of its mission. According to stakeholders, IES has substantially improved the quality of education research since its inception over a decade ago. However, GAO identified concerns with IES's ability to produce timely and relevant research. For example, IES's efforts are slow to respond to stakeholders' needs, in part, because the time IES's products have spent in peer review has substantially increased in recent years--from an average of 117 days in fiscal year 2011 to 150 days in fiscal year 2013--and IES does not monitor some aspects of these timeframes. In addition, IES does not have a structured process for incorporating stakeholder input into its research agenda, which previous GAO work has shown to be key to sound federal research programs. Lastly, IES's performance measures do not fully reflect its current programs, which is not consistent with leading practices GAO has identified for performance management. For example, IES does not publicly report on the overall performance of the Regional Educational Laboratories (REL) program, which constitutes one of the agency's largest investments. IES officials said that they have begun to develop new performance measures for all of their programs, but these will not be in place until fiscal year 2015.
Although Education's research and technical assistance groups have taken steps to disseminate relevant research to the education field, IES does not always assess these efforts. Some stakeholders raised concerns about the dissemination of relevant products from the RELs and Research and Development Centers (R & D Center). For example, they told GAO that these groups do not always adapt their products for use by both policymaker and practitioner audiences. Further, IES has not fully assessed REL and R & D Center dissemination efforts. As a result, IES does not know if these efforts are effective in meeting their mandated goal of providing usable research and information to stakeholders. GAO's prior work on information dissemination suggests that further assessment could help inform IES's oversight of the RELs and R & D Centers to improve these groups' dissemination to key audiences.
IES coordinates with other relevant federal research agencies to increase the use of research evidence in federal decision-making. For example, IES and the National Science Foundation recently released guidelines to help improve the quality of evidence resulting from federally-funded education research, which stakeholders said will benefit the education field. Within the department, IES plans evaluations of Education programs through coordination with various other offices. However, Education lacks statutory authority to combine evaluation funds for Elementary and Secondary Education Act (ESEA) programs, which officials said limits the department's ability to conduct high-quality evaluations of programs it considers most important. GAO's prior work has shown that many Education programs, especially smaller programs, have not been evaluated, limiting the ability of Congress to make informed policy decisions. For example, in 2009, GAO reported that 11 of Education's teacher quality programs had not been evaluated in more than 7 years. Officials said that the ability to combine evaluation funds would allow the department to conduct needed evaluations of ESEA programs.
section_title: What GAO Recommends
paragraphs: GAO recommends that Congress consider granting Education authority to combine funds authorized for evaluation of ESEA programs, and IES use available data to manage its peer review process, develop a structured process to gather stakeholder input, develop performance measures that reflect all key agency activities, and assess REL and R & D Center dissemination strategies. Education agreed with our recommendations.

section_title: Letter
paragraphs: For more than 100 years, the federal government has invested in  conducting education research and collecting education data. More  recently, the Education Sciences Reform Act of 2002 (ESRA) established  the Institute of Education Sciences (IES) and outlines a broad mission for  IES to expand fundamental knowledge and understanding of education  and to provide this information to a wide variety of stakeholders, including  parents, educators, researchers, policymakers, and the general public.  As the primary research and evaluation arm of the Department of  Education (Education), IES is charged with providing information about  educational policies, programs, and practices that improve academic  achievement and access to educational opportunities for all students. In  fiscal year 2013, IES had a budget of just under $600 million, which it  used to support a range of research, data collection, and evaluation  activities.
Specifically, IES funds projects on topics that range from studying the  effectiveness of basic reading or math initiatives, to evaluating the impact  of federal grant programs, to collecting student performance data through  the National Assessment of Educational Progress, a nationally  representative measure of student academic performance. You requested  that we review IES’s performance in addressing its critical and wide- ranging mission. This report examines: (1) the extent to which IES has  demonstrated its ability to support high-quality research and fulfill its  mission; (2) the extent to which selected Education research and  technical assistance groups disseminate relevant products to the  education field; and (3) how IES coordinates its activities with other  relevant federal research entities and within Education.
To assess the extent to which IES has demonstrated its ability to support  high-quality research and fulfill its mission, we reviewed relevant federal  laws and regulations as well as agency documents describing IES’s  performance measures and key agency processes. We conducted  interviews with officials from IES and a range of stakeholder groups  representing researchers, policymakers, and practitioners, as well as  several members of IES’s advisory board—the National Board for  Education Sciences (NBES). We obtained IES data for the most recent  four years, fiscal years 2010 through 2013, on the average number of  work days from the submission of the initial manuscript to final approval  for reports sponsored by IES that underwent external peer review before  being publicly released. We assessed the reliability of IES data by  obtaining written responses from agency officials knowledgeable about  the data. We determined that the data were sufficiently reliable for the  purposes of this report. To address the extent to which selected  Education research and technical assistance groups disseminate relevant  products, we obtained documents from each of these groups: the  Regional Educational Laboratories (REL), the Comprehensive Technical  Assistance Centers (Comprehensive Centers), and the Research and  Development Centers (R & D Center). We analyzed these documents to  identify what is known about the relevance, dissemination, and  utilization of the research and products these groups produce. We also  met with the directors of the RELs and Comprehensive Centers as well as  program staff at IES and Education’s Office of Elementary and Secondary  Education (OESE) who administer these programs. We administered a  short survey via e-mail to the directors of IES’s R & D Centers to obtain  additional information. We fielded this survey from April 2013 to May 2013  and achieved a 94 percent (17 of 18) response rate. We selected these  three research and technical assistance groups because they are all  authorized by the same law that established IES. To describe how IES  coordinates with other relevant federal research agencies and within the  department, we reviewed relevant federal laws and regulations and other  relevant documents. We also interviewed Education officials responsible  for planning evaluations of departmental programs. Finally, we met with  officials from two federal research organizations that conduct education- related research—the Eunice Kennedy Shriver National Institute of Child  Health and Human Development (NICHD) and the National Science  Foundation (NSF)—to learn about their processes and coordination with  IES. For all three objectives, we compared agency documents and  procedures to GAO’s criteria on internal controls, performance  measurement and reporting, and collaboration, as well as a framework  GAO developed to identify key elements of sound federal research and  evaluation programs. Appendix I describes our scope and methodology  in more detail.
We conducted this performance audit from July 2012 to December 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Prior to the establishment of IES, federal education research was carried  out by the Office of Educational Research and Improvement, a federal  agency that—according to observers both within and outside the federal  government—was challenged by frequent leadership changes, competing  interests, and lack of a focused mission. The 2002 reauthorization of the  Elementary and Secondary Education Act (ESEA) made numerous  references to the use of research-based evidence in educational decision  making and federal education programs. With its enactment later in 2002,  ESRA established IES, an arm of the Department of Education dedicated  to the improvement of federal education research, statistics, evaluation,  information, and dissemination.
IES’s mission is “to provide national leadership in expanding fundamental  knowledge and understanding of education from early childhood through  postsecondary study, in order to provide parents, educators, students,  researchers, policymakers, and the general public with reliable  information about (1) the condition and progress of education in the  United States, including early childhood education; (2) educational  practices that support learning and improve academic achievement and  access to educational opportunities for all students; and (3) the  effectiveness of Federal and other education programs.” Furthermore, in  carrying out its mission, ESRA calls upon IES to “compile statistics,  develop products, and conduct research, evaluations, and wide  dissemination activities in areas of demonstrated national need.”
ESRA authorizes IES to conduct and support many different types of  research and evaluations in support of its mission. Specifically, ESRA  contains several key provisions related to the management, core  functions, and processes of IES: all research conducted by IES is to use scientifically based research  standards that include, where appropriate, making claims of causal  relationships only in random assignment experiments; education evaluations conducted by IES are to employ experimental  designs using random assignment, when feasible; all research, statistics, and evaluation reports conducted by or  supported through IES must be subjected to rigorous peer review  before being published or otherwise made available to the public;  the Director of IES will be appointed by the President, by and with  consent of the Senate, for a 6-year term and will propose the  Institute’s overall research priorities and report biennially to Congress  and others on the Institute’s activities, among other duties; and a board of directors is established—the NBES—whose duties include,  among other things, (1) advising and consulting with the Director of  IES regarding its policies and approving the Director’s overall  research priorities, (2) reviewing and approving procedures for peer  review, and (3) reviewing the work of IES to ensure the consistency of  scientifically valid research.
All of IES’s work is carried out by four Centers, created by law, that exist  within IES. See table 1 for more information on these Centers and their  key activities.
Education administers programs that support education research and  technical assistance through grants and contracts involving several  research groups—the RELs and R & D Centers within IES and the  Comprehensive Centers within OESE (see fig. 1).
The National Center for Education Evaluation and Regional Assistance  (NCEERA) administers the REL program, a network of 10 regional  entities that conduct applied research; develop and disseminate research  and products; and conduct technical assistance and other activities to  support the needs of state and local educational agencies in their region.  The RELs were first established in 1965 under Title IV of ESEA with the  broad goal of supporting general educational improvement efforts in their  regions. Since IES was created and began to administer the REL  program, the RELs have completed two contract cycles, and the current  RELs were launched in 2012 under new 5-year contracts.
ESRA also includes specific requirements for the administration of  National R & D Centers, which are designed to address areas of national  need and each of which addresses at least one of the broad research  topics outlined in the law. R & D Centers are also responsible for the  production of rigorous evidence and dissemination of products that  provide practical solutions to important education problems in the United  States. Each R & D Center is funded for no more than 5 years, and in  fiscal year 2013, the National Center for Education Research (NCER) and  the National Center for Special Education Research (NCSER)  administered 18 R & D Centers in total.
The Comprehensive Centers Program is a network of 22 technical  assistance grantees that help to increase the capacity of state  educational agencies (SEA) to assist districts and schools in meeting  student achievement goals. Established in 1994, Comprehensive Centers  provide services primarily to SEAs to enable them to assist school  districts and schools, especially low-performing schools. Comprehensive  Centers provide training and technical assistance in (1) the  implementation and administration of programs authorized under ESEA  and (2) the use of research-based information and strategies. Selected  Comprehensive Centers focus on specific content areas and produce  research-based information and products for use by SEAs.
In addition to IES, other entities conduct education-related research and  evaluations, and ESRA includes general requirements for the Director of  IES to coordinate its research and evaluation work with these entities,  both within Education and across the rest of the federal government.
Within Education, the Office of Planning, Evaluation, and Policy  Development (OPEPD) conducts analyses and program evaluations on  behalf of the department. Other federal agencies, such as NSF and  NICHD—part of the National Institutes of Health—also support education- related research and the directors of these agencies serve as nonvoting  ex officio members on the NBES. In addition, many nongovernmental  organizations conduct education-related research and evaluations that  inform general knowledge and understanding of educational policies,  programs, and practice. For example, foundations such as the Bill &  Melinda Gates Foundation, The William T. Grant Foundation, and The  Spencer Foundation, conduct education research or program evaluation  in specific topic areas to advance the public interest or their organization’s  mission or goals. Additionally, other research organizations such as  Mathematica Policy Research, MDRC, and SRI International, contract  with the federal government or other clients to conduct agreed upon  research or program evaluations on their behalf.
section_title: IES Supports High- Quality Research, but Lacks Key Processes and Performance Measures in Some Areas
subsections: section_title: IES Supports High-Quality Research, but Does Not Monitor the Timeliness of Some Aspects of Its Peer Review Process
paragraphs: IES has substantially improved the education research field. In 2007, the  Office of Management and Budget (OMB) assessed IES’s research and  concluded that IES had transformed the quality and rigor of research  within Education and increased the demand for scientifically based  evidence of effectiveness in the education field as a whole. Likewise,  many stakeholders told us that IES’s research standards had improved  Education’s research and had a positive influence on education research  generally. More specifically, several stakeholders told us that IES  products, such as its publications of education statistics reports, were  useful for their work. In addition, one regional comprehensive center  director told us that a Practice Guide IES released on dropout prevention  has become the framework that one of the states in its region is using for  dropout prevention efforts statewide. While IES’s research grants and  evaluations have resulted in many randomized controlled studies since  the agency was established over 10 years ago, its research standards  also include guidelines for the implementation of other rigorous research  methodologies, and it has recently funded studies using those  methodologies, such as regression discontinuity or single-case designs.  Further, IES officials also described a variety of other types of research  that the agency supports, such as data analyses and correlational  analyses. IES’s support of these multiple types of methodologies allows it  to better meet its various stakeholders’ needs.
At the same time, IES’s research can be of limited usefulness to  policymakers and practitioners if it is not released in a timely manner.  Some stakeholders told us that research and evaluations supported by  IES may not be completed soon enough to inform the decision making of  policymakers and practitioners on important questions, which is a key  component of its mission. For example, officials from one constituency- based organization for policymakers told us that IES’s evaluation of  Education’s Race to the Top and School Improvement Grant Programs  are of great interest to states, but by the time the results of these  evaluations are released in 2014, states will not have much time to  implement lessons learned from these studies before their program’s  funding expires.
IES’s peer review process may also exacerbate timeliness concerns. In  order to ensure the high quality of IES’s work, ESRA requires IES- supported research reports to be peer reviewed before being published.
In recent years however, the time it takes to complete this review process  substantially increased, from an average of 117 days in fiscal year 2011  to 175 days in fiscal year 2012 and 150 days in fiscal year 2013. As  long ago as 2008, peer review timeliness was a concern of the NBES and  at that time, they recommended that IES establish procedures to ensure  the timely receipt of reports and revisions from its contractors. When  asked for explanations for the recent increase in its peer review  timeframes, senior IES officials told us that the timeliness of IES  contractor responses to peer review comments may still be a factor. They  also cited factors such as the complexity of the reports reviewed recently  and the time it took IES’s Centers to work with its contractors on suitable  responses to peer review comments. However, IES officials told us that  while the peer review office within IES monitors the time its own staff take  to review reports, the peer review office does not monitor the time IES’s  Centers or contractors take to respond to peer review comments. Such  monitoring would allow IES to take steps to mitigate delays, such as by  holding contractors more accountable. In accordance with federal internal  control standards, program managers should have access to and use  operational data to determine whether they are meeting their agencies’  goals for accountability for effective and efficient use of resources.
section_title: IES Has Taken Steps to Meet Stakeholders’ Needs, but Lacks a Formal Process to Gather Input on Its Research Agenda
paragraphs: To ensure that its research addresses the needs of a range of  stakeholders and to address concerns about how relevant its research is  to these stakeholders, IES is soliciting feedback from practitioners and  redesigning some existing programs. Current IES officials said that IES  has in the past 10 years established the quality of IES-supported  research, and that they are continuing to prioritize engagement with  policymakers and practitioners. Several stakeholders with whom we  spoke also noted that IES has recently devoted more attention to  policymaker and practitioner outreach to improve its relevance to the  education field. For example, IES recently convened a group of 17 state  and local education officials and other stakeholders to discuss the  strengths and weaknesses of products from the RELs and the What  Works Clearinghouse, a program that IES administers to evaluate the  merits of and disseminate education research evidence. According to  IES, at this meeting participants discussed their opinions on the usability,  relevance, and accessibility of current REL and What Works  Clearinghouse products. In addition, in June 2013, IES officials told us  they were planning to convene a small group of researchers to assess  potential gaps in the research conducted by two of its Centers: NCER and  NCSER. IES also recently held a discussion with its board of directors  about how to make its research more relevant to policymakers and  practitioners.
IES has also reorganized several of its existing grant programs and  initiated a new grant program to support research that will better target  and understand the needs of the education field. For example, one of  these grant programs focuses on collaborations between researchers and  state or local educational agencies, and in July 2013, six new grants were  awarded for collaborative research projects, such as the design of a  randomized postsecondary developmental education experiment in Texas  and the development of a system to monitor students’ social and  emotional learning in a school district in Nevada. Additionally, IES is  planning to fund a new R & D Center in fiscal year 2014 on knowledge  utilization to study how education researchers can make their work more  relevant and useful to practitioners and how practitioners can make  productive decisions based on research evidence.
Despite IES’s recent efforts to increase the relevance of its research, IES  does not have a structured process for incorporating feedback from  policymakers and practitioners into its research agenda. ESRA requires  that IES’s board of directors be composed of education researchers and  other stakeholders, including educators, policymakers, parents, or school  administrators. However, according to IES officials, IES has had  difficulty with the nomination and approval process for appointing board  members and, according to one long time board member, the NBES has,  at times, had difficulty retaining members from these stakeholder groups.  Further, according to IES officials, IES does not have in place any other  ongoing, structured outreach to stakeholders for input to its research  agenda, though the senior leaders of IES’s Centers and the Director of  IES periodically engage with policymakers and practitioners on specific  topics or projects. Inconsistent outreach by IES may have contributed to  gaps in its research. For example, stakeholders said that there is a  shortage of research using mixed methodologies that could allow for  shorter turnaround times among IES-funded research projects.
While obtaining and integrating policymaker and practitioner feedback  into a research agenda can be difficult, other research agencies have  external groups of practitioner-stakeholders to which they routinely turn  for perspectives on the overall direction of their research. Officials at  NICHD, for example, told us that, in addition to its statutorily required  National Advisory Council, they consult with an independent group called  the Friends of NICHD, comprised of organizations that use NICHD  research, to gather input on NICHD research plans and agendas.  Similarly, in addition to the National Science Board, NSF officials told us  that several advisory committees, composed of outside experts and  stakeholders, regularly provide recommendations regarding the direction  of the agency’s research. Though there is no single or ideal way for  government agencies to conduct research, we previously developed a  framework to identify key elements that promote a sound federal research  program, using guidelines from several leading national research  organizations. Within that framework, we found that agencies should  establish a structured process for developing their research and  evaluation priorities that considers key stakeholders’ input. For example,  we reported on one federal research agency that has procedures that call  for routinely consulting with stakeholders—including policymakers and  key institutions that influence public policy—about past research  accomplishments and program effectiveness and impact.
section_title: IES’s Performance Measures Do Not Reflect Its Current Programs
paragraphs: IES cannot demonstrate the impact of its efforts to improve the quality  and relevance of its research in some areas because its performance  measures have not been updated to reflect its current programs. Since  1993, all federal executive branch agencies have been required to set  strategic goals, measure performance, and report on the degree to which  goals were met in an effort to ensure government accountability and  enhance public awareness about agencies’ accomplishments. In  addition, according to federal internal control standards and leading  practices on performance management, agencies should establish  performance measures for their activities and continually compare actual  performance data against these goals. However, IES officials told us  that IES’s current performance measures, which were developed after the  agency was created by law in 2002, no longer capture the scope of IES’s  current research and priorities. Furthermore, in some cases, senior IES  officials told us these performance measures are no longer relevant to  managing the agency’s operations. For example, one performance  measure relies on the results of a survey of potential users of the What  Works Clearinghouse, but IES officials told us they have never conducted  this survey because they do not feel it would yield enough useful  information to be worth the investment. To measure the effectiveness of  its research grants, IES currently reports on a measure that counts the  number of IES-supported interventions that have been determined to be  effective in improving student outcomes in particular areas. While a new  performance measure was reported in Education’s fiscal year 2014  budget request for IES that better reflects the results of its recent  research, this measure still does not include certain areas, such as  research on the organization and management of schools and education  systems. In addition, IES does not publicly report on the overall  performance of the REL program, which constitutes one of the agency’s  largest investments. As we have reported, without performance measures  that include targets or goals to demonstrate results, agencies may be at  risk for failing to achieve their goals. IES officials told us they have  begun work on revising their agency’s overall performance measures.  Officials told us that they plan to include revised performance measures  in Education’s fiscal year 2015 budget request for IES, and that they have  begun discussions with OMB to establish these new measures. As of  August 2013, IES officials told us they intend for the new performance  measures they are developing to include all programs, including the REL  program and its new grant programs for researcher-practitioner  partnerships.
section_title: Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts
subsections: section_title: Groups Have Produced Research and Products for Use by Policymakers and Practitioners, but Relevance and Dissemination Could Be Improved
paragraphs: RELs, Comprehensive Centers, and R & D Centers have taken various  steps to provide relevant research to the education field. According to  statute, RELs and Comprehensive Centers are required disseminate  information that can be used by practitioners and policymakers to improve  academic achievement. Similarly, IES requires R & D Centers to  disseminate research to policymakers and practitioners to improve  teaching and learning, and ultimately, student achievement. To identify  topics of relevance to the education field, all three groups have engaged  policymakers and practitioners in planning research activities. For  example, beginning in 2012, RELs were required to conduct their work  through new or existing partnerships of practitioners, policymakers, and  others—called research alliances, which would work together to use data  and research to address specific problems in education. According to IES  officials, REL projects must be based on needs identified and agreed to  by the research alliances. For example, the REL that includes the Silicon  Valley area works with a research alliance consisting of local school  districts, county education officials, foundation leaders, university faculty,  and intervention specialists to boost math achievement. This alliance  began because local districts were concerned about low student  achievement in math, given its importance for the local technology  industry. With input from the alliance, this REL is developing and  analyzing strategies to ensure adequate preparation in math with the goal  of preparing students for postsecondary education.
Despite efforts by these groups to ensure the relevance of their work,  stakeholders, including teachers and policymakers, have raised concerns  about the applicability of some of the research and products these groups  have produced, as well as their timeliness. For example, a stakeholder  group we spoke with that represents local school districts, as well as two  superintendents we spoke with, said they did not find REL research as  relevant or as timely as other sources they turn to for research  information. Additionally, teachers we spoke with told us it would be  helpful if IES-supported groups produced more products that synthesize  research findings so that they are more applicable to classroom practice,  similar to research-based products they use from professional  associations or other intermediary organizations. Stakeholders also noted  that the timeliness of REL products has been a concern. REL research  findings cannot be released to the public until they have cleared the IES  peer review process, which as we noted earlier, has taken longer in  recent years. Some stakeholders expressed concern that in the  absence of timely and applicable information from IES, other entities may  provide practitioners and policymakers with research-based information  that may not be conducted with comparable research standards or be as  objective and unbiased as IES-supported research. In addition, the  research topics and the products produced by the R & D Centers  primarily reflect the priorities of researchers, according to many of the  researchers we spoke with, even though the Centers have multiple  audiences, including policymakers and practitioners. Stakeholder groups  representing policymakers and practitioners also said that the R & D  Centers could do more to adapt their research findings to formats readily  accessible by these audiences, such as by producing nontechnical  reports and shorter research summaries.
All three groups use a range of methods to disseminate their research  evidence and products, such as publications and conferences. For  example, in our survey of R & D Center directors, we found the common  methods they used to disseminate their research were academic journal  publications and presentations at conferences, as well as hosting  conferences. Comprehensive Centers disseminate products and tools  they produce by sharing them on their websites, via e-mail distribution  lists, and with other Comprehensive Centers. In addition, Comprehensive  Centers create summaries of current research for their state clients, and  also develop other technical assistance resources specifically for  teachers, such as professional development courses.
Although largely successful in reaching academicians, REL and R & D  Center dissemination efforts do not always ensure that research reaches  policymakers and practitioners. Regarding RELs, researchers and several  groups representing policymakers and practitioners told us that REL work  was not reaching these audiences, and teachers we spoke with were  generally unfamiliar with RELs. A few stakeholders also noted that REL  productivity varied widely, and that some have produced very few  reports. Stakeholders gave several reasons why RELs may struggle to  reach practitioner audiences. For example, some intermediaries—such as  industry associations—we spoke with said their organizations help to  disseminate research information to policymakers and practitioners.  However, some noted that further efforts are needed to leverage  intermediary groups to better market REL and R & D Center work to  reach IES’s target audiences.
section_title: IES Has Not Fully Assessed Groups’ Efforts to Disseminate Relevant Research
paragraphs: IES requires RELs to report some information about relevance and  dissemination that could be used to evaluate their efforts, but this  information has not been collected in a consistent manner and is  therefore difficult for IES to use to improve program management. For  example, RELs use a stakeholder feedback survey instrument to capture  feedback from participants on relevance and other aspects of the  products, activities, and events they sponsor. This survey includes a set  of questions for five different types of REL activities and products, such  as technical assistance workshops or research alliance participation.  However, as administered in 2012—the first year of the current RELs’  operation—RELs were not consistent in how they administered the  survey. For example, some RELs administered the complete survey for  their activity or product type, while other RELs selected specific questions  for respondents. The results, therefore, are not comparable across all  REL activities and products. Further, the data collected only applies to  participants who elected to provide feedback, rather than all participants  in REL programs or potential users.
In addition to the feedback survey, RELs are required to annually report  to IES on a series of indicators that they developed collaboratively with  IES in March 2012. However, this preliminary information is not sufficient  for IES’s management of RELs’ efforts to produce and disseminate  relevant research. The REL program has six expected outcomes, one of  which is to build a body of knowledge in topics that address regional  needs. To assess performance on this and the other five program  outcomes, RELs must report to IES on 24 indicators. For example,  regarding dissemination, RELs must report the number of events held  and the total number of attendees at these events. However, IES has not  established performance targets or goals for these indicators to help  RELs prioritize their activities or assess their performance in these areas.  Were IES to establish targets or goals, the RELs would likely have more  incentive to perform at the agency’s desired level, and IES would be  better positioned to determine if the RELs are meeting its expectations  and hold them more accountable for performance. Our previous work has  indicated agencies successful in measuring performance had  performance measures including targets or goals that (1) demonstrate  results; (2) are limited to the vital few; (3) cover multiple priorities; and (4)  provide useful information for decision making.
Regarding the R & D Centers, IES has few requirements for tracking  relevance and dissemination, although some Centers collect additional  information that could be helpful for assessing performance in these  areas. In combination with other factors, IES assesses the relevance of  the R & D Centers’ proposed research projects when making funding  decisions, but according to officials, this is the only formal assessment of  the relevance of R & D Center work. However, although they are not  required to do so, most R & D Center directors responding to our survey  said they use quantitative data to gauge their relevance to policymakers  and practitioners. For example, one R & D Center has conducted national  surveys of certain constituent groups, such as state policymakers, asking  about the relevance of the Center’s findings and products for their use. In  their annual performance reports, IES asks R & D Centers to report  whether they disseminated research information, but does not specify  what information they should provide, such as how they disseminated  information or the effectiveness of these efforts. Though it can be difficult  to demonstrate how research programs with broad dissemination goals— such as the R & D Centers—are able to inform others and contribute to  outcome-oriented goals, there are several evaluation strategies to assess  how programs with great breadth and flexibility contribute to agency  goals. IES has not, however, set utilization or dissemination goals and  objectives, or investigated whether dissemination strategies were  effective in improving the efforts of these groups.
According to officials, IES has no plans to conduct formal evaluations for  the current group of RELs and R & D Centers to comprehensively assess  their relevance and dissemination activities. IES is still in the process of  conducting a mandated evaluation of the prior group of RELs, whose  contracts ended in 2011. According to study design documents and IES  officials, this IES-contracted evaluation will include information from a  customer satisfaction survey of a representative sample of state and  district staff regarding their views of the relevance and usefulness of REL  work. The evaluation will also include an assessment by expert panelists  of the quality and relevance of selected REL reports. However, it will not  include an assessment of REL dissemination activities. This evaluation  began in 2009 and an interim report was initially expected to be released  in spring 2012; however, the interim report was delayed and was released  in late September 2013. IES has no further plans to evaluate the REL  program beyond the forthcoming evaluation of the prior group. Officials  told us although there was a clear requirement in ESRA for this  evaluation, they did not believe that the law required any subsequent  evaluation of the REL program. In addition, as of August 2013, IES had  no plans to conduct a formal evaluation of the R & D Center program, and  there is no requirement to do so. According to IES officials, they do not  plan to evaluate them because they do not view them as a program, but  rather as one vehicle for supporting IES’s programs of research, as with  its education research grants. However, research on dissemination  practices funded by other federal research programs has shown that  dissemination efforts should be assessed periodically to improve further  efforts to communicate key messages to target audiences.
As required by ESRA, IES has previously evaluated the Comprehensive  Centers. A 2011 IES-contracted evaluation reviewed how well the  Comprehensive Centers addressed state client needs, as well as the  quality, relevance, and usefulness of their assistance, among other  things. The evaluation included a survey of SEA administrators and a  review of Comprehensive Center products. Evaluation data were  collected annually in 3 of the 5 program years in which the Centers  operated. Comprehensive Centers were rated higher in each successive  year, and in 2008-2009, 56 percent of state managers surveyed said that  technical assistance from the Comprehensive Centers served the state’s  purposes completely. For the state managers who said the  Comprehensive Centers did not completely meet their needs, the main  reason cited was that Center staff had limited time to work with their state.  The evaluation also included a quality assessment of a sample of  Comprehensive Center projects by expert panelists and project  participants. Each sampled project was independently rated by a panel of  experts using a 5-point scale. SEA staff who participated in these projects  (e.g., by serving on a work group associated with the project) were asked  via survey to rate the relevance and usefulness of each project. For the  sample of projects assessed, the evaluation showed that the  Comprehensive Centers’ technical assistance was rated higher on each  measure in each successive year, from 2006-2007 to 2008-2009. On  average, across each of the 3 years, expert panels rated sampled project  materials from “moderate” to “high” for quality, and project participants  rated the sampled projects “high” for relevance and usefulness. While  these results indicate a positive trend in Comprehensive Centers’  performance, this evaluation covered the prior group of Comprehensive  Centers and IES recently released a new Request for Proposals to  evaluate the current group.
section_title: IES Coordinates with Other Federal Agencies, but Fragmented Evaluation Funding Poses Challenges for Education’s Evaluation Efforts
subsections: section_title: IES Coordinates with Other Agencies to Increase the Use of Evidence in Federal Decision Making and Research Planning
paragraphs: IES coordinates with relevant federal research agencies on projects to  increase federal agencies’ use of research evidence in guiding funding  decisions. According to federal agency officials and stakeholders we  spoke with, NICHD and NSF are the two other primary federal agencies  conducting education-related research. IES officials meet regularly with  their NICHD and NSF counterparts to jointly sponsor projects and ensure  their efforts are complementary. For example, IES co-led a joint  Education-NSF working group to develop common evidence guidelines  for education research and development. These guidelines are intended  to help guide NSF’s and IES’s respective decisions about investments in  education research and clarify for potential grantees and peer reviewers  the evidence expected from each type of study. Several researchers we  interviewed, as well as officials from three federal agencies we spoke  with, said these guidelines will benefit the education field. In addition,  developing common evidence guidelines is consistent with our key  practices for enhancing and sustaining collaboration among federal  agencies. In particular, we have found that establishing compatible  policies, procedures, and other means helps to align the partner  agencies’ activities, core processes, and resources to accomplish a  common outcome.
Consistent with this key practice, IES also coordinates with other federal  agencies to leverage its expertise in planning federal research agendas  on topics such as early learning and prisoner re-entry, as well as  increasing the quality of and access to data on education outcomes. For  example, IES sits on the Early Learning Policy Board, a joint Education- Health and Human Services body, to discuss ways to coordinate  federally-funded research on early childhood topics. IES’s statistics  Center leads the Interagency Working Group on Expanded Measures of  Enrollment and Attainment, to improve national data on education,  training, and credentials for work. This effort includes OMB, the Council of  Economic Advisors, and other federal statistical agencies, such as the  Census Bureau and the Bureau of Labor Statistics at the Department of  Labor. In addition, IES has worked with the Defense Advanced Research  Projects Agency at the Department of Defense on its Small Business  Innovation Research Program, and in December 2012 they released a  joint solicitation to support the development and evaluation of education  technology games.
IES’s efforts to help develop a systematic, transparent evidence review  process and incorporate a tiered evidence framework for awarding grants  for Education’s Investing in Innovation Fund grant program have had an  impact on the use of evidence in federal decision making both within and  outside the department. According to an official from the Administration of  Children and Families at the Department of Health and Human Services,  for some of its grant programs, the Administration of Children and  Families has recently used tiered evidence standards similar to those IES  helped to develop for Education’s Investing in Innovation Fund, reserving  the majority of funds for applicants proposing to implement service  models with evidence of effectiveness. IES is also working with Education  program offices to incorporate tiered evidence standards in awarding their  grants, such as the Strengthening Institutions Program in the Office of  Postsecondary Education.
section_title: Coordination between the RELs and Comprehensive Centers Has Improved
paragraphs: In addition to working with other federal agencies, IES coordinates within  Education to facilitate coordination between the REL and Comprehensive  Center programs. There is a statutory requirement for each  Comprehensive Center to coordinate its activities and collaborate with the  RELs in its region, as well as with certain other entities. Prior to the  current award cycle, IES and OESE restructured REL and  Comprehensive Center regions so that the number of Comprehensive  Centers in each REL region was reduced and aligned more closely with  REL regions. According to OESE officials, its Comprehensive Center  grantees and states found the lack of alignment between the regions in  the prior contract cycle to be a barrier to coordination. Further, IES has  provided guidance regarding its expectations for coordination between  RELs and Comprehensive Centers. For example, IES has encouraged  RELs to partner with Comprehensive Centers and has allowed RELs to  budget resources for these collaborative efforts.
According to directors of these groups, coordination has improved with  the current group of RELs and Comprehensive Centers, which began in  2012. One REL Director we spoke with told us that the realignment of  the regions has facilitated improved coordination between the groups. In  addition, some directors said that the RELs’ new structure of research  alliances has improved coordination. For example, some Comprehensive  Centers are members of REL research alliances and have partnered with  RELs to host “bridge” events, where state officials and researchers meet  to discuss research findings relevant to issues of policy and practice.
REL directors told us that although there can still be confusion among  some SEA officials about the appropriate role and tasks performed by  each, RELs and Comprehensive Centers have used several strategies to  reduce confusion and improve coordination. For example, some groups  have conducted joint visits with SEA officials so that they can describe  their respective roles and responsibilities and specifically discuss how  each group can address the state’s various technical assistance needs. In  addition, some directors described their groups’ efforts to sequence their  work for their state clients to better meet their needs. In some instances,  for example, the Comprehensive Center may conduct initial planning work  for a project, and the REL would later assist with any aspects of the  project requiring data analysis or original research. For example, one  Comprehensive Center director told us that, through joint efforts with the  REL in their region, it developed a framework for measuring teacher  effectiveness. For this project, the REL formed a research alliance with  other Comprehensive Centers and SEAs interested in piloting the  framework. The Comprehensive Center is helping states implement the  pilot, and the REL will conduct an evaluation of the pilot projects. This  Center Director told us that it will help the states adjust their strategies  based on the findings from the pilot evaluation.
section_title: Education’s Inability to Combine Evaluation Funding Poses Challenges for Evaluation Planning Efforts
paragraphs: IES coordinates within the department to plan evaluations of Education  programs. Beginning in 2010, OPEPD and IES have jointly led an annual  department-wide evaluation planning process to identify evaluation  projects to conduct. Although there is no official guidance delineating  these roles, Education officials said that the annual evaluation planning  process is used to help determine which of these offices will conduct each  project. According to Education, this planning process has helped to  improve the quality of evaluations conducted, and made evaluations more  responsive to the information needs of the department. For example, IES  is conducting an implementation and quasi-experimental evaluation of the  Race to the Top Program. An IES official reported that input from the  Education program office running this program helped ensure that the  evaluation’s data collection instruments focused on implementation  issues most important to Education and state education leaders. IES is  also working with this program office to select topics of greatest  immediate interest to Education for short implementation briefs to be  released as the study progresses.
However, efforts to coordinate internally and prioritize evaluation research  projects through this annual process are challenged in part by statutory  requirements related to ESEA evaluation funding. For these programs,  evaluation funds are typically set aside as a percentage of program  funding or as national activities funds. With some exceptions, ESEA  authorizes Education to reserve up to 0.5 percent of the amount  appropriated to carry out each program or project authorized under that  Act to conduct evaluations. However, according to Education officials,  due to the statutory limitation, the funds available for evaluation are often  insufficient to conduct high-quality evaluations of these programs. This  is particularly true with respect to certain discretionary grant programs  where overall program funding is itself relatively small. As a result,  OPEPD and IES officials said that some evaluations, including high- priority evaluations, may not occur. For example, IES officials told us that  the department is interested in conducting evaluations of the Mathematics  and Science Partnerships and Promise Neighborhoods programs, but  there is not sufficient evaluation funding under these programs due to the  statutory limitation and Education does not have the authority to combine  evaluation funds across the department and use them to conduct  evaluations of these or other high-priority programs.
We have previously reported that many Education programs, especially  smaller programs, have not been evaluated, which can limit the ability of  Congress to make informed decisions about which programs to continue,  expand, modify, consolidate, or eliminate. For example, in 2009, we  reported that 11 of Education’s programs focused on teacher quality had  been operating for over seven years and had never been evaluated. In  addition, according to OPEPD officials, ideas for some high-priority  evaluations that department officials have discussed during the annual  evaluation planning process are never developed into plans because  officials know that evaluation funds are not available to carry them out.  However, according to officials from OPEPD, beginning in fiscal year  2015, Education will maintain a list of all of the department’s evaluation  needs should funds for conducting these evaluations become available.  According to a senior IES official, if the department were able to combine  funds authorized to be used for evaluation it would have more flexibility to  conduct appropriate evaluations of any of ESEA programs that need  evaluation. The President’s Fiscal Year 2014 Budget request for  Education contained a proposal to increase Education’s flexibility to  conduct program evaluations by allowing the department to use funds  from certain programs across the department for this purpose. An OMB  staff member told us that the ability to combine evaluation funds and  prioritize evaluations is critical in the current environment of constrained  funding to help make the best use of limited resources and to fill gaps in  topic areas where research is needed. In fiscal year 2012, Congress  granted the Department of Labor temporary authority to combine  program-specific funds to support evaluations of its programs. The  Department of Labor is required to provide a plan to the congressional  appropriations committees before combining its funds in this manner. This  plan is to describe the specific evaluations it plans to carry out using  these funds and helps ensure transparency and accountability in the  department’s use of this authority. In the Department of Labor’s fiscal  year 2014 budget justification, it reported that the availability of flexible  evaluation funds has allowed the department to evaluate large and high- priority programs, as well as programs that may not have been fully  evaluated previously.
section_title: Conclusions
paragraphs: Since its creation more than a decade ago, IES has made significant  contributions to strengthening the rigor of the education research field and  has promoted the use of scientifically based research in our nation’s  education system. However, IES’s ability to release timely information to  policymakers and practitioners is exacerbated by its peer review process  which, while promoting rigorous research, can take a substantial amount  of time. Without better use of available data to manage this peer review  process, it will remain difficult for IES officials to identify potential causes  of these delays and develop strategies to mitigate these challenges. More  recently, the agency has begun working to make its products more  relevant to a range of stakeholders, particularly policymakers and  practitioners. Producing education research that is relevant for these  stakeholders is critical to inform decision making in Congress and in the  field about key federal investments in education. Despite progress IES  has made in ensuring that its research is relevant to its stakeholders, IES  does not have a formal, structured process for engaging with these  groups in the research-agenda setting process. Until such a process is in  place, IES limits its ability to ensure the relevance, and ultimately,  usefulness of its research to the field. IES’s relevance to the field is  unclear in other ways as well. For example, without publicly reported  performance measures for key investments—such as the RELs or new  high-priority grant programs—stakeholders, such as Congress and the  public, are unable to determine the effectiveness of IES’s activities or  evaluate the merits of the federal investment in these programs.
IES’s research and technical assistance grantees and contractors take a  number of steps to help ensure they produce and disseminate relevant  research and products. However, without assessing the effectiveness of  these groups’ dissemination efforts, IES may miss opportunities to think  more strategically about dissemination and ensure that these groups are  using the most effective strategies to reach their target audiences.  Providing Education with the authority to combine funds authorized for  evaluations under ESEA would allow the department more flexibility to  prioritize and conduct effective evaluations of programs that represent  key federal investments and critical subject areas. Without this authority,  it will be difficult for Education to make the best use of its limited  resources and to support Congress in making informed decisions about  which programs to continue, expand, modify, consolidate, or eliminate.
section_title: Matter for Congressional Consideration
paragraphs: To ensure that Education can conduct critical program evaluation work,  Congress should consider granting Education authority to combine funds  authorized for evaluations of ESEA programs and target them to high- priority evaluations, with appropriate measures to ensure transparency  and accountability for how the funds will be used.
section_title: Recommendations for Executive Action
paragraphs: To improve the management and accountability of IES’s research and  evaluation efforts, we are making the following recommendations:  1.  IES should use available data to routinely monitor all stages of its  peer review process and identify opportunities to improve the  timeliness of its reviews.  2.  IES should develop a structured process to systematically gather  input from policymakers and practitioners and use this input when  developing its research agenda.  3.  IES should develop performance measures, including targets and  goals that clearly demonstrate results, and that reflect its current  programs and all key agency activities, such as the performance of  the RELs and its new grant programs supporting researcher- practitioner partnerships.  4.  In order to identify leading practices and target areas for  improvement, IES should assess the effectiveness of REL and R & D  Center dissemination strategies by, for example, collecting consistent  data and lessons learned from these groups to inform future  dissemination efforts.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Education for  review and comment. Their comments are reproduced in appendix II.  Education officials also provided technical comments, which we  incorporated into the final report, as appropriate.
Education generally agreed with our recommendations and has several  plans in place to address them. Specifically, IES noted that it continues to  be committed to producing high-quality reports while shortening the time  period for its peer review process, and stated that it will develop a system  to more comprehensively monitor all of the stages in its report production  process so that it is able to identify and address potential problems more  easily and quickly. To further emphasize its commitment to obtaining  practitioner and policymaker input, IES also said it will begin an annual  solicitation for practitioner and policymaker input on its research agenda  by posting an invitation on its web site and widely advertising the  opportunity through its IES Newsflash email notices.
Additionally, as we noted in our report, IES commented that it is currently  working to align its performance measures with its current investments,  including developing measures for the REL program. Finally, IES agreed  that it is important to measure and assess the effectiveness of IES- supported dissemination activities. IES noted it will build knowledge in this  area through its fiscal year 2014 request for applications for a new R & D  Center focused on knowledge utilization to develop measures for how  practitioners use research evidence to guide what they do in schools and  classrooms. We support IES’s continued focus and emphasis on  advancing knowledge and providing usable information to improve  education policy and practice.
We are sending copies of this report to relevant congressional  committees and the Secretary of Education. In addition, the report will be  available at no charge on GAO’s website at http://www.gao.gov.
If you or our staffs have any questions about this report, please contact  me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our review examined (1) the extent to which the Institute for Education  Sciences (IES) demonstrates its ability to support high-quality research  and fulfill its mission; (2) the extent to which selected Department of  Education (Education) research and technical assistance groups  disseminate relevant products to the field; and (3) how IES coordinates its  activities with other relevant federal research entities and within  Education. For all three objectives, we reviewed the Education Sciences  Reform Act of 2002 (ESRA) to identify statutory requirements related to  IES management, processes, and procedures. We obtained and  analyzed agency documents detailing IES’s processes and performance.  We also conducted interviews with senior IES officials, including  Commissioners at all four IES Centers as well as selected program staff,  several current and former members of IES’s advisory board, the National  Board for Education Sciences, and selected former senior IES officials to  gather additional perspectives on IES’s management and performance  over time.
Specifically, to assess the extent to which IES has demonstrated its ability  to support high-quality research and fulfill its mission, we reviewed  documents and data from IES’s Standards and Review Office, which  administers the peer review of IES reports and grant applications. In part  to evaluate IES’s timeliness in producing relevant research, we obtained  IES data for the most recent 4 years, fiscal years 2010 through 2013, on  the average number of work days from the submission of the initial  manuscript to final approval for reports sponsored by IES that underwent  external peer review before being publicly released. We assessed the  reliability of IES data by obtaining written responses from agency officials  knowledgeable about the data. We determined that the data were  sufficiently reliable for the purposes of this report. To further assess IES’s  overall processes and operations, we applied a framework GAO  previously developed to evaluate the soundness of federal research and  evaluation programs. This framework includes five key phases of the  research and evaluation process: (1) agenda setting, (2) selecting  research, (3) designing research, (4) conducting research, and (5)  disseminating research results. While we assessed IES’s activities in  each of the five phases, we primarily focused on the first and fifth phases  of this process: agenda setting and disseminating research, as these two  phases most directly addressed our research questions.
Additionally, we reviewed publicly reported performance information  contained in Education’s most recent budget request to Congress for IES  to compare performance trends over time and to assess the  completeness of IES’s performance measures against GAO’s leading  practices for performance management and federal standards for internal  control in the areas of information and communications, and performance  reporting and monitoring. We also interviewed officials at the research  and evaluation arm of the Administration for Children and Families at the  Department of Health and Human Services, and at the Office of  Management and Budget that were recommended to us as  knowledgeable of federal research and performance reporting to obtain  their insight on IES’s research, evaluation, and performance  measurement. Lastly, we observed two public meetings of the National  Board for Education Sciences that took place during the course of our  review on October 5, 2012, and February 22, 2013.
We also conducted more than 40 interviews with several categories of  IES stakeholders: (1) education researchers; (2) organizations  representing education policymakers and state-level administrators; and  (3) practitioner organizations, individual teachers, and district leaders. To  identify appropriate stakeholders in all three categories, we obtained  recommendations on individuals or groups to contact from key  organizations, contacted those individuals or groups, and obtained further  recommendations from them on additional individuals or groups to  contact. We continued to ask for names from the subsequent individuals  and organizations until we began getting duplicate referrals. We also  identified individuals and groups to contact through our prior work on  education research and internal expertise. For the education researchers,  we sought to identify education researchers that were considered to be  knowledgeable about education research and the activities of IES and its  predecessor organizations. For the education policymaker organizations,  we ensured representation from groups that represent the wide range of  topical areas in which IES funds research and evaluation, such as early  childhood education, special education, and postsecondary education.  We interviewed teachers and superintendents because they are part of  IES’s target population of end-users, though they had various levels of  familiarity with IES prior to our interview.
We conducted two discussion groups with experienced education  researchers at the 2013 Annual Conference of the American Educational  Research Association (AERA). We assembled senior researchers for  these discussion groups with the help of AERA staff. Invitations to  participate in these discussion groups were provided to selected  members of the AERA Fellows program that were culled by AERA staff to  ensure representation from a balanced range of subject matter areas.  During these discussion groups, we met with a total of 16 senior  education researchers and discussed issues such as IES’s research  standards, peer review process, and the relevance and dissemination  practices of IES overall, as well as those of selected IES research and  technical assistance groups, specifically—the Regional Educational  Laboratories (REL) and Research & Development Centers (R & D  Center).
To address our second research objective, we focused on three types of  research and technical assistance groups. Two are administered by  IES—the RELs and R & D Centers—and one is administered by the  Office of Elementary and Secondary Education (OESE)—the  Comprehensive Technical Assistance Centers (Comprehensive Centers).  We selected these three research and technical assistance groups  because they are all authorized by the same law that established IES.  Moreover, there is a statutory requirement for the Comprehensive  Centers to coordinate their activities and to collaborate with the RELs,  among others. Specifically, to assess the extent to which these groups  disseminate relevant products to the education field, we obtained and  analyzed the most recent annual performance report submitted to IES  and OESE, respectively, from each of these groups. We analyzed these  documents to identify what quantitative and qualitative information IES  collects about the relevance, dissemination, and utilization of the research  and products these groups produce. We also interviewed Directors of the  RELs and the Comprehensive Centers as well as program staff at IES  and OESE that administer the programs to discuss relevance,  dissemination, and coordination activities. To further understand the  Comprehensive Centers activities related to the production and  dissemination of relevant research and products to the field, we reviewed  a contractor-led evaluation of the Comprehensive Center Program. In  particular, we reviewed findings related to relevance of the products these  groups produced and the levels of satisfaction reported by their  customers. We also reviewed early study design documents and an  interim report for an ongoing evaluation of the REL program. We  reviewed the scope and methodology for each of these evaluations and  determined they were sufficiently reliable for our reporting purposes. We  also obtained information from IES staff responsible for overseeing these  evaluations to obtain additional information.
We administered a short survey to the Directors of IES’s 18 R & D  Centers that were operational in the most recent full program year (2011- 2012) at the time of our review. We asked questions about activities the   R & D Centers undertake to disseminate relevant research to the field  and what qualitative and quantitative data they collect on their activities.  We conducted pretests to check that (1) the questions were clear and  unambiguous, (2) terminology was used correctly, (3) the survey did not  place an undue burden on respondents, (4) the information could be  feasibly obtained, and (5) the survey was comprehensive and unbiased.  We conducted two pretests with R & D Centers, one grantee of the  National Center for Education Research and one grantee of the National  Center for Special Education Research. We made changes to the survey  after both pretests based on the feedback we received. We sent the  survey by e-mail in an attached Microsoft Word form on April 2, 2013.  Surveys were completed by 17 of 18 R & D Center Directors, for a 94  percent response rate, by May 10, 2013. Because this was not a sample  survey, it has no sampling errors. However, the practical difficulties of  conducting any survey may introduce errors, commonly referred to as  nonsampling errors. For example, difficulties in interpreting a particular  question, sources of information available to respondents, or entering  data into a database or analyzing them can introduce unwanted variability  into the survey results. We took steps in developing the questionnaire,  collecting the data, and analyzing them to minimize such nonsampling  error.
To describe how IES coordinates with other relevant federal research  agencies and within the department, we reviewed pertinent documents  and statutory requirements for coordination contained in ESRA and  compared agency efforts to criteria we previously developed for practices  agencies can use to help enhance and sustain interagency collaboration.  To identify relevant federal research agencies, we obtained  recommendations on federal research agencies that conduct education- related research during interviews with knowledgeable stakeholders. We  continued to ask for recommendations during stakeholder interviews until  we established that the vast majority of recommendations identified two  other federal research agencies: the education directorate at the National  Science Foundation (NSF) and the Eunice Kennedy Shriver National  Institute of Child Health and Human Development (NICHD), an arm of the  National Institutes of Health. While stakeholders and officials at IES, NSF,  and NICHD noted some key differences between these agencies, such as  the types of research funded, there was general consensus that there  were clear commonalities in the research missions and some of the topics  of research funded by all three agencies. As such, we interviewed officials  at NSF and NICHD about their processes and procedures as well as their  coordination with IES, asking for examples of coordination activities and  any potential areas for improvement. Additionally, during our interviews  with IES officials, we asked about their coordination activities with NSF  and NICHD. To describe IES’s coordination within the department, we  reviewed agency documents such as those describing the annual  evaluation planning process at Education and conducted interviews with  IES officials and officials responsible for evaluation and research within  the department’s Office of Program Evaluation and Policy Development.  Finally, we also asked about coordination activities between the REL and  Comprehensive Centers programs during our interviews with program  staff administering these programs, as well as during our interviews with  Directors of these groups, to assess the extent and nature of the  coordination between these technical assistance programs.
We conducted this performance audit from July 2012 to December 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Education
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Scott Spicer, Assistant Director;  Lucas Alvarez, Nora Boretti, and Dana Hopings made significant  contributions to this report. Also contributing to this report were Sandra  Baxter, Deborah Bland, David Chrisinger, Elizabeth Curda, Helen  Desaulniers, Sheila McCoy, Jean McSween, Mimi Nguyen, James  Rebbe, and Sarah Veale.
section_title: Related GAO Products
paragraphs: Education Research: Preliminary Observations on the Institute of  Education Sciences’ Research and Evaluation Efforts. GAO-13-852T.  Washington, D.C.: September 10, 2013.
Managing for Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Government  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.
Employment and Training Administration: More Actions Needed to  Improve Transparency and Accountability of Its Research Program.  GAO-11-285. Washington, D.C.: March 15, 2011.
Program Evaluation: Experienced Agencies Follow a Similar Model for  Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.
Department of Education: Improved Dissemination and Timely Product  Release Would Enhance the Usefulness of the What Works  Clearinghouse. GAO-10-644. Washington, D.C.: July 23, 2010.
Employment and Training Administration: Increased Authority and  Accountability Could Improve Research Program. GAO-10-243.  Washington, D.C.: January 29, 2010.
Program Evaluation: Strategies for Assessing How Information  Dissemination Contributes to Agency Goals. GAO-02-923. Washington,  D.C.: September 30, 2002.
Education Research: Education Should Improve Assessments of R&D  Centers, Regional Labs, and Comprehensive Centers. GAO-02-190.  Washington, D.C.: January 24, 2002.