Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts
Released: 2013-12-05T00:00:00

Letter

Background

Prior to the establishment of IES, federal education research was carried out by the Office of Educational Research and Improvement, a federal agency that—according to observers both within and outside the federal government—was challenged by frequent leadership changes, competing interests, and lack of a focused mission. The 2002 reauthorization of the Elementary and Secondary Education Act (ESEA) made numerous references to the use of research-based evidence in educational decision making and federal education programs. With its enactment later in 2002, ESRA established IES, an arm of the Department of Education dedicated to the improvement of federal education research, statistics, evaluation, information, and dissemination.

IES’s mission is “to provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study, in order to provide parents, educators, students, researchers, policymakers, and the general public with reliable information about (1) the condition and progress of education in the United States, including early childhood education; (2) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (3) the effectiveness of Federal and other education programs.” Furthermore, in carrying out its mission, ESRA calls upon IES to “compile statistics, develop products, and conduct research, evaluations, and wide dissemination activities in areas of demonstrated national need.”

ESRA authorizes IES to conduct and support many different types of research and evaluations in support of its mission. Specifically, ESRA contains several key provisions related to the management, core functions, and processes of IES: all research conducted by IES is to use scientifically based research standards that include, where appropriate, making claims of causal relationships only in random assignment experiments; education evaluations conducted by IES are to employ experimental designs using random assignment, when feasible; all research, statistics, and evaluation reports conducted by or supported through IES must be subjected to rigorous peer review before being published or otherwise made available to the public; the Director of IES will be appointed by the President, by and with consent of the Senate, for a 6-year term and will propose the Institute’s overall research priorities and report biennially to Congress and others on the Institute’s activities, among other duties; and a board of directors is established—the NBES—whose duties include, among other things, (1) advising and consulting with the Director of IES regarding its policies and approving the Director’s overall research priorities, (2) reviewing and approving procedures for peer review, and (3) reviewing the work of IES to ensure the consistency of scientifically valid research.

All of IES’s work is carried out by four Centers, created by law, that exist within IES. See table 1 for more information on these Centers and their key activities.

Education administers programs that support education research and technical assistance through grants and contracts involving several research groups—the RELs and R & D Centers within IES and the Comprehensive Centers within OESE (see fig. 1).

The National Center for Education Evaluation and Regional Assistance (NCEERA) administers the REL program, a network of 10 regional entities that conduct applied research; develop and disseminate research and products; and conduct technical assistance and other activities to support the needs of state and local educational agencies in their region. The RELs were first established in 1965 under Title IV of ESEA with the broad goal of supporting general educational improvement efforts in their regions. Since IES was created and began to administer the REL program, the RELs have completed two contract cycles, and the current RELs were launched in 2012 under new 5-year contracts.

ESRA also includes specific requirements for the administration of National R & D Centers, which are designed to address areas of national need and each of which addresses at least one of the broad research topics outlined in the law. R & D Centers are also responsible for the production of rigorous evidence and dissemination of products that provide practical solutions to important education problems in the United States. Each R & D Center is funded for no more than 5 years, and in fiscal year 2013, the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER) administered 18 R & D Centers in total.

The Comprehensive Centers Program is a network of 22 technical assistance grantees that help to increase the capacity of state educational agencies (SEA) to assist districts and schools in meeting student achievement goals. Established in 1994, Comprehensive Centers provide services primarily to SEAs to enable them to assist school districts and schools, especially low-performing schools. Comprehensive Centers provide training and technical assistance in (1) the implementation and administration of programs authorized under ESEA and (2) the use of research-based information and strategies. Selected Comprehensive Centers focus on specific content areas and produce research-based information and products for use by SEAs.

In addition to IES, other entities conduct education-related research and evaluations, and ESRA includes general requirements for the Director of IES to coordinate its research and evaluation work with these entities, both within Education and across the rest of the federal government.

Within Education, the Office of Planning, Evaluation, and Policy Development (OPEPD) conducts analyses and program evaluations on behalf of the department. Other federal agencies, such as NSF and NICHD—part of the National Institutes of Health—also support education- related research and the directors of these agencies serve as nonvoting ex officio members on the NBES. In addition, many nongovernmental organizations conduct education-related research and evaluations that inform general knowledge and understanding of educational policies, programs, and practice. For example, foundations such as the Bill & Melinda Gates Foundation, The William T. Grant Foundation, and The Spencer Foundation, conduct education research or program evaluation in specific topic areas to advance the public interest or their organization’s mission or goals. Additionally, other research organizations such as Mathematica Policy Research, MDRC, and SRI International, contract with the federal government or other clients to conduct agreed upon research or program evaluations on their behalf.

IES Supports High- Quality Research, but Lacks Key Processes and Performance Measures in Some Areas

IES Supports High-Quality Research, but Does Not Monitor the Timeliness of Some Aspects of Its Peer Review Process

IES has substantially improved the education research field. In 2007, the Office of Management and Budget (OMB) assessed IES’s research and concluded that IES had transformed the quality and rigor of research within Education and increased the demand for scientifically based evidence of effectiveness in the education field as a whole. Likewise, many stakeholders told us that IES’s research standards had improved Education’s research and had a positive influence on education research generally. More specifically, several stakeholders told us that IES products, such as its publications of education statistics reports, were useful for their work. In addition, one regional comprehensive center director told us that a Practice Guide IES released on dropout prevention has become the framework that one of the states in its region is using for dropout prevention efforts statewide. While IES’s research grants and evaluations have resulted in many randomized controlled studies since the agency was established over 10 years ago, its research standards also include guidelines for the implementation of other rigorous research methodologies, and it has recently funded studies using those methodologies, such as regression discontinuity or single-case designs. Further, IES officials also described a variety of other types of research that the agency supports, such as data analyses and correlational analyses. IES’s support of these multiple types of methodologies allows it to better meet its various stakeholders’ needs.

At the same time, IES’s research can be of limited usefulness to policymakers and practitioners if it is not released in a timely manner. Some stakeholders told us that research and evaluations supported by IES may not be completed soon enough to inform the decision making of policymakers and practitioners on important questions, which is a key component of its mission. For example, officials from one constituency- based organization for policymakers told us that IES’s evaluation of Education’s Race to the Top and School Improvement Grant Programs are of great interest to states, but by the time the results of these evaluations are released in 2014, states will not have much time to implement lessons learned from these studies before their program’s funding expires.

IES’s peer review process may also exacerbate timeliness concerns. In order to ensure the high quality of IES’s work, ESRA requires IES- supported research reports to be peer reviewed before being published.

In recent years however, the time it takes to complete this review process substantially increased, from an average of 117 days in fiscal year 2011 to 175 days in fiscal year 2012 and 150 days in fiscal year 2013. As long ago as 2008, peer review timeliness was a concern of the NBES and at that time, they recommended that IES establish procedures to ensure the timely receipt of reports and revisions from its contractors. When asked for explanations for the recent increase in its peer review timeframes, senior IES officials told us that the timeliness of IES contractor responses to peer review comments may still be a factor. They also cited factors such as the complexity of the reports reviewed recently and the time it took IES’s Centers to work with its contractors on suitable responses to peer review comments. However, IES officials told us that while the peer review office within IES monitors the time its own staff take to review reports, the peer review office does not monitor the time IES’s Centers or contractors take to respond to peer review comments. Such monitoring would allow IES to take steps to mitigate delays, such as by holding contractors more accountable. In accordance with federal internal control standards, program managers should have access to and use operational data to determine whether they are meeting their agencies’ goals for accountability for effective and efficient use of resources.

IES Has Taken Steps to Meet Stakeholders’ Needs, but Lacks a Formal Process to Gather Input on Its Research Agenda

To ensure that its research addresses the needs of a range of stakeholders and to address concerns about how relevant its research is to these stakeholders, IES is soliciting feedback from practitioners and redesigning some existing programs. Current IES officials said that IES has in the past 10 years established the quality of IES-supported research, and that they are continuing to prioritize engagement with policymakers and practitioners. Several stakeholders with whom we spoke also noted that IES has recently devoted more attention to policymaker and practitioner outreach to improve its relevance to the education field. For example, IES recently convened a group of 17 state and local education officials and other stakeholders to discuss the strengths and weaknesses of products from the RELs and the What Works Clearinghouse, a program that IES administers to evaluate the merits of and disseminate education research evidence. According to IES, at this meeting participants discussed their opinions on the usability, relevance, and accessibility of current REL and What Works Clearinghouse products. In addition, in June 2013, IES officials told us they were planning to convene a small group of researchers to assess potential gaps in the research conducted by two of its Centers: NCER and NCSER. IES also recently held a discussion with its board of directors about how to make its research more relevant to policymakers and practitioners.

IES has also reorganized several of its existing grant programs and initiated a new grant program to support research that will better target and understand the needs of the education field. For example, one of these grant programs focuses on collaborations between researchers and state or local educational agencies, and in July 2013, six new grants were awarded for collaborative research projects, such as the design of a randomized postsecondary developmental education experiment in Texas and the development of a system to monitor students’ social and emotional learning in a school district in Nevada. Additionally, IES is planning to fund a new R & D Center in fiscal year 2014 on knowledge utilization to study how education researchers can make their work more relevant and useful to practitioners and how practitioners can make productive decisions based on research evidence.

Despite IES’s recent efforts to increase the relevance of its research, IES does not have a structured process for incorporating feedback from policymakers and practitioners into its research agenda. ESRA requires that IES’s board of directors be composed of education researchers and other stakeholders, including educators, policymakers, parents, or school administrators. However, according to IES officials, IES has had difficulty with the nomination and approval process for appointing board members and, according to one long time board member, the NBES has, at times, had difficulty retaining members from these stakeholder groups. Further, according to IES officials, IES does not have in place any other ongoing, structured outreach to stakeholders for input to its research agenda, though the senior leaders of IES’s Centers and the Director of IES periodically engage with policymakers and practitioners on specific topics or projects. Inconsistent outreach by IES may have contributed to gaps in its research. For example, stakeholders said that there is a shortage of research using mixed methodologies that could allow for shorter turnaround times among IES-funded research projects.

While obtaining and integrating policymaker and practitioner feedback into a research agenda can be difficult, other research agencies have external groups of practitioner-stakeholders to which they routinely turn for perspectives on the overall direction of their research. Officials at NICHD, for example, told us that, in addition to its statutorily required National Advisory Council, they consult with an independent group called the Friends of NICHD, comprised of organizations that use NICHD research, to gather input on NICHD research plans and agendas. Similarly, in addition to the National Science Board, NSF officials told us that several advisory committees, composed of outside experts and stakeholders, regularly provide recommendations regarding the direction of the agency’s research. Though there is no single or ideal way for government agencies to conduct research, we previously developed a framework to identify key elements that promote a sound federal research program, using guidelines from several leading national research organizations. Within that framework, we found that agencies should establish a structured process for developing their research and evaluation priorities that considers key stakeholders’ input. For example, we reported on one federal research agency that has procedures that call for routinely consulting with stakeholders—including policymakers and key institutions that influence public policy—about past research accomplishments and program effectiveness and impact.

IES’s Performance Measures Do Not Reflect Its Current Programs

IES cannot demonstrate the impact of its efforts to improve the quality and relevance of its research in some areas because its performance measures have not been updated to reflect its current programs. Since 1993, all federal executive branch agencies have been required to set strategic goals, measure performance, and report on the degree to which goals were met in an effort to ensure government accountability and enhance public awareness about agencies’ accomplishments. In addition, according to federal internal control standards and leading practices on performance management, agencies should establish performance measures for their activities and continually compare actual performance data against these goals. However, IES officials told us that IES’s current performance measures, which were developed after the agency was created by law in 2002, no longer capture the scope of IES’s current research and priorities. Furthermore, in some cases, senior IES officials told us these performance measures are no longer relevant to managing the agency’s operations. For example, one performance measure relies on the results of a survey of potential users of the What Works Clearinghouse, but IES officials told us they have never conducted this survey because they do not feel it would yield enough useful information to be worth the investment. To measure the effectiveness of its research grants, IES currently reports on a measure that counts the number of IES-supported interventions that have been determined to be effective in improving student outcomes in particular areas. While a new performance measure was reported in Education’s fiscal year 2014 budget request for IES that better reflects the results of its recent research, this measure still does not include certain areas, such as research on the organization and management of schools and education systems. In addition, IES does not publicly report on the overall performance of the REL program, which constitutes one of the agency’s largest investments. As we have reported, without performance measures that include targets or goals to demonstrate results, agencies may be at risk for failing to achieve their goals. IES officials told us they have begun work on revising their agency’s overall performance measures. Officials told us that they plan to include revised performance measures in Education’s fiscal year 2015 budget request for IES, and that they have begun discussions with OMB to establish these new measures. As of August 2013, IES officials told us they intend for the new performance measures they are developing to include all programs, including the REL program and its new grant programs for researcher-practitioner partnerships.

Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts

Groups Have Produced Research and Products for Use by Policymakers and Practitioners, but Relevance and Dissemination Could Be Improved

RELs, Comprehensive Centers, and R & D Centers have taken various steps to provide relevant research to the education field. According to statute, RELs and Comprehensive Centers are required disseminate information that can be used by practitioners and policymakers to improve academic achievement. Similarly, IES requires R & D Centers to disseminate research to policymakers and practitioners to improve teaching and learning, and ultimately, student achievement. To identify topics of relevance to the education field, all three groups have engaged policymakers and practitioners in planning research activities. For example, beginning in 2012, RELs were required to conduct their work through new or existing partnerships of practitioners, policymakers, and others—called research alliances, which would work together to use data and research to address specific problems in education. According to IES officials, REL projects must be based on needs identified and agreed to by the research alliances. For example, the REL that includes the Silicon Valley area works with a research alliance consisting of local school districts, county education officials, foundation leaders, university faculty, and intervention specialists to boost math achievement. This alliance began because local districts were concerned about low student achievement in math, given its importance for the local technology industry. With input from the alliance, this REL is developing and analyzing strategies to ensure adequate preparation in math with the goal of preparing students for postsecondary education.

Despite efforts by these groups to ensure the relevance of their work, stakeholders, including teachers and policymakers, have raised concerns about the applicability of some of the research and products these groups have produced, as well as their timeliness. For example, a stakeholder group we spoke with that represents local school districts, as well as two superintendents we spoke with, said they did not find REL research as relevant or as timely as other sources they turn to for research information. Additionally, teachers we spoke with told us it would be helpful if IES-supported groups produced more products that synthesize research findings so that they are more applicable to classroom practice, similar to research-based products they use from professional associations or other intermediary organizations. Stakeholders also noted that the timeliness of REL products has been a concern. REL research findings cannot be released to the public until they have cleared the IES peer review process, which as we noted earlier, has taken longer in recent years. Some stakeholders expressed concern that in the absence of timely and applicable information from IES, other entities may provide practitioners and policymakers with research-based information that may not be conducted with comparable research standards or be as objective and unbiased as IES-supported research. In addition, the research topics and the products produced by the R & D Centers primarily reflect the priorities of researchers, according to many of the researchers we spoke with, even though the Centers have multiple audiences, including policymakers and practitioners. Stakeholder groups representing policymakers and practitioners also said that the R & D Centers could do more to adapt their research findings to formats readily accessible by these audiences, such as by producing nontechnical reports and shorter research summaries.

All three groups use a range of methods to disseminate their research evidence and products, such as publications and conferences. For example, in our survey of R & D Center directors, we found the common methods they used to disseminate their research were academic journal publications and presentations at conferences, as well as hosting conferences. Comprehensive Centers disseminate products and tools they produce by sharing them on their websites, via e-mail distribution lists, and with other Comprehensive Centers. In addition, Comprehensive Centers create summaries of current research for their state clients, and also develop other technical assistance resources specifically for teachers, such as professional development courses.

Although largely successful in reaching academicians, REL and R & D Center dissemination efforts do not always ensure that research reaches policymakers and practitioners. Regarding RELs, researchers and several groups representing policymakers and practitioners told us that REL work was not reaching these audiences, and teachers we spoke with were generally unfamiliar with RELs. A few stakeholders also noted that REL productivity varied widely, and that some have produced very few reports. Stakeholders gave several reasons why RELs may struggle to reach practitioner audiences. For example, some intermediaries—such as industry associations—we spoke with said their organizations help to disseminate research information to policymakers and practitioners. However, some noted that further efforts are needed to leverage intermediary groups to better market REL and R & D Center work to reach IES’s target audiences.

IES Has Not Fully Assessed Groups’ Efforts to Disseminate Relevant Research

IES requires RELs to report some information about relevance and dissemination that could be used to evaluate their efforts, but this information has not been collected in a consistent manner and is therefore difficult for IES to use to improve program management. For example, RELs use a stakeholder feedback survey instrument to capture feedback from participants on relevance and other aspects of the products, activities, and events they sponsor. This survey includes a set of questions for five different types of REL activities and products, such as technical assistance workshops or research alliance participation. However, as administered in 2012—the first year of the current RELs’ operation—RELs were not consistent in how they administered the survey. For example, some RELs administered the complete survey for their activity or product type, while other RELs selected specific questions for respondents. The results, therefore, are not comparable across all REL activities and products. Further, the data collected only applies to participants who elected to provide feedback, rather than all participants in REL programs or potential users.

In addition to the feedback survey, RELs are required to annually report to IES on a series of indicators that they developed collaboratively with IES in March 2012. However, this preliminary information is not sufficient for IES’s management of RELs’ efforts to produce and disseminate relevant research. The REL program has six expected outcomes, one of which is to build a body of knowledge in topics that address regional needs. To assess performance on this and the other five program outcomes, RELs must report to IES on 24 indicators. For example, regarding dissemination, RELs must report the number of events held and the total number of attendees at these events. However, IES has not established performance targets or goals for these indicators to help RELs prioritize their activities or assess their performance in these areas. Were IES to establish targets or goals, the RELs would likely have more incentive to perform at the agency’s desired level, and IES would be better positioned to determine if the RELs are meeting its expectations and hold them more accountable for performance. Our previous work has indicated agencies successful in measuring performance had performance measures including targets or goals that (1) demonstrate results; (2) are limited to the vital few; (3) cover multiple priorities; and (4) provide useful information for decision making.

Regarding the R & D Centers, IES has few requirements for tracking relevance and dissemination, although some Centers collect additional information that could be helpful for assessing performance in these areas. In combination with other factors, IES assesses the relevance of the R & D Centers’ proposed research projects when making funding decisions, but according to officials, this is the only formal assessment of the relevance of R & D Center work. However, although they are not required to do so, most R & D Center directors responding to our survey said they use quantitative data to gauge their relevance to policymakers and practitioners. For example, one R & D Center has conducted national surveys of certain constituent groups, such as state policymakers, asking about the relevance of the Center’s findings and products for their use. In their annual performance reports, IES asks R & D Centers to report whether they disseminated research information, but does not specify what information they should provide, such as how they disseminated information or the effectiveness of these efforts. Though it can be difficult to demonstrate how research programs with broad dissemination goals— such as the R & D Centers—are able to inform others and contribute to outcome-oriented goals, there are several evaluation strategies to assess how programs with great breadth and flexibility contribute to agency goals. IES has not, however, set utilization or dissemination goals and objectives, or investigated whether dissemination strategies were effective in improving the efforts of these groups.

According to officials, IES has no plans to conduct formal evaluations for the current group of RELs and R & D Centers to comprehensively assess their relevance and dissemination activities. IES is still in the process of conducting a mandated evaluation of the prior group of RELs, whose contracts ended in 2011. According to study design documents and IES officials, this IES-contracted evaluation will include information from a customer satisfaction survey of a representative sample of state and district staff regarding their views of the relevance and usefulness of REL work. The evaluation will also include an assessment by expert panelists of the quality and relevance of selected REL reports. However, it will not include an assessment of REL dissemination activities. This evaluation began in 2009 and an interim report was initially expected to be released in spring 2012; however, the interim report was delayed and was released in late September 2013. IES has no further plans to evaluate the REL program beyond the forthcoming evaluation of the prior group. Officials told us although there was a clear requirement in ESRA for this evaluation, they did not believe that the law required any subsequent evaluation of the REL program. In addition, as of August 2013, IES had no plans to conduct a formal evaluation of the R & D Center program, and there is no requirement to do so. According to IES officials, they do not plan to evaluate them because they do not view them as a program, but rather as one vehicle for supporting IES’s programs of research, as with its education research grants. However, research on dissemination practices funded by other federal research programs has shown that dissemination efforts should be assessed periodically to improve further efforts to communicate key messages to target audiences.

As required by ESRA, IES has previously evaluated the Comprehensive Centers. A 2011 IES-contracted evaluation reviewed how well the Comprehensive Centers addressed state client needs, as well as the quality, relevance, and usefulness of their assistance, among other things. The evaluation included a survey of SEA administrators and a review of Comprehensive Center products. Evaluation data were collected annually in 3 of the 5 program years in which the Centers operated. Comprehensive Centers were rated higher in each successive year, and in 2008-2009, 56 percent of state managers surveyed said that technical assistance from the Comprehensive Centers served the state’s purposes completely. For the state managers who said the Comprehensive Centers did not completely meet their needs, the main reason cited was that Center staff had limited time to work with their state. The evaluation also included a quality assessment of a sample of Comprehensive Center projects by expert panelists and project participants. Each sampled project was independently rated by a panel of experts using a 5-point scale. SEA staff who participated in these projects (e.g., by serving on a work group associated with the project) were asked via survey to rate the relevance and usefulness of each project. For the sample of projects assessed, the evaluation showed that the Comprehensive Centers’ technical assistance was rated higher on each measure in each successive year, from 2006-2007 to 2008-2009. On average, across each of the 3 years, expert panels rated sampled project materials from “moderate” to “high” for quality, and project participants rated the sampled projects “high” for relevance and usefulness. While these results indicate a positive trend in Comprehensive Centers’ performance, this evaluation covered the prior group of Comprehensive Centers and IES recently released a new Request for Proposals to evaluate the current group.

IES Coordinates with Other Federal Agencies, but Fragmented Evaluation Funding Poses Challenges for Education’s Evaluation Efforts

IES Coordinates with Other Agencies to Increase the Use of Evidence in Federal Decision Making and Research Planning

IES coordinates with relevant federal research agencies on projects to increase federal agencies’ use of research evidence in guiding funding decisions. According to federal agency officials and stakeholders we spoke with, NICHD and NSF are the two other primary federal agencies conducting education-related research. IES officials meet regularly with their NICHD and NSF counterparts to jointly sponsor projects and ensure their efforts are complementary. For example, IES co-led a joint Education-NSF working group to develop common evidence guidelines for education research and development. These guidelines are intended to help guide NSF’s and IES’s respective decisions about investments in education research and clarify for potential grantees and peer reviewers the evidence expected from each type of study. Several researchers we interviewed, as well as officials from three federal agencies we spoke with, said these guidelines will benefit the education field. In addition, developing common evidence guidelines is consistent with our key practices for enhancing and sustaining collaboration among federal agencies. In particular, we have found that establishing compatible policies, procedures, and other means helps to align the partner agencies’ activities, core processes, and resources to accomplish a common outcome.

Consistent with this key practice, IES also coordinates with other federal agencies to leverage its expertise in planning federal research agendas on topics such as early learning and prisoner re-entry, as well as increasing the quality of and access to data on education outcomes. For example, IES sits on the Early Learning Policy Board, a joint Education- Health and Human Services body, to discuss ways to coordinate federally-funded research on early childhood topics. IES’s statistics Center leads the Interagency Working Group on Expanded Measures of Enrollment and Attainment, to improve national data on education, training, and credentials for work. This effort includes OMB, the Council of Economic Advisors, and other federal statistical agencies, such as the Census Bureau and the Bureau of Labor Statistics at the Department of Labor. In addition, IES has worked with the Defense Advanced Research Projects Agency at the Department of Defense on its Small Business Innovation Research Program, and in December 2012 they released a joint solicitation to support the development and evaluation of education technology games.

IES’s efforts to help develop a systematic, transparent evidence review process and incorporate a tiered evidence framework for awarding grants for Education’s Investing in Innovation Fund grant program have had an impact on the use of evidence in federal decision making both within and outside the department. According to an official from the Administration of Children and Families at the Department of Health and Human Services, for some of its grant programs, the Administration of Children and Families has recently used tiered evidence standards similar to those IES helped to develop for Education’s Investing in Innovation Fund, reserving the majority of funds for applicants proposing to implement service models with evidence of effectiveness. IES is also working with Education program offices to incorporate tiered evidence standards in awarding their grants, such as the Strengthening Institutions Program in the Office of Postsecondary Education.

Coordination between the RELs and Comprehensive Centers Has Improved

In addition to working with other federal agencies, IES coordinates within Education to facilitate coordination between the REL and Comprehensive Center programs. There is a statutory requirement for each Comprehensive Center to coordinate its activities and collaborate with the RELs in its region, as well as with certain other entities. Prior to the current award cycle, IES and OESE restructured REL and Comprehensive Center regions so that the number of Comprehensive Centers in each REL region was reduced and aligned more closely with REL regions. According to OESE officials, its Comprehensive Center grantees and states found the lack of alignment between the regions in the prior contract cycle to be a barrier to coordination. Further, IES has provided guidance regarding its expectations for coordination between RELs and Comprehensive Centers. For example, IES has encouraged RELs to partner with Comprehensive Centers and has allowed RELs to budget resources for these collaborative efforts.

According to directors of these groups, coordination has improved with the current group of RELs and Comprehensive Centers, which began in 2012. One REL Director we spoke with told us that the realignment of the regions has facilitated improved coordination between the groups. In addition, some directors said that the RELs’ new structure of research alliances has improved coordination. For example, some Comprehensive Centers are members of REL research alliances and have partnered with RELs to host “bridge” events, where state officials and researchers meet to discuss research findings relevant to issues of policy and practice.

REL directors told us that although there can still be confusion among some SEA officials about the appropriate role and tasks performed by each, RELs and Comprehensive Centers have used several strategies to reduce confusion and improve coordination. For example, some groups have conducted joint visits with SEA officials so that they can describe their respective roles and responsibilities and specifically discuss how each group can address the state’s various technical assistance needs. In addition, some directors described their groups’ efforts to sequence their work for their state clients to better meet their needs. In some instances, for example, the Comprehensive Center may conduct initial planning work for a project, and the REL would later assist with any aspects of the project requiring data analysis or original research. For example, one Comprehensive Center director told us that, through joint efforts with the REL in their region, it developed a framework for measuring teacher effectiveness. For this project, the REL formed a research alliance with other Comprehensive Centers and SEAs interested in piloting the framework. The Comprehensive Center is helping states implement the pilot, and the REL will conduct an evaluation of the pilot projects. This Center Director told us that it will help the states adjust their strategies based on the findings from the pilot evaluation.

Education’s Inability to Combine Evaluation Funding Poses Challenges for Evaluation Planning Efforts

IES coordinates within the department to plan evaluations of Education programs. Beginning in 2010, OPEPD and IES have jointly led an annual department-wide evaluation planning process to identify evaluation projects to conduct. Although there is no official guidance delineating these roles, Education officials said that the annual evaluation planning process is used to help determine which of these offices will conduct each project. According to Education, this planning process has helped to improve the quality of evaluations conducted, and made evaluations more responsive to the information needs of the department. For example, IES is conducting an implementation and quasi-experimental evaluation of the Race to the Top Program. An IES official reported that input from the Education program office running this program helped ensure that the evaluation’s data collection instruments focused on implementation issues most important to Education and state education leaders. IES is also working with this program office to select topics of greatest immediate interest to Education for short implementation briefs to be released as the study progresses.

However, efforts to coordinate internally and prioritize evaluation research projects through this annual process are challenged in part by statutory requirements related to ESEA evaluation funding. For these programs, evaluation funds are typically set aside as a percentage of program funding or as national activities funds. With some exceptions, ESEA authorizes Education to reserve up to 0.5 percent of the amount appropriated to carry out each program or project authorized under that Act to conduct evaluations. However, according to Education officials, due to the statutory limitation, the funds available for evaluation are often insufficient to conduct high-quality evaluations of these programs. This is particularly true with respect to certain discretionary grant programs where overall program funding is itself relatively small. As a result, OPEPD and IES officials said that some evaluations, including high- priority evaluations, may not occur. For example, IES officials told us that the department is interested in conducting evaluations of the Mathematics and Science Partnerships and Promise Neighborhoods programs, but there is not sufficient evaluation funding under these programs due to the statutory limitation and Education does not have the authority to combine evaluation funds across the department and use them to conduct evaluations of these or other high-priority programs.

We have previously reported that many Education programs, especially smaller programs, have not been evaluated, which can limit the ability of Congress to make informed decisions about which programs to continue, expand, modify, consolidate, or eliminate. For example, in 2009, we reported that 11 of Education’s programs focused on teacher quality had been operating for over seven years and had never been evaluated. In addition, according to OPEPD officials, ideas for some high-priority evaluations that department officials have discussed during the annual evaluation planning process are never developed into plans because officials know that evaluation funds are not available to carry them out. However, according to officials from OPEPD, beginning in fiscal year 2015, Education will maintain a list of all of the department’s evaluation needs should funds for conducting these evaluations become available. According to a senior IES official, if the department were able to combine funds authorized to be used for evaluation it would have more flexibility to conduct appropriate evaluations of any of ESEA programs that need evaluation. The President’s Fiscal Year 2014 Budget request for Education contained a proposal to increase Education’s flexibility to conduct program evaluations by allowing the department to use funds from certain programs across the department for this purpose. An OMB staff member told us that the ability to combine evaluation funds and prioritize evaluations is critical in the current environment of constrained funding to help make the best use of limited resources and to fill gaps in topic areas where research is needed. In fiscal year 2012, Congress granted the Department of Labor temporary authority to combine program-specific funds to support evaluations of its programs. The Department of Labor is required to provide a plan to the congressional appropriations committees before combining its funds in this manner. This plan is to describe the specific evaluations it plans to carry out using these funds and helps ensure transparency and accountability in the department’s use of this authority. In the Department of Labor’s fiscal year 2014 budget justification, it reported that the availability of flexible evaluation funds has allowed the department to evaluate large and high- priority programs, as well as programs that may not have been fully evaluated previously.

Conclusions

Since its creation more than a decade ago, IES has made significant contributions to strengthening the rigor of the education research field and has promoted the use of scientifically based research in our nation’s education system. However, IES’s ability to release timely information to policymakers and practitioners is exacerbated by its peer review process which, while promoting rigorous research, can take a substantial amount of time. Without better use of available data to manage this peer review process, it will remain difficult for IES officials to identify potential causes of these delays and develop strategies to mitigate these challenges. More recently, the agency has begun working to make its products more relevant to a range of stakeholders, particularly policymakers and practitioners. Producing education research that is relevant for these stakeholders is critical to inform decision making in Congress and in the field about key federal investments in education. Despite progress IES has made in ensuring that its research is relevant to its stakeholders, IES does not have a formal, structured process for engaging with these groups in the research-agenda setting process. Until such a process is in place, IES limits its ability to ensure the relevance, and ultimately, usefulness of its research to the field. IES’s relevance to the field is unclear in other ways as well. For example, without publicly reported performance measures for key investments—such as the RELs or new high-priority grant programs—stakeholders, such as Congress and the public, are unable to determine the effectiveness of IES’s activities or evaluate the merits of the federal investment in these programs.

IES’s research and technical assistance grantees and contractors take a number of steps to help ensure they produce and disseminate relevant research and products. However, without assessing the effectiveness of these groups’ dissemination efforts, IES may miss opportunities to think more strategically about dissemination and ensure that these groups are using the most effective strategies to reach their target audiences. Providing Education with the authority to combine funds authorized for evaluations under ESEA would allow the department more flexibility to prioritize and conduct effective evaluations of programs that represent key federal investments and critical subject areas. Without this authority, it will be difficult for Education to make the best use of its limited resources and to support Congress in making informed decisions about which programs to continue, expand, modify, consolidate, or eliminate.

Matter for Congressional Consideration

To ensure that Education can conduct critical program evaluation work, Congress should consider granting Education authority to combine funds authorized for evaluations of ESEA programs and target them to high- priority evaluations, with appropriate measures to ensure transparency and accountability for how the funds will be used.

Recommendations for Executive Action

To improve the management and accountability of IES’s research and evaluation efforts, we are making the following recommendations: 1. IES should use available data to routinely monitor all stages of its peer review process and identify opportunities to improve the timeliness of its reviews. 2. IES should develop a structured process to systematically gather input from policymakers and practitioners and use this input when developing its research agenda. 3. IES should develop performance measures, including targets and goals that clearly demonstrate results, and that reflect its current programs and all key agency activities, such as the performance of the RELs and its new grant programs supporting researcher- practitioner partnerships. 4. In order to identify leading practices and target areas for improvement, IES should assess the effectiveness of REL and R & D Center dissemination strategies by, for example, collecting consistent data and lessons learned from these groups to inform future dissemination efforts.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Education for review and comment. Their comments are reproduced in appendix II. Education officials also provided technical comments, which we incorporated into the final report, as appropriate.

Education generally agreed with our recommendations and has several plans in place to address them. Specifically, IES noted that it continues to be committed to producing high-quality reports while shortening the time period for its peer review process, and stated that it will develop a system to more comprehensively monitor all of the stages in its report production process so that it is able to identify and address potential problems more easily and quickly. To further emphasize its commitment to obtaining practitioner and policymaker input, IES also said it will begin an annual solicitation for practitioner and policymaker input on its research agenda by posting an invitation on its web site and widely advertising the opportunity through its IES Newsflash email notices.

Additionally, as we noted in our report, IES commented that it is currently working to align its performance measures with its current investments, including developing measures for the REL program. Finally, IES agreed that it is important to measure and assess the effectiveness of IES- supported dissemination activities. IES noted it will build knowledge in this area through its fiscal year 2014 request for applications for a new R & D Center focused on knowledge utilization to develop measures for how practitioners use research evidence to guide what they do in schools and classrooms. We support IES’s continued focus and emphasis on advancing knowledge and providing usable information to improve education policy and practice.

We are sending copies of this report to relevant congressional committees and the Secretary of Education. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or our staffs have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our review examined (1) the extent to which the Institute for Education Sciences (IES) demonstrates its ability to support high-quality research and fulfill its mission; (2) the extent to which selected Department of Education (Education) research and technical assistance groups disseminate relevant products to the field; and (3) how IES coordinates its activities with other relevant federal research entities and within Education. For all three objectives, we reviewed the Education Sciences Reform Act of 2002 (ESRA) to identify statutory requirements related to IES management, processes, and procedures. We obtained and analyzed agency documents detailing IES’s processes and performance. We also conducted interviews with senior IES officials, including Commissioners at all four IES Centers as well as selected program staff, several current and former members of IES’s advisory board, the National Board for Education Sciences, and selected former senior IES officials to gather additional perspectives on IES’s management and performance over time.

Specifically, to assess the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, we reviewed documents and data from IES’s Standards and Review Office, which administers the peer review of IES reports and grant applications. In part to evaluate IES’s timeliness in producing relevant research, we obtained IES data for the most recent 4 years, fiscal years 2010 through 2013, on the average number of work days from the submission of the initial manuscript to final approval for reports sponsored by IES that underwent external peer review before being publicly released. We assessed the reliability of IES data by obtaining written responses from agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. To further assess IES’s overall processes and operations, we applied a framework GAO previously developed to evaluate the soundness of federal research and evaluation programs. This framework includes five key phases of the research and evaluation process: (1) agenda setting, (2) selecting research, (3) designing research, (4) conducting research, and (5) disseminating research results. While we assessed IES’s activities in each of the five phases, we primarily focused on the first and fifth phases of this process: agenda setting and disseminating research, as these two phases most directly addressed our research questions.

Additionally, we reviewed publicly reported performance information contained in Education’s most recent budget request to Congress for IES to compare performance trends over time and to assess the completeness of IES’s performance measures against GAO’s leading practices for performance management and federal standards for internal control in the areas of information and communications, and performance reporting and monitoring. We also interviewed officials at the research and evaluation arm of the Administration for Children and Families at the Department of Health and Human Services, and at the Office of Management and Budget that were recommended to us as knowledgeable of federal research and performance reporting to obtain their insight on IES’s research, evaluation, and performance measurement. Lastly, we observed two public meetings of the National Board for Education Sciences that took place during the course of our review on October 5, 2012, and February 22, 2013.

We also conducted more than 40 interviews with several categories of IES stakeholders: (1) education researchers; (2) organizations representing education policymakers and state-level administrators; and (3) practitioner organizations, individual teachers, and district leaders. To identify appropriate stakeholders in all three categories, we obtained recommendations on individuals or groups to contact from key organizations, contacted those individuals or groups, and obtained further recommendations from them on additional individuals or groups to contact. We continued to ask for names from the subsequent individuals and organizations until we began getting duplicate referrals. We also identified individuals and groups to contact through our prior work on education research and internal expertise. For the education researchers, we sought to identify education researchers that were considered to be knowledgeable about education research and the activities of IES and its predecessor organizations. For the education policymaker organizations, we ensured representation from groups that represent the wide range of topical areas in which IES funds research and evaluation, such as early childhood education, special education, and postsecondary education. We interviewed teachers and superintendents because they are part of IES’s target population of end-users, though they had various levels of familiarity with IES prior to our interview.

We conducted two discussion groups with experienced education researchers at the 2013 Annual Conference of the American Educational Research Association (AERA). We assembled senior researchers for these discussion groups with the help of AERA staff. Invitations to participate in these discussion groups were provided to selected members of the AERA Fellows program that were culled by AERA staff to ensure representation from a balanced range of subject matter areas. During these discussion groups, we met with a total of 16 senior education researchers and discussed issues such as IES’s research standards, peer review process, and the relevance and dissemination practices of IES overall, as well as those of selected IES research and technical assistance groups, specifically—the Regional Educational Laboratories (REL) and Research & Development Centers (R & D Center).

To address our second research objective, we focused on three types of research and technical assistance groups. Two are administered by IES—the RELs and R & D Centers—and one is administered by the Office of Elementary and Secondary Education (OESE)—the Comprehensive Technical Assistance Centers (Comprehensive Centers). We selected these three research and technical assistance groups because they are all authorized by the same law that established IES. Moreover, there is a statutory requirement for the Comprehensive Centers to coordinate their activities and to collaborate with the RELs, among others. Specifically, to assess the extent to which these groups disseminate relevant products to the education field, we obtained and analyzed the most recent annual performance report submitted to IES and OESE, respectively, from each of these groups. We analyzed these documents to identify what quantitative and qualitative information IES collects about the relevance, dissemination, and utilization of the research and products these groups produce. We also interviewed Directors of the RELs and the Comprehensive Centers as well as program staff at IES and OESE that administer the programs to discuss relevance, dissemination, and coordination activities. To further understand the Comprehensive Centers activities related to the production and dissemination of relevant research and products to the field, we reviewed a contractor-led evaluation of the Comprehensive Center Program. In particular, we reviewed findings related to relevance of the products these groups produced and the levels of satisfaction reported by their customers. We also reviewed early study design documents and an interim report for an ongoing evaluation of the REL program. We reviewed the scope and methodology for each of these evaluations and determined they were sufficiently reliable for our reporting purposes. We also obtained information from IES staff responsible for overseeing these evaluations to obtain additional information.

We administered a short survey to the Directors of IES’s 18 R & D Centers that were operational in the most recent full program year (2011- 2012) at the time of our review. We asked questions about activities the R & D Centers undertake to disseminate relevant research to the field and what qualitative and quantitative data they collect on their activities. We conducted pretests to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the survey did not place an undue burden on respondents, (4) the information could be feasibly obtained, and (5) the survey was comprehensive and unbiased. We conducted two pretests with R & D Centers, one grantee of the National Center for Education Research and one grantee of the National Center for Special Education Research. We made changes to the survey after both pretests based on the feedback we received. We sent the survey by e-mail in an attached Microsoft Word form on April 2, 2013. Surveys were completed by 17 of 18 R & D Center Directors, for a 94 percent response rate, by May 10, 2013. Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such nonsampling error.

To describe how IES coordinates with other relevant federal research agencies and within the department, we reviewed pertinent documents and statutory requirements for coordination contained in ESRA and compared agency efforts to criteria we previously developed for practices agencies can use to help enhance and sustain interagency collaboration. To identify relevant federal research agencies, we obtained recommendations on federal research agencies that conduct education- related research during interviews with knowledgeable stakeholders. We continued to ask for recommendations during stakeholder interviews until we established that the vast majority of recommendations identified two other federal research agencies: the education directorate at the National Science Foundation (NSF) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), an arm of the National Institutes of Health. While stakeholders and officials at IES, NSF, and NICHD noted some key differences between these agencies, such as the types of research funded, there was general consensus that there were clear commonalities in the research missions and some of the topics of research funded by all three agencies. As such, we interviewed officials at NSF and NICHD about their processes and procedures as well as their coordination with IES, asking for examples of coordination activities and any potential areas for improvement. Additionally, during our interviews with IES officials, we asked about their coordination activities with NSF and NICHD. To describe IES’s coordination within the department, we reviewed agency documents such as those describing the annual evaluation planning process at Education and conducted interviews with IES officials and officials responsible for evaluation and research within the department’s Office of Program Evaluation and Policy Development. Finally, we also asked about coordination activities between the REL and Comprehensive Centers programs during our interviews with program staff administering these programs, as well as during our interviews with Directors of these groups, to assess the extent and nature of the coordination between these technical assistance programs.

We conducted this performance audit from July 2012 to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Education

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Scott Spicer, Assistant Director; Lucas Alvarez, Nora Boretti, and Dana Hopings made significant contributions to this report. Also contributing to this report were Sandra Baxter, Deborah Bland, David Chrisinger, Elizabeth Curda, Helen Desaulniers, Sheila McCoy, Jean McSween, Mimi Nguyen, James Rebbe, and Sarah Veale.

Related GAO Products

Education Research: Preliminary Observations on the Institute of Education Sciences’ Research and Evaluation Efforts. GAO-13-852T. Washington, D.C.: September 10, 2013.

Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Government Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program. GAO-11-285. Washington, D.C.: March 15, 2011.

Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.

Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse. GAO-10-644. Washington, D.C.: July 23, 2010.

Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program. GAO-10-243. Washington, D.C.: January 29, 2010.

Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals. GAO-02-923. Washington, D.C.: September 30, 2002.

Education Research: Education Should Improve Assessments of R&D Centers, Regional Labs, and Comprehensive Centers. GAO-02-190. Washington, D.C.: January 24, 2002.