Evidence-Based Policymaking: Selected Agencies Coordinate Activities, but Could Enhance Collaboration
Released: 2019-12-04T00:00:00

Letter

Background

Federal Evidence-Building

According to OMB guidance, evidence can consist of quantitative or qualitative information and may be derived from a variety of sources. Those sources include foundational fact-finding (e.g., aggregate indicators, exploratory studies, descriptive statistics, and other research), performance measurement, policy analysis, and program evaluation. OMB recommends that agencies build a portfolio of high-quality, credible sources of evidence—rather than a single source—to support decision- making. Further, since different sources of evidence have varying degrees of credibility, the use of evidence in decision-making requires an understanding of what conclusions can—and cannot—be drawn from the information.

Evidence-building can be viewed as a cycle of activities that can help decision makers obtain the evidence they need to address policy questions or identify the questions they should address. As illustrated in figure 1, the following four activities comprise the evidence-building cycle: assessing existing evidence to determine its sufficiency and if additional evidence is needed to further understand results and inform decision-making; prioritizing among the identified needs which new evidence to generate, when, and how; generating new evidence, by collecting, analyzing, and synthesizing sources of data and research results; and using that evidence to support learning and decision-making processes.

Our prior work highlights long-standing challenges agencies continue to face in generating some sources of evidence—developing performance measures for federal programs and conducting evaluations of their programs. Our work also identified variations in the use of evidence for decision-making by agency leaders and managers.

Fragmentation of Federal Evidence-Building Activities

Fragmentation refers to those circumstances in which more than one federal agency (or organization within an agency) is involved in the same activity and opportunities exist to improve implementation of that activity.

The Commission on Evidence-Based Policymaking found that evidence- building activities are fragmented in the federal government. For example, it found that within agencies, many organizations have evidence-building responsibilities, including statistical agencies and programs, evaluation and policy research offices, performance management offices, policy analysis offices, and program administrators. In addition, the commission highlighted challenges the federal government faces in fully addressing cross-cutting research and policy questions when evidence-building activities span multiple agencies. The commission’s final report noted that this fragmentation (see sidebar) can lead to duplication of effort or missed opportunities for collaboration. The commission’s report stated that when activities are fragmented within an agency or across the federal government, they should be coordinated to improve the capacity to fully address a specific research or policy question.

Similarly, our past work highlights the importance of coordination and collaboration to reduce or better manage fragmentation, overlap, and duplication. We found that uncoordinated or poorly coordinated efforts can waste scarce funds and limit their effectiveness. Even when efforts are coordinated, enhancements to those efforts can lead to improvements in effectiveness. As noted earlier, our work also identified leading practices that can help agencies enhance and sustain their implementation of collaborative efforts.

Efforts to Improve Federal Evidence-Building

Congress and OMB have taken actions to strengthen federal evidence- building activities and improve coordination of those activities during the last decade. Figure 2 provides a timeline of selected actions. Appendix II provides additional detail regarding the selected actions.

Selected Agencies Have Taken Actions that Align with Congressional and OMB Direction to Strengthen Evidence- Building

Selected Agencies Implemented Evidence- Based Approaches in Response to Congressional Direction

Tiered evidence grants seek to incorporate evidence of effectiveness into grant making. Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services. Grantees generally are required to evaluate their service models as a condition for the receipt of grant funds. spent on home visiting models with sufficient evidence of their effectiveness.

To support this requirement, the program incorporated activities across each element of the evidence-building cycle. For example, through its Home Visiting Evidence of Effectiveness review, HHS annually assesses existing evidence about the effectiveness of new and existing home visiting models to identify those that meet criteria for inclusion in the program. The most recent review, in October 2018, identified 20 models that met HHS’s criteria for an evidence-based early childhood home visiting model. Of those, HHS determined that 18 models were eligible for MIECHV grantees to select for implementation. In addition, based on statutory requirements, officials prioritized the generation of new evidence to assess the program’s results in certain areas, including child health and development, and child maltreatment. The program generated this evidence through program evaluations assessing both program implementation and results. For example, an impact evaluation of four home visiting models published in January 2019 found that these models may reduce household aggression. Because child abuse has been shown to be associated with negative long-term outcomes, reducing household aggression could benefit children as they grow older.

In another example of the use of tiered evidence, the Department of Labor’s (DOL) Workforce Innovation Fund, established in 2011, intends to generate long-term improvements in the performance of the public workforce system. The fund established and funded projects in three different tiers: 1. those that proposed new and untested approaches, with little or no evidence of effectiveness; 2. those with promising approaches that were tested and existing evidence suggested could be effective; and 3. those that adapted proven approaches, supported by ample and robust evidence.

To further build DOL’s base of evidence on the effectiveness of evidence- based approaches, it required grantees to plan for third-party evaluations of their programs.

During the first grant round in 2012, the Workforce Innovation Fund awarded 26 grants, including one for approximately $1.4 million in tier one funding to the Pasco-Hernando Workforce Board in Florida. This grant supported making one-stop services, such as employment workshops and workforce program orientations, more accessible to job seekers by providing online access. In addition, the grant supported offering virtual case management and business services through a call-in Employment Support Center to individuals who found it difficult to access these services in person. According to a 2016 case study of this project conducted by DOL, users of the online one-stop accessed services nearly twice as much during this 3-year grant period when compared to the prior 3-year period. In addition, the case study found there was a 53 percent increase in job placements during this 3-year grant period.

Selected Agencies Have Taken Evidence-Building Actions that Align with OMB Direction for Cross- Agency Priority Goal Implementation

The selected agencies’ evidence-building activities also aligned with implementation actions outlined by OMB for selected cross-agency priority (CAP) goals. As required by the GPRA Modernization Act of 2010, at least every 4 years, OMB is to coordinate with other agencies to develop and implement CAP goals. Two current CAP goals, established in March 2018 in the President’s Management Agenda, place a particular focus on evidence-building activities.

Leveraging data as a strategic asset. OMB and agency efforts to implement this goal included developing a long-term, enterprise-wide federal data strategy to better govern and leverage the federal government’s data. Published in June 2019, this strategy established 10 principles and 40 practices intended to leverage the value of federal data assets while protecting security, privacy, and confidentiality. Officials at each of the five selected agencies described actions taken by their agencies that aligned with the federal data strategy’s principles and practices.

Federal Evidence Clearinghouses According to the Office of Management and Budget (OMB), evidence or “what works" clearinghouses are repositories that synthesize evaluation findings in ways that make research more useful to decision makers, researchers, and service organizations. These repositories provide tools for understanding what service models are ready for replication or expansion and disseminating results. grade. Officials told us in September 2018 that preliminary evidence suggested the model could help close the literacy gap for the target population. In addition, officials told us they intended to disseminate the final results to stakeholders to help inform their decision-making about the approach. To do so, Education officials developed a communication plan to share this evidence via the OELA website, its Facebook account, the National Clearinghouse for English Language Acquisition (see sidebar), and a listserv of more than 10,000 recipients, among other means. As of September 2019, this study had not been completed. Therefore Education has not implemented its communication plan.

Results-oriented accountability for grants. One of the four strategies for this CAP goal focuses on the achievement of grant program goals and objectives. In October 2019, OMB staff told us that the strategy aims to hold grant recipients accountable for promising performance practices that support the achievement of those goals and objectives while streamlining compliance requirements for those grant programs that demonstrate results. According to the September 2019 quarterly update for this goal, initial efforts for this strategy involved developing performance management processes to help grant-making entities improve their ability to monitor, and ultimately improve, the performance of grantees. The update stated that OMB and the Chief Financial Officers Council completed efforts in fiscal year 2019 that included soliciting information from agencies on their current grants performance management practices and identifying emerging and innovative performance practices. Subsequent efforts for this goal involved hosting monthly grants practitioner sessions (called Innovation Exchange Sessions) to share new ideas and approaches to grants management, which began in May 2019. The September 2019 session focused on data- driven decision-making for grants.

We identified actions that each of the selected agencies took, aligned with the intent of this CAP goal, to better assess the performance of their grant programs. Officials at each agency told us that they took steps to further incorporate evidence-building requirements into their grant programs. They told us they did this based in part on their experiences in implementing the evidence-based approaches, such as the tiered evidence grants described earlier in this report.

For example, officials at the Corporation for National and Community Service (CNCS) described their incorporation of evidence-building requirements into the agency’s AmeriCorps State and National program. Agency officials told us that grantees have been required to evaluate their programs since 2005. In recent years, CNCS embedded the evidence generated by these evaluations into their grant-making activities. For instance, its grant announcement for 2019 stated that AmeriCorps State and National applications would be scored, in part, based on the reported empirical evidence supporting the applicants’ proposed projects. In addition, the announcement required applicants proposing projects in the education focus area to choose one of 13 models that had previously demonstrated effectiveness. According to CNCS officials, this was based on evidence generated in previous projects supported by AmeriCorps State and National grants or CNCS’s Social Innovation Fund.

Selected Agencies’ Component Organizations and Programs Developed Learning Agendas Aligned with OMB Guidance

Although the Evidence Act’s requirements apply to the agency-wide level, OMB’s guidance strongly encourages lower-level organizations within agencies to develop and implement their own learning agendas (see side bar). We found instances where officials developed learning agendas at lower organizational levels within several of the selected agencies prior to the issuance of the June 2019 OMB guidance. These learning agendas covered individual component agencies, bureaus, offices, and programs.

Learning Agendas According to Office of Management and Budget (OMB) guidance for implementing the Evidence Act, a learning agenda is to define and prioritize relevant questions and identify strategies for building evidence to answer them. In developing a learning agenda, an agency should involve key leaders and stakeholders, to help (1) meet their evidence needs for decision-making and (2) coordinate evidence-building activities across the agency.

For example, from September 2016 to June 2017, the U.S. Agency for International Development (USAID) conducted a landscape analysis of learning agendas, in which officials identified 15 documented, office-, bureau-, or initiative-wide learning agenda processes at different stages of development within USAID. This included an office-wide learning agenda developed by the Center of Excellence on Democracy, Human Rights, and Governance (DRG). According to USAID, DRG seeks to elevate and integrate democracy, human rights, and governance issues within USAID’s overall development portfolio. According to DRG’s 2017 learning agenda, its development was informed by ongoing DRG research and evaluation efforts, and consultations with a range of internal stakeholders, including USAID staff from other bureaus and missions. The learning agenda included a set of 11 questions across five thematic areas, as illustrated in figure 3.

DRG outlined steps it planned to take throughout 2017 to address each question, such as assessing existing evidence, identifying any gaps, and conducting new research and evaluation activities to fill those gaps. For example, DRG commissioned a study to help answer a question about the effects of human rights awareness campaigns. The study, published in September 2017, synthesized the results of a literature review to identify (1) characteristics of effective campaigns, and (2) typical causes of unintended negative consequences of human rights awareness campaigns and ways to avoid them.

Selected Agencies Established Processes to Coordinate Fragmented Evidence-Building Activities, but Processes to Prioritize New Evidence Did Not Always Reflect Leading Practices

Selected Agencies Established Processes to Coordinate Fragmented Evidence-Building Activities

We found that evidence-building activities are fragmented within each of the five selected agencies and occur at multiple levels and entities within and across the agencies. As illustrated in figure 4, this fragmented approach to evidence-building includes separate component agencies or offices with responsibilities for building specific sources of evidence, such as performance information, evaluations, and statistical data.

For example, at the Department of Labor (DOL), different organizations at the department level are responsible for certain evidence-building activities. This includes the Bureau of Labor Statistics (collecting statistical data), Office of the Chief Evaluation Officer (conducting program evaluations) and Performance Management Center (developing performance information).

In addition, some evidence-building activities are dispersed throughout agencies and occur at multiple organizational levels (see figure 5).

For example, at the Department of Health and Human Services (HHS), evidence-building activities are generally managed at the component agency level (referred to as divisions). The divisions manage their own offices and programs, which include evidence-building responsibilities. For instance, within the Administration for Children and Families (ACF), an operating division within HHS—the Office of Planning, Research, and Evaluation—is responsible for ACF-related evidence-building activities. These activities include program evaluations, research syntheses, descriptive and exploratory studies, data analyses, and performance management activities.

Officials at the selected agencies said that evidence-building activities are fragmented and occur at lower levels for a variety of reasons. First, this approach helps ensure that decision makers at different levels within the organization have the evidence they need to inform decisions. Second, officials stated that many times these evidence-building activities have been undertaken in response to direction from Congress—for example, through provisions in laws or related committee reports directed at a component agency or program. Third, agency officials said they have undertaken these activities based on OMB direction, such as memorandums or budget guidance. This has encouraged agencies to take actions at different organizational levels.

However, each of the selected agencies had established processes for coordinating their evidence-building activities. For example, officials at each agency established one or more processes intended to regularly coordinate the assessment and prioritization of evidence needs across the agency, as described later in this report.

Agency officials also described other efforts to coordinate evidence- building activities, but these efforts were either ad hoc (i.e., they did not occur regularly) or not comprehensive in nature (i.e., they did not focus broadly across different sources of evidence or did not cover the entire agency). For example, in August 2017, the Corporation for National and Community Service (CNCS) published the results of an assessment of existing evidence—results from research and evaluation activities conducted between fiscal years 2015 and 2016—in its State of the Evidence report. However, CNCS has not conducted a similar analysis or issued a similar report since that time. Moreover, the assessment did not cover all of the agency’s activities. While the report included evidence related to its programs, CNCS did not assess evidence related to other activities, such as internal management functions including information technology or human capital management.

We identified instances in which effective coordination helped selected agencies better manage their fragmented evidence-building activities. For example, the U.S. Agency for International Development (USAID) developed an agency-wide Private Sector Engagement learning agenda, published in May 2019. This learning agenda is intended to guide and coordinate crosscutting efforts to develop evidence of effective approaches for engaging the private sector to help partner countries meet development goals and ultimately move beyond the need for foreign assistance. This learning agenda includes establishing performance measures to monitor progress on engagement with the private sector, and further evaluate the results of its activities. The coordinated evidence- building approach established by this learning agenda can help USAID better focus limited resources on building new evidence in this crosscutting area for use across the agency, thereby reducing any unwarranted overlap or duplication of effort.

Effectively-coordinated processes can help agencies ensure they are comprehensively and systematically looking across their organizations to leverage their existing evidence and focus limited resources on building new evidence. They can also help agencies manage their fragmented evidence-building activities to improve effectiveness and reduce the potential for any unwarranted overlapping or duplicative efforts. Such processes can help ensure agencies are well positioned to meet forthcoming Evidence Act requirements related to assessing and prioritizing evidence across the entire agency.

Selected Agencies Use Similar Approaches to Assess Evidence Needs That Reflect Leading Practices for Collaboration

Selected Agencies Established Similar Approaches to Assess Existing Evidence

Each of the five selected agencies established a similar approach for assessing existing evidence and identifying gaps or other evidence needs across the agency. Agency officials said that these approaches responded to OMB guidance for agencies to conduct annual strategic reviews. Specifically, in its guidance for implementing the GPRA Modernization Act of 2010, OMB established an annual process in which each agency is to review progress in achieving strategic objectives— goals that reflect the outcome or impact the agency is seeking to achieve—established in its strategic plan. According to OMB’s guidance, as a part of those reviews, the assessment of existing evidence should inform agency decisions about where to focus limited available resources to build new evidence to fulfill any identified needs.

OMB’s guidance encourages agencies to leverage existing decision- making processes, such as the budget development process, to implement these reviews. Each of the five selected agencies conducts strategic reviews and associated evidence assessments in similar ways, through a variety of existing decision-making processes:

CNCS and HHS use their budget formulation processes;

Education incorporates strategic objective reviews into existing quarterly reviews of progress in meeting goals;

DOL uses a stand-alone strategic review process; and

USAID leverages an existing review process conducted at lower levels (i.e., its missions).

Officials at selected agencies identified instances in which they used their agency strategic reviews to (1) assess a variety of existing sources of evidence—a portfolio of evidence—to determine progress toward a strategic objective, and (2) identify the need for additional evidence, as illustrated by the following examples.

Assessing a portfolio of evidence. DOL’s guidance for its strategic review process directs its component agencies to assess a variety of evidence sources to determine results and risks or challenges that may affect future outcomes. This includes performance information, program evaluations, risk assessments, and findings from reports by us and the department’s Office of Inspector General (OIG), among other sources. In its fiscal year 2018 Annual Performance Report, DOL identified different sources of evidence to demonstrate the effectiveness of some of its programs, and challenges related to others, for its strategic objective to create customer-focused workforce solutions for American workers. For example, it cited statistics and performance data to provide context and some quantitative results related to this objective. It also shared the results from several program evaluations, including a 2017 impact evaluation that suggested DOL’s Adult and Dislocated Worker programs were effective at increasing participants’ earnings and employment.

DOL’s performance report also highlighted that its OIG identified aspects of several programs that support this objective as Top Management and Performance Challenges for Fiscal Year 2018. One of those challenges related to maintaining the integrity of Foreign Labor Certification Programs. DOL’s performance report stated that balancing the quality review of applications with employers’ needs for timely processing has been a challenge for years. Based on the totality of evidence, DOL identified this strategic objective as a focus area for improvement for fiscal year 2018.

Identifying evidence needs. In its Strategic Plan for Fiscal Years 2018-22, Education established a strategic objective to increase high- quality education options and empower students and parents to choose an option that meets their needs. To implement this strategic objective, the strategic plan states that the department will encourage state and local education agencies to expand school choice by administering programs that increase education options, such as the Charter Schools Program (CSP). One of the performance measures Education uses to assess the program and progress on this strategic objective is the aggregate number of charter schools that are open, operating, and supported by CSP.

Education officials told us that they identified limitations with this measure through the department’s strategic review process, and the need for additional evidence. As an aggregate count, the measure did not allow the department to accurately identify underlying changes in individual charter schools served by the program or the results and activities of CSP. For example, Education officials set a goal to increase the number of CSP-supported charter schools by 150 for the 2017-2018 school year. However, Education reported a decrease of four charter schools for this time period. To better understand CSP’s performance, Education officials told us they needed additional evidence to assess other aspects of the program’s performance.

Education officials identified additional sources of evidence within the department that they could use to understand the program’s performance. These included statistics from Education’s National Center for Education Statistics (NCES) on the total number of charter schools that opened and closed over the same time period, and annual performance reports from grantees. According to information on Performance.gov, these additional sources of information showed that, in the 2017-2018 school year, 134 new charter schools supported by CSP opened, and 101 charter schools expanded under a CSP grant.

These actions illustrate an instance of effective coordination of evidence- building activities to manage fragmentation and reduce the risk of duplication. Education officials looked across the agency and leveraged existing evidence generated by different organizational units—CSP and NCES—to better understand program performance. Had this not occurred, CSP might have collected data that duplicated what was already generated by NCES.

Selected Agencies’ Processes to Assess Existing Evidence Reflect Leading Practices for Collaboration

Agencies’ assessments of the sufficiency of their existing evidence— conducted via processes for their strategic reviews—reflect the four leading collaboration practices. Although OMB’s guidance provides flexibility in how the reviews are conducted, it also sets specific expectations for who should lead the process, who should participate in the process, and the types of roles and responsibilities for these individuals. Table 1 provides illustrative examples of the selected agencies’ evidence assessment processes that reflect leading practices for collaboration.

Some Agency Processes to Prioritize New Evidence to Generate Reflect Leading Practices for Collaboration

Selected Agencies Established a Range of Processes to Coordinate Evidence Prioritization

Unlike the similar processes they use for assessing existing evidence and identifying needs, the five selected agencies use a variety of processes to prioritize new evidence to generate. Agency officials told us that much of this prioritization takes place at lower organizational levels. For example, at HHS, the department’s component agencies—11 operating divisions and 14 staff divisions—generally lead their own evidence-building processes, through which they prioritize which evidence to generate.

Officials from HHS’s Office of the Assistant Secretary for Planning and Evaluation told us that this decentralized model is due to the size and complexity of the department, and that it respects the unique needs of the divisions. According to these officials, a 2017 review by this office found variation in the processes that the components use for this purpose. HHS officials said that most components prioritize their evidence needs through their budget formulation processes.

Officials at each of the selected agencies identified one or more processes intended to coordinate the prioritization of evidence needs across the entire organization. Table 2 describes these processes.

We identified instances in which officials used these processes to more effectively focus limited resources to build new evidence through coordination across the agency. For example, CNCS officials described an instance in which agency leadership used the agency’s budget formulation process to prioritize evidence-building activities to address knowledge gaps about the AmeriCorps National Civilian Community Corps (NCCC) program. According to CNCS officials, through the agency’s evidence assessment processes, they found that the agency did not have evidence to fully assess the impact of NCCC programs on members and communities.

Moreover, existing evidence showed that NCCC had experienced a decline in the number of qualified applicants and the retention of its members since 2014. To better understand the performance and results of this program, CNCS officials told us that agency leadership approved funding in fiscal years 2018 and 2019 for NCCC to undertake a multi-year impact evaluation. This evaluation, which is being conducted in conjunction with CNCS’s Office of Research and Evaluation and an independent contractor, is expected to examine the member retention, leadership development, and community impact of NCCC programming.

Officials at each of the selected agencies told us that they were considering how best to meet Evidence Act requirements to take a systematic and coordinated approach to prioritizing evidence-building activities, such as through learning agendas. For example, as described in table 3, Education created a new body in March 2019—the Evidence Leadership Group—to coordinate its evidence-building activities.

Education officials told us that in establishing this new group, they took into consideration our leading practices for collaboration.

Evidence Prioritization Processes at Four Agencies Reflect Leading Practices for Collaboration to Varying Extents

As described in table 3, all five selected agencies identified one or more leadership models for their evidence prioritization processes.

We found that all five of the selected agencies involved at least some relevant participants in their evidence prioritization processes, as summarized in table 4. Our past work related to evidence-building activities identified a wide range of relevant participants to involve. Within agencies, these participants include agency leadership, program staff, and those with functional management responsibilities including budget, human capital, and information technology. External stakeholders include Congress, other federal agencies, state and local governments, grant recipients, and regulated entities.

The five selected agencies include a range of relevant internal participants, although the evidence prioritization process at CNCS does not always include key internal stakeholders. CNCS’s budget hearings involve discussions about prioritizing evidence, but primarily focus on budget formulation decisions. Therefore, agency leaders and budget officials are consistently involved in the hearings, but others, such as the Director of the Office of Research and Evaluation, are not. Involving all key internal stakeholders helps ensure that those involved in a collaborative effort can commit resources, make decisions, and share their knowledge, skills, and abilities. This can also help ensure that the evidence that will be subsequently generated will be useful to decision makers across the organization.

Education and USAID established expectations to seek input from external stakeholders in their evidence prioritization processes. Education’s charter for its recently-established Evidence Leadership Group states that the group is to engage a wide array of external stakeholders in its work. Similarly, for the evidence prioritization activities that occur through USAID’s program cycle and learning agendas, related guidance sets expectations to involve or obtain the perspectives of external stakeholders. As USAID developed its Self-Reliance learning agenda, it sought input from external stakeholders including officials from other federal agencies, organizations that implement USAID programs, and experts in international development, among others.

Three of the selected agencies, however, do not always have mechanisms in place to involve, or consider the evidence needs of, a range of external stakeholders in their evidence prioritization processes. Officials at CNCS, HHS, and DOL told us that, because they consider their prioritization processes to cover internal management purposes and decisions, including external stakeholders is not appropriate. Officials at these three agencies described ways in which they sought input on evidence needs from some stakeholders, such as from interactions with grant recipients and external researchers. However, these agencies have not developed an approach to collect and consider input on evidence needs from all relevant stakeholders to inform their prioritization processes.

Our past work highlights the importance of engaging key external stakeholders, especially Congress, to better understand and meet their evidence needs. Engaging external stakeholders can also create a shared understanding of competing demands facing the agency and ensure that their efforts and resources are targeted at the highest priorities across the agency. Moreover, through this engagement, agencies may find that external stakeholders have, or are aware of, existing evidence that helps the agency meet its needs or provide a fuller picture of performance. Involving a full range of relevant stakeholders in the process for prioritizing new evidence to generate would help each of the selected agencies ensure it is meeting the evidence needs of decision makers within and external to the agency.

Four of the selected agencies—Education, HHS, DOL, and USAID—fully define roles and responsibilities for those involved in their evidence prioritization processes, while the process at CNCS partially reflects this practice, as described in table 5.

CNCS officials said that the primary focus of the agency’s process is budget formulation. Therefore, roles and responsibilities are generally related to that purpose instead of the evidence prioritization activities that also take place during that process.

Clearly defining roles and responsibilities can ensure all participants are aware of and agree upon (1) who will have what responsibilities, (2) how they will organize their joint and individual evidence-building efforts, and (3) how they will make decisions.

As described in table 6, Education and USAID’s processes reflect this practice, while those at CNCS, DOL, and HHS reflect it in part.

Officials at CNCS, HHS, and DOL gave different reasons for why their written guidance and agreements related to evidence prioritization processes do not fully reflect this leading practice.

CNCS’s and HHS’s written guidance primarily focuses on their budget formulation processes, since this is where their evidence prioritization activities take place. Thus, these guidance documents contain information on leadership, participants, and roles and responsibilities related to budget formulation activities, but not all relevant details related to evidence prioritization.

Officials at DOL stated that they do not want to take a “one-size-fits- all” approach to developing learning agendas within the department. They told us they had not developed specific written guidance for that process to provide flexibility to component agencies to develop processes that work best for them in developing their learning agendas.

As we have previously found, documenting a clear and compelling rationale to work together—and how that work will be done and by whom—is a key factor in successful collaboration. By incorporating this leading practice into their existing guidance, CNCS, HHS, and DOL would have greater assurance that they are effectively collaborating to prioritize evidence needs.

Conclusions

Decision makers need evidence to help them address pressing governance challenges faced by the federal government. Agencies undertake a range of efforts at different organizational levels to build evidence to meet their own decision-making needs, as well as those of others, such as Congress. However, these evidence-building activities are fragmented within agencies. Through a more comprehensive and coordinated framework, Evidence Act implementation provides opportunities to improve the effectiveness of federal evidence-building activities.

The five selected agencies have taken steps to improve the coordination of evidence-building activities across their organizations, with Education’s and USAID’s evidence-building activities reflecting the leading practices for collaboration. CNCS, DOL, and HHS would have greater assurance that they are comprehensively considering evidence needs across their individual organizations by fully incorporating leading collaboration practices into their agency-wide efforts to prioritize new evidence to generate. These actions could also help ensure these agencies are better managing fragmented evidence-building activities and more effectively focusing their limited resources to generate evidence to meet decision makers’ needs. In addition, improved coordination could reduce the potential for any unwarranted overlap and duplication in their efforts, and better position the agencies to meet the Evidence Act’s requirements and related implementation actions outlined in OMB’s guidance.

Recommendations for Executive Action

We are making a total of seven recommendations, including three to CNCS, two to HHS, and two to DOL. Specifically: The Chief Executive Officer of CNCS should develop an approach to ensure that all relevant participants are involved in the agency-wide process for prioritizing evidence needs. (Recommendation 1)

The Chief Executive Officer of CNCS should define roles and responsibilities for all relevant participants involved in the agency-wide process for prioritizing evidence needs. (Recommendation 2)

The Chief Executive Officer of CNCS should revise written guidance for the agency-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 3)

The Secretary of Health and Human Services should develop an approach to ensure that all relevant participants are involved in the department-wide process for prioritizing evidence needs. (Recommendation 4)

The Secretary of Health and Human Services should revise written guidance for the department-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 5)

The Secretary of Labor should develop an approach to ensure that all relevant participants are involved in the department-wide process for prioritizing evidence needs. (Recommendation 6)

The Secretary of Labor should revise written guidance for the department-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 7)

Agency Comments and Our Evaluation

We provided a draft of this product for comment to OMB and the five selected agencies—CNCS, Education, HHS, DOL, and USAID. CNCS, Education, HHS, DOL and USAID provided written comments, which are summarized below and reproduced in appendixes V, VI, VII, VIII, and IX, respectively. In addition, CNCS, Education, HHS, USAID, and OMB provided technical comments, which we incorporated as appropriate.

In its written comments, CNCS neither agreed nor disagreed with the three recommendations we directed to it. The agency stated that it believes the planned actions included in its Strategic Evidence Plan, published in September 2019, address those recommendations. The plan includes a goal to strengthen how the agency prioritizes and uses evidence, and outlines various actions intended to achieve that goal. The plan does not include sufficient details to enable us to assess the extent to which its implementation would fully address the issues identified in our review and covered by our recommendations.

Education stated in its written comments that the department is committed to maximizing the performance of its programs, and it views building, using, and disseminating evidence as critical to those efforts. Education also outlined planned and proposed actions that it believes would further its evidence-building activities.

In its written comments, HHS did not concur with the two recommendations we directed to it. In response to both recommendations, HHS stated that the department had developed an approach for including all relevant participants in its process for prioritizing evidence needs. However, according to an HHS official in November 2019, HHS had not yet finalized the approach, and therefore was unable to provide any additional information about it. Thus we could not assess the extent to which HHS’s stated actions would address our recommendations.

DOL agreed with the two recommendations we directed to it, and in its written comments described an action it plans to take to address them. We will monitor DOL’s action, which we believe would likely address our recommendations, if effectively implemented.

USAID, in its written comments, reiterated the agency’s commitment to a comprehensive and integrated approach for its evidence-building activities. In the draft of this report we sent to USAID for its review in October 2019, we included a recommendation to USAID that it ensure that all relevant participants are involved in agency-wide processes for prioritizing evidence needs. USAID subsequently provided documentation that it had not provided previously that showed the agency had taken various steps to seek the input of a range of external stakeholders. We determined that these actions addressed our draft recommendation. Thus, we removed the draft recommendation from our report.

We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Chief Executive Officer of the Corporation for National and Community Service, the Secretary of the Department of Education, the Secretary of the Department of Health and Human Services, the Secretary of the Department of Labor, the Administrator of the U.S. Agency for International Development, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix X.

Appendix I: Objectives, Scope, and Methodology

This report responds to a request that we review the coordination of federal evidence-building activities. This report (1) describes activities selected agencies have taken that align with congressional and Office of Management and Budget (OMB) direction to strengthen evidence- building, and (2) examines the extent to which selected agencies’ processes for assessing and prioritizing evidence needs reflect leading practices for collaboration.

To address both objectives, we analyzed agency documents about federal evidence-building activities and interviewed relevant staff at OMB and officials at five selected agencies: the Departments of Education, Health and Human Services, and Labor; the Corporation for National and Community Service; and the U.S. Agency for International Development.

We selected these five agencies based on their experiences incorporating evidence-building activities into program design and implementation. These experiences include evidence-based approaches such as pay for success projects, performance partnerships, and tiered evidence grants. At the time we made our selection, these five agencies had designed or implemented evidence-based approaches to a greater extent than other agencies we identified.

The agencies we selected vary in size—as measured by budget authority and employees—and organizational structure (see table 7).

For the first objective, we reviewed information from the five selected agencies and identified examples of evidence-building activities within each agency since 2010. We then determined if these examples illustrated actions that aligned with evidence-building statutory requirements and directions from OMB including guidance, memorandums, and activities outlined in the President’s Management Agenda. To do so, we reviewed relevant laws and OMB guidance.

For the second objective, we evaluated processes each selected agency had established to take a coordinated approach to assessing and prioritizing evidence needs across the agency. We compared these processes to four selected leading practices for collaboration identified in our prior work (see table 8).

We selected these four collaboration practices because our past work on evidence-building activities, such as analysis of performance information and program evaluations, has similarly identified them as key approaches related to evidence-building. Table 9 illustrates this alignment for selected past reports.

We conducted this performance audit from April 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Actions Taken by Congress and OMB to Strengthen Federal Evidence-Building Activities and Improve Coordination

The Office of Management and Budget (OMB) has issued several memorandums and other key policy documents that encourage agencies to take actions to strengthen their capacity to build evidence. For example, in a July 2013 memorandum, OMB encouraged agencies to identify proposals for building evidence in their budget requests. Such proposals could be used to improve existing programs or inform decisions about new programs. The OMB guidance highlighted several evidence- based approaches for agencies to consider, including pay for success, performance partnerships, and tiered evidence grants, described further in the text box below.

Examples of Evidence-Based Program Approaches Identified in Office of Management and Budget (OMB) Guidance Pay for success. Pay for success is a contracting mechanism under which final payment is contingent upon achieving specific outcomes. The government specifies performance outcomes in pay for success contracts and generally includes a requirement that contractors assess program outcomes or impacts through an independent evaluation. The evaluators may also generate and analyze performance data to inform program management and improvement during implementation.

Performance partnerships. Performance partnerships allow federal agencies to provide grant recipients flexibility in how they use funding across two or more programs along with additional flexibilities. In exchange, the recipient commits to improve and assess progress toward agreed-upon outcomes by developing and using evidence.

Tiered evidence grants. Tiered evidence grants seek to incorporate evidence of effectiveness into grant making. Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services. The grant generally requires grantees to evaluate their service models as a condition for the receipt of grant funds.

In addition, Congress passed laws aimed at strengthening and better coordinating evidence-building activities, which OMB reinforced through related guidance to implement those laws.

GPRA Modernization Act (GPRAMA). GPRAMA established a framework aimed at taking a more crosscutting and integrated approach to improve government performance. Requirements included in that framework, such as cross-agency priority (CAP) goals and strategic reviews, were intended to strengthen evidence-building activities and improve coordination.

CAP goals. At least every 4 years, OMB is to coordinate with other agencies to develop and implement CAP goals. These goals are to address issues in a limited number of policy areas requiring action across multiple agencies, or management improvements that are needed across the government. The President’s Management Agenda, released in March 2018, established the third set of CAP goals since GPRAMA was enacted. Implementation of each CAP goal can involve evidence-building activities; however, two goals in particular are to focus on them, as described further in the text box.

Cross-Agency Priority (CAP) Goals Focused on Evidence-Building Leveraging data as a strategic asset. The President’s Management Agenda highlights several root causes for the challenges the federal government faces. One root cause is that agencies do not consistently apply data-driven decision-making practices. This agenda states that agencies need to make smarter use of data and evidence to orient decisions and accountability around service and results. The administration established this CAP goal to improve the use of data in decision- making to increase the federal government’s effectiveness.

Results-oriented accountability for grants. According to the June 2019 update for this goal, the federal government uses grants to invest approximately $700 billion each year in mission-critical needs. However, the report states that grant managers report spending 40 percent of their time using antiquated processes to monitor compliance instead of analyzing data to improve results. The administration established this CAP goal to maximize the value of grant funding by applying a risk- based, data-driven framework that balances compliance requirements with demonstrating successful results.

A strategic objective is a type of goal that reflects the outcome or impact the agency is seeking to achieve. The agency is to identify the strategies—the portfolio of organizations, regulations, tax expenditures, programs, policies, and other activities—within and external to the agency that contribute to each strategic objective. As a set, the agency’s strategic objectives are to encompass all of its activities.

Strategic reviews. In its guidance for implementing GPRAMA, OMB established an annual process in which each agency is to review progress in achieving the strategic objectives established in its strategic plans (see sidebar). To do so, OMB’s guidance directs agencies to assess existing sources of evidence to understand the progress made toward each strategic objective and identify where additional evidence is needed to determine effectiveness. In addition, OMB’s guidance states that another purpose of strategic reviews is to strengthen collaboration. It notes that the reviews can do so by identifying and addressing crosscutting challenges and fragmentation.

The Foreign Aid Transparency and Accountability Act of 2016 (FATAA). Among other things, FATAA requires the President to establish guidelines for establishing measurable goals, performance metrics, and monitoring and evaluation plans for federal foreign assistance. In January 2018, OMB issued guidelines for federal agencies that administer foreign assistance—which includes the Departments of Labor and Health and Human Services, and the U.S. Agency for International Development. Among other things, the guidelines provide direction on strengthening evidence-building activities, such as establishing annual monitoring and evaluation plans, and disseminating findings and lessons learned. Agencies were directed to align their monitoring and evaluation policies with the guidelines by January 2019.

The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act). In June and July 2019, OMB released its initial guidance on implementing the Evidence Act. Among other things, this guidance provides direction to agencies on developing evidence-building plans, also known as learning agendas (see text box below). According to OMB, these plans will serve as the driving force for other evidence- building activities required by the Evidence Act.

Prior to the enactment of the Foundations for Evidence-Based Policymaking Act (Evidence Act), both the Office of Management and Budget (OMB) and the Commission on Evidence-Based Policymaking highlighted and recommended the use of learning agendas by federal agencies to strengthen and coordinate their evidence- building activities.

According to OMB’s guidance for implementing the Evidence Act, a learning agenda is to define and prioritize relevant questions and identify strategies for building evidence to answer them. A federal agency developing a learning agenda should involve key leaders and stakeholders to help (1) meet their evidence needs for decision-making, and (2) coordinate evidence-building activities across an agency.

OMB’s guidance stated that the Evidence Act emphasizes the need for collaboration and coordination of agency staff and activities to achieve successful implementation. The guidance provides time frames for a phased approach to implement several Evidence Act requirements. For example, although learning agendas are not required to be published until February 2022, OMB’s guidance includes several interim milestones and deliverables to build toward the final published version.

5 U.S.C. §§ 306, 312.

OMB embedded portions of Evidence Act implementation guidance in its 2019 update to Cir. No. A-11. In it, OMB noted that many of the Evidence Act’s provisions support the Federal Performance Framework for Improving Program and Service Delivery (Part 6 of the Circular), which provides guidance for implementing GPRAMA and other related laws and policies. OMB Cir. No. A-11, at § 200.2 (2019).

Appendix III: Examples of Evidence-Building Approaches at Five Selected Agencies

Evidence-Building Approaches Used by Selected Agencies

We identified 20 examples of the five selected agencies’ incorporating evidence-based approaches in their program design and implementation. Table 10 describes each of these examples.

Selected Agencies’ Use of Evidence- Based Approaches Aligned with OMB Direction

OMB’s July 2013 memorandum stated that agencies’ use of evidence- based approaches could help strengthen agencies’ abilities to improve program performance by using experimentation and innovation to test new approaches for service delivery. In addition, it noted that these approaches can be used to (1) generate new knowledge, and (2) apply existing evidence about approaches found to be effective.

Generate new knowledge. OMB guidance notes that new knowledge can be used to improve existing programs or inform decisions about new ones. For example, Education designed the First in the World program to generate evidence about effective strategies for improving college completion rates for underrepresented, underprepared, or low-income students. Program officials told us that, prior to the issuance of the 2014 grant solicitation for the program’s first year, Education had limited evidence of effective approaches. As noted in the solicitation, Education sought to expand its evidence base about effective approaches through the first round of grant awards.

Using a tiered evidence approach, the program awarded grants to institutions of higher education to implement and evaluate the effectiveness of approaches, such as coaching or advisement services, intended to increase the number of these students who complete postsecondary education. The first round awarded grant funds to projects in a single evidence tier to test and evaluate the effectiveness of approaches. Education officials told us that after the program’s first year, they conducted a literature review to identify approaches that were supported by some evidence of their effectiveness. Using this evidence, Education created a second tier for the 2015 grant awards, for which grantees could receive increased funding by implementing one of the program designs identified in the literature review.

Officials told us they intend to publish the final results of First in the World grant recipient evaluations in Education’s What Works Clearinghouse. Evaluation results will not be available until after the completion of the grant periods, the first of which ended in September 2019. However, Education officials told us that the evidence they have generated to date has improved their understanding of services that could potentially help at-risk students complete post-secondary education.

Apply effective approaches. To meet increased demand for services in a constrained resource environment, OMB’s guidance encourages agencies to allocate resources to programs and approaches backed by strong evidence of effectiveness. In addition, OMB’s guidance encourages agencies to “scale up” effective program approaches by expanding them to a larger or different group of recipients. For example, USAID created the Development Innovation Ventures program in 2010 as a tiered evidence grant competition to create a portfolio of innovative approaches to reducing global poverty. This program provides funding in three tiers, with greater funding provided to those approaches with greater evidence of effectiveness. These three tiers (which USAID referred to as stages) were as follows: 1. Proof of concept. The program provided smaller grants to test the viability of an innovative approach; 2. Testing and positioning for scale. Grantees determined, through rigorous assessments, whether their approach could achieve greater results and also be implemented successfully at a larger scale; and 3. Scaling. The program funded the expanded implementation of an effective approach within one country or replicated that approach in another country.

For example, from 2013 to 2015, the Development Innovation Ventures program awarded stage two funding to a nonprofit organization in India. The organization designed a methodology to help primary school students improve reading skills by grouping students according to skill level, instead of age or grade and tailoring lessons to their learning level. Evidence generated through randomized control trials showed that the approach was effective. Based on that evidence, in 2017, the program awarded stage three funding to replicate the approach in Zambia.

Appendix IV: Additional Examples of Selected Agencies’ Coordination of Evidence-Building Activities

Earlier in this report, we discussed agency-wide evidence assessment and prioritization processes established by the five selected agencies. In addition to those processes, officials described other actions they have taken to coordinate fragmented evidence-building activities across organizational levels (see table 11). Some of these actions were ad hoc (i.e., they did not occur regularly) or not comprehensive in nature (i.e., they did not focus broadly across different sources of evidence or did not cover the entire agency).

Appendix V: Comments from the Corporation for National and Community Service

Appendix VI: Comments from the Department of Education

Appendix VII: Comments from the Department of Health and Human Services

Appendix VIII: Comments from the Department of Labor

Appendix IX: Comments from the U.S. Agency for International Development

Appendix X: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director), Daniel Webb (Analyst-in-Charge), Amanda Prichard, Kelly Turner, and Brian Wanlass made significant contributions to this report. Valerie Caracelli, Jacqueline Chapin, Ann Czapiewski, Steven Putansu, and Andrew J. Stephens also made key contributions.