Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early Implementation and Inform Future Federal Efforts
Released: 2016-09-21T00:00:00

Letter

Background

According to OMB’s 2013 memorandum on the Next Steps in the Evidence and Innovation Agenda …evidence includes evaluation results, performance measures, and other relevant data analytics and research studies, with a preference for high-quality experimental and quasi-experimental studies.

Since 2010, OMB has used its annual budget guidance to encourage federal agencies to use evidence to consider the effectiveness of their programs and to institutionalize the use of evidence to foster innovation rooted in research and rigorous evaluation. In a 2013 memorandum, OMB identified areas in which agencies could improve their use of evidence (see sidebar), including (1) strengthening evaluation capacity and proposing new evaluations; (2) developing high-quality, low-cost evaluations and rapid, iterative experimentation; (3) using innovative outcome-focused grant design; and (4) increasing agency capacity to use evidence. In 2015, OMB directed agencies to provide an overview of evidence-building strategies and identify related priorities with their fiscal year 2017 budget submissions.

To incorporate evidence into grantmaking, the White House and OMB have encouraged agencies to use tiered evidence grants, which link funds to an evidence framework. When applying for grants, grantees demonstrate the strength of evidence on their service models and agencies award funding based on the level of evidence (see figure 1). Smaller awards are used to test new and innovative service models; larger awards are used to scale service models with strong evidence.

Grantees generally evaluate their service model during the grant period. As a result, tiered-evidence grant programs have a goal of identifying evidence-based service models that can be replicated. According to an OMB memorandum on evidence and innovation, the goal is that, over time, service models move up tiers as evidence becomes stronger. OMB staff said that other goals of tiered evidence grants include devoting more resources to service models with stronger evidence, allowing for testing of innovations, and encouraging rigorous evaluation.

From fiscal year 2010 to 2015, agencies have awarded approximately $4.1 billion to the tiered evidence grants we reviewed. As shown in figure 2, each agency uses its own names and descriptions for its evidence tiers. While Education, DOL, and CNCS use a three-tier model, HHS uses a two-tiered model, where the agency makes a larger overall investment in replicating evidence-based practices by awarding more grants in its high tier. For example, HHS awards 75 percent of funds to replicating service models with moderate or strong evidence for the Teen Pregnancy Prevention Program, and awards 25 percent of funds to rigorously evaluate promising approaches.

Tiered Evidence Grants Require Evidence and Evaluation Requirements throughout the Grant Life Cycle

While there is substantial variation among grant types, competitively awarded federal grants generally follow a life cycle of pre-award (announcement and application), award, implementation, and closeout. To implement tiered evidence grants, the four agencies in our review add evidence and evaluation requirements throughout the federal grant life cycle. These added requirements include: (1) assessing the evidence base and identifying evidence-based approaches, (2) implementing evidence-based approaches with fidelity, (3) conducting independent evaluations, and (4) disseminating evaluation results, as shown in figure 3.

The four federal agencies in our review had varied approaches to identify evidence-based approaches for tiered evidence grants. Education, DOL, and CNCS first developed definitions of each evidence tier that grantees must meet to qualify for a grant award as well as the program priorities for the grant competition. Evidence definitions included requirements, such as the type and number of studies on the service models, as well as the scale of service models studied. The rigor and scale of the evidence requirements increased from tier to tier. For example,

Preliminary evidence tier. CNCS’s Social Innovation Fund’s preliminary evidence tier was designed for service models that have preliminary evidence based on a reasonable hypothesis supported by credible research findings.

Moderate evidence tier. Education’s Investing in Innovation Fund’s moderate evidence tier required at least one experimental study of effectiveness of the grantees’ chosen service model, or one quasi- experimental study that was conducted on a large and multi-site sample. Grantees could validate their service model by testing the service model in a new location, testing it on a new or bigger population, or tweaking an aspect of their service to see if it still works on their target population.

Strong evidence tier. DOL’s Workforce Innovation Fund’s strong evidence tier was designed to support significant expansion of service models that had previously demonstrated strong evidence of positive results and will be evaluated using a randomized controlled trial evaluation.

For HHS, the evidence tiers for the Teen Pregnancy Prevention Program are defined in statute. For example, in the appropriation providing funding for the Teen Pregnancy Prevention Program for fiscal year 2016, 75 percent of amounts provided for the grant were to be used for a strong evidence tier to replicate programs that have been proven effective through rigorous evaluation to reduce teenage pregnancy, and behavioral risk factors underlying teenage pregnancy. The other 25 percent were to be used for a preliminary evidence tier to conduct research and demonstration projects to develop and test additional models and strategies.

After the four agencies in our review defined the evidence tiers, they used the standards to assess the available evidence on the service models. The agencies varied in terms of when they assessed the evidence:

Before announcement of grant opportunity. For the Teen Pregnancy Prevention Program and the Federal Home Visiting Program, HHS reviewed the available evidence on teen pregnancy prevention and home visiting models before it announced the grant opportunities and identified a list of approved evidence-based models. Tier one (strong evidence tier) grantees that replicated evidence-based models selected an evidence- based model from the approved list that met the needs of their target populations when applying for the grant. As of July 2016, the Teen Pregnancy Prevention Program included 44 service models and the Federal Home Visiting Program included 17 service models. HHS periodically updates the approved models.

During review of grant applications. Grantees of DOL’s Workforce Innovation Fund and Education’s Investing in Innovation Fund identified in their grant applications the evidence-based service models they planned to use. DOL and Education officials assessed the evidence that supported the service models when they reviewed the grant applications and make award decisions. For example, Education’s Institute of Educational Sciences’ What Works Clearinghouse—the entity within Education that is focused on research and evaluation—reviewed the citations of the studies that supported Investing in Innovation Fund grant applications to determine whether they met the evidence requirements.

After grant award is made. For the Social Innovation Fund, CNCS made awards to intermediary organizations, which then made awards to subgrantees to implement service models. CNCS provided the intermediary organizations with guidance on the levels of evidence, which they used to assess the evidence for their subawards. CNCS reviewed and confirmed the intermediaries’ decisions before they made the final awards.

Grantees were responsible for delivering their evidence-based service models with fidelity—in a manner consistent with how the program was designed. For example, in its guidance to Teen Pregnancy Prevention Program grantees, HHS defined fidelity as the degree to which a program is implemented with adherence to its core components – the key ingredients related to achieving the outcomes associated with the program model. The core components included a model’s content and how it is delivered. Fidelity is important because delivering a service model as it was designed increases the likelihood that the participants served will experience similar positive outcomes to those found in the original evaluation study.

The five grant programs we reviewed had evaluation requirements, but they generally had two different evaluation approaches. For DOL, Education, and CNCS tiered evidence grants, every grantee was required to contract with a third party evaluator to conduct an independent evaluation of its service model and submit it to the respective federal grantmaking agency at the end of the grant period. DOL also commissioned a crosscutting analysis of the findings from, and experiences with, the evaluations of the projects under the Workforce Innovation Fund.

For HHS, the Federal Home Visiting and Teen and Pregnancy Prevention Programs used different evaluation approaches for each tier that included both grantee and federal government-led evaluations. For the preliminary evidence tier to test promising approaches of both grant programs, grantees were required to conduct an independent evaluation. For the strong evidence tier of the Federal Home Visiting Program, grantees receiving competitive awards or tribal grants were also required to conduct grantee-led evaluations.Also, state formula grantees are encouraged to conduct grantee-led evaluations. For the strong evidence tier to replicate evidence-based service models of the Teen Pregnancy Prevention Program, all grantees receiving a grant award of greater than $1 million were required to conduct an independent evaluation. HHS also is conducting programwide evaluations of both grant programs (see sidebar).

When conducting independent evaluations, grantees we reviewed hired an independent evaluator to conduct evaluations. This work included developing an evaluation plan, collecting and analyzing data, and completing an evaluation report to communicate the service model results. The four federal agencies we reviewed contracted with a separate external national evaluation coordinator to provide technical assistance to the independent evaluators and grantees on their evaluations. The national evaluation coordinators developed guidance on evaluations, reviewed evaluation plans, provided ongoing assistance on conducting the evaluations, and provided feedback on the final reports. The four agencies we reviewed encouraged grantees to conduct rigorous evaluations, such as randomized controlled trials or studies using quasi-experimental designs. We have previously reported that when developing an evaluation design, an evaluator should select appropriate measures and comparisons that will permit valid conclusions. The evaluator should explore a variety of options available to collecting and analyzing information, and choose alternatives that will best address the evaluation objectives within available resources.

During and after the grant closeout stage, the four agencies we reviewed disseminated the evaluation results to various audiences. For example, Education and CNCS posted links to completed evaluations from the Investing in Innovation Fund and the Social Innovation Fund, respectively, online. Grantees in our review told us that they also took steps to communicate their evaluation results by, for example, posting links to them on their websites and presenting at conferences. Grantees can use their evaluation results to support applications for future tiered evidence grant awards. The Teen Pregnancy Prevention Program and the Federal Home Visiting Program included grantee evaluations in their evidence reviews.

Agencies Identified Potential Benefits of Tiered Evidence Grants and Described Implementation Challenges

Federal Agencies Identified Potential Benefits of Tiered Evidence Grants

Officials that we interviewed from the four agencies in our review identified the following potential benefits of tiered evidence grants:

Provide incentives to implement service models that are supported by evidence. According to OMB staff, tiered evidence grants provide more flexibility than some traditional grants—such as those for which program models are defined in statute or regulation. Tiered evidence grants target funds to the most effective interventions based on the evidence of the service model’s effectiveness without being limited to a particular intervention or operating model. Agencies generally define outcomes for the target population in the notice of funding availability. Grantees use service models with the most robust evidence base to meet a program’s desired outcomes. Further, tiered evidence grants create incentives for grantees to use evidence-based service models because they award larger funding amounts to approaches with more evidence and grantees may be motivated to use evidence-based models because of the evidence supporting their outcomes.

Build the evidence base. Tiered evidence grants generally require grantees to conduct rigorous, independent evaluations, which will contribute to the evidence base for addressing a given social challenge. Tiered evidence grant awards play a role in expanding the research on service models to include new geographic regions or variations in the target population and in the services themselves. These evaluations will be added to agencies’ databases of research studies (see text box). They are available to the public, policymakers, and service organizations to inform future efforts and contribute to the increased use of evidence- based models over time.

Build grantee capacity for evaluation. Experience with tiered evidence grants and third-party evaluations can help grantees build capacity for evaluation. According to a national assessment of the Social Innovation Fund, there has been substantial improvement in grantees’ internal evaluation capacity over time.

Apply evidence and evaluation requirements to other grant programs. Three of the four the agencies in our review applied evidence and evaluation requirements from the tiered evidence grant model to other grant programs. For example, CNCS has incorporated evidence requirements from the Social Innovation Fund into its existing AmeriCorps State and National grant program. AmeriCorps increased the total number of points assigned to evaluation and evidence effectiveness in the review and scoring process for the service model proposed by the grant applicants. Education amended its Education Department General Administrative Regulations, which outline requirements for its grant programs, to include definitions of four levels of evidence, priorities, and selection criteria related to evidence and evaluation. Education officials said that any program office can use the definitions to add evidence or evaluation requirements to their competitive grants in a manner consistent with the purpose and goals of the program. Education has also developed a process to consider the use of evidence and evaluation requirements in competitive grants when developing its annual spending plan (see textbox).

Education’s Use of Evidence in Grant Competitions According to Education officials, the agency is building on lessons from the Investing in Innovation Fund and increasing its use of evidence and evaluation requirements in other competitive grants, as appropriate.

Education has an Agency Priority Goal to increase the use and generation of credible evidence on what works and what does not work in education. By September 30, 2017, Education aims for at least 20 percent of new competitive grant dollars to support evidence-based strategies. Also, by September 30, 2017, Education will increase by 20 the number of Education-funded project evaluations that provide credible evidence about what works in education. In fiscal year 2016, Education encouraged program offices in building their spending plans to use evidence and evaluation definitions in agency regulations to the fullest extent practicable. In spending plans, program offices should identify competitions where it would be appropriate to ask applicants to (1) submit evidence of effectiveness in support of their proposed projects, or (2) demonstrate a plan for rigorously evaluating their proposed project once it is implemented.

Agencies and Grantees Faced Challenges in Implementing Evidence and Evaluation Requirements for Tiered Evidence Grants

Evidence and evaluation requirements add a layer of activities to tiered evidence grants compared to some other federal grants. Officials from the five federal grant programs and 18 grantee organizations in our review reported that they faced challenges in implementing aspects of the evidence and evaluation requirements. At our facilitated discussion on tiered evidence grants, agency officials agreed that the primary challenges are: identifying evidence-based service models, implementing them with fidelity, communicating evaluation results.

As discussed below and in the following section, the four agencies we reviewed took steps to address challenges in implementing evidence and evaluation requirements.

Agency and Grantee Capacity to Identify Evidence-based Service Models

Agency officials agreed at our facilitated discussion that grantees faced challenges in their capacity to understand the evidence base and agencies faced challenges in their capacity to review grant applications.

Grantee capacity to understand the evidence base. At our facilitated discussion, agency officials agreed that some grantees did not have the technical skills and infrastructure to understand the evidence base and select the evidence-based model that would best fit their target populations. For example, an official at our facilitated discussion said grantees faced challenges in selecting an evidence-based model that was a good fit for their communities. When a model did not fit a community, grantees had to make many adaptations to the evidence- based model. As a result, grantees were less likely to achieve the greatest possible impact for the population because the service model was not implemented as it was designed.

Agency capacity to review grant applications. The agencies that managed the five grant programs in our review also faced challenges in reviewing the evidence and evaluation portions of grant applications. The agencies generally used panels of expert reviewers and program staff to review grant applications. Reviewing evidence and evaluation requires specific technical knowledge and skills, such as reviewers that have experience with rigorous evaluation. Education officials told us that the Investing in Innovation Fund has used reviewers with evaluation expertise to review and score the evaluation criteria. They said that given the success of using these reviewers, the Investing in Innovation Fund officials work with other Education programs that have rigorous evaluation requirements to help structure their peer reviews so they may also use these evaluation experts. Also, three of the five grant programs in our review ran multiple grant competitions simultaneously—one for each tier—which required additional capacity from their program and evaluation offices. Further, during our facilitated discussion, agency officials said that it is challenging to provide technical assistance sufficient to meet the varied needs of applicants during the application phase. For example, the officials told us that technical assistance must be broad so it is applicable to all organizations that are applying for grants. They said that while one-on-one technical assistance would benefit applicants, their agencies lack the capacity to help every single applicant, and it is challenging for those agencies to ensure that similar technical assistance is available to all applications that need it. In many cases, intensive pre-application technical assistance is not permissible under grants policy, as it would be impossible to ensure that no one applicant has a competitive advantage.

Grantee Capacity to Implement Evidence-based Models with Fidelity

Agency officials at our facilitated discussion agreed there are two potential challenges with maintaining fidelity for evidence-based service models:

Grantees delivering service models as intended. At our facilitated discussion, agency officials said that grantees faced challenges in collecting quality data and using the data to monitor whether they are implementing their service model with fidelity. Some of the grantees we interviewed said that to monitor fidelity, they had to build new systems to collect data—such as attendance data—on how their service models were implemented. In addition to using data to monitor fidelity, agency officials that participated in our facilitated discussion said that grantees faced challenges in using data during the grant period to improve the implementation of evidence-based service models. For example, officials said some grantees still see performance data simply as a reporting requirement, and do not use the data to make decisions about their programs.

Managing adaptations to service models. Within tiered evidence grants, grantees generally deliver existing service models to new groups of individuals, at times in a new location with varying degrees of similarity to the population studied in the original evaluation. Therefore, grantees may need to make adaptations to the evidence-based model to best serve their population. For example, a Teen Pregnancy Prevention Program grantee we interviewed said that its organization made four adaptations to the evidence-based model selected because the model was studied in the 1990s and the curriculum was out of date. The adaptations included updating information on HIV/AIDS for medical accuracy, modifying curriculum scenarios to be more inclusive and culturally relevant, adding basic sexual health information to reach a younger population, and changing when the program would be delivered from a weekend program to during the school day.

At our facilitated discussion, agency officials said there was a tradeoff between implementing the evidence-based model with fidelity and adjusting the model to respond to new information or specific circumstances. If adaptations compromise the core components of a model, the grantees may be unable to attribute their results to the service model and may not have comparable outcomes. At our facilitated discussion, one agency official explained that this is challenging because grantees are accustomed to serving people as their highest priority, and their instincts are to make changes to meet the needs of their populations without considering how the changes would affect implementation fidelity or the evaluation of their service models. Model owners can also play a role in managing fidelity of a service model (see text box).

Service Model Owners Add a Layer of Complexity to Tiered Evidence Grants Department of Health and Human Services (HHS) officials said the implementation and scale-up of evidence-based models is complex because it involves model developers that own the evidence-based models. Grantees pay a fee to implement the model. The model developers generally provide grantees with the content of the service model, training, and technical assistance.

HHS officials said that working with model developers has created some challenges for tiered evidence grants. Because they have a role in approving adaptations and have an incentive to approve adaptations, grantees would be more likely to use their models again in the future. There is a need for the federal agencies to have a role in approving adaptations and enhancements to provide another level of review and oversight. For example, the Federal Home Visiting program changed its oversight process so HHS approves any adaptations in addition to the model developers’ approval.

Teen Pregnancy Prevention Program grantees we interviewed also described challenges working with model developers including additional reporting requirements. One grantee also said that it would be helpful for HHS to negotiate fees in advance of the grant announcement so each grantee does not need to individually negotiate fees with the model developer.

Grantee Challenges in Conducting Evaluations

All of the grantees included in our review reported that they faced challenges in fulfilling requirements for rigorous evaluation in tiered evidence grants, for example, when planning for their independent evaluations.

Identifying and hiring an independent evaluator. Some grantees had not previously worked with an independent evaluator and were not familiar with the qualifications they should look for in an evaluation. For example, they faced challenges in developing a description of the requirements and hiring an evaluator with the appropriate experiences and skill set. State and local governments also found it difficult to procure an evaluator within the grant’s timeframe.

Developing an evaluation plan, including choosing the appropriate evaluation type. In some cases, grantees are required to submit an evaluation plan as part of their grant application. At our facilitated discussion, agency officials said grantee capacity to develop an evaluation plan varies widely and some grantees lack understanding of what evaluations are and the related planning involved. Grantees may need to work with independent evaluators on their evaluation plans in advance of submitting their grant applications. However, grantees may not have the financial resources to obtain evaluator assistance with grant applications. Grantees we interviewed that had an existing relationship with an evaluator said that evaluators’ assistance in developing their grant applications facilitated their ability to apply for a tiered evidence grant award. Both grantees and evaluators faced challenges in choosing the evaluation type best suited to the service model. At our facilitated discussion, agency officials said that some evaluators used the evaluation type with which they had the most experience instead of the type that would be most appropriate.

Budgeting for evaluations. At our facilitated discussion, agency officials agreed that it can be challenging for some grantees to budget for evaluations because they want to dedicate as much funding as possible to serving their target populations. One agency official also noted that grantees may be unfamiliar with the resources needed to conduct rigorous evaluations, such as randomized controlled trials and quasi- experimental designs.

We have previously reported that randomized controlled trial evaluations are often difficult to carry out because evaluators must be able to maintain a treatment and control group. Similarly, officials from agencies we reviewed, and some grantees, also described challenges that were specific to conducting a randomized controlled trial evaluation. For example, some grantees from the programs in our review face the following challenges:

Recruiting sufficient numbers of participants for a treatment and control group. Some grantees faced recruitment challenges because they needed to recruit twice the number of people they planned to serve to maintain a control group for a randomized controlled trial. One grantee said it is particularly challenging to recruit participants for a randomized controlled trial because participants must agree to participate in the programing knowing there is a chance they will not receive the new service and instead receive the traditional services offered.

Retaining participants for data collection. Some grantees experienced challenges in retaining participants to collect data for the evaluation after the service delivery ended, particularly for the control group. For example, for a Teen Pregnancy Prevention Program grantee, many of the participants graduated from high school and moved away during the evaluation period. This created challenges for the grantee to locate participants and administer follow-up surveys.

Navigating potential ethical issues with using a control group. Some grantees faced challenges in managing potential ethical issues in using a control group. We have previously reported that in randomized controlled trials there can be concerns that the control group should not be harmed by withholding needed services. A Workforce Innovation Fund grantee expressed concerns about using a control group and possible backlash from the community for withholding the service from individuals in the control group who need the services. The grantee said it planned to advertise the program as a lottery to help address this challenge.

Grantees also faced systemic challenges to conducting evaluations:

Accessing administrative data. Some grantees that we interviewed reported encountering difficulties obtaining administrative data collected by federal, state, or local agencies to measure outcomes for the evaluation because of legal or privacy reasons. For example, a Workforce Innovation Fund grantee said that it was challenging or not possible to get access to administrative data, including unemployment data and wage data, from the states where it was implementing its service model. Challenges in accessing administrative data have led to grantees obtaining information for some states and not others, leading to uneven reporting. We have previously reported on challenges agencies and stakeholders face in using administrative data to conduct evaluations of programs.

External policy changes. Some grantees faced challenges resulting from external policy changes because they potentially prevented a tiered evidence grant award from operating as initially intended. For example, Education officials and Investing in Innovation Fund grantees said that changes in state-level student assessment policies created challenges to conducting rigorous evaluations. An Investing in Innovation Fund grantee told us that its analyses for its evaluation were complicated by changes in the state testing regime in the state of California.

Conducting evaluations within a 3- to 5-year grant period. Tiered evidence grants generally have an initial planning year, up to 3 years to implement the service model, and an additional year for data analysis and reporting. Agency officials agreed during our facilitated discussion that it can be challenging to conduct an evaluation within this 3-5 year grant period because some data may be unavailable or not measured until some timeframe after the service model is delivered. For example, one DOL Workforce Innovation Fund grantee used unemployment insurance wage record data to measure the employment and earnings outcomes of participants. According to DOL officials, unemployment wage record data typically are unavailable for two to three quarters after they are reported. So, for later cohorts of participants, there was not enough time after program completion to obtain available data before the end of the 5 year grant period. As a result, the grantee’s evaluation will not include unemployment data for the participants in the later cohorts. Education officials also described challenges in conducting evaluations during the grant period for the Investing in Innovation Fund and said the department has taken steps to address this challenge. In 2013, Education updated its Education Department General Administrative Regulations to allow grant periods for competitive grants to be extended beyond 5 years with funding for data collection and analysis. Education approves extensions on a case-by-case basis when extending the project will result in better data and higher quality evaluation. For example, according to Education officials, a 2010 Investing in Innovation Fund grantee is implementing a service model in middle and high schools to increase high school graduation rates. Education awarded an evaluation grant award to continue data collection and analysis for an additional four years beyond the original grant period to track students’ high school completion and graduation rates.

Agency Challenges in Providing Oversight and Technical Assistance

At our facilitated discussion, agency officials generally agreed that their agencies also faced challenges in overseeing and providing technical assistance to tiered evidence grant programs. The agencies provided extensive coaching and technical assistance to walk grantees through the evidence and evaluation requirements in tiered evidence grants, which requires additional capacity to implement evaluation requirements compared to other federal grant programs. At the facilitated discussion, agency officials emphasized that agency grant officers for tiered evidence grants need both knowledge of evidence and evaluation as well as the ability to provide ongoing coaching to grantees to help them meet the evidence and evaluation requirements. The officials said that in some cases agencies faced challenges in shifting the culture among their grant officers from a compliance orientation to the addition of a coaching model of oversight to help build grantees’ capacity.

Grantee and Agency Challenges in Communicating Evaluation Results

Officials from the four agencies in our review expressed concern about how to accurately summarize and communicate complex and nuanced evaluation findings to different audiences.

Confusion around evaluation results. At our facilitated discussion, officials agreed that there is confusion among policymakers around evaluation results of randomized controlled trials or quasi-experimental studies. These evaluation types generally measure several specific outcomes and can show a positive, negative, or null finding. A positive finding occurs when being in the treatment group is associated with a positive outcome, relative to control group. A negative finding occurs when being in the treatment group is associated with a negative outcome, relative to control group. A null finding occurs when there is no statistical difference in outcome between treatment and control groups, which may occur when a sample size is too small to be statistically significant. Mixed results, where some findings are positive, some are negative and some are null, are also common. At our facilitated discussion, agency officials expressed concerns that policymakers may use negative, null, or mixed evaluation results from one grant award as a justification for discontinuing an entire tiered evidence grant program. They told us that there is a risk that policymakers will interpret evaluation results based on one outcome to mean that a service model is or is not working. However, the officials said that evaluation results can be nuanced, and negative, null, and mixed findings can be used to consider what led to the findings and to adjust the service model to improve future results. Similarly, at our facilitated discussion, agency officials expressed concerns about the risk of a single study being interpreted to represent outcomes of an entire tiered evidence grant program. Further, the officials said it is a challenge to communicate that null or negative findings have a valuable role in building the evidence base because they can identify areas where a model can be altered to improve outcomes for its target population. One official told us that before they did not know whether the service models were working and at least now they will know what does not work.

Expectations about grantees moving up evaluation tiers. Another challenge is the expectation from policymakers that grantees will move up from one evidence tier to the next evidence tier. For example, according to agency officials that attended our facilitated discussion, there is an expectation that a grantee that receives a preliminary evidence tier award will qualify for a moderate evidence award at the end of the grant period. In practice, evaluation results have led to few of the hundreds of tiered evidence grantees moving up tiers, in part, because a single evaluation may not be enough to qualify for highest level of evidence tiers. Moreover, in some cases, grant recipients moved down tiers. Such was the case with Education’s Investing in Innovation Fund, where some grantees applied for a lower tier in subsequent rounds to test a variation or adaptation of their existing service model. For example, a teacher professional development service model focused content on reading moved from the validation (moderate evidence) tier to the development (preliminary evidence) tier to test an online version of the professional development model. OMB staff said that moving up tiers to scale an evidence-based service model and moving down tiers to test an adaptation of a service model both highlight successful outcomes of the tiered evidence grant model.

Agency Officials and Grantees Identified Key Factors That Facilitate the Use of Tiered Evidence Grants

Officials from the four agencies in our review and grantees identified five key factors that facilitate the use of tiered evidence grants by addressing challenges in administering the evidence and evaluation requirements in tiered evidence grants (see text box). We have also previously reported on strategies agencies can use to facilitate the use of evaluation in program management and policy making, including demonstrated leadership support for evaluation and engaging stakeholders throughout the evaluation process.

Comprehensive Agency Oversight Helped Grantees Fulfill Evidence and Evaluation Requirements

Officials from the five grant programs in our review said that comprehensive oversight throughout the grant life cycle is essential to address grantee capacity challenges and facilitate the successful use of tiered evidence grants. Oversight of grantees is also important to reasonably assure that grants are used for their intended purposes and that risks of fraud, waste, and abuse are minimized. Although federal oversight of grantees is not unique to tiered evidence grants, at our facilitated discussion, agency officials said that their agencies provided closer oversight on the evidence and evaluation requirements. For example, they approved evaluation plans and adaptations to service models. Agencies took several steps to create oversight mechanisms to address challenges grantees face with evidence and evaluation requirements:

Cooperative agreements provided for closer oversight. Both cooperative agreements and grants provide funding to carry out approved federal activities, but a cooperative agreement allows for substantial programmatic involvement from the federal agency. Officials from three of the four agencies in our review—Education, HHS, and CNCS—said they structured their programs as cooperative agreements because they allowed the agencies to more closely oversee grantees compared to other grant programs, especially in how grantees adhered to the fidelity of service models. Cooperative agreements are used instead of grants when substantial involvement is expected between the agency and the recipient.

Clear and upfront requirements and expectations for evaluations.

The four agencies in our review included specific requirements for independent evaluations in the notice of funding opportunities for tiered evidence grants to help ensure that grantees would be prepared to meet the evidence standards for their evaluations. For example, Education required grantees in the Investing in Innovation Fund’s Scale-up (strong evidence) and Validation (moderate evidence) tiers to identify the program’s elements that can be replicated by other entities, and in a variety of contexts for a variety of students.

Agency approval of evaluation plans. An evaluation plan spells out the research questions and methodology of an evaluation. Officials from the four agencies we reviewed told us that completing an evaluation plan is essential because it provides the groundwork for a robust evaluation, including quality data collection, analysis, and reporting. For example, for the Workforce Innovation Fund, DOL required grantees to submit an initial evaluation design report prepared by an independent evaluator within 9 months after the grant award. DOL and the program’s national evaluation coordinator assessed the content of initial evaluation reports and provided comments. Grantees then worked with DOL and the national evaluation coordinator to have their final evaluation design reports approved within 11 months of the grant award.

Using a pilot period. Similarly, at our facilitated discussion, agency officials said that a planning and piloting year at the beginning of the grant period facilitates grantee implementation of tiered evidence grants because it allows time for grantees to plan and set up the infrastructure for their service models and evaluations. Some grantees said that it was easier for grantees in moderate or strong evidence tiers to develop an evaluation plan because their service models were more fully defined from the beginning of the grant’s implementation. Some grantees said they found it particularly helpful to use part of the grant period as a pilot program. For example, a Teen Pregnancy Prevention grantee said they used the first year of the grant to test its service model in three different environments.

Reporting requirements to ensure fidelity to the service delivery model. The four agencies we reviewed used mechanisms to address challenges with implementing service models with fidelity, including requiring grantees to report specific data elements and conducting in- person observations. For example, to help ensure grantees implemented their service models with fidelity, HHS’s Teen Pregnancy Prevention Program required grantees to collect data on fidelity measures reflecting the extent to which community-based organizations, schools, and other organizations adhered to the service models’ core elements when administering the models in classrooms and other settings (see text box). Agency officials at our facilitated discussion said these measures are very important to ensure quality implementation. Some grantees also said they found the measures to be useful. For example, a Teen Pregnancy Prevention Program grantee said that the independent observations were particularly helpful during the pilot year of program implementation because they helped to identify inconsistencies in how the service providers delivered the messages in the curriculum at different locations. The grantee organization addressed the inconsistencies across different locations by having service providers from different organizations teach together for the later years of the project to ensure lessons were taught in a consistent manner.

Fidelity Monitoring for the HHS Teen Pregnancy Prevention Program Teen Pregnancy Prevention Program grantee data on outcomes to monitor whether they implemented their evidence-based service model with fidelity, include: attendance data from participants for all sessions completed; a facilitator self-assessment or fidelity monitoring log for each session implemented; information on planned and unplanned adaptations; and observation data on service model administrators from independent observers for 5 to 10 percent of all sessions implemented.

HHS reported that for the fiscal year 2010-2014 cohort 95 percent of all sessions were implemented with high fidelity and 92 percent of all sessions that were independently observed were rated as either very high or high quality.

Agency approval and dissemination of adaptations to service models. To address the challenge of grantees making adaptations to approved service models, the four agencies we reviewed required grantees to obtain approval for making adaptations. For example, HHS assigned some project officers for the Teen Pregnancy Prevention Program to serve as model leads for individual evidence-based programs. This process allowed the project officers to be aware of all approved adaptations that had been made to the model, and to share this information with all other grantees that are implementing that model. In addition, HHS compiled and analyzed all approved adaptations from the program’s first round of grantees from 2010 to 2015. It also published a summary of adaptations that had been made by the grantees for each model so future grantees and others could learn from former grantees. However, some grantees said that it was time consuming to obtain approval for adaptations. For example, a Workforce Innovation Fund grantee stated that it felt bound to achieve the outcome goals in its evaluation plan and that it was difficult and time consuming to obtain modifications that deviated from the grant’s application.

Providing funding for evaluations. All five tiered evidence grants we reviewed allowed for grantees to spend grant funds on evaluations; however, agencies structured the funding differently. Officials from the four agencies in our review said the very fact their authorizing statutes allowed for federal funds to be used for various evaluation activities was a facilitating factor. Further, the officials stated that some programs are not a good fit for tiered evidence grants because they are prohibited from using federal funds for evaluation purposes and require all funds to provide services. Since independent evaluations were new to some grantees, agency officials at our facilitated discussion said they had to work closely with grantees to ensure they provided sufficient funding for evaluations. The officials said that grantees are accustomed to quickly providing services after receiving grant funding, and that grantees faced challenges in budgeting for independent evaluations.

Agency Technical Assistance Throughout the Grant Lifecycle Helped Build Grantee Capacity

The four agencies in our review provided evaluation technical assistance from the design stage to the analysis and reporting stages of evaluations to help ensure that grantees conducted rigorous evaluations. Some grantees stated that technical assistance and guidance helped them understand and apply the evidence and evaluation requirements. One Teen Pregnancy Prevention Program grantee said that technical assistance and guidance was the key facilitating factor in the success with evidence gathering and evaluation. According to HHS officials, without technical assistance, at least some of the evaluations would not have met standards for rigor, yielding very little information about the impacts of the program model. Further, agency officials at our facilitated discussion said they have observed an increase in grantees’ capacity to apply for and implement tiered evidence grants since they were first used in 2010. A CNCS-commissioned independent evaluation of the Social Innovation Fund found evidence of improved organizational capacity among grantees as a result of their participation in the Social Innovation Fund. According to this evaluation, between 2009 and 2014, Social Innovation Fund grantees expanded organizational capacities and behaviors related to selection of grantees, support for grantees, evaluation, scaling up, and collaboration. Likewise, an HHS report on the Federal Home Visiting Program found that technical assistance enhanced grantee capacity in various areas, such as designing and modifying data systems and disseminating evaluation findings. Below are specific ways the four agencies provided technical assistance.

Agency and evaluation contractor guidance. The four agencies we reviewed developed a range of tools and guidance on evidence and evaluation to address challenges grantees faced implementing evidence and evaluation requirements and to build grantee capacity. For example, CNCS developed a number of resources to assist grantees as they moved through each stage of the evaluation process, from planning the evaluation to using evaluation results for action and improvement. The guidance consisted of written materials and seminars that the agency makes available on its website, and provides information on some of the more common challenges that grantees cited in planning and managing an evaluation. For example, one of the guidance documents addressed how to budget for an evaluation and identify approaches for creating an evaluation budget.

The four agencies also provided guidance on identifying and measuring the core components of specific service models to help grantees monitor implementation fidelity. For example, the evaluation technical assistance provider for Education’s Investing in Innovation Fund developed a fidelity tracking tool to help grantees identify their core components and the data needed to determine if they have been implemented. In another example, for DOL’s Workforce Innovation Fund, the program’s national evaluation coordinator developed a 74-page toolkit to help grantees plan for and manage an evaluation. The publication covers a broad range of topics, including identifying and hiring an independent evaluator and protecting the rights of the service model’s participants.

Technical assistance at each stage of the grant lifecycle. The four agencies we reviewed provided comprehensive technical assistance to grantees throughout the grant lifecycle. Technical assistance can help grantees deliver services to their intended populations more efficiently to improve outcomes. For example, the four agencies we reviewed provided assistance to grantees before awarding grants. This assistance included webinars and guidance available for organizations interested in applying for their tiered evidence grant programs. For example, before releasing the notice of funding announcement for its Teen Pregnancy Prevention Program, HHS provided webinars and guidance to help potential grantees understand the evidence and evaluation requirements to complete the application.

Some grantees said that their prior experience with federal grants and evaluations facilitated participating in tiered evidence grant programs, technical assistance was still beneficial in helping them understand the evidence and evaluation requirements specific to their programs. For example, a Workforce Innovation Fund grantee with prior randomized controlled trial experience said it still found DOL’s evaluation assistance helpful. The four agencies also provided assistance to grantees on disseminating evaluation results. For example, officials from HHS’s Federal Home Visiting Program stated that grantees lacked the capacity to disseminate their findings to a broader audience because historically they have been more focused on providing services. As a result, HHS developed a dissemination tool kit for grantees on communicating evaluation results.

Assistance to address unexpected external events. Officials from the four agencies and some grantees emphasized that tiered evidence grants can face challenges conducting rigorous evaluations based on unexpected external events. Agency officials at our facilitated discussion agreed it was important to provide comprehensive technical assistance for this type of challenge. Similarly, one Investing in Innovation Fund grantee reported that California did not give its standardized tests during one of the grant period years, which affected the outcome data for the evaluation. Education worked with the grantee to ensure that the evaluation’s methodology would not be compromised because of the change.

Networks for grantees. At our facilitated discussion, agency officials said that their agencies provided networks for grantees to communicate with each other and share lessons learned. For example, the four agencies organized in-person conferences to bring grantees together. Some of the grantees we interviewed said it was particularly helpful to meet at the conferences with grant recipients who had participated in prior award years. For example, a recipient of a Workforce Innovation Fund grant in 2015 stated that it was helpful to have a roundtable discussion with recipients from prior funding rounds because they shared lessons learned such as developing an evaluation plan.

Agencies Built Their Capacity to Monitor and Assist Grantees

We also identified factors that helped the four agencies we reviewed to address the challenges in administering the evidence and evaluation requirements in tiered evidence grants. Agency officials at our facilitated discussion said the following factors were essential to the implementation of tiered evidence grants.

Build capacity to assess the evidence and review applications. At our facilitated discussion, officials from the four agencies stated that they had to increase staff capacity before and during the grant application period and that the application period required an upsurge to review applications. For example, the program office for the Investing in Innovation Fund worked with the Education’s Institute of Education Sciences to review studies submitted in support of applications. Education officials said that it can be challenging to recruit an adequate number of external reviewers to review applications against the selection criteria. To address this challenge, Education’s office that administers the Investing in Innovation Fund coordinated with other Education offices to ensure that the application review occurred at a different time from those for other Education grant programs. The office also used academic journals and an electronic mailing list of education experts to recruit more qualified reviewers, according to Education officials.

Contracting with evaluation providers to address federal capacity issues. Evaluation organizations served as national technical assistance providers to work with both grantees and their independent evaluators on evaluation issues. Agency officials at our facilitated discussion stated that contracting with evaluation providers was essential because the agencies often lacked sufficient federal staffing to provide technical assistance. In addition to providing grantees and their independent evaluators assistance, as previously discussed, evaluation providers boosted the capacity of agencies, according to federal officials. For example, for the Workforce Innovation Fund, DOL officials also reported contracting with an evaluation provider to provide technical assistance to the individual third party evaluators and to review individual grantee evaluations to assess cross-innovation outcomes. The evaluation provider will review the individual grantees’ evaluation reports and synthesize the findings into lessons that substantially add to the body of knowledge about cost- effective practices for workforce development and the workforce system.

Program Management and Evaluation Officials Collaborated to Assist Grantees

Agency officials at our facilitated discussion said that tiered evidence grants have encouraged program management and evaluation offices to leverage resources to focus on the shared goal of providing evidence and evaluation assistance to grantees. Education officials stated that the Investing in Innovation Fund would not have been possible without the Institute of Education Sciences because Education program management officials do not have the expertise needed to review the evidence and evaluation requirements in grant applications. In addition, HHS officials said they added an evaluation specialist in the Office of Adolescent Health to manage the federal evaluation contracts, oversee evaluation technical assistance to grantees, and coordinate evaluation activities with the Teen Pregnancy Program.

Increased collaboration between program management and evaluation staff can represent a culture shift in agencies, because, according to agency officials at our facilitated discussion, program management and evaluation staff do not always collaborate when administering grants. The officials said it was important for program management officials to stay abreast of key developments related to particular service models. For example, an official said that engagement between the program and evaluation offices creates a richer understanding of the evidence base for officials administering the programs, such as being familiar with recent studies related to various service components or models.

Leadership Commitment to a Culture of Continuous Learning Facilitated Agencies’ use of Tiered Evidence Grants

At our facilitated discussion, agency officials said three aspects of high- level leadership commitment are essential to supporting tiered evidence grants.

Providing additional time and resources. Because tiered evidence grants require more hands-on assistance from agencies, leaders were willing to invest in the people and resources that are necessary to support both federal officials and grantees. An Education official said that, while it is easy to support the concept of evidence in policy making, there can be resistance once agencies realize the extent to which they must invest in the infrastructure needed to successfully administer such programs. At our facilitated discussion, agency officials said that policymakers may be unaware of the extent to which evidence and evaluation requirements add time and resources to the federal grant- making process. They added that agency leadership has a role to play in ensuring that policymakers are aware of this. It is important that policymakers recognize the extent of agency resources needed when establishing tiered evidence grant programs.

Understanding that grantees are taking a risk. Both agency officials at our facilitated discussion and grantees that we interviewed said that mixed or null findings could portray a service model or an entire organization negatively, and that this could affect their reputation in the community and with policymakers. A commitment to evidence-based practices helps to address this reputational risk for grantees because grantees feel comfortable testing approaches and adding to the evidence base regardless of whether they have positive, mixed, or null findings. Agency officials at our facilitated discussion said they understood upfront that some evaluation findings may not have positive results and may portray some service models or grantees negatively. They agreed that key agency leaders should be committed to standing by the results of evaluations. In cases where the results were negative or mixed, agency officials want to better understand the reasons why.

Convincing policymakers that evidence is a worthwhile investment.

Agency officials at our facilitated discussion agreed that tiered evidence grants present the opportunity to demonstrate to agency leaders and policymakers the benefits and potential of evidence in supporting the effective use of resources. Turnover makes it challenging for leadership to commit to evidence-based practices because new leadership may not be familiar with evidence or evaluation requirements, which can take time to embed into an agency’s culture. To address this challenge, Education officials said they are currently working to embed the importance of evidence, including tiered evidence grants, into briefing materials for the next presidential administration. Education has further shown leadership support for tiered evidence grants through one of their agency priory goals for fiscal year 2016-2017: to increase the use and generation of credible evidence on what works and what does not work in education.

Beyond agency leadership commitment, agency officials at our facilitated discussion agreed that they consider a program’s appropriation amount when deciding whether to create a tiered funding approach. For example, one agency official told us it may be impractical for agencies to create the evidence tiers, funding announcements, and oversight and technical assistance mechanisms for smaller grant programs.

Agencies’ Collaboration and Sharing of Lessons Learned on Tiered Evidence Grants Relied on Informal Networks

The four agencies in our review collaborated informally on tiered evidence grants and officials said that they found these informal networks useful. Individual agencies organized informal meetings and workshops on tiered evidence grants that occurred as needed. For example, CNCS convened multiple meetings with other agencies administering tiered evidence grants. CNCS officials told us that they saw a benefit to making those meetings more regular and structured. For example, CNCS officials said that collaboration helped address specific topic areas of interest related to tiered evidence grants, such as approaches to conducting evidence reviews, provision of evaluation capacity building services to grantees, and approaches in communicating evaluation results. The meetings can also be used to discuss issues of importance to the sector, such as the need for using consistent evidence definitions across agencies. Education also hosted workshops with speakers from various agencies who described how tiered evidence grants function and how agencies implement the evidence and evaluation requirements, according to Education officials.

In addition, OMB has taken steps to convene agencies on issues related to tiered evidence grants. For example, in November 2013, OMB hosted a workshop on innovative, evidence-focused grant programs that included tiered evidence grants. OMB staff said OMB continues to provide coordination and facilitate focused meetings on issues as they arise, and has made materials and communication mechanisms available to the agencies. For example, OMB created a web portal and discussion board for agency officials to share information on evidence and evaluation requirements, amongst other topics. However, some of the information on the portal is not current and its focus is on the broader use of evidence in policymaking. OMB staff said that OMB addresses issues on tiered evidence grants as they arise and under the broader umbrella of evidence-backed policy, and that no formal collaboration mechanism is needed. However, officials from the four agencies in our review said that they do not actively use the site for issues that arise specific to tiered evidence grants. An official from one agency stated that it is unclear whether the site is meant to be used to organize events, as a resource repository, or a means to organize a community of practice. Another official from a different agency said the site is designed is to provide guidance to agencies on developing their annual budgets. Officials from three programs in our review stated that they do not use the site for tiered evidence grants.

At our facilitated discussion, agency officials generally agreed that they would benefit from more formal collaboration on tiered evidence grants and lessons learned from implementing them. The officials stated that they have benefitted from collaborative efforts to date and are looking for additional ways to communicate about specific issues related to tiered evidence grants. For example, an Education official stated there is not an easy way to locate and contact officials from other agencies who are working on tiered evidence grant programs and that it would be helpful to have a mechanism to identify federal officials working on similar tiered evidence grant programs. According to the official, such a mechanism would provide agencies a forum to answer questions and share lessons learned.

By relying on ad-hoc collaboration, agencies using tiered evidence grants may miss opportunities to capture and share lessons learned that could strengthen and improve tiered evidence grant making government-wide. Further, federal officials who are new to evidence-based grant making may not be able to tap into the informal network. In our prior work on collaboration, we reported that collaborative mechanisms, such as interagency groups or collaboration technology, can be used to develop policies, implement programs, and share information. Specifically, more formal collaborative mechanisms can be effective to ensure collaboration continues to address staff turnover in leadership positions.

For example, in response to a recommendation in our prior work on Pay for Success programs, another relatively new tool for incorporating evidence into policymaking, OMB developed an Interagency Learning Network with representatives from 10 agencies to share lessons learned, hone policy, and strengthen implementation. Creating a similar mechanism for tiered evidence grants could help ensure current collaboration efforts are continued, and transcend staff turnover and that key lessons learned are captured. Further, formalized collaboration on tiered evidence grants could continue momentum on agency efforts to institutionalize evidence into policy-making.

Conclusions

Tiered evidence grants are an important component of congressional and federal agency efforts to increase the use of evidence in policymaking and distribute federal funds. Tiered evidence grants offer a means for agencies to evaluate the effectiveness of their programs and for Congress and agencies to fund programs based on proven and emerging service delivery models. The lessons agencies have learned in implementing tiered evidence grants could be applied to other federal grant programs that use evidence and evaluation requirements as a condition for receipt of federal grant funds. Agencies and grantees reported a number of challenges in administering tiered evidence grants and have also developed strategies for addressing these challenges.

Federal agencies that have administered tiered evidence grant programs found that informal inter-agency collaboration helped them address challenges and share lessons learned. These collaborative efforts to date have relied on informal networks that have emerged during implementation. A formal mechanism to facilitate communication and sharing of lessons learned across federal agencies could ensure that the beneficial informal collaborative efforts to date continue in a manner that transcends changes in staffing and individual grant programs. The absence of a formal mechanism to facilitate agency collaboration and to share and document lessons learned presents a missed opportunity to capitalize on what is known about early implementation of tiered evidence grant programs. A formal mechanism to facilitate tiered evidence grant collaboration and lessons learned could also help maintain momentum for continuous learning about broader government-wide efforts to institutionalize the use of evidence to consider the effectiveness of federal grant programs.

Recommendation for Executive Action

To facilitate collaboration and identify and broadly disseminate information on leading practices and lessons learned, the Director of OMB should establish a formal means for agencies to collaborate on tiered evidence grants. This could include creating a formal working group or providing collaboration technologies, such as an electronic mailing list or web portals.

Agency Comments and our Evaluation

We provided a draft of this product to the Office of Management and Budget; Departments of Education, Health and Human Services, and Labor; and the Corporation for National and Community Service for comment.

The Office of Management and Budget had no comments on the draft report. We received written comments on the draft report from the Department of Health and Human Services, which are reproduced in appendix III. The Departments of Education, Health and Human Services, Labor, and the Corporation for National and Community Service provided technical comments that were incorporated into the draft as appropriate.

We are sending copies of this report to the Director of the Office of Management and Budget; Secretaries of Education, Health and Human Services, and DOL; the Chief Executive Officer of the Corporation for National and Community Service; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

We conducted this work under the GPRA Modernization Act of 2010 (GPRAMA), which requires us to review the implementation of the Act at several critical junctures. This report is part of our response to that mandate. The objectives of this report were to describe 1) key features that tiered evidence grants add to federal grant processes, (2) benefits and challenges of using tiered evidence grants, (3) key factors to facilitate the use of tiered evidence grants, and (4) to assess the extent to which the Office of Management and Budget (OMB) and federal agencies are collaborating on evidence and evaluation requirements in tiered evidence grants.

To meet our objectives, we initially identified nine tiered evidence grant programs by conducting a literature review, reviewing annual budget documents, and validating the list with the OMB. We focused on programs that have operated long enough to show implementation results and lessons learned—specifically those established prior to 2013. We then narrowed the grant programs to the five that focus on domestic populations: Investing in Innovation Fund at the Department of Education (Education);

Teen Pregnancy Prevention Program at the Department of Health and Human Services (HHS);

Maternal Infant and Early Childhood Home Visiting (Federal Home Visiting) program at HHS;

Workforce Innovation Fund at the Department of Labor (DOL); and

Social Innovation Fund at the Corporation for National and Community Service (CNCS).

OMB verified that the five programs we identified represented the universe of tiered evidence grant programs that met our selection criteria.

We also interviewed officials from two other programs—Education’s Supporting Effective Educator Development (SEED) and DOL’s Trade Adjustment Assistance Community College and Career Training (TAACCCT). While these programs include evidence and evaluation features, they did not meet the full definition of tiered evidence grants for purposes of this report. SEED requires grantees to show moderate evidence on their service models but did not link the size of the grant award to the strength of evidence. TAACCCT included a “break the cap” award based on the level of evidence in its first funding round, but did not include evidence tiers in the subsequent funding rounds.

To represent the tiered evidence grantee perspective, we selected three programs for grantee interviews: the Investing in Innovation Fund, Teen Pregnancy Prevention Program, and Workforce Innovation Fund. We selected programs based on design features, including the size of the grant award, whether programs had a matching funding requirement, the timing of when the agency reviewed the evidence, and the number of evidence tiers. For the size of the grant award, we selected one large grant program, given that the majority of tiered evidence grant programs are relatively small compared to other grant programs. We defined large as greater than $1 billion in cumulative grant awards. We selected one grant program that has a matching requirement because most tiered evidence grants do not have such a requirement. Further, we decided to select one grant program that reviewed evidence components of grant applications at different times—both before grant solicitation and after grant solicitation. Moreover, we selected programs to include models with three evidence tiers and two evidence tiers.

We selected grantees in three metropolitan areas—the Washington, D.C./Baltimore area, New Orleans, and the San Francisco Bay Area— using non-generalizable sampling. First, we grouped cities with grantees into the same metropolitan area if they were within an approximately 1 hour driving distance. We focused on metropolitan areas that had grantees in the three programs we selected for site visits (the Investing in Innovation Fund, Teen Pregnancy Prevention Program, and Workforce Innovation Fund). We then focused on metropolitan areas that had at least 10 total grantees to ensure we would have a sufficient number of grantees to interview in each location. We also focused on metropolitan areas that had grantees in at least two evidence tiers. Finally, we selected grantees from three geographic regions (East Coast, West Coast, and South) to provide for geographic dispersion.

We interviewed two to three grant recipients for each evidence tier in the three programs across our three locations. To select individual grantees, we focused on grant recipients that have had sufficient time to implement programs and identify lessons learned. For the Investing in Innovation Fund and Teen Pregnancy Prevention Program, to the extent possible, we focused on grantees that received funds from the initial round of funding (2010 for both programs). If we could not further differentiate recipients beyond this criterion, we randomly selected grantees to interview. The Workforce Innovation Fund had only six recipients across our three locations. We selected all of them for interviews. We interviewed 18 grantees in total. When summarizing statements from grantees in our report, we defined “some grantees” as at least three grantees.

To identify the key features tiered evidence grants add to the federal grant making life cycle, we reviewed federal notice of funding announcements that described the evidence and evaluation requirements for tiered evidence grant requirements. We also interviewed officials from Education, HHS, DOL, and CNCS regarding how the tiered evidence grant-making life cycle differs from traditional grant programs.

To describe the benefits and challenges of using tiered evidence grants and factors for addressing the challenges, we reviewed agency documents, including reports on lessons learned, evaluation reports, and agency guidance. We interviewed officials from Education, HHS, DOL, and CNCS, as well as the grantees from the Investing in Innovation Fund, Teen Pregnancy Prevention Program, and Workforce Innovation Fund. We also interviewed national evaluation organizations that agencies contracted with to provide technical assistance to grantees and conduct evaluations across multiple grantees. We synthesized the list of challenges and factors for addressing the challenges. We then provided this list to key federal officials to provide the basis for a facilitated discussion to validate our list. For the facilitated discussion, we hosted an in-person meeting with officials from the five programs we selected for analysis.

To assess the extent to which OMB and federal agencies collaborated on evidence and evaluation requirements in tiered evidence grants, we interviewed officials from Education, HHS, DOL, CNCS, and OMB on current collaboration mechanisms and plans for collaboration moving forward. We also asked about interagency collaboration on tiered evidence grants during our facilitated discussion. For criteria, we used our key practices for enhancing and sustaining collaboration and implementing interagency collaborative mechanisms.

We conducted this performance audit from September 2015 to September 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Evidence Tiers at the Department of Education

The Department of Education’s (Education) Investing in Innovation Fund is one example of how a federal agency defines the tiers for its tiered evidence grant program (see table 1). Education included the definitions of its three evidence tiers in the notice of finding availability announcements for the grant program.

Appendix III: Comments from the Department of Health and Human Services

Appendix IV: GAO Contact and Staff Acknowledgement

GAO Contact

Michelle Sager, 202-512-6806 or sagerm@gao.gov.

Staff Acknowledgements

In addition to the contact named above, Jeff Arkin (Assistant Director), Barbara Lancaster (Analyst-in-Charge), Amy Bowser, Jeff DeMarco, Robert Gebhart, Steven Putansu, and Robert Robinson made key contributions to this work.