Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early Implementation and Inform Future Federal Efforts
Published: 2016-09-21T00:00:00

Fast Facts:
Are federal grants rewarding results?
A new policy tool called "tiered evidence grantmaking" allows federal agencies to award smaller amounts of grant funding to test promising ideas, and larger amounts to replicate practices with strong evidence of success. To help support innovation and reward results, agencies collect information on how health care, education, and social service projects are performing throughout the life of their grants.
We recommended that the Office of Management and Budget establish a way for agencies to work together and share leading practices as they attempt to tie resources to results.

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government spends more than $600 billion a year on grants to fund a wide range of programs and services, including those related to social services, education, and health care. To better integrate evidence and rigorous evaluation in federal grantmaking, the Office of Management and Budget (OMB) has encouraged federal agencies to use tiered evidence grant programs.
The GPRA Modernization Act of 2010 includes a provision for GAO to periodically review its implementation. The objectives of this report are to describe (1) key features of tiered evidence grants, (2) benefits and challenges of using tiered evidence grants, and (3) key factors to facilitate their use, and (4) to assess the extent to which federal agencies collaborate on tiered evidence grants.
To address these objectives GAO identified the five domestic-focused tiered evidence grant programs that were established prior to 2013. GAO reviewed key documents and interviewed officials from these programs. GAO also interviewed grant recipients from three of the grant programs. GAO selected these grant programs using various criteria, such as the number of evidence tiers and their total amount of funding.
section_title: What GAO Found
paragraphs: Tiered evidence grants are a new policy tool federal agencies are using to incorporate evidence of effectiveness into grantmaking. Under this approach, agencies establish tiers of grant funding based on the level of evidence grantees provide on their models for providing social, educational, health, or other services. Smaller awards are used to test new and innovative service models; larger awards are used to scale service models with strong evidence. To implement tiered evidence grants, agencies add evidence and evaluation requirements throughout the federal grant life cycle, including conducting independent evaluations of the grantees' service models and disseminating the evaluation results.
Note: Some programs have two tiers—preliminary evidence and strong evidence.
Agency officials identified several potential benefits of using tiered evidence grants, such as providing incentives for grantees to implement service models supported by evidence and conducting evaluations to build the evidence base. Officials from the agencies in GAO's review and grantees also identified various challenges with tiered evidence grants. In some cases the agencies identified factors to mitigate the challenges. For example, grantees told GAO that they encountered challenges drafting evaluation plans (which describe the methodology and are generally required for the grant applications). As an example of how agencies addressed this challenge, the Department of Labor contracted with a program evaluation firm to provide grantees with technical assistance and increased the time for grantees to draft evaluation plans that accurately reflected their service models.
GAO has previously reported on collaborative mechanisms, such as interagency groups, that can be used to implement programs and share lessons learned. However, currently there is no formal mechanism administered by OMB for agencies to collaborate on tiered evidence grants. By relying on ad hoc collaboration, agencies may miss opportunities to capture and share lessons learned that could strengthen tiered evidence grantmaking.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB establish a formal means for federal agencies to collaborate on tiered evidence grants. OMB had no comments on the recommendation.

section_title: Letter
paragraphs: Grants are an important tool the federal government uses to achieve  national objectives. The federal government spends more than $600  billion a year on federal grants to state and local governments, nonprofits,  and educational institutions to fund a wide range of programs and  services, including those related to social services, education, and health  care. To better integrate evidence and rigorous evaluation in budget,  management, operational, and policy decisions, the Office of  Management and Budget (OMB) has encouraged federal agencies to  expand or improve the use of grant program designs that focus federal  dollars on effective practices while encouraging innovation in service  delivery. As part of this effort, several federal agencies have  implemented tiered evidence grant programs. Under this approach,  agencies establish tiers of grant funding based on the level of evidence of  effectiveness provided for a grantee’s service model. Agencies award  smaller amounts to promising service models with a smaller evidence  base, while providing larger amounts to those with more supporting  evidence. Tiered evidence grantees are also generally required to  evaluate their service models as a condition for the receipt of grant funds.
Proponents of tiered evidence grants contend that they create incentives  for grantees to use approaches backed by strong evidence of  effectiveness, encourage learning and feedback loops to inform future  investment decisions, and provide some funding to test innovative  approaches. The evidence and evaluation requirements for tiered  evidence grants represent a deliberate approach to using evidence that  may require different policies, practices for outcome and performance  measurement, and capacities for agencies and grantee organizations.
We are required to review the implementation of the GPRA Modernization  Act of 2010 (GPRAMA) at several critical junctures. This report is part of  our response to that mandate. The objectives of this report are to  describe (1) key features that tiered evidence grants add to federal grant  processes, (2) benefits and challenges of using tiered evidence grants,  and (3) key factors to facilitate the use of tiered evidence grants, and      (4) to assess the extent to which OMB and federal agencies are  collaborating on evidence and evaluation requirements in tiered evidence  grants.
To meet our objectives, we identified the universe of domestic-focused  tiered evidence grant programs that were established prior to 2013: the  Department of Education’s (Education) Investing in Innovation Fund, the  Department of Health and Human Services (HHS) Teen Pregnancy  Prevention Program and Maternal Infant, and Early Childhood Home  Visiting (Federal Home Visiting) program, the Department of Labor’s  (DOL) Workforce Innovation Fund, and the Corporation for National and  Community Services (CNCS) Social Innovation Fund. We identified these  programs by conducting a literature review and by reviewing annual  agency budget documents. OMB verified that the five programs we  identified represented the universe of tiered evidence grant programs  meeting our selection criteria.
To address the first three objectives, we reviewed and analyzed relevant  documents from the five grant programs—such as notices of funding  availability, evaluation reports, and agency guidance—and interviewed  agency officials. To obtain insights on grantee perspectives, we selected  three of the five grant programs: Education’s Investing in Innovation  Fund, HHS’s Teen Pregnancy Prevention Program, and DOL’s Workforce  Innovation Fund. We interviewed two to three grantees from each funding  level for each program. We conducted these interviews in three  metropolitan areas: Washington, D.C./Baltimore, New Orleans, and the  San Francisco Bay Area.
To address the fourth objective, we interviewed officials from Education,  DOL, HHS, and CNCS, as well as OMB staff on current collaboration  mechanisms and any future plans for collaboration. For criteria, we used  our prior work on key practices for enhancing and sustaining collaboration  and implementing interagency collaborative mechanisms.
We also held a facilitated discussion with officials from the five grant  programs we reviewed. The discussion focused on the challenges in  using tiered evidence grants, the key factors for agencies to consider in  addressing these challenges, and interagency collaboration on tiered  evidence grants.
We conducted this performance audit from September 2015 to  September 2016 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: According to OMB’s 2013 memorandum on  the Next Steps in the Evidence and  Innovation Agenda   …evidence includes evaluation results,  performance measures, and other relevant  data analytics and research studies, with a  preference for high-quality experimental and  quasi-experimental studies.
Since 2010, OMB has used its annual budget guidance to encourage  federal agencies to use evidence to consider the effectiveness of their  programs and to institutionalize the use of evidence to foster innovation  rooted in research and rigorous evaluation. In a 2013 memorandum,  OMB identified areas in which agencies could improve their use of  evidence (see sidebar), including (1) strengthening evaluation capacity  and proposing new evaluations; (2) developing high-quality, low-cost  evaluations and rapid, iterative experimentation; (3) using innovative  outcome-focused grant design; and (4) increasing agency capacity to use  evidence. In 2015, OMB directed agencies to provide an overview of  evidence-building strategies and identify related priorities with their fiscal  year 2017 budget submissions.
To incorporate evidence into grantmaking, the White House and OMB  have encouraged agencies to use tiered evidence grants, which link  funds to an evidence framework. When applying for grants, grantees  demonstrate the strength of evidence on their service models and  agencies award funding based on the level of evidence (see figure 1).  Smaller awards are used to test new and innovative service models;  larger awards are used to scale service models with strong evidence.
Grantees generally evaluate their service model during the grant period.  As a result, tiered-evidence grant programs have a goal of identifying  evidence-based service models that can be replicated. According to an  OMB memorandum on evidence and innovation, the goal is that, over  time, service models move up tiers as evidence becomes stronger. OMB  staff said that other goals of tiered evidence grants include devoting more  resources to service models with stronger evidence, allowing for testing of  innovations, and encouraging rigorous evaluation.
From fiscal year 2010 to 2015, agencies have awarded approximately  $4.1 billion to the tiered evidence grants we reviewed. As shown in figure  2, each agency uses its own names and descriptions for its evidence  tiers. While Education, DOL, and CNCS use a three-tier model, HHS  uses a two-tiered model, where the agency makes a larger overall  investment in replicating evidence-based practices by awarding more  grants in its high tier. For example, HHS awards 75 percent of funds to  replicating service models with moderate or strong evidence for the Teen  Pregnancy Prevention Program, and awards 25 percent of funds to  rigorously evaluate promising approaches.
section_title: Tiered Evidence Grants Require Evidence and Evaluation Requirements throughout the Grant Life Cycle
paragraphs: While there is substantial variation among grant types, competitively  awarded federal grants generally follow a life cycle of pre-award  (announcement and application), award, implementation, and closeout.  To implement tiered evidence grants, the four agencies in our review add  evidence and evaluation requirements throughout the federal grant life  cycle. These added requirements include: (1) assessing the evidence  base and identifying evidence-based approaches, (2) implementing  evidence-based approaches with fidelity, (3) conducting independent  evaluations, and (4) disseminating evaluation results, as shown in figure  3.
The four federal agencies in our review had varied approaches to identify  evidence-based approaches for tiered evidence grants. Education, DOL,  and CNCS first developed definitions of each evidence tier that grantees  must meet to qualify for a grant award as well as the program priorities for  the grant competition. Evidence definitions included requirements, such  as the type and number of studies on the service models, as well as the  scale of service models studied. The rigor and scale of the evidence  requirements increased from tier to tier. For example,
Preliminary evidence tier. CNCS’s Social Innovation Fund’s preliminary  evidence tier was designed for service models that have preliminary  evidence based on a reasonable hypothesis supported by credible  research findings.
Moderate evidence tier. Education’s Investing in Innovation Fund’s  moderate evidence tier required at least one experimental study of  effectiveness of the grantees’ chosen service model, or one quasi- experimental study that was conducted on a large and multi-site  sample. Grantees could validate their service model by testing the  service model in a new location, testing it on a new or bigger population,  or tweaking an aspect of their service to see if it still works on their target  population.
Strong evidence tier. DOL’s Workforce Innovation Fund’s strong  evidence tier was designed to support significant expansion of service  models that had previously demonstrated strong evidence of positive  results and will be evaluated using a randomized controlled trial  evaluation.
For HHS, the evidence tiers for the Teen Pregnancy Prevention Program  are defined in statute. For example, in the appropriation providing funding  for the Teen Pregnancy Prevention Program for fiscal year 2016, 75  percent of amounts provided for the grant were to be used for a strong  evidence tier to replicate programs that have been proven effective  through rigorous evaluation to reduce teenage pregnancy, and behavioral  risk factors underlying teenage pregnancy. The other 25 percent were to  be used for a preliminary evidence tier to conduct research and  demonstration projects to develop and test additional models and  strategies.
After the four agencies in our review defined the evidence tiers, they used  the standards to assess the available evidence on the service models.  The agencies varied in terms of when they assessed the evidence:
Before announcement of grant opportunity. For the Teen Pregnancy  Prevention Program and the Federal Home Visiting Program, HHS  reviewed the available evidence on teen pregnancy prevention and home  visiting models before it announced the grant opportunities and identified  a list of approved evidence-based models. Tier one (strong evidence tier)  grantees that replicated evidence-based models selected an evidence- based model from the approved list that met the needs of their target  populations when applying for the grant. As of July 2016, the Teen  Pregnancy Prevention Program included 44 service models and the  Federal Home Visiting Program included 17 service models. HHS  periodically updates the approved models.
During review of grant applications. Grantees of DOL’s Workforce  Innovation Fund and Education’s Investing in Innovation Fund identified  in their grant applications the evidence-based service models they  planned to use. DOL and Education officials assessed the evidence that  supported the service models when they reviewed the grant applications  and make award decisions. For example, Education’s Institute of  Educational Sciences’ What Works Clearinghouse—the entity within  Education that is focused on research and evaluation—reviewed the  citations of the studies that supported Investing in Innovation Fund grant  applications to determine whether they met the evidence requirements.
After grant award is made. For the Social Innovation Fund, CNCS  made awards to intermediary organizations, which then made awards to  subgrantees to implement service models. CNCS provided the  intermediary organizations with guidance on the levels of evidence, which  they used to assess the evidence for their subawards. CNCS reviewed  and confirmed the intermediaries’ decisions before they made the final  awards.
Grantees were responsible for delivering their evidence-based service  models with fidelity—in a manner consistent with how the program was  designed. For example, in its guidance to Teen Pregnancy Prevention  Program grantees, HHS defined fidelity as the degree to which a program  is implemented with adherence to its core components – the key  ingredients related to achieving the outcomes associated with the  program model. The core components included a model’s content and  how it is delivered. Fidelity is important because delivering a service  model as it was designed increases the likelihood that the participants  served will experience similar positive outcomes to those found in the  original evaluation study.
The five grant programs we reviewed had evaluation requirements, but  they generally had two different evaluation approaches. For DOL,  Education, and CNCS tiered evidence grants, every grantee was required  to contract with a third party evaluator to conduct an independent  evaluation of its service model and submit it to the respective federal  grantmaking agency at the end of the grant period. DOL also  commissioned a crosscutting analysis of the findings from, and  experiences with, the evaluations of the projects under the Workforce  Innovation Fund.
For HHS, the Federal Home Visiting and Teen and Pregnancy Prevention  Programs used different evaluation approaches for each tier that included  both grantee and federal government-led evaluations. For the preliminary  evidence tier to test promising approaches of both grant programs,  grantees were required to conduct an independent evaluation. For the  strong evidence tier of the Federal Home Visiting Program, grantees  receiving competitive awards or tribal grants were also required to  conduct grantee-led evaluations.Also, state formula grantees are  encouraged to conduct grantee-led evaluations. For the strong evidence  tier to replicate evidence-based service models of the Teen Pregnancy  Prevention Program, all grantees receiving a grant award of greater than  $1 million were required to conduct an independent evaluation. HHS also  is conducting programwide evaluations of both grant programs (see  sidebar).
When conducting independent evaluations, grantees we reviewed hired  an independent evaluator to conduct evaluations. This work included  developing an evaluation plan, collecting and analyzing data, and  completing an evaluation report to communicate the service model  results. The four federal agencies we reviewed contracted with a separate  external national evaluation coordinator to provide technical assistance to  the independent evaluators and grantees on their evaluations. The  national evaluation coordinators developed guidance on evaluations,  reviewed evaluation plans, provided ongoing assistance on conducting  the evaluations, and provided feedback on the final reports. The four  agencies we reviewed encouraged grantees to conduct rigorous  evaluations, such as randomized controlled trials or studies using  quasi-experimental designs. We have previously reported that when  developing an evaluation design, an evaluator should select appropriate  measures and comparisons that will permit valid conclusions. The  evaluator should explore a variety of options available to collecting and  analyzing information, and choose alternatives that will best address the  evaluation objectives within available resources.
During and after the grant closeout stage, the four agencies we reviewed  disseminated the evaluation results to various audiences. For example,  Education and CNCS posted links to completed evaluations from the  Investing in Innovation Fund and the Social Innovation Fund, respectively,  online. Grantees in our review told us that they also took steps to  communicate their evaluation results by, for example, posting links to  them on their websites and presenting at conferences. Grantees can use  their evaluation results to support applications for future tiered evidence  grant awards. The Teen Pregnancy Prevention Program and the Federal  Home Visiting Program included grantee evaluations in their evidence  reviews.
section_title: Agencies Identified Potential Benefits of Tiered Evidence Grants and Described Implementation Challenges
subsections: section_title: Federal Agencies Identified Potential Benefits of Tiered Evidence Grants
paragraphs: Officials that we interviewed from the four agencies in our review  identified the following potential benefits of tiered evidence grants:
Provide incentives to implement service models that are supported  by evidence. According to OMB staff, tiered evidence grants provide  more flexibility than some traditional grants—such as those for which  program models are defined in statute or regulation. Tiered evidence  grants target funds to the most effective interventions based on the  evidence of the service model’s effectiveness without being limited to a  particular intervention or operating model. Agencies generally define  outcomes for the target population in the notice of funding availability.  Grantees use service models with the most robust evidence base to meet  a program’s desired outcomes. Further, tiered evidence grants create  incentives for grantees to use evidence-based service models because  they award larger funding amounts to approaches with more evidence  and grantees may be motivated to use evidence-based models because  of the evidence supporting their outcomes.
Build the evidence base. Tiered evidence grants generally require  grantees to conduct rigorous, independent evaluations, which will  contribute to the evidence base for addressing a given social challenge.  Tiered evidence grant awards play a role in expanding the research on  service models to include new geographic regions or variations in the  target population and in the services themselves. These evaluations will  be added to agencies’ databases of research studies (see text box). They  are available to the public, policymakers, and service organizations to  inform future efforts and contribute to the increased use of evidence- based models over time.
Build grantee capacity for evaluation. Experience with tiered evidence  grants and third-party evaluations can help grantees build capacity for  evaluation. According to a national assessment of the Social Innovation  Fund, there has been substantial improvement in grantees’ internal  evaluation capacity over time.
Apply evidence and evaluation requirements to other grant  programs. Three of the four the agencies in our review applied evidence  and evaluation requirements from the tiered evidence grant model to  other grant programs. For example, CNCS has incorporated evidence  requirements from the Social Innovation Fund into its existing  AmeriCorps State and National grant program. AmeriCorps increased  the total number of points assigned to evaluation and evidence  effectiveness in the review and scoring process for the service model  proposed by the grant applicants. Education amended its Education  Department General Administrative Regulations, which outline  requirements for its grant programs, to include definitions of four levels of  evidence, priorities, and selection criteria related to evidence and  evaluation. Education officials said that any program office can use the  definitions to add evidence or evaluation requirements to their  competitive grants in a manner consistent with the purpose and goals of  the program. Education has also developed a process to consider the  use of evidence and evaluation requirements in competitive grants when  developing its annual spending plan (see textbox).
Education’s Use of Evidence in Grant Competitions      According to Education officials, the agency is building on lessons from the Investing in  Innovation Fund and increasing its use of evidence and evaluation requirements in other  competitive grants, as appropriate.
Education has an Agency Priority Goal to increase the use and generation of credible  evidence on what works and what does not work in education. By September 30,  2017, Education aims for at least 20 percent of new competitive grant dollars to  support evidence-based strategies. Also, by September 30, 2017, Education will  increase by 20 the number of Education-funded project evaluations that provide  credible evidence about what works in education.  In fiscal year 2016, Education encouraged program offices in building their spending  plans to use evidence and evaluation definitions in agency regulations to the fullest  extent practicable. In spending plans, program offices should identify competitions  where it would be appropriate to ask applicants to (1) submit evidence of  effectiveness in support of their proposed projects, or (2) demonstrate a plan for  rigorously evaluating their proposed project once it is implemented.
section_title: Agencies and Grantees Faced Challenges in Implementing Evidence and Evaluation Requirements for Tiered Evidence Grants
paragraphs: Evidence and evaluation requirements add a layer of activities to tiered  evidence grants compared to some other federal grants. Officials from the  five federal grant programs and 18 grantee organizations in our review  reported that they faced challenges in implementing aspects of the  evidence and evaluation requirements. At our facilitated discussion on  tiered evidence grants, agency officials agreed that the primary  challenges are:  identifying evidence-based service models,  implementing them with fidelity,  communicating evaluation results.
As discussed below and in the following section, the four agencies we  reviewed took steps to address challenges in implementing evidence and  evaluation requirements.
subsections: section_title: Agency and Grantee Capacity to Identify Evidence-based Service Models
paragraphs: Agency officials agreed at our facilitated discussion that grantees faced  challenges in their capacity to understand the evidence base and  agencies faced challenges in their capacity to review grant applications.
Grantee capacity to understand the evidence base. At our facilitated  discussion, agency officials agreed that some grantees did not have the  technical skills and infrastructure to understand the evidence base and  select the evidence-based model that would best fit their target  populations. For example, an official at our facilitated discussion said  grantees faced challenges in selecting an evidence-based model that  was a good fit for their communities. When a model did not fit a  community, grantees had to make many adaptations to the evidence- based model. As a result, grantees were less likely to achieve the  greatest possible impact for the population because the service model  was not implemented as it was designed.
Agency capacity to review grant applications. The agencies that  managed the five grant programs in our review also faced challenges in  reviewing the evidence and evaluation portions of grant applications. The  agencies generally used panels of expert reviewers and program staff to  review grant applications. Reviewing evidence and evaluation requires  specific technical knowledge and skills, such as reviewers that have  experience with rigorous evaluation. Education officials told us that the  Investing in Innovation Fund has used reviewers with evaluation  expertise to review and score the evaluation criteria. They said that given  the success of using these reviewers, the Investing in Innovation Fund  officials work with other Education programs that have rigorous  evaluation requirements to help structure their peer reviews so they may  also use these evaluation experts. Also, three of the five grant programs  in our review ran multiple grant competitions simultaneously—one for  each tier—which required additional capacity from their program and  evaluation offices. Further, during our facilitated discussion, agency  officials said that it is challenging to provide technical assistance  sufficient to meet the varied needs of applicants during the application  phase. For example, the officials told us that technical assistance must  be broad so it is applicable to all organizations that are applying for  grants. They said that while one-on-one technical assistance would  benefit applicants, their agencies lack the capacity to help every single  applicant, and it is challenging for those agencies to ensure that similar  technical assistance is available to all applications that need it. In many  cases, intensive pre-application technical assistance is not permissible  under grants policy, as it would be impossible to ensure that no one  applicant has a competitive advantage.
section_title: Grantee Capacity to Implement Evidence-based Models with Fidelity
paragraphs: Agency officials at our facilitated discussion agreed there are two  potential challenges with maintaining fidelity for evidence-based service  models:
Grantees delivering service models as intended. At our facilitated  discussion, agency officials said that grantees faced challenges in  collecting quality data and using the data to monitor whether they are  implementing their service model with fidelity. Some of the grantees we  interviewed said that to monitor fidelity, they had to build new systems to  collect data—such as attendance data—on how their service models  were implemented. In addition to using data to monitor fidelity, agency  officials that participated in our facilitated discussion said that grantees  faced challenges in using data during the grant period to improve the  implementation of evidence-based service models. For example, officials  said some grantees still see performance data simply as a reporting  requirement, and do not use the data to make decisions about their  programs.
Managing adaptations to service models. Within tiered evidence  grants, grantees generally deliver existing service models to new groups  of individuals, at times in a new location with varying degrees of similarity  to the population studied in the original evaluation. Therefore, grantees  may need to make adaptations to the evidence-based model to best  serve their population. For example, a Teen Pregnancy Prevention  Program grantee we interviewed said that its organization made four  adaptations to the evidence-based model selected because the model  was studied in the 1990s and the curriculum was out of date. The  adaptations included updating information on HIV/AIDS for medical  accuracy, modifying curriculum scenarios to be more inclusive and  culturally relevant, adding basic sexual health information to reach a  younger population, and changing when the program would be delivered  from a weekend program to during the school day.
At our facilitated discussion, agency officials said there was a tradeoff  between implementing the evidence-based model with fidelity and  adjusting the model to respond to new information or specific  circumstances. If adaptations compromise the core components of a  model, the grantees may be unable to attribute their results to the service  model and may not have comparable outcomes. At our facilitated  discussion, one agency official explained that this is challenging because  grantees are accustomed to serving people as their highest priority, and  their instincts are to make changes to meet the needs of their populations  without considering how the changes would affect implementation fidelity  or the evaluation of their service models. Model owners can also play a  role in managing fidelity of a service model (see text box).
Service Model Owners Add a Layer of Complexity to Tiered Evidence Grants Department of Health and Human Services (HHS) officials said the implementation and  scale-up of evidence-based models is complex because it involves model developers  that own the evidence-based models.  Grantees pay a fee to implement the model. The  model developers generally provide grantees with the content of the service model,  training, and technical assistance.
HHS officials said that working with model developers has created some challenges for  tiered evidence grants. Because they have a role in approving adaptations and have an  incentive to approve adaptations, grantees would be more likely to use their models  again in the future. There is a need for the federal agencies to have a role in approving  adaptations and enhancements to provide another level of review and oversight.  For  example, the Federal Home Visiting program changed its oversight process so HHS  approves any adaptations in addition to the model developers’ approval.
Teen Pregnancy Prevention Program grantees we interviewed also described  challenges working with model developers including additional reporting requirements.  One grantee also said that it would be helpful for HHS to negotiate fees in advance of  the grant announcement so each grantee does not need to individually negotiate fees  with the model developer.
section_title: Grantee Challenges in Conducting Evaluations
paragraphs: All of the grantees included in our review reported that they faced  challenges in fulfilling requirements for rigorous evaluation in tiered  evidence grants, for example, when planning for their independent  evaluations.
Identifying and hiring an independent evaluator. Some grantees had  not previously worked with an independent evaluator and were not  familiar with the qualifications they should look for in an evaluation. For  example, they faced challenges in developing a description of the  requirements and hiring an evaluator with the appropriate experiences  and skill set. State and local governments also found it difficult to procure  an evaluator within the grant’s timeframe.
Developing an evaluation plan, including choosing the appropriate  evaluation type. In some cases, grantees are required to submit an  evaluation plan as part of their grant application. At our facilitated  discussion, agency officials said grantee capacity to develop an  evaluation plan varies widely and some grantees lack understanding of  what evaluations are and the related planning involved. Grantees may  need to work with independent evaluators on their evaluation plans in  advance of submitting their grant applications. However, grantees may  not have the financial resources to obtain evaluator assistance with grant  applications. Grantees we interviewed that had an existing relationship  with an evaluator said that evaluators’ assistance in developing their  grant applications facilitated their ability to apply for a tiered evidence  grant award. Both grantees and evaluators faced challenges in choosing  the evaluation type best suited to the service model. At our facilitated  discussion, agency officials said that some evaluators used the  evaluation type with which they had the most experience instead of the  type that would be most appropriate.
Budgeting for evaluations. At our facilitated discussion, agency officials  agreed that it can be challenging for some grantees to budget for  evaluations because they want to dedicate as much funding as possible  to serving their target populations. One agency official also noted that  grantees may be unfamiliar with the resources needed to conduct  rigorous evaluations, such as randomized controlled trials and quasi- experimental designs.
We have previously reported that randomized controlled trial evaluations  are often difficult to carry out because evaluators must be able to  maintain a treatment and control group. Similarly, officials from agencies  we reviewed, and some grantees, also described challenges that were  specific to conducting a randomized controlled trial evaluation. For  example, some grantees from the programs in our review face the  following challenges:
Recruiting sufficient numbers of participants for a treatment and  control group. Some grantees faced recruitment challenges because  they needed to recruit twice the number of people they planned to serve  to maintain a control group for a randomized controlled trial. One grantee  said it is particularly challenging to recruit participants for a randomized  controlled trial because participants must agree to participate in the  programing knowing there is a chance they will not receive the new  service and instead receive the traditional services offered.
Retaining participants for data collection. Some grantees experienced  challenges in retaining participants to collect data for the evaluation after  the service delivery ended, particularly for the control group. For  example, for a Teen Pregnancy Prevention Program grantee, many of  the participants graduated from high school and moved away during the  evaluation period. This created challenges for the grantee to locate  participants and administer follow-up surveys.
Navigating potential ethical issues with using a control group. Some  grantees faced challenges in managing potential ethical issues in using a  control group. We have previously reported that in randomized controlled  trials there can be concerns that the control group should not be harmed  by withholding needed services. A Workforce Innovation Fund grantee  expressed concerns about using a control group and possible backlash  from the community for withholding the service from individuals in the  control group who need the services. The grantee said it planned to  advertise the program as a lottery to help address this challenge.
Grantees also faced systemic challenges to conducting evaluations:
Accessing administrative data. Some grantees that we interviewed  reported encountering difficulties obtaining administrative data collected  by federal, state, or local agencies to measure outcomes for the  evaluation because of legal or privacy reasons. For example, a  Workforce Innovation Fund grantee said that it was challenging or not  possible to get access to administrative data, including unemployment  data and wage data, from the states where it was implementing its  service model. Challenges in accessing administrative data have led to  grantees obtaining information for some states and not others, leading to  uneven reporting. We have previously reported on challenges agencies  and stakeholders face in using administrative data to conduct evaluations  of programs.
External policy changes. Some grantees faced challenges resulting  from external policy changes because they potentially prevented a tiered  evidence grant award from operating as initially intended. For example,  Education officials and Investing in Innovation Fund grantees said that  changes in state-level student assessment policies created challenges to  conducting rigorous evaluations. An Investing in Innovation Fund grantee  told us that its analyses for its evaluation were complicated by changes in  the state testing regime in the state of California.
Conducting evaluations within a 3- to 5-year grant period. Tiered  evidence grants generally have an initial planning year, up to 3 years to  implement the service model, and an additional year for data analysis  and reporting. Agency officials agreed during our facilitated discussion  that it can be challenging to conduct an evaluation within this 3-5 year  grant period because some data may be unavailable or not measured  until some timeframe after the service model is delivered. For example,  one DOL Workforce Innovation Fund grantee used unemployment  insurance wage record data to measure the employment and earnings  outcomes of participants. According to DOL officials, unemployment  wage record data typically are unavailable for two to three quarters after  they are reported. So, for later cohorts of participants, there was not  enough time after program completion to obtain available data before the  end of the 5 year grant period. As a result, the grantee’s evaluation will  not include unemployment data for the participants in the later cohorts.  Education officials also described challenges in conducting evaluations  during the grant period for the Investing in Innovation Fund and said the  department has taken steps to address this challenge. In 2013,  Education updated its Education Department General Administrative  Regulations to allow grant periods for competitive grants to be extended  beyond 5 years with funding for data collection and analysis. Education  approves extensions on a case-by-case basis when extending the project  will result in better data and higher quality evaluation. For example,  according to Education officials, a 2010 Investing in Innovation Fund  grantee is implementing a service model in middle and high schools to  increase high school graduation rates. Education awarded an evaluation  grant award to continue data collection and analysis for an additional four  years beyond the original grant period to track students’ high school  completion and graduation rates.
section_title: Agency Challenges in Providing Oversight and Technical Assistance
paragraphs: At our facilitated discussion, agency officials generally agreed that their  agencies also faced challenges in overseeing and providing technical  assistance to tiered evidence grant programs. The agencies provided  extensive coaching and technical assistance to walk grantees through the  evidence and evaluation requirements in tiered evidence grants, which  requires additional capacity to implement evaluation requirements  compared to other federal grant programs. At the facilitated discussion,  agency officials emphasized that agency grant officers for tiered evidence  grants need both knowledge of evidence and evaluation as well as the  ability to provide ongoing coaching to grantees to help them meet the  evidence and evaluation requirements. The officials said that in some  cases agencies faced challenges in shifting the culture among their grant  officers from a compliance orientation to the addition of a coaching model  of oversight to help build grantees’ capacity.
section_title: Grantee and Agency Challenges in Communicating Evaluation Results
paragraphs: Officials from the four agencies in our review expressed concern about  how to accurately summarize and communicate complex and nuanced  evaluation findings to different audiences.
Confusion around evaluation results. At our facilitated discussion,  officials agreed that there is confusion among policymakers around  evaluation results of randomized controlled trials or quasi-experimental  studies. These evaluation types generally measure several specific  outcomes and can show a positive, negative, or null finding. A positive  finding occurs when being in the treatment group is associated with a  positive outcome, relative to control group. A negative finding occurs  when being in the treatment group is associated with a negative  outcome, relative to control group. A null finding occurs when there is no  statistical difference in outcome between treatment and control groups,  which may occur when a sample size is too small to be statistically  significant. Mixed results, where some findings are positive, some are  negative and some are null, are also common. At our facilitated  discussion, agency officials expressed concerns that policymakers may  use negative, null, or mixed evaluation results from one grant award as a  justification for discontinuing an entire tiered evidence grant program.  They told us that there is a risk that policymakers will interpret evaluation  results based on one outcome to mean that a service model is or is not  working. However, the officials said that evaluation results can be  nuanced, and negative, null, and mixed findings can be used to consider  what led to the findings and to adjust the service model to improve future  results. Similarly, at our facilitated discussion, agency officials expressed  concerns about the risk of a single study being interpreted to represent  outcomes of an entire tiered evidence grant program. Further, the  officials said it is a challenge to communicate that null or negative  findings have a valuable role in building the evidence base because they  can identify areas where a model can be altered to improve outcomes for  its target population. One official told us that before they did not know  whether the service models were working and at least now they will know  what does not work.
Expectations about grantees moving up evaluation tiers. Another  challenge is the expectation from policymakers that grantees will move  up from one evidence tier to the next evidence tier. For example,  according to agency officials that attended our facilitated discussion,  there is an expectation that a grantee that receives a preliminary  evidence tier award will qualify for a moderate evidence award at the end  of the grant period. In practice, evaluation results have led to few of the  hundreds of tiered evidence grantees moving up tiers, in part, because a  single evaluation may not be enough to qualify for highest level of  evidence tiers. Moreover, in some cases, grant recipients moved down  tiers. Such was the case with Education’s Investing in Innovation Fund,  where some grantees applied for a lower tier in subsequent rounds to  test a variation or adaptation of their existing service model. For example,  a teacher professional development service model focused content on  reading moved from the validation (moderate evidence) tier to the  development (preliminary evidence) tier to test an online version of the  professional development model. OMB staff said that moving up tiers to  scale an evidence-based service model and moving down tiers to test an  adaptation of a service model both highlight successful outcomes of the  tiered evidence grant model.
section_title: Agency Officials and Grantees Identified Key Factors That Facilitate the Use of Tiered Evidence Grants
paragraphs: Officials from the four agencies in our review and grantees identified five  key factors that facilitate the use of tiered evidence grants by addressing  challenges in administering the evidence and evaluation requirements in  tiered evidence grants (see text box). We have also previously reported  on strategies agencies can use to facilitate the use of evaluation in  program management and policy making, including demonstrated  leadership support for evaluation and engaging stakeholders throughout  the evaluation process.
subsections: section_title: Comprehensive Agency Oversight Helped Grantees Fulfill Evidence and Evaluation Requirements
paragraphs: Officials from the five grant programs in our review said that  comprehensive oversight throughout the grant life cycle is essential to  address grantee capacity challenges and facilitate the successful use of  tiered evidence grants. Oversight of grantees is also important to  reasonably assure that grants are used for their intended purposes and  that risks of fraud, waste, and abuse are minimized. Although federal  oversight of grantees is not unique to tiered evidence grants, at our  facilitated discussion, agency officials said that their agencies provided  closer oversight on the evidence and evaluation requirements. For  example, they approved evaluation plans and adaptations to service  models. Agencies took several steps to create oversight mechanisms to  address challenges grantees face with evidence and evaluation  requirements:
Cooperative agreements provided for closer oversight. Both  cooperative agreements and grants provide funding to carry out  approved federal activities, but a cooperative agreement allows for  substantial programmatic involvement from the federal agency. Officials  from three of the four agencies in our review—Education, HHS, and  CNCS—said they structured their programs as cooperative agreements  because they allowed the agencies to more closely oversee grantees  compared to other grant programs, especially in how grantees adhered to  the fidelity of service models. Cooperative agreements are used instead  of grants when substantial involvement is expected between the agency  and the recipient.
Clear and upfront requirements and expectations for evaluations.
The four agencies in our review included specific requirements for  independent evaluations in the notice of funding opportunities for tiered  evidence grants to help ensure that grantees would be prepared to meet  the evidence standards for their evaluations. For example, Education  required grantees in the Investing in Innovation Fund’s Scale-up (strong  evidence) and Validation (moderate evidence) tiers to identify the  program’s elements that can be replicated by other entities, and in a  variety of contexts for a variety of students.
Agency approval of evaluation plans. An evaluation plan spells out the  research questions and methodology of an evaluation. Officials from the  four agencies we reviewed told us that completing an evaluation plan is  essential because it provides the groundwork for a robust evaluation,  including quality data collection, analysis, and reporting. For example, for  the Workforce Innovation Fund, DOL required grantees to submit an  initial evaluation design report prepared by an independent evaluator  within 9 months after the grant award. DOL and the program’s national  evaluation coordinator assessed the content of initial evaluation reports  and provided comments. Grantees then worked with DOL and the  national evaluation coordinator to have their final evaluation design  reports approved within 11 months of the grant award.
Using a pilot period. Similarly, at our facilitated discussion, agency  officials said that a planning and piloting year at the beginning of the  grant period facilitates grantee implementation of tiered evidence grants  because it allows time for grantees to plan and set up the infrastructure  for their service models and evaluations. Some grantees said that it  was easier for grantees in moderate or strong evidence tiers to develop  an evaluation plan because their service models were more fully defined  from the beginning of the grant’s implementation. Some grantees said  they found it particularly helpful to use part of the grant period as a pilot  program. For example, a Teen Pregnancy Prevention grantee said they  used the first year of the grant to test its service model in three different  environments.
Reporting requirements to ensure fidelity to the service delivery  model. The four agencies we reviewed used mechanisms to address  challenges with implementing service models with fidelity, including  requiring grantees to report specific data elements and conducting in- person observations. For example, to help ensure grantees implemented  their service models with fidelity, HHS’s Teen Pregnancy Prevention  Program required grantees to collect data on fidelity measures reflecting  the extent to which community-based organizations, schools, and other  organizations adhered to the service models’ core elements when  administering the models in classrooms and other settings (see text box).  Agency officials at our facilitated discussion said these measures are  very important to ensure quality implementation. Some grantees also  said they found the measures to be useful. For example, a Teen  Pregnancy Prevention Program grantee said that the independent  observations were particularly helpful during the pilot year of program  implementation because they helped to identify inconsistencies in how  the service providers delivered the messages in the curriculum at  different locations. The grantee organization addressed the  inconsistencies across different locations by having service providers  from different organizations teach together for the later years of the  project to ensure lessons were taught in a consistent manner.
Fidelity Monitoring for the HHS Teen Pregnancy Prevention Program  Teen Pregnancy Prevention Program grantee data on outcomes to monitor whether  they implemented their evidence-based service model with fidelity, include:  attendance data from participants for all sessions completed;  a facilitator self-assessment or fidelity monitoring log for each session  implemented;  information on planned and unplanned adaptations; and  observation data on service model administrators from independent observers for 5  to 10 percent of all sessions implemented.
HHS reported that for the fiscal year 2010-2014 cohort 95 percent of all sessions were  implemented with high fidelity and 92 percent of all sessions that were independently  observed were rated as either very high or high quality.
Agency approval and dissemination of adaptations to service  models. To address the challenge of grantees making adaptations to  approved service models, the four agencies we reviewed required  grantees to obtain approval for making adaptations. For example, HHS  assigned some project officers for the Teen Pregnancy Prevention  Program to serve as model leads for individual evidence-based  programs. This process allowed the project officers to be aware of all  approved adaptations that had been made to the model, and to share this  information with all other grantees that are implementing that model. In  addition, HHS compiled and analyzed all approved adaptations from the  program’s first round of grantees from 2010 to 2015. It also published a  summary of adaptations that had been made by the grantees for each  model so future grantees and others could learn from former grantees.  However, some grantees said that it was time consuming to obtain  approval for adaptations. For example, a Workforce Innovation Fund  grantee stated that it felt bound to achieve the outcome goals in its  evaluation plan and that it was difficult and time consuming to obtain  modifications that deviated from the grant’s application.
Providing funding for evaluations. All five tiered evidence grants we  reviewed allowed for grantees to spend grant funds on evaluations;  however, agencies structured the funding differently. Officials from the  four agencies in our review said the very fact their authorizing statutes  allowed for federal funds to be used for various evaluation activities was  a facilitating factor. Further, the officials stated that some programs are  not a good fit for tiered evidence grants because they are prohibited from  using federal funds for evaluation purposes and require all funds to  provide services. Since independent evaluations were new to some  grantees, agency officials at our facilitated discussion said they had to  work closely with grantees to ensure they provided sufficient funding for  evaluations. The officials said that grantees are accustomed to quickly  providing services after receiving grant funding, and that grantees faced  challenges in budgeting for independent evaluations.
section_title: Agency Technical Assistance Throughout the Grant Lifecycle Helped Build Grantee Capacity
paragraphs: The four agencies in our review provided evaluation technical assistance  from the design stage to the analysis and reporting stages of evaluations  to help ensure that grantees conducted rigorous evaluations. Some  grantees stated that technical assistance and guidance helped them  understand and apply the evidence and evaluation requirements. One  Teen Pregnancy Prevention Program grantee said that technical  assistance and guidance was the key facilitating factor in the success  with evidence gathering and evaluation. According to HHS officials,  without technical assistance, at least some of the evaluations would not  have met standards for rigor, yielding very little information about the  impacts of the program model. Further, agency officials at our facilitated  discussion said they have observed an increase in grantees’ capacity to  apply for and implement tiered evidence grants since they were first used  in 2010. A CNCS-commissioned independent evaluation of the Social  Innovation Fund found evidence of improved organizational capacity  among grantees as a result of their participation in the Social Innovation  Fund. According to this evaluation, between 2009 and 2014, Social  Innovation Fund grantees expanded organizational capacities and  behaviors related to selection of grantees, support for grantees,  evaluation, scaling up, and collaboration. Likewise, an HHS report on  the Federal Home Visiting Program found that technical assistance  enhanced grantee capacity in various areas, such as designing and  modifying data systems and disseminating evaluation findings. Below are  specific ways the four agencies provided technical assistance.
Agency and evaluation contractor guidance. The four agencies we  reviewed developed a range of tools and guidance on evidence and  evaluation to address challenges grantees faced implementing evidence  and evaluation requirements and to build grantee capacity. For example,  CNCS developed a number of resources to assist grantees as they  moved through each stage of the evaluation process, from planning the  evaluation to using evaluation results for action and improvement. The  guidance consisted of written materials and seminars that the agency  makes available on its website, and provides information on some of the  more common challenges that grantees cited in planning and managing  an evaluation. For example, one of the guidance documents addressed  how to budget for an evaluation and identify approaches for creating an  evaluation budget.
The four agencies also provided guidance on identifying and measuring  the core components of specific service models to help grantees monitor  implementation fidelity. For example, the evaluation technical assistance  provider for Education’s Investing in Innovation Fund developed a fidelity  tracking tool to help grantees identify their core components and the data  needed to determine if they have been implemented. In another example,  for DOL’s Workforce Innovation Fund, the program’s national evaluation  coordinator developed a 74-page toolkit to help grantees plan for and  manage an evaluation. The publication covers a broad range of topics,  including identifying and hiring an independent evaluator and protecting  the rights of the service model’s participants.
Technical assistance at each stage of the grant lifecycle. The four  agencies we reviewed provided comprehensive technical assistance to  grantees throughout the grant lifecycle. Technical assistance can help  grantees deliver services to their intended populations more efficiently to  improve outcomes. For example, the four agencies we reviewed provided  assistance to grantees before awarding grants. This assistance included  webinars and guidance available for organizations interested in applying  for their tiered evidence grant programs. For example, before releasing  the notice of funding announcement for its Teen Pregnancy Prevention  Program, HHS provided webinars and guidance to help potential  grantees understand the evidence and evaluation requirements to  complete the application.
Some grantees said that their prior experience with federal grants and  evaluations facilitated participating in tiered evidence grant programs,  technical assistance was still beneficial in helping them understand the  evidence and evaluation requirements specific to their programs. For  example, a Workforce Innovation Fund grantee with prior randomized  controlled trial experience said it still found DOL’s evaluation assistance  helpful. The four agencies also provided assistance to grantees on  disseminating evaluation results. For example, officials from HHS’s  Federal Home Visiting Program stated that grantees lacked the capacity  to disseminate their findings to a broader audience because historically  they have been more focused on providing services. As a result, HHS  developed a dissemination tool kit for grantees on communicating  evaluation results.
Assistance to address unexpected external events. Officials from the  four agencies and some grantees emphasized that tiered evidence  grants can face challenges conducting rigorous evaluations based on  unexpected external events. Agency officials at our facilitated discussion  agreed it was important to provide comprehensive technical assistance  for this type of challenge. Similarly, one Investing in Innovation Fund  grantee reported that California did not give its standardized tests during  one of the grant period years, which affected the outcome data for the  evaluation. Education worked with the grantee to ensure that the  evaluation’s methodology would not be compromised because of the  change.
Networks for grantees. At our facilitated discussion, agency officials  said that their agencies provided networks for grantees to communicate  with each other and share lessons learned. For example, the four  agencies organized in-person conferences to bring grantees together.  Some of the grantees we interviewed said it was particularly helpful to  meet at the conferences with grant recipients who had participated in  prior award years. For example, a recipient of a Workforce Innovation  Fund grant in 2015 stated that it was helpful to have a roundtable  discussion with recipients from prior funding rounds because they shared  lessons learned such as developing an evaluation plan.
section_title: Agencies Built Their Capacity to Monitor and Assist Grantees
paragraphs: We also identified factors that helped the four agencies we reviewed to  address the challenges in administering the evidence and evaluation  requirements in tiered evidence grants. Agency officials at our facilitated  discussion said the following factors were essential to the implementation  of tiered evidence grants.
Build capacity to assess the evidence and review applications. At  our facilitated discussion, officials from the four agencies stated that they  had to increase staff capacity before and during the grant application  period and that the application period required an upsurge to review  applications. For example, the program office for the Investing in  Innovation Fund worked with the Education’s Institute of Education  Sciences to review studies submitted in support of applications.  Education officials said that it can be challenging to recruit an adequate  number of external reviewers to review applications against the selection  criteria. To address this challenge, Education’s office that administers the  Investing in Innovation Fund coordinated with other Education offices to  ensure that the application review occurred at a different time from those  for other Education grant programs. The office also used academic  journals and an electronic mailing list of education experts to recruit more  qualified reviewers, according to Education officials.
Contracting with evaluation providers to address federal capacity  issues. Evaluation organizations served as national technical assistance  providers to work with both grantees and their independent evaluators on  evaluation issues. Agency officials at our facilitated discussion stated that  contracting with evaluation providers was essential because the agencies  often lacked sufficient federal staffing to provide technical assistance. In  addition to providing grantees and their independent evaluators  assistance, as previously discussed, evaluation providers boosted the  capacity of agencies, according to federal officials. For example, for the  Workforce Innovation Fund, DOL officials also reported contracting with  an evaluation provider to provide technical assistance to the individual  third party evaluators and to review individual grantee evaluations to  assess cross-innovation outcomes. The evaluation provider will review  the individual grantees’ evaluation reports and synthesize the findings  into lessons that substantially add to the body of knowledge about cost- effective practices for workforce development and the workforce system.
section_title: Program Management and Evaluation Officials Collaborated to Assist Grantees
paragraphs: Agency officials at our facilitated discussion said that tiered evidence  grants have encouraged program management and evaluation offices to  leverage resources to focus on the shared goal of providing evidence and  evaluation assistance to grantees. Education officials stated that the  Investing in Innovation Fund would not have been possible without the  Institute of Education Sciences because Education program management  officials do not have the expertise needed to review the evidence and  evaluation requirements in grant applications. In addition, HHS officials  said they added an evaluation specialist in the Office of Adolescent  Health to manage the federal evaluation contracts, oversee evaluation  technical assistance to grantees, and coordinate evaluation activities with  the Teen Pregnancy Program.
Increased collaboration between program management and evaluation  staff can represent a culture shift in agencies, because, according to  agency officials at our facilitated discussion, program management and  evaluation staff do not always collaborate when administering grants. The  officials said it was important for program management officials to stay  abreast of key developments related to particular service models. For  example, an official said that engagement between the program and  evaluation offices creates a richer understanding of the evidence base for  officials administering the programs, such as being familiar with recent  studies related to various service components or models.
section_title: Leadership Commitment to a Culture of Continuous Learning Facilitated Agencies’ use of Tiered Evidence Grants
paragraphs: At our facilitated discussion, agency officials said three aspects of high- level leadership commitment are essential to supporting tiered evidence  grants.
Providing additional time and resources. Because tiered evidence  grants require more hands-on assistance from agencies, leaders were  willing to invest in the people and resources that are necessary to  support both federal officials and grantees. An Education official said  that, while it is easy to support the concept of evidence in policy making,  there can be resistance once agencies realize the extent to which they  must invest in the infrastructure needed to successfully administer such  programs. At our facilitated discussion, agency officials said that  policymakers may be unaware of the extent to which evidence and  evaluation requirements add time and resources to the federal grant- making process. They added that agency leadership has a role to play in  ensuring that policymakers are aware of this. It is important that  policymakers recognize the extent of agency resources needed when  establishing tiered evidence grant programs.
Understanding that grantees are taking a risk. Both agency officials at  our facilitated discussion and grantees that we interviewed said that  mixed or null findings could portray a service model or an entire  organization negatively, and that this could affect their reputation in the  community and with policymakers. A commitment to evidence-based  practices helps to address this reputational risk for grantees because  grantees feel comfortable testing approaches and adding to the evidence  base regardless of whether they have positive, mixed, or null findings.  Agency officials at our facilitated discussion said they understood upfront  that some evaluation findings may not have positive results and may  portray some service models or grantees negatively. They agreed that  key agency leaders should be committed to standing by the results of  evaluations. In cases where the results were negative or mixed, agency  officials want to better understand the reasons why.
Convincing policymakers that evidence is a worthwhile investment.
Agency officials at our facilitated discussion agreed that tiered evidence  grants present the opportunity to demonstrate to agency leaders and  policymakers the benefits and potential of evidence in supporting the  effective use of resources. Turnover makes it challenging for leadership  to commit to evidence-based practices because new leadership may not  be familiar with evidence or evaluation requirements, which can take time  to embed into an agency’s culture. To address this challenge, Education  officials said they are currently working to embed the importance of  evidence, including tiered evidence grants, into briefing materials for the  next presidential administration. Education has further shown leadership  support for tiered evidence grants through one of their agency priory  goals for fiscal year 2016-2017: to increase the use and generation of  credible evidence on what works and what does not work in education.
Beyond agency leadership commitment, agency officials at our facilitated  discussion agreed that they consider a program’s appropriation amount  when deciding whether to create a tiered funding approach. For example,  one agency official told us it may be impractical for agencies to create the  evidence tiers, funding announcements, and oversight and technical  assistance mechanisms for smaller grant programs.
section_title: Agencies’ Collaboration and Sharing of Lessons Learned on Tiered Evidence Grants Relied on Informal Networks
paragraphs: The four agencies in our review collaborated informally on tiered evidence  grants and officials said that they found these informal networks useful.  Individual agencies organized informal meetings and workshops on tiered  evidence grants that occurred as needed. For example, CNCS convened  multiple meetings with other agencies administering tiered evidence  grants. CNCS officials told us that they saw a benefit to making those  meetings more regular and structured. For example, CNCS officials said  that collaboration helped address specific topic areas of interest related to  tiered evidence grants, such as approaches to conducting evidence  reviews, provision of evaluation capacity building services to grantees,  and approaches in communicating evaluation results. The meetings can  also be used to discuss issues of importance to the sector, such as the  need for using consistent evidence definitions across agencies. Education  also hosted workshops with speakers from various agencies who  described how tiered evidence grants function and how agencies  implement the evidence and evaluation requirements, according to  Education officials.
In addition, OMB has taken steps to convene agencies on issues related  to tiered evidence grants. For example, in November 2013, OMB hosted  a workshop on innovative, evidence-focused grant programs that included  tiered evidence grants. OMB staff said OMB continues to provide  coordination and facilitate focused meetings on issues as they arise, and  has made materials and communication mechanisms available to the  agencies. For example, OMB created a web portal and discussion board  for agency officials to share information on evidence and evaluation  requirements, amongst other topics. However, some of the information on  the portal is not current and its focus is on the broader use of evidence in  policymaking. OMB staff said that OMB addresses issues on tiered  evidence grants as they arise and under the broader umbrella of  evidence-backed policy, and that no formal collaboration mechanism is  needed. However, officials from the four agencies in our review said that  they do not actively use the site for issues that arise specific to tiered  evidence grants. An official from one agency stated that it is unclear  whether the site is meant to be used to organize events, as a resource  repository, or a means to organize a community of practice. Another  official from a different agency said the site is designed is to provide  guidance to agencies on developing their annual budgets. Officials from  three programs in our review stated that they do not use the site for tiered  evidence grants.
At our facilitated discussion, agency officials generally agreed that they  would benefit from more formal collaboration on tiered evidence grants  and lessons learned from implementing them. The officials stated that  they have benefitted from collaborative efforts to date and are looking for  additional ways to communicate about specific issues related to tiered  evidence grants. For example, an Education official stated there is not an  easy way to locate and contact officials from other agencies who are  working on tiered evidence grant programs and that it would be helpful to  have a mechanism to identify federal officials working on similar tiered  evidence grant programs. According to the official, such a mechanism  would provide agencies a forum to answer questions and share lessons  learned.
By relying on ad-hoc collaboration, agencies using tiered evidence grants  may miss opportunities to capture and share lessons learned that could  strengthen and improve tiered evidence grant making government-wide.  Further, federal officials who are new to evidence-based grant making  may not be able to tap into the informal network. In our prior work on  collaboration, we reported that collaborative mechanisms, such as  interagency groups or collaboration technology, can be used to develop  policies, implement programs, and share information. Specifically, more  formal collaborative mechanisms can be effective to ensure collaboration  continues to address staff turnover in leadership positions.
For example, in response to a recommendation in our prior work on Pay  for Success programs, another relatively new tool for incorporating  evidence into policymaking, OMB developed an Interagency Learning  Network with representatives from 10 agencies to share lessons learned,  hone policy, and strengthen implementation. Creating a similar  mechanism for tiered evidence grants could help ensure current  collaboration efforts are continued, and transcend staff turnover and that  key lessons learned are captured. Further, formalized collaboration on  tiered evidence grants could continue momentum on agency efforts to  institutionalize evidence into policy-making.
section_title: Conclusions
paragraphs: Tiered evidence grants are an important component of congressional and  federal agency efforts to increase the use of evidence in policymaking  and distribute federal funds. Tiered evidence grants offer a means for  agencies to evaluate the effectiveness of their programs and for  Congress and agencies to fund programs based on proven and emerging  service delivery models. The lessons agencies have learned in  implementing tiered evidence grants could be applied to other federal  grant programs that use evidence and evaluation requirements as a  condition for receipt of federal grant funds. Agencies and grantees  reported a number of challenges in administering tiered evidence grants  and have also developed strategies for addressing these challenges.
Federal agencies that have administered tiered evidence grant programs  found that informal inter-agency collaboration helped them address  challenges and share lessons learned. These collaborative efforts to date  have relied on informal networks that have emerged during  implementation. A formal mechanism to facilitate communication and  sharing of lessons learned across federal agencies could ensure that the  beneficial informal collaborative efforts to date continue in a manner that  transcends changes in staffing and individual grant programs. The  absence of a formal mechanism to facilitate agency collaboration and to  share and document lessons learned presents a missed opportunity to  capitalize on what is known about early implementation of tiered evidence  grant programs. A formal mechanism to facilitate tiered evidence grant  collaboration and lessons learned could also help maintain momentum for  continuous learning about broader government-wide efforts to  institutionalize the use of evidence to consider the effectiveness of federal  grant programs.
section_title: Recommendation for Executive Action
paragraphs: To facilitate collaboration and identify and broadly disseminate  information on leading practices and lessons learned, the Director of  OMB should establish a formal means for agencies to collaborate on  tiered evidence grants. This could include creating a formal working group  or providing collaboration technologies, such as an electronic mailing list  or web portals.
section_title: Agency Comments and our Evaluation
paragraphs: We provided a draft of this product to the Office of Management and  Budget; Departments of Education, Health and Human Services, and  Labor; and the Corporation for National and Community Service for  comment.
The Office of Management and Budget had no comments on the draft  report. We received written comments on the draft report from the  Department of Health and Human Services, which are reproduced in  appendix III. The Departments of Education, Health and Human Services,  Labor, and the Corporation for National and Community Service provided  technical comments that were incorporated into the draft as appropriate.
We are sending copies of this report to the Director of the Office of  Management and Budget; Secretaries of Education, Health and Human  Services, and DOL; the Chief Executive Officer of the Corporation for  National and Community Service; and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Office  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We conducted this work under the GPRA Modernization Act of 2010  (GPRAMA), which requires us to review the implementation of the Act at  several critical junctures. This report is part of our response to that  mandate. The objectives of this report were to describe 1) key features  that tiered evidence grants add to federal grant processes, (2) benefits  and challenges of using tiered evidence grants, (3) key factors to facilitate  the use of tiered evidence grants, and (4) to assess the extent to which  the Office of Management and Budget (OMB) and federal agencies are  collaborating on evidence and evaluation requirements in tiered evidence  grants.
To meet our objectives, we initially identified nine tiered evidence grant  programs by conducting a literature review, reviewing annual budget  documents, and validating the list with the OMB. We focused on  programs that have operated long enough to show implementation results  and lessons learned—specifically those established prior to 2013. We  then narrowed the grant programs to the five that focus on domestic  populations:  Investing in Innovation Fund at the Department of Education (Education);
Teen Pregnancy Prevention Program at the Department of Health and  Human Services (HHS);
Maternal Infant and Early Childhood Home Visiting (Federal Home  Visiting) program at HHS;
Workforce Innovation Fund at the Department of Labor (DOL); and
Social Innovation Fund at the Corporation for National and Community  Service (CNCS).
OMB verified that the five programs we identified represented the  universe of tiered evidence grant programs that met our selection criteria.
We also interviewed officials from two other programs—Education’s  Supporting Effective Educator Development (SEED) and DOL’s Trade  Adjustment Assistance Community College and Career Training  (TAACCCT). While these programs include evidence and evaluation  features, they did not meet the full definition of tiered evidence grants for  purposes of this report. SEED requires grantees to show moderate  evidence on their service models but did not link the size of the grant  award to the strength of evidence. TAACCCT included a “break the cap”  award based on the level of evidence in its first funding round, but did not  include evidence tiers in the subsequent funding rounds.
To represent the tiered evidence grantee perspective, we selected three  programs for grantee interviews: the Investing in Innovation Fund, Teen  Pregnancy Prevention Program, and Workforce Innovation Fund. We  selected programs based on design features, including the size of the  grant award, whether programs had a matching funding requirement, the  timing of when the agency reviewed the evidence, and the number of  evidence tiers. For the size of the grant award, we selected one large  grant program, given that the majority of tiered evidence grant programs  are relatively small compared to other grant programs. We defined large  as greater than $1 billion in cumulative grant awards. We selected one  grant program that has a matching requirement because most tiered  evidence grants do not have such a requirement. Further, we decided to  select one grant program that reviewed evidence components of grant  applications at different times—both before grant solicitation and after  grant solicitation. Moreover, we selected programs to include models with  three evidence tiers and two evidence tiers.
We selected grantees in three metropolitan areas—the Washington,  D.C./Baltimore area, New Orleans, and the San Francisco Bay Area— using non-generalizable sampling. First, we grouped cities with grantees  into the same metropolitan area if they were within an approximately 1  hour driving distance. We focused on metropolitan areas that had  grantees in the three programs we selected for site visits (the Investing in  Innovation Fund, Teen Pregnancy Prevention Program, and Workforce  Innovation Fund). We then focused on metropolitan areas that had at  least 10 total grantees to ensure we would have a sufficient number of  grantees to interview in each location. We also focused on metropolitan  areas that had grantees in at least two evidence tiers. Finally, we selected  grantees from three geographic regions (East Coast, West Coast, and  South) to provide for geographic dispersion.
We interviewed two to three grant recipients for each evidence tier in the  three programs across our three locations. To select individual grantees,  we focused on grant recipients that have had sufficient time to implement  programs and identify lessons learned. For the Investing in Innovation  Fund and Teen Pregnancy Prevention Program, to the extent possible,  we focused on grantees that received funds from the initial round of  funding (2010 for both programs). If we could not further differentiate  recipients beyond this criterion, we randomly selected grantees to  interview. The Workforce Innovation Fund had only six recipients across  our three locations. We selected all of them for interviews. We  interviewed 18 grantees in total. When summarizing statements from  grantees in our report, we defined “some grantees” as at least three  grantees.
To identify the key features tiered evidence grants add to the federal  grant making life cycle, we reviewed federal notice of funding  announcements that described the evidence and evaluation requirements  for tiered evidence grant requirements. We also interviewed officials from  Education, HHS, DOL, and CNCS regarding how the tiered evidence  grant-making life cycle differs from traditional grant programs.
To describe the benefits and challenges of using tiered evidence grants  and factors for addressing the challenges, we reviewed agency  documents, including reports on lessons learned, evaluation reports, and  agency guidance. We interviewed officials from Education, HHS, DOL,  and CNCS, as well as the grantees from the Investing in Innovation Fund,  Teen Pregnancy Prevention Program, and Workforce Innovation Fund.  We also interviewed national evaluation organizations that agencies  contracted with to provide technical assistance to grantees and conduct  evaluations across multiple grantees. We synthesized the list of  challenges and factors for addressing the challenges. We then provided  this list to key federal officials to provide the basis for a facilitated  discussion to validate our list. For the facilitated discussion, we hosted an  in-person meeting with officials from the five programs we selected for  analysis.
To assess the extent to which OMB and federal agencies collaborated on  evidence and evaluation requirements in tiered evidence grants, we  interviewed officials from Education, HHS, DOL, CNCS, and OMB on  current collaboration mechanisms and plans for collaboration moving  forward. We also asked about interagency collaboration on tiered  evidence grants during our facilitated discussion. For criteria, we used our  key practices for enhancing and sustaining collaboration and  implementing interagency collaborative mechanisms.
We conducted this performance audit from September 2015 to  September 2016 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Evidence Tiers at the Department of Education
paragraphs: The Department of Education’s (Education) Investing in Innovation Fund  is one example of how a federal agency defines the tiers for its tiered  evidence grant program (see table 1). Education included the definitions  of its three evidence tiers in the notice of finding availability  announcements for the grant program.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgement
subsections: section_title: GAO Contact
paragraphs: Michelle Sager, 202-512-6806 or sagerm@gao.gov.
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Jeff Arkin (Assistant Director),  Barbara Lancaster (Analyst-in-Charge), Amy Bowser, Jeff DeMarco,  Robert Gebhart, Steven Putansu, and Robert Robinson made key  contributions to this work.