Performance Partnerships: Agencies Need to Better Identify Resource Contributions to Sustain Disconnected Youth Pilot Programs and Data to Assess Pilot Results
Released: 2017-04-18T00:00:00

Letter

Background

Performance partnerships, as we reported in December 2014, are a type of hybrid approach to grant consolidation in which grant recipients can obtain flexibility to use funds awarded across multiple federal programs in exchange for greater accountability for results. Figure 1 provides an overview of the performance partnership model.

Grant consolidation can create opportunities to eliminate federal programs that are overlapping or outdated, or for which the balance between costs and benefits received either do not (or no longer) justify federal spending. According to prior research by the former U.S. Advisory Commission on Intergovernmental Relations, grant consolidations are generally suitable when categorical programs are too small to have much impact or to be worth the cost of administration, or when multiple programs exist in functional areas that have a large number of programs (including health, education, and social services), or where there is fragmentation (including justice, natural resources, and occupational health and safety). Grant consolidations generally take either a block grant or a hybrid approach. A block grant approach is usually broad in scope, intended to increase state and local flexibility, and generally give recipients greater discretion to identify problems or to design programs addressing those problems using funding from the block grant. A hybrid approach, such as a performance partnership, can consolidate a number of narrower categorical programs while retaining strong standards and accountability for discrete but related federal performance goals.

Since the 1990s, the federal government has taken steps to explore and establish performance partnerships. For example, the National Performance Review (NPR) identified performance partnerships as a tool for helping federal agencies reform the existing federal grant system, which it stated, among other issues, featured too many funding categories, an emphasis on remediating rather than preventing problems, and no clear focus on measurable outcomes. It noted that performance partnerships could improve federal grant making in situations in which the federal government intends to deliver services at the state or local level, agrees with state or local partners on goals and objectives, and progress toward goals and objectives can be measured. In February 1995, the President’s Budget for fiscal year 1996 proposed 6 performance partnerships spanning 7 federal agencies—the Departments of Agriculture, Education, HHS, HUD, DOL, and Transportation, and EPA— that it stated were aimed at combining funding streams, eliminating overlapping authorities, and turning agencies’ focus to outcomes as the basic measure of success. In 1996, Congress provided EPA authority to create PPGs. More recently, the Performance Partnership Pilots for Disconnected Youth were authorized in January 2014.

For our December 2014 report, we determined that EPA’s PPGs and the disconnected youth pilots were the only 2 existing federal performance partnerships, and for this report we confirmed that they remain the only ones authorized to date.

Performance Partnership Grants within the Environmental Protection Agency’s National Environmental Performance Partnership System

According to EPA, the relationship between EPA and the states has long been complex, due in part to the division of roles and responsibilities under federal environmental statutes. Prior to EPA’s creation in 1970, states provided the majority of environmental management controls, such as establishing standards for the amount of pollutants that can be released into air or water and developing public health and natural resources regulations. Subsequently, EPA became a partner with states and localities in environmental management. Most major federal environmental statutes, including the Clean Water Act, permit EPA to allow states under certain circumstances to implement key programs and to enforce their requirements.

Several efforts to explore and improve relationships between EPA and the states led to the creation of NEPPS and PPGs. In 1993, EPA and the states convened the Task Force to Enhance State Capacity to generate ideas for improving their partnership. The task force reported that new federal environmental statutes had increased the environmental management responsibilities being borne by states at a time when they were facing declining resources. In addition, the task force reported that EPA and the states faced difficulty in working together on issues of day- to-day program management, which strained their relationship. The task force made a number of recommendations, including that EPA and the states establish a new framework and policy for their relations and a joint process for strategic planning and the integration of both sides’ priorities. In May 1995, EPA and the states established NEPPS to address the task force’s recommendations. As we reported in December 2014, NEPPS is a performance-based system designed to direct scarce public resources toward improving environmental results, allow states greater flexibility to achieve those results, and enhance accountability to the public and taxpayers. A key element of NEPPS, upon its establishment, was EPA’s commitment to give states with strong environmental performance greater flexibility and autonomy in running their environmental programs. percent of the $1.08 billion in environmental program grants EPA awarded in fiscal year 2016.

Performance Partnership Pilots for Disconnected Youth

As we reported in February 2008, while most youth successfully transition to adulthood, some become disconnected from school and work, and experience challenges in making this transition. Their disconnection may result from incarceration, aging out of foster care, dropping out of school, or homelessness. Some of these youth are more likely than others to remain low-income and lose jobs during economic downturns, and to engage in criminal activity, antisocial behavior, and teenage parenting. Direct services intended to assist youth in transitioning to adulthood are provided at the local level with the support of federal, state, and local governments, and private funding sources. A range of local entities, such as community-based organizations—which are generally non-profit entities that provide social services—and charter schools, in urban and rural communities nationwide, help to provide such services.

Multiple federal agencies play a role in providing funding and assistance to local programs that serve disconnected youth, which can create challenges for local service providers. In February 2008, we reported that the White House Task Force for Disadvantaged Youth identified 12 federal agencies that funded over 300 programs to assist local communities in serving disadvantaged youth in fiscal year 2003. In conducting that work, we also interviewed the directors of 39 local programs serving disconnected youth, and those whose programs received multiple federal grants from multiple federal agencies told us they experienced difficulties in working across varying reporting requirements, funding cycles, and eligibility requirements. The directors also reported experiencing challenges working across varying program goals and sharing information about their clients that participate in multiple federal grants.

The Performance Partnership Pilots for Disconnected Youth seek to identify cost-effective strategies for providing services that can address these types of challenges and achieve better results through making better use of budgetary resources. Although implementation of the first round of pilots began in 2015, their genesis dates back to early 2011. Figure 2 identifies key events in the development and implementation of the disconnected youth pilots.

According to federal officials involved in the pilots, the concept for the disconnected youth pilots came in response to a February 2011 Presidential memorandum. It directed federal agencies to work with state, local, and tribal governments to identify and develop strategies for eliminating administrative, regulatory, and legislative barriers to achieving results in federally funded programs and increase access to flexibilities needed to produce the same or better outcomes at lower cost. Officials from Education, HHS, and DOL told us that following the memorandum’s issuance, representatives from their agencies met with representatives from state, local, and tribal governments to discuss policy areas in which they thought additional flexibilities could improve outcomes. They identified programs for disconnected youth as an area that would benefit from such flexibilities.

In 2012, the Administration and agencies took several steps aimed at better coordinating and integrating programs focused on disconnected youth. In February 2012, the President’s Budget for fiscal year 2013 requested authority for a new performance partnership pilot initiative to test approaches to improve outcomes for disconnected youth. Shortly thereafter, in March 2012, OMB, Education, HHS, HUD, DOJ, and DOL established the Interagency Forum for Disconnected Youth with the goal of improving outcomes for disconnected youth through enhanced interagency and intergovernmental collaboration. In June 2012, Education published a request for information in the Federal Register seeking ideas and information on effective approaches for improving outcomes for disconnected youth. The Interagency Forum on Disconnected Youth used the responses to the request for information to develop initial design considerations for the disconnected youth pilots, according to documentation of the design considerations.

The first round of the Performance Partnership Pilots for Disconnected Youth was authorized in the Departments of Labor, Health and Human Services and Education and Related Agencies Appropriations Act for fiscal year 2014. Enacted in January 2014, it authorized federal agencies appropriated funds thereunder to select and implement a round of up to 10 pilots designed to improve outcomes for disconnected youth that may run for 5 fiscal years (through fiscal year 2018, which ends September 30, 2018). The act defines disconnected youth as individuals between the ages of 14 and 24 who are low-income and either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in or at risk of dropping out of an educational institution. The pilots are to involve 2 or more federal programs administered by 1 or more federal agencies. The act provides agencies authority to use discretionary funding made available in the act and waive statutory, regulatory, or administrative requirements related to the use of that funding. The agencies involved in the first round of pilots—Education, HHS, DOL, CNCS, and IMLS—issued a request for public comment on the pilot application process in July 2014 that sought feedback on information applicants should include in their applications, criteria agencies should use in evaluating applications, and technical assistance for entities preparing applications. Officials from Education told us that the agencies incorporated public comments from the request in a notice inviting applications, which they subsequently issued in November 2014. The federal agencies designated 9 pilot locations as finalists for the first round in September 2015 and publicly announced the locations in October 2015 (see fig. 3).

Subsequent appropriations laws authorized 2 additional rounds of disconnected youth pilots and broadened the scope of the effort.

Second Round. In December 2014, a second round of up to 10 locations for disconnected youth pilots was authorized, again with a 5- year timeframe for implementation (through fiscal year 2019), and agencies were authorized to use funds made available in the fiscal year 2015 appropriations to participate in previously authorized pilots. Education published a notice of proposed priorities, requirements, definitions, and selection criteria for the second and future rounds of pilots in the Federal Register in October 2015. The notice proposed additional priorities for projects serving specific high- need subpopulations of disconnected youth, changed application requirements to reduce burden on applicants, and asked for comments on how federal agencies could improve future pilot competitions. In April 2016, the agencies participating in the pilots published final priorities, requirements, definitions, and selection criteria. That same month, they published a notice inviting applications for the second round of pilots. The notice established a June 2016 deadline for application submissions, but in July 2016 the agencies published a notice reopening the application process. Agencies took this action to allow applicants additional time to prepare and submit their applications. According to Education officials, in September 2016 federal agencies designated 1 second round applicant as a pilot finalist. The agencies expect to announce the pilot publicly once the agencies and the finalist have signed a performance partnership agreement.

Third Round. The third round of disconnected youth pilots, again consisting of up to 10 locations with a 5-year timeframe (through fiscal year 2020) was authorized in December 2015, and agencies were authorized to use funds made available in the fiscal year 2016 appropriations act to participate in previously authorized pilots. DOJ and HUD were authorized to participate in this round of pilots. In addition, this authorization established that new pilots selected for the second round using fiscal year 2015 funds and the subsequent third round must include communities that have recently experienced civil unrest. In August 2016, the agencies published a notice inviting applications for the third round of pilots. The application period closed in October 2016, and in January 2017 the agencies designated 6 applicants as third round pilot finalists.

Performance Partnership Initiatives Exhibit Four Key Characteristics Somewhat Differently

We identified 4 key characteristics that PPGs and the disconnected youth pilots share: 1. documented agreement outlining goals, roles, and responsibilities; 2. flexibility in the use of funds across multiple federal programs; 3. additional flexibilities, such as expanded program participant eligibility or streamlined reporting requirements; and 4. accountability for results.

The following sections describe each of the shared key characteristics, providing illustrative examples from selected states with PPGs and selected pilot locations and any benefits or challenges associated with these key characteristics as described by participants. More detailed information about how the key characteristics are exhibited in the 2 initiatives and additional illustrative examples are contained in appendix II (PPGs) and appendix III (disconnected youth pilots).

Performance Partnership Goals, Roles, and Responsibilities Are Generally Documented in Agreements

Federal agencies and non-federal grant recipients generally document in an agreement what is entailed by their partnership. The document establishes the various goals the partners seek to achieve through their partnership. It also lays out the roles and responsibilities of each partner.

PPG Work Plans Document Goals, Roles, and Responsibilities

A PPG generally involves an EPA regional office and a state agency, such as a state environmental, health, or agricultural agency. Figure 4 provides the general structure of this partnership.

EPA and state agencies define the scope of their partnership in a PPG work plan. For programs authorized under federal environmental statutes, EPA generally is responsible for establishing program policy and guidance and oversight, and states generally are responsible for carrying out day-to-day program operations. Therefore, PPG work plans, like other EPA program grant work plans, identify an agreed-upon set of planned work activities the state agency will undertake and their timeframes for completion, as well as information about the EPA strategic goals and objectives that the activities are expected to help meet. EPA guidance states that a PPG work plan should result from negotiations between EPA and state program managers and staff and reflect joint planning, priority setting, and mutual agreement between the 2 sides.

For example, the work plan for state fiscal year 2015 for EPA’s and the California Department of Pesticide Regulation’s (DPR) PPG defined their partnership for a PPG that spanned state fiscal years 2013 to 2016. The document identified program areas in which DPR would undertake work. These included areas such as enforcing pesticide laws and ensuring worker safety from pesticides. DPR linked each of these program areas to the specific EPA strategic plan goals and objectives they supported. For instance, DPR’s work in the area of enforcing pesticide laws was linked to EPA’s strategic plan goal of protecting human health and the environment by enforcing laws and ensuring compliance, as well as its related strategic objective of enforcing environmental laws to achieve compliance. The work plan also identified specific work activities DPR planned to complete by the end of the state fiscal year. For example, in the program area of enforcing pesticide laws, DPR agreed to conduct 182 oversight inspections of the use of pesticides in agricultural operations.

Officials from EPA and state agencies involved in the PPGs in our review described as a benefit how the 2 sides work together to develop PPG work plans, noting that the partnerships have strengthened their collaborative relationships. For example, officials from EPA Region 2 and the New York State Department of Environmental Conservation (DEC) said that EPA’s National Program Manager Guidance—biennial guidance from EPA program offices that establishes priorities and key actions to accomplish—serves as a framework for the activities that DEC will conduct through the PPG. Officials from Region 2 and DEC annually discuss how DEC priorities can be addressed within the framework and what appropriate goals and targets are for identified priorities. Officials from the New York DEC told us that the good working relationship they have with EPA Region 2 officials allows them to effectively work together to adjust or reconcile competing priorities when unexpected challenges arise.

Disconnected Youth Pilots Use Performance Partnership Agreements to Document Goals, Roles, and Responsibilities

In contrast with the 1-on-1 partnerships in PPGs, first round disconnected youth pilots involve 2 or more federal partners and, in most cases, multiple grant recipients whose joint application was selected to participate in the pilot. The general structure for this partnership is shown in figure 5.

The federal and non-federal organizations involved in each first round disconnected youth pilot defined the scope of their partnership in a performance partnership agreement. These agreements establish the terms and conditions under which the federal and non-federal partners will participate in the pilot and identify the specific outcomes the 2 sides will seek to achieve.

The roles and responsibilities that the partners assume in developing and implementing the agreements are specific to the initiative. The federal agencies involved in developing the initiative, through a separate interagency agreement, established the following roles for federal partners in individual pilots:

Lead agency: The lead agency is responsible for managing the performance partnership agreement. OMB designated Education as the lead agency for all 9 first round pilots. During the negotiations of the first round partnership agreements, Education coordinated the negotiations on behalf of, and in partnership with, all federal agencies involved, and worked with non-federal partners to finalize planned pilot goals and related performance measures. In addition, Education, as the lead agency, provides and oversees start-up grants to pilots.

Consulting agency: The consulting agency leads pilot monitoring on behalf of the involved federal agencies. It does so by providing feedback on pilot performance reporting and facilitating communication among federal agencies and non-federal partners.

Participating agency: The participating agency provides support to the lead and consulting agency by, as appropriate, providing feedback on pilot performance reporting and assistance to the other federal agencies and non-federal partners to address any implementation issues.

Non-federal partners assume roles and responsibilities established in the performance partnership agreements. The partnership agreements for each of the first round pilots designate a state, local, or tribal government entity as the pilot lead. The pilot lead is responsible for ensuring that the pilot is carried out in accordance with applicable federal requirements and oversees the proper use of all federal funds. The agreement also identifies any additional non-federal partners, such as another government entity or a non-profit community organization, involved in the pilot and their roles and responsibilities.

For example, in the Chicago pilot, the federal partners are Education as the lead agency, HHS as the consulting agency, and DOL as a participating agency. The non-federal partners, and their roles, are:

The Chicago Department of Family and Support Services, the pilot lead, works to connect Chicago residents and families to resources that build stability, support their well-being, and empower them to thrive.

The Chicago Cook Workforce Partnership, a pilot partner, will consult with the Department of Family and Support Services in implementation and oversight of the pilot. The organization is a collaborative effort between Cook County, Illinois and the City of Chicago designed to align the 2 entities’ efforts in delivering services under the Workforce Innovation and Opportunity Act (WIOA).

The federal and non-federal partners in each first round pilot worked during their development of a performance partnership agreement to identify the pilot’s intended outcomes. In the partnership agreements, these intended outcomes take the form of quantitative goals and measures. Federal agencies require that at least 1 set of goals and measures address educational outcomes and a second set address employment outcomes. The 4 pilots in our review established education and employment goals and measures tied to their particular service interventions. The Oklahoma pilot, for example, is structured to help youth with foster care experience in the Oklahoma City Public Schools complete high school, attend college, and enter the workforce. Its partners established an education goal for 80 percent of youth who complete at least 6 months in the program to attain a high school diploma or its equivalent. The partners established an interim measure—that 85 percent of participants will be absent from school for 15 days or fewer during the school year—to track progress toward the goal. According to the pilot’s application, increased school attendance for participants is likely to lead to an improved high school graduation rate for them.

Similar to PPGs, officials from federal and non-federal partners involved in disconnected youth pilots told us in our interviews that their partnerships have strengthened collaborative relationships with each other. For example, officials from the pilot lead for the Chicago pilot told us that the non-federal partners worked closely with HHS, the pilot’s consulting agency, and DOL, a participating agency, to gain a better understanding of the types of flexibilities they could use to implement the pilot. In addition, partners told us that the partnerships have strengthened collaboration among multiple non-federal partners working together at the state, tribal, or local level to implement disconnected youth pilots. For instance, officials from the pilot lead for the Oklahoma pilot told us that they have used the pilot development process to convene a wide range of organizations involved in addressing the needs of youth in foster care in Oklahoma City, the pilot’s target population, to establish a network broader than just those officially in the pilot. This convening enabled the pilot lead to identify organizations with which it previously had not worked that could contribute to improved outcomes for foster youth and bring them into the new network.

However, federal and non-federal partners in disconnected youth pilots also told us that at times the multiple-partner structure of the pilots, along with their new and unique nature, has caused complications and delays in the pilots’ design and implementation. Officials we spoke with from several of the federal agencies involved in developing the initiative after its 2014 authorization said that, given their lack of familiarity with a legal provision like this, they had to spend time and effort to reach a common interpretation of the provision and how it could be implemented. This resulted in a longer than usual process for the agencies to develop and issue the notice inviting applications for the first pilot round. The first pilot round was authorized in January 2014 and Education released the notice inviting applications in November 2014. Federal and non-federal partners involved in pilots in our review told us that there were additional challenges related to negotiating and finalizing the partnership agreements, given the numerous parties involved. This led to additional time being spent on finalizing the agreements. The agreements for 6 of the 9 pilots were signed by their non-federal partners in December 2015 and January 2016, with the other 3 being signed by their non-federal partners between February and April 2016.

Non-federal partners in the selected pilots also identified challenges in developing and coming to agreement on the goals and measures. Officials from the Eastern Kentucky pilot told us that it took time to work with federal agencies to agree on the goals and measures that were included in the final partnership agreement. The non-federal partners wanted to set goals for pilot participants to improve their academic performance to the average level of students in the Kentucky Highlands Promise Zone. Federal partners, however, requested that those goals be set at the average level of students in all of Kentucky—which is a higher average than of those just in the Promise Zone. The partners had discussions around the issue and agreed to establish goals at the higher level of students across the state, with recognition that these were stretch goals. Despite these challenges, several non-federal partners also told us that they see benefit in being able to establish goals and measures tailored to interventions, and not having to use standard federal performance measures, which may not always be useful in determining outcomes among their target populations. For example, officials from the Ysleta del Sur Pueblo pilot told us that they established performance measures for their pilot that will allow them to better determine the educational and employment outcomes of the tribal youth they plan to serve.

Performance Partnerships Are Characterized by Funding and Other Flexibilities

Performance partnerships provide non-federal partners with flexibility in how they use funds from multiple federal programs. Partnerships can also provide non-federal partners with additional flexibilities, such as expanded program participant eligibility or streamlined reporting requirements. The non-federal partners can use these flexibilities hand-in-hand to tailor efforts to more effectively achieve their goals, as well as reduce their administrative burden.

PPGs Provide Funding and Other Flexibilities to Ease Administrative Burden for States

PPGs permit state agencies to request that funding they receive from 2 or more EPA program grants be combined into a single award. This is intended to enable state agencies to, among other things, consider trade- offs across the breadth of their environmental program funding and exercise flexibility to direct resources to their most pressing priorities. Specifically, once a state agency has requested and received selected EPA program grants in a PPG, it can choose to use the funds to support any activity that is eligible under at least 1 of the grants included in the PPG. PPGs also streamline administrative requirements so that state agencies can realize cost savings through reduced administrative burden in areas such as grant applications, cost sharing, and financial reporting.

PPGs we examined in selected states exhibit variation in how state agencies chose to receive combined EPA program grant funding and exercise additional flexibilities. We selected states in which environmental agencies received either a small or a large number of grants combined in their PPGs. On the small end, we selected New York, whose DEC received funding from 3 grants, and California, whose DPR received funding from 4 grants in 1 environmental area (water and pesticides, respectively) in their PPGs in fiscal year 2016. On the large end, we selected Alabama, whose Department of Environmental Management (DEM) received funding from 9 grants, and Utah, whose Department of Environmental Quality (DEQ) received funding from 10 grants across multiple environmental program areas in their PPGs in fiscal year 2016. Figure 6 shows the grants in DEQ’s PPG in fiscal year 2016. Details on the PPGs in Alabama, California, and New York are included in appendix II.

According to EPA guidance on PPGs, states can take different approaches to exercising their flexibility to direct resources to their most pressing priorities covered by any activity that is eligible under at least 1 of the grants included in the PPG. A state can propose using this funding to pool resources from multiple programs consolidated into the PPG to implement projects or initiatives that cross traditional program barriers. For example, a state can propose to conduct inspections to assess compliance across air, water, and hazardous waste management requirements if it has included grant programs in those areas in its PPG. A state can also, based on its environmental priorities, propose increasing resources and effort in 1 program area while decreasing resources and effort in a second program area. For example, if a state has identified that its needs in addressing water pollution are greater than its needs in addressing air pollution, and it has included relevant water and air grant programs in its PPG, it can propose to strategically increase resources and effort for water pollution activities while decreasing them for air pollution activities. States that intend to exercise programmatic flexibilities must explain the reasons for and expected benefits of the flexibilities in their PPG application.

State agencies involved in PPGs also see great benefit in their ability to make use of flexibilities that reduce administrative burden related to grant applications, cost sharing, and financial reporting, according to the state agency officials whom we interviewed.

Grant applications. A state agency can submit a single application covering all of the grants it is seeking to consolidate in its PPG rather than a separate application for each. Officials from the Alabama DEM told us that the streamlined PPG grant application requirement has allowed the DEM to submit a single PPG application rather than 9 individual grant applications, which it had to do prior to adopting the PPG. This has reduced the amount of administrative work that DEM staff must complete, thereby allowing them to focus on other activities.

Cost-sharing. Certain EPA program grants require state agencies to provide a portion of program costs in order to receive the grant. Some grants require states to provide a certain percentage of total expenditures under the grant, known as a match requirement, while some others require states to spend non-federal funds for work conducted under the grant in an amount at least equal to those spent in a previous year, known as a maintenance of effort requirement. When a state combines grants in a PPG, it does not have to meet the individual cost-share requirements of the grants included in the PPG; instead, the state’s cost-share for a PPG is not less than the sum of the minimum required under each of the underlying grants included in it. According to EPA guidance on PPGs, the ability to meet cost- sharing requirements in the aggregate can be valuable when a state has more than adequate resources to meet the match required of 1 program included in the PPG but not enough for a second included program. The state can use excess match resources to cover the program that cannot meet its match requirement. Officials from the Utah DEQ said that the ability to meet match requirements in the aggregate is one of the most useful aspects of a PPG. They explained that multiple grants they include in their PPG have match requirements. Because of the DEQ’s line-item budget structure, the officials stated it would be challenging for them to meet the match requirements of specific program grants if they were not included in a PPG. However, because the PPG allows match requirements to be met at an aggregate level, DEQ’s expenditures in particular program areas can be added together to meet the overall match requirement.

Financial reporting. A state agency can report on expenditures within the PPG in the aggregate, covering all grants consolidated in the PPG, rather than for each grant individually. Officials from each of the states in our review reported that they benefit from streamlined financial reporting. For example, officials from the Alabama DEM said that they provide 1 annual financial report for their PPG to EPA rather than the 9 that they were required to provide for individual grants before they adopted the PPG. They stated that this change has significantly reduced DEM’s administrative burden.

Officials from EPA regional offices and state agencies we interviewed told us that existing organizational silos within EPA and state agencies can limit a state’s willingness and ability either to include EPA program grants across multiple environmental areas in a PPG or, in cases in which states have done so, to take full advantage of available funding flexibilities. For example, although officials from the New York DEC told us that their agency has been able to use Water Pollution Control and Water Non- Point Source Management funds to create a more integrated, comprehensive clean water program by including them in a PPG, they said that their agency’s structure makes it difficult for the agency to include programs from other environmental areas in a PPG and thereby take advantage of additional programmatic flexibility. They explained that their agency organizes its operating divisions by environmental areas, such as water and air. The management of a cross-area PPG would require coordination across divisions, which would require the divisions to make changes in the way they operate. The costs associated with making the changes necessary to administer such a PPG, the officials stated, would likely negate the benefits of the potential additional programmatic flexibility. Furthermore, officials from the Alabama DEM, which receives a PPG combining program grants across environmental areas, told us that once the agency receives a PPG from EPA it generally distributes the funding associated with each of the underlying grants to the program offices responsible for implementing them. In essence, the agency reverses the combining of the funding at the state level and uses it in much the same way it would if it received the funding from the underlying grants outside of a PPG. The officials explained that their agency uses PPG funding for the individual program-specific activities for which they were originally approved because officials within the relevant program offices at both EPA Region 4 and DEM have wanted to maintain control over the program funds they are responsible for managing and overseeing.

Disconnected Youth Pilots Use Flexibilities to Tailor Services to Target Populations

Similar to PPGs, the disconnected youth pilots enable non-federal partners to combine funds from federal agencies’ programs and obtain additional flexibilities, but we found that for the disconnected youth pilots we reviewed, these flexibilities were generally used to tailor service interventions to the specific needs of their target populations rather than to reduce administrative burden. Specifically, the authorization for the first round pilots allows for the combining of discretionary funding that Education, HHS, DOL, CNCS, and IMLS received through the fiscal year 2014 appropriations act, and the waiver of statutory, regulatory, or administrative requirements affecting target populations, as proposed by the grantees, to carry out the pilots. This authorization included 2 safeguards on the use of these flexibilities, both which require written determinations by the head of an agency. First, an agency can participate in a pilot and combine funds only after its head provides a written determination that the agency’s participation will not result in denying or restricting the eligibility of individuals for any of the services that are funded by the agency’s programs or funds being used in the pilot, and that vulnerable populations who receive such services will not be otherwise adversely affected by the agency’s participation. In the notice inviting applications for first round pilots, applicants were advised that where a program’s funds are not suitable for combining (referred to as “blending”) in a pilot given these constraints, the applicant may nevertheless consider how to coordinate (referred to as “braiding”) such funding in a pilot to promote more effective and efficient outcomes even though the funds would maintain a separate identity and remain subject to the program requirements for which the funds were appropriated. Second, an agency also can waive program requirements associated with funds being used in a pilot, but only after its agency head issues a written determination that the granting of such waivers (1) are consistent with the statutory purposes of the underlying federal program and other provisions of the pilot authority, including that individuals will not be denied or restricted eligibility for services, (2) are necessary to achieve the outcomes of the pilot and no broader in scope than is necessary to do so, and (3) will result in either realizing efficiencies (by simplifying reporting or reducing administrative barriers) or increasing the ability of individuals to obtain access to services.

In addition, for the first pilot round federal agencies awarded separate start-up grants to provide funding of up to $700,000 to each pilot to finance evaluations, capacity building, technical assistance and other related activities to support the pilot. According to officials from OMB, funds available for these purposes from CNCS, DOL, and Education were used for these start-up grants. Education officials told us that start-up grants also were in part intended to provide an incentive to non-federal partners to participate in and implement the disconnected youth pilots. Non-federal partners stated that the start-up grants were a key incentive to become involved with the pilots, as they represent a significant amount of new funding for them, which could help them work across traditional program lines, among other things.

The first round pilots vary in the extent to which they use combined federal funds and waivers from selected program requirements in their pilots, as noted below and further illustrated in appendix III. Of the 9 pilots:

2, including Ysleta del Sur Pueblo, are combining all of the federal funds from all of the federal programs they are including in their pilots,

2, including Eastern Kentucky, are combining federal funds from some of the federal programs they are including as well as coordinating the use of federal funds from other federal programs, and

5, including Chicago and Oklahoma, are not using any combined funding but are instead coordinating the use of federal funds from multiple federal programs.

Non-federal partners from the 2 pilots selected in our review that are combining all or some federal funds—Eastern Kentucky and Ysleta del Sur Pueblo—told us that they consider the ability to combine funds to be a benefit, as it allows them to implement more effective programs and services for disconnected youth. In addition, the use of combined funds reduces non-federal partners’ financial reporting burden. To ensure accountability for the proper use of combined funds and start-up grants in pilots, non-federal partners provide financial reports on their use of these funds to Education as the lead agency. In contrast, when a pilot has coordinated funds, the relevant non-federal partner(s) will report on the use of those funds separately, as prescribed by the originating agency per its normal guidelines for the program. Ysleta del Sur Pueblo officials told us that they see this reporting process as a key benefit of its pilot. Officials said the tribe was motivated to apply to become a pilot site to gain administrative benefits such as reducing the amount of reporting they normally would have provided to CNCS and IMLS—the 2 agencies with program funds being used in the pilot. Since the tribe is combining all funding involved in the pilot, it instead reports to Education on its use of those funds.

In addition, 8 of the 9 first round disconnected youth pilots, including the 4 pilots selected for this review, requested and were granted waivers of selected requirements for at least 1 of the federal programs included in their pilots. These waivers provide pilots with additional flexibilities to tailor allowable activities, participant eligibility, and reporting requirements to better meet the needs of disconnected youth, according to the notice inviting applications for the first pilot round. Furthermore, non-federal partners in the 4 pilots in our review told us that, among other things, the waivers enable them to change eligibility requirements or the allowable use of select federal funds, which will allow them to implement innovative approaches tailored to disconnected youth. For these reasons, all of the federal and non-federal partners with whom we spoke told us that the flexibilities possible through the pilots—the ability to combine funds and obtain waivers from selected federal program requirements—are one of their biggest benefits.

The Eastern Kentucky pilot illustrates variation in the use of combined and coordinated funding and waivers. As illustrated in figure 7, to improve the academic performance of disconnected youth, the pilot is using combined funding across DOL’s WIOA Title I Youth program and Education’s Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) and Promise Neighborhoods programs and coordinating the use of funding from Education’s Full Service Community Schools program.

The Eastern Kentucky pilot was also granted 5 waivers that allow the non-federal partners to change eligibility requirements and the allowable use of certain federal funds. Officials from Partners for Education at Berea College (PFE), a pilot partner, told us that the various federal programs serving disconnected youth in Eastern Kentucky were focused on different aspects of the disconnected youth population. The non- federal partners had a difficult time providing comprehensive services to disconnected youth in the region since they had to serve distinct segments of the population with individual federal grants. Officials said that the waivers should help the non-federal partners expand the reach of their services. One of these waivers is related to a requirement in Education’s GEAR UP program, which is designed to help low-income students prepare for and succeed in post-secondary education. Traditionally, GEAR UP grant recipients can use the funds to provide mentoring, outreach, and other services to students for 6 years, usually between seventh grade and the completion of high school. They can also provide those services for a seventh year as long as during that year the student is enrolled in post-secondary education. The officials from PFE told us that the waiver allows them to provide services to youth who, during their seventh year, are not enrolled in post-secondary education, to help them connect to post-secondary education. Officials from the pilot told us that using some combined funds allows them to hire staff who are not tied to a specific program and therefore can work across the programs involved in the pilot. In addition, the officials from PFE told us that the start-up grant is allowing the non-federal partners to establish a data collection system to share information about the participants across partners. This would allow them to use those data to identify effective strategies and support the evaluation of the pilot. The waivers, the pilot officials added, should help the non-federal partners expand the reach of their services.

Federal and non-federal partners involved in disconnected youth pilots told us that in some cases the 2 sides faced challenges in coming to agreement on the use of funds in pilots. For example, the non-federal partners in the Chicago pilot initially proposed combining Head Start funds from HHS with WIOA Title I Youth funds from DOL. According to documentation of HHS’s review of this request, HHS denied the request because officials were concerned that combining Head Start funds could adversely affect vulnerable populations—one of the restrictions for combining funds in the pilot authorization—by potentially diverting funds away from services for children in Head Start toward activities primarily targeted at teenage and young adult mothers, who are primarily served by the WIOA program. HHS agreed that the pilot could coordinate Head Start funds, as is illustrated in figure 8, since this would ensure that the funds retain their cost allocation requirements and therefore allow HHS officials to ensure accountability for the funds’ use to support services for children in Head Start.

Performance Partnerships Use Performance Reporting to Ensure Accountability for Results

Performance partnerships use performance reporting to ensure accountability for results. Non-federal partners periodically report to federal partners on their progress towards the goals established in the partnership document. As was described in the prior section, non-federal partners also ensure accountability for the use of funds through financial reporting processes.

PPGs Use Performance Reports and Data to Ensure Accountability for Results

To monitor progress toward the work activities and goals and measures established during performance planning and included in the work plans discussed above, federal and non-federal partners in PPGs engage in performance reporting. Related to PPGs, we reported in July 2016 that, according to EPA policies and officials, after EPA approves a work plan for an EPA grant, grantees generally submit information on their progress and results to EPA in 2 ways: (1) performance reports, which are generally written and describe the grantees’ progress toward the planned grant results in their work plans and (2) program-specific data, which is generally numeric and which grantees electronically submit on certain program measures that EPA tracks in various program databases.

Performance reports. These reports describe the grantees’ progress toward the planned grant results in their work plans, such as using grant funds to provide technical assistance to local officials. EPA grantees are to submit these reports at least annually. EPA policies include general guidelines about what performance reports should include, such as a comparison between planned and actual grant results, but allow the frequency, content, and format of performance reports to vary by program and grant. According to EPA officials, EPA project officers monitor these reports to review grantee progress toward agreed-upon program results. PPGs enable state agencies to submit a single performance report for all the programs included in their PPGs, according to EPA and state agency officials with whom we spoke. We reviewed the most recently available end-of-year performance reports that the Utah DEQ submitted to EPA for federal fiscal year 2014 and the California DPR submitted to EPA for state fiscal year 2014-2015. For its report, DPR provided information about how it addressed each planned activity in its work plan. For example, the report stated that DPR conducted 253 oversight inspections of the use of pesticides in agricultural operations, exceeding its target of 182 oversight inspections. According to information in the report, EPA, after it received the report from the department, reviewed the material and provided comments, as needed.

Program-specific data. Grantees electronically submit data on certain program measures, such as the number of hazardous waste violations issued or the acres of brownfield properties made ready for reuse, which EPA tracks in various program databases. According to EPA policy and program officials, program officials monitor these data to track and report program accomplishments, at the regional and agency levels, and, as applicable, to assess the agency’s progress in meeting its performance targets in support of agency strategic goals. According to EPA officials, generally grantees or EPA program officials—depending on the database—are to enter grant results, such as the number of enforcement actions, into EPA’s program-specific data systems at agreed-upon intervals, such as quarterly. These requirements may be part of a grant’s terms and conditions.

Disconnected Youth Pilots Rely on Reports and Evaluations to Ensure Accountability

For disconnected youth pilots, non-federal partners are to submit quarterly reports on progress made toward the goals and measures established in the pilots’ performance partnership agreements to Education in its role as the lead agency. This performance reporting covers the entire pilot and the programs included in it, regardless if the funds involved are combined or coordinated, according to Education officials. Education will then share the reports with the relevant consulting and participating agencies for their review. If the quarterly report shows that the pilot is facing challenges in making progress towards its goals, federal officials we met with told us that the agencies can work with the non-federal partners to address the challenges, which could include amending waivers, providing technical assistance, or requiring the pilot to develop a corrective action plan.

Federal and non-federal partners in the disconnected youth pilots are also taking steps to conduct evaluations of pilot outcomes. DOL has responsibility for leading a national evaluation of the pilots and contracted with Mathematica Policy Research (Mathematica) to conduct it. Officials from DOL involved in the national evaluation told us that they have sought input, and involved individuals, from the other federal agencies in the ongoing design of the national evaluation, and that they will also have a chance to review the final evaluation plan. Officials from DOL and Mathematica told us that the national evaluation will focus on 4 major areas: technical assistance, implementation, outcomes, and impacts. According to officials from DOL and Mathematica, they have made, and will continue to make, adjustments to the design of the national evaluation to better align with the data collection efforts for the individual pilot evaluations and avoid duplicating efforts.

In addition, the federal agencies established a competitive preference priority for first round pilot applicants whereby the agencies awarded extra points to applicants who proposed conducting a site-specific evaluation using a randomized control trial or quasi-experimental approach of at least one of the pilot’s components. Each of the pilots for the first round is conducting a site-specific program evaluation, according to DOL officials. The Ysleta del Sur Pueblo pilot, for instance, is using an impact evaluation to test the effectiveness of its Tigua Leadership Curriculum on 1 of the 2 cohorts of youth it plans to serve in the pilot. The evaluation includes treatment and comparison groups, though the pilot will provide the Tigua Leadership Curriculum to both groups. It will provide the services to the comparison group after the final data collection. Ysleta del Sur Pueblo will collect data on both groups, which will be evaluated to determine whether the new services had a positive effect on improving youth attitudes towards staying in school, completing high school, and understanding the connection between education and career development opportunities. This evaluation will be conducted in addition to the pilot’s collection and reporting of performance against specific performance measures for participants’ educational and employment outcomes.

Federal and non-federal officials with whom we spoke identified key benefits of the national-level and site-specific evaluations. Education officials said that both the site-specific and the national-level evaluations will help federal agencies to determine if there is a need for broader legislative authority to allow more grant recipients in different locations to propose waivers similar to those received by the first round pilots. Officials from the Chicago pilot lead told us that the evaluations will help both the federal and non-federal partners learn more about what works within their program, which could help them improve it in the future. However, officials from the Oklahoma pilot lead told us that while evaluations are important for tracking if pilots are meeting their intended results, they were unsure whether the site-specific and national-level evaluations would be able to determine what, if any, impact the ability to consolidate funds and use waivers would have on improved outcomes for disconnected youth. They said that the pilot’s 3-year timeframe may be too short to allow federal or non-federal partners to clearly determine whether the pilot has improved outcomes for disconnected youth or created meaningful systems change. For example, they said that the partners will likely be unable to determine whether the pilot has improved youth involvement in post-secondary education or employment outcomes before the pilot’s timeframe, and the funding to support the evaluations, expires.

In Designing, Implementing, and Evaluating the Disconnected Youth Pilots, Federal Agencies Have Taken Steps Intended to Ensure Success, but Have Opportunities for Improvement

Federal Agencies Generally Took Actions Consistent with Leading Practices for Interagency Collaboration

The federal agencies involved in the disconnected youth pilots have taken a number of actions consistent with leading practices for interagency collaboration identified in our prior reports. In September 2012, we reported that federal agencies have used a variety of mechanisms to implement interagency collaborative efforts, which can be used to address a range of purposes including policy development; program implementation; oversight and monitoring; information sharing and communication; and building organizational capacity, such as staffing and training. We noted that although collaborative mechanisms differ in complexity and scope, they all benefit from certain leading practices, which raise issues to consider when implementing these mechanisms. Table 1 provides examples of how actions taken by federal agencies in designing and implementing the disconnected youth pilots generally were consistent with the 7 leading practices identified in our September 2012 report.

Opportunities Exist to Improve Planning for Financial and Staff Resource Contributions

Although federal agencies generally have taken actions consistent with leading practices for interagency collaboration, we identified additional actions they could take in relation to several leading practices to better support the success of the individual disconnected youth pilots as well as the overall initiative. In response to our findings, the agencies already have taken steps to address issues we identified in the areas of written guidance and agreements and participants during the course of our audit. However, their planning for and management of financial and staff resources are not yet in line with leading practices.

Written Guidance and Agreements. We previously found that written agreements for interagency groups are most effective when updated and monitored regularly to reflect the roles and responsibilities of current participants. Such agreements can help strengthen agency commitment to working collaboratively and provide a clear delineation of activities to be undertaken by individual agencies. Additionally, updated written agreements can serve as a source of current information in the case of staff transitions. In assessing the federal agencies’ efforts related to written guidance and agreements, we found that the 2015 interagency agreement covered roles and responsibilities for the federal agencies for the first 2 pilot rounds, but as of July 2016, it had not been revised to reflect the third pilot round or the authorization for HUD to participate in the initiative. In discussing these findings with Education officials in July 2016, they stated several reasons why this had not happened. First, they did not expect established roles and responsibilities to change in future pilot rounds. Second, they stated that federal agencies had been focused on implementing the first round pilots and preparing for the second and third round pilots. Finally, they told us that the process for updating the agreement was cumbersome, as it required the approval of the heads of all agencies involved in the initiative. Subsequently, in December 2016, Education officials told us that the agencies, in response to the issues we identified, were taking steps to modify the interagency agreement to cover the third pilot round and include HUD’s roles and responsibilities. Moreover, they agreed to streamline the process for making future changes to the agreement. In January 2017, Education shared a draft update to the interagency agreement reflecting these changes, which are pending final approval by the relevant agencies.

Participants. We previously found that it is important to ensure that all relevant participants have been included in a collaborative effort. For all agencies authorized to participate in a particular initiative, such as the disconnected youth pilots, their collective involvement helps ensure that someone can commit resources and make decisions on behalf of their agency, and contribute to the outcomes of the collaborative effort through their individual knowledge, skills, and abilities. We identified a few instances in which officials from federal agencies with program funds being used in individual pilots were not notified of their funds’ planned use. For example, the non-federal partners in both the Oklahoma and Seattle pilots plan to use AmeriCorps funding. However, in a June 2016 meeting with CNCS officials about their involvement in the pilots, the agency’s key point of contact for the disconnected youth pilots told us that she was not aware of those pilots’ planned use of the funding. When we raised this issue in a subsequent meeting with Education officials in July 2016, they told us that in these instances they did not notify CNCS because their process only involved notifying an agency in cases in which the non-federal partners proposed combining an agency’s funds or requested a waiver of requirements related to that agency’s programs in a pilot.

The non-federal partners in the Seattle and Oklahoma pilots did not propose combining CNCS funds or request waivers of AmeriCorps program requirements. Therefore, according to Education officials, the established process did not require them to notify CNCS about the use of AmeriCorps funds in those pilots because CNCS did not need to approve anything. Because the funds are being coordinated, the recipients are still required to adhere to the requirements of the program for which the funds were appropriated, including reporting on the funds in accordance with the program’s underlying requirements. As such, CNCS officials told us that the use of AmeriCorps program funds in those 2 pilots would be covered by the program’s usual grant oversight processes. In response to our observations and to address any potential future issues about involving relevant agency officials in individual pilots, Education officials told us in December 2016 that they have revised their processes. Moving forward, they will notify relevant agency officials of instances in which their programs and funds are proposed for inclusion in a pilot, regardless of whether funds are combined or coordinated or waivers are sought.

Resources. We previously found that collaborating agencies should identify the various resources, including financial and staff resources, needed to initiate and sustain their collaborative effort. Collaboration can take time and resources in order to accomplish activities such as building trust among the participants, setting up the ground rules for the process, attending meetings, conducting project work, and monitoring and evaluating the results of work performed. Moreover, relying on agencies to participate can present challenges for collaborative mechanisms. Our past work has also found that, in cases where staff participation was insufficient, collaboration often failed to meet key objectives and achieve intended outcomes. Consequently, it is important for groups to ensure that they identify and leverage sufficient funding and staffing to accomplish the objectives.

Agencies do not have a full understanding of their future resource needs—in terms of individual agency funds and staffing contributions—to maintain the pilot initiative through September 2020, when the third round is currently scheduled to end. Federal and non-federal partners in the disconnected youth pilots told us that these resources are important for its success. Agency officials told us that, because much of their early focus was on designing and implementing the initiative in short time frames, decisions related to resource contributions generally were made as needed to ensure near-term progress. Now that agencies have designated finalists for the second and third rounds, they can better identify and plan for the resources they will need to contribute to support the overall initiative through its lifetime (September 2020). However, the agencies—including OMB which has responsibility for coordinating agencies’ overall efforts to implement the disconnected youth pilots— have not yet fully identified how those resources will be provided. Key aspects of the pilot initiative rely on funding and staff contributed by individual agencies. As such, it is important for each agency to understand what resources it is expected to provide so that it can plan accordingly. By fully identifying and planning for the specific financial and staff resource contributions described below, the agencies will have greater assurance that those contributions will sustain success in their collaborative efforts and the overall pilot initiative.

Financial Resources. As was previously mentioned, the funding for federal grants included in individual pilots is provided through appropriations. According to agency officials involved in the pilots, they identified additional activities that they considered key to the success of the initiative. As highlighted in Table 1, these include the start-up grants, general technical assistance, and the national and site-specific evaluations. Education and DOL leveraged various mechanisms, including contracts and interagency transfers, and other agencies contributed funds, to the extent possible, to support these activities. According to agency officials, their agencies consider potential contributions on a year-to-year basis, depending on the available resources in eligible programs. They told us that their agencies have needed to make trade-off decisions among competing priorities to contribute funds to these activities each year.

As a result, funding for some of these activities has decreased over time. For example, although pilots could receive up to $700,000 in start-up grants for the first round, they may receive up to $350,000 for the second round and up to $250,000 for the third round due to decreased contributions of funding for such purposes from among the federal agencies. As was noted earlier, the offer of start-up grants was a key incentive for non-federal partners from the 4 selected pilots as they considered participating in the initiative. Agencies already have made their contributions for the start-up grants for the 3 rounds currently authorized, so this may not be an issue moving forward unless further rounds are authorized. However, the contract for general technical assistance has the potential, through annual extensions which require additional funding, to run through 2020, when the third round pilots are due to end. Moreover, although the contract for the national evaluation and related evaluation technical assistance runs through 2020, work related to completing the evaluation—which is necessitated by the legislation authorizing the pilots—will need to continue after this date if the national evaluation is expanded to include the pilots from the second and third rounds. To do so, DOL officials told us that they may be able to extend the prior performance to continue the work under the existing contract or otherwise award a new contract following federal acquisition procedures. Regardless of what approach is taken, continued performance would require additional funding.

Staff Resources. Incomplete information about existing staff investments in the disconnected youth pilots to date, and uncertainty in the composition of future pilot rounds has limited agencies’ abilities to identify long-term staffing needs. Most agencies are not tracking their current staff support to the first round pilots, which could help them understand potential staff investments for future rounds. When we asked agency officials for estimates of staff time spent supporting the pilot initiative, they could not provide overall estimates for their agencies. Similarly, when we asked the key points of contact at each agency—those generally most involved in the disconnected youth pilots—about their own time spent on the initiative, they could only provide rough estimates. Officials explained that several factors made it difficult to track staff investment. The number of staff involved and their related time demands varied depending on the stage of the process and the activities that were being undertaken. Moreover, at different points in the past, the agencies were designing and implementing multiple rounds concurrently. In addition, in interviews with officials from each of the agencies involved in the pilots during summer 2016, officials told us that their agencies face difficulties anticipating future staffing needs since the exact number and structure of the second and third round pilots were unknown at the time. Officials said their agencies could not sufficiently plan for staff needs beyond what had been established in the first round of pilots without knowing the number of pilots that would be selected for each round, and which federal agencies and programs would be involved.

However, the selection processes for both the second and third rounds are now complete. As was mentioned previously, the agencies designated 1 applicant as a second round pilot finalist in September 2016, and 6 applicants as third round pilot finalists in January 2017. Therefore, agencies should have a better sense of their involvement in all 3 rounds. Moreover, because all 3 rounds will soon be in the implementation phase, it should be easier for agencies to track current staff investments to better project needed staffing contributions moving forward.

Federal Agencies Generally Took Actions Consistent with Leading Practices for Pilot Design, but Could Improve Plans for Assessing Scalability of the Disconnected Youth Pilots

Federal agency efforts for the disconnected youth pilots were also generally consistent with practices for effective pilot design. A well- developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations in addition to helping to ensure that time and resources are used effectively. In April 2016, we identified 5 leading practices that, taken together, form a framework for effective pilot design. By following these leading practices, agencies can promote a consistent and effective pilot design process. Examples of actions federal agencies took in line with these practices are illustrated in table 2.

Although federal agencies generally took actions consistent with leading practices for effective pilot design, we identified an additional action they could take in relation to assessing scalability. We previously found that, as part of their design, agencies should have criteria or standards for identifying lessons about the pilot to inform decisions about scalability and whether, how, and when to integrate pilot activities into overall efforts. To assess scalability, criteria should relate to the similarity or comparability of the pilot to the range of circumstances and population expected in full implementation. The criteria or standards can be based on lessons from past experiences or other related efforts known to influence implementation and performance as well as on literature reviews and stakeholder input, among other sources. The criteria and standards should be observable and measureable events, actions, or characteristics that provide evidence that the pilot objectives have been met. Choosing well-regarded criteria against which to make comparisons can lead to strong, defensible conclusions.

Although the federal agencies have identified a variety of data to collect through performance reporting and the national and pilot-level program evaluations, they did not identify criteria or standards for assessing scalability of the flexibilities being tested by the pilots as part of the pilot or evaluation design processes. The agencies conducted outreach to stakeholders to learn about the needs of organizations assisting disconnected youth, established key goals and objectives for the organizations implementing the pilots and included them in partnership agreements, which also detail data collection methodology, reporting requirements, and interim measures. They also developed evaluation plan reporting templates for the pilots. However, according to DOL officials, federal agencies decided not to require any new data collection by the pilots for several reasons, including the potential costs to pilots.

Therefore, the pilots are only reporting performance data that they would routinely collect from their program participants, with the possibility of reporting any new data they may be collecting for their local evaluations. DOL officials also told us that they and their contractor for the national evaluation are not yet thinking about the potential scalability of the various flexibilities being tested by the pilots because it is too early in the implementation period—in the first year of the 5-year pilots. They said that the national evaluation is intended to review the flexibilities that each pilot uses and determine what recommendations can be made to decision makers, such as Congress, for broader application. Officials then plan to combine the information on flexibilities with final outcome data and evidence from the site-specific impact evaluations to assess and synthesize, as appropriate, the individual pilots. DOL officials also said that they plan to examine the partnership structures for the pilots to determine the effectiveness of the structures. While the plans DOL officials have laid out could, and likely will, provide useful information and insights into the success of the individual pilots and overall initiative, they may not be collecting the data needed to inform conclusions about scaling the flexibilities tested by the pilots without first determining the criteria or standards for such an assessment. Going forward, such data would better position Congress to decide whether and to what extent the flexibilities tested by the pilots should be integrated into broader efforts.

Conclusions

Performance partnerships are part of a broader federal effort to align federal grantmaking priorities with state and local government needs in addressing key national objectives. One of the 2 existing partnership initiatives—the multi-agency disconnected youth pilots—allows partners to collaborate across organizational lines primarily to leverage programmatic flexibilities that enable them to combine funding from across several federal grant programs (and agencies) for interventions aimed at improving outcomes among disconnected youth. In establishing the disconnected youth pilots, federal agencies generally took actions consistent with leading practices for interagency collaboration and pilot design. For example, agencies established and documented key roles and responsibilities, including those related to leadership, in an interagency agreement. In addition, they established goals and objectives for the individual pilots and overall initiative, and have plans to monitor performance and evaluate results. However, agencies—including OMB, which has responsibility for coordinating agencies’ overall efforts to implement the disconnected youth pilots—have not fully identified the key financial and staff resources each agency will need to contribute over the lifetime of the initiative. Doing so would help ensure they are able to provide the support needed for successful implementation of the pilots and to sustain their collaborative efforts. In addition, the agencies have not identified criteria or standards to assess the scalability of the flexibilities being tested by the pilots. Without them, agencies may not collect the data they—and ultimately Congress—will need to determine whether and how to implement successful approaches more broadly.

Recommendations for Executive Action

To help ensure that the pilot programs for disconnected youth can be effectively implemented over the lifetime of the initiative, the Director of OMB should coordinate with relevant federal agencies to identify and estimate expected annual financial and staff resource contributions from each agency, including during the implementation and evaluation phases of the pilots.

To ensure that federal agencies involved in the disconnected youth pilots are able to evaluate pilot outcomes and ultimately communicate to Congress whether and to what extent the flexibilities tested by the pilots should be integrated into broader efforts, the Director of OMB should coordinate with relevant federal agencies to identify criteria or standards for assessing scalability, and collect data needed to address those criteria or standards.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of the Office of Management and Budget, Secretary of the Department of Education, Secretary of the Department of Health and Human Services, Acting Secretary of the Department of Housing and Urban Development, Attorney General of the Department of Justice, Acting Secretary of the Department of Labor, Chief Operating Officer of the Corporation for National and Community Service, Administrator of the Environmental Protection Agency, and Acting Director of the Institute of Museum and Library Services for review and comment.

In comments provided by email, OMB’s Liaison to GAO stated that OMB neither agreed nor disagreed with the recommendations in this report. OMB staff also provided oral comments in which they asked us to clarify that (1) OMB’s role is to coordinate agencies’ overall efforts to implement the disconnected youth pilots, (2) the resource issues identified in our report involve agencies better identifying and planning for their individual contributions to the pilot initiative, and (3) our discussion of scalability is focused on the flexibilities being tested by the pilots. We revised the report accordingly to provide these clarifications.

DOL, Education, EPA, HHS, and IMLS provided technical comments, which we incorporated as appropriate. CNCS, DOJ, and HUD informed us that they had no comments.

We are sending copies of this report to the appropriate congressional committees, the heads of each of the federal agencies included in this review, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or by email at bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The GPRA Modernization Act of 2010 (GPRAMA) put into place a framework intended to increase the use of performance information and other evidence in federal decision making. GPRAMA also requires us to periodically report on how its implementation is affecting performance management at federal agencies, including whether agencies are using performance management to improve the efficiency and effectiveness of their programs. According to the Office of Management and Budget (OMB), because performance partnerships require federal agencies and their grant recipients to manage toward agreed upon outcomes, they can help the 2 sides to collect information and evidence about what works and therefore how to employ federal resources more efficiently. We therefore conducted this review under our GPRAMA reporting requirement. There are currently 2 sets of federal performance partnerships that have been authorized. The Environmental Protection Agency’s (EPA) Performance Partnership Grants (PPG) under its National Environmental Performance Partnership System have been in place for 20 years. The multiple-agency Performance Partnership Pilots for Disconnected Youth (disconnected youth pilots) were authorized in January 2014. Over the past 3 years, 8 federal agencies—OMB; the Departments of Education (Education), Health and Human Services (HHS), Housing and Urban Development (HUD), Justice (DOJ), and Labor (DOL); the Corporation for National and Community Service (CNCS); and the Institute for Museum and Library Services (IMLS)—have worked to implement the pilots. This report identifies the key characteristics of those 2 existing performance partnership initiatives. It also provides a more in-depth review of the design, implementation, and evaluation of 1 of the 2—the Performance Partnership Pilots for Disconnected Youth.

To address these objectives, we identified current federal performance partnerships by reviewing relevant literature, including of our past work on grants management and material produced by public sector and nonprofit organizations; searching public laws for references to “performance partnerships”; and interviewing former federal officials knowledgeable about the federal government’s efforts to develop and implement performance partnerships. Based on this work, we determined that EPA’s PPGs and the disconnected youth pilots were the only 2 federal performance partnerships currently authorized.

To identify key characteristics of performance partnership initiatives and how these key characteristics are exhibited, we collected, reviewed, and analyzed documents about the overall performance partnership initiatives, such as authorizing legislation, regulations, and notices inviting applications, as well as from selected individual performance partnerships within them, including applications, performance partnership agreements, and grant work plans.

To further illustrate how these key characteristics are exhibited by the 2 performance partnerships, we selected a non-generalizable sample of 4 states with EPA PPGs and 4 disconnected youth pilots for in-depth review. To select a sample of states with PPGs, we used data that EPA provided to us on PPG use in fiscal year 2014. We selected states based on the number of grants they included in their PPG, the number of years they had a PPG, and the EPA region in which they are located. We chose 1 state with a high total number of years in a PPG and a high total number of grants in a PPG; a second with a high total number of years in a PPG and a low total number of grants in a PPG; a third with a low total number of years in a PPG and a high total number of grants in a PPG; and a fourth with a low total number of years in a PPG with a low total number of grants in a PPG. We defined high and low as being within the top or bottom quartile of states in each category. We ensured that each of our selections was from a different EPA region since variations in management practices and general environmental needs across EPA regions may impact how states use PPGs. The states we selected based on these criteria were Utah, New York, Alabama, and California.

To select a sample of disconnected youth pilots, we considered pilots with various locations (urban, rural, tribal), consulting agencies, and federal programs used. First, we grouped pilots by their urban, rural, and tribal designations to ensure that we could examine pilots in different environments. We selected the only 2 pilots designated as rural and tribal—Eastern Kentucky and Ysleta del Sur Pueblo. Next, we grouped pilots by the consulting agencies responsible for overseeing them to ensure that we could examine whether pilot oversight differs among federal agencies. Four agencies serve as consulting agencies— Education, DOL, HHS, and CNCS. Education and CNCS serve as the consulting agencies for Eastern Kentucky and Ysleta del Sur Pueblo, respectively, so we then selected Chicago from the group of urban pilots left for selection since it is the only pilot for which HHS serves as the consulting agency. Finally, we looked to ensure that each federal agency’s programs were being used in at least 1 of our selected pilots. Since all of the agencies had at least 1 program being used in our first 3 selections, we selected Oklahoma as our final pilot, as DOL is its consulting federal agency and it is using programs from 4 of the 5 agencies.

To obtain perspectives on the key characteristics of these performance partnership initiatives, including reporting benefits and challenges they may present, we conducted semi-structured interviews with officials involved in them. For PPGs, we met with relevant officials at EPA headquarters and regional offices, as well as officials from state environmental agencies in the 4 selected states. These included officials from EPA’s Office of Congressional and Intergovernmental Affairs; EPA Regions 2, 4, 8, and 9; and the Alabama Department of Environmental Management, the California Department of Pesticide Regulation, the New York State Department of Environmental Conservation, and the Utah Department of Environmental Quality. For the disconnected youth pilots, we conducted semi-structured interviews with officials from each of the federal agencies currently involved in them—OMB, Education, HHS, HUD, DOJ, DOL, CNCS, and IMLS. We also conducted semi-structured interviews with representatives from a federal and non-federal partner for each of the individual partnerships. For the disconnected youth pilots, these included Education and Partners for Education at Berea College (Eastern Kentucky); CNCS and the Ysleta del Sur Pueblo tribal government (Ysleta del Sur Pueblo); HHS and the Chicago Department of Family Support Services (Chicago); and DOL and the Oklahoma Department of Human Services (Oklahoma).

To assess federal agencies’ efforts to design, implement, and evaluate the disconnected youth pilots, we obtained and reviewed key documents, including requests for information and public comment, preliminary design papers, interagency agreements, and evaluation plans. In addition, we interviewed officials from OMB, Education, HHS, HUD, DOJ, DOL, CNCS, and IMLS about their collaboration and design of the pilot initiative. We then assessed agencies’ efforts in these areas against leading practices to determine the extent to which these efforts reflect leading practices for interagency collaboration and effective pilot design. These leading practices were developed in our prior work.

We conducted this performance audit from July 2015 to April 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Additional Information on the Environmental Protection Agency’s Performance Partnership Grants

The Environmental Protection Agency’s (EPA) performance partnership grants (PPG) permit state agencies to request that funding they receive from 2 or more EPA program grants be combined into a single award. This is intended to enable state agencies to, among other things, consider trade-offs across the breadth of their environmental program funding and exercise flexibility to direct resources to their most pressing priorities. Specifically, once a state agency has requested and received selected EPA program grants in a PPG, it can choose to use the funds to support any activity that is eligible under at least 1 of the grants included in the PPG. PPGs also streamline administrative requirements so that states agencies can realize cost savings through reduced administrative burden in areas such as grant applications, cost sharing, and financial reporting.

The number of grants that states request to include in PPGs varies. For example, in fiscal year 2016, the 39 states that had PPGs managed by their environmental or health agency included between 2 to 16 grants in their PPGs, according to EPA data. Approximately half of these agencies included 6 or fewer grants in their PPGs, with the other half including 6 or more.

To illustrate variation in how state agencies chose to receive combined EPA program grant funding in PPGs, we selected to review 4 in greater depth that combined either a small or a large number of grants in their PPGs. On the small end, we selected New York, whose Department of Environmental Conservation received funding from 3 grants (illustrated in figure 9), and California, whose Department of Pesticide Regulation received funding from 4 grants in 1 environmental area (water and pesticides, respectively, illustrated in figure 10) in their PPGs in fiscal year 2016.

On the large end, we selected Alabama, whose Department of Environmental Management received funding from 9 grants (illustrated in figure 11 below), and Utah, whose Department of Environmental Quality received funding from 10 grants across multiple environmental program areas in their PPGs in fiscal year 2016 (previously illustrated in figure 6).

Appendix III: Additional Information on the Multi-Agency Performance Partnership Pilots for Disconnected Youth

The multi-agency disconnected youth pilots enable non-federal partners to combine funds from federal agencies’ programs and obtain additional flexibilities to tailor service interventions to the specific needs of their target populations. Three rounds of pilots have been authorized, and the first round of 9 pilots approved is underway. Specifically, the authorization for the first round pilots allows for the combining of discretionary funding received by the Departments of Education (Education), Health and Human Services (HHS), and Labor (DOL), as well as the Corporation for National and Community Service (CNCS) and the Institute of Museum and Library Services (IMLS) through the fiscal year 2014 appropriations act. In the notice inviting applications for first round pilots, applicants were advised that where a program’s funds are not suitable for combining (referred to as “blending”) in a pilot given the constraints under the pilot authorization, the applicant may nevertheless consider how to coordinate (referred to as “braiding”) such funding in a pilot to promote more effective and efficient outcomes even though the funds would maintain a separate identity and remain subject to the program requirements for which the funds were appropriated.

In addition, for the first pilot round federal agencies awarded separate start-up grants to provide funding of up to $700,000 to each pilot to finance evaluations, capacity building, technical assistance and other related activities to support the pilot. According to officials from OMB, funds available for these purposes from CNCS, DOL and Education, were used for these start-up grants.

The first round pilots vary in the extent to which they are using combined federal funds in their pilots. Our selection of case study pilots also reflects this variation, as noted below. Two of the 9 pilots are combining all federal funds from all of the federal programs they are including in their pilots. For example, the Ysleta del Sur Pueblo pilot is consolidating funding from CNCS’s AmeriCorps program and IMLS’s Native American Library Services Enhancement Grants program, as illustrated in figure 12.

According to Ysleta del Sur Pueblo’s pilot application, while the tribe has had success in using the CNCS and IMLS programs individually to meet short-term outcomes for tribal youth, such as cultural and traditional engagement and enhanced knowledge of the services and programs the tribe offers, it has been unable to use the programs to provide a continuum of services to help tribal youth meet longer-term goals, such as staying in school, enrolling in post-secondary education, and obtaining gainful employment. The application states that the tribe’s ability to consolidate the 2 programs’ funding will allow its Empowerment and Economic Development Departments to better collaborate and more effectively provide youth services in a single, wraparound program. In addition, officials from Ysleta del Sur Pueblo told us that the start-up grant the tribe received will further enhance its ability to implement its pilot. The tribe’s application stated that, among other things, the funds will help the tribe to hire staff and support the evaluation.

Five pilots are not using any combined funding but are instead coordinating the use of federal funds from multiple federal programs. These include the Oklahoma and Chicago pilots we selected to review in- depth. For example, the Oklahoma pilot is coordinating funds from 3 federal programs: DOL’s WIOA Title I Youth program, HHS’s Now Is the Time: Healthy Transitions program, and CNCS’s AmeriCorps program. Figure 13 illustrates the structure of the Oklahoma pilot.

Officials from Oklahoma DHS told us that the non-federal partners involved in the pilot expressed a preference to manage and report on the use of their federal funds independently. They told us that the organizations determined that they would be able to operate the pilot as intended by coordinating their use of funds. They had no prior experience combining funds across organizations, and Oklahoma DHS, as the pilot lead, did not want to track other organizations’ funds. Oklahoma DHS officials also told us that start-up grants have been beneficial in allowing the non-federal partners to hire a staff member to oversee pilot operations, support collaboration among partners, and prepare for and conduct an evaluation of the pilot.

The structure of the Chicago pilot, illustrated in figure 8, along with a discussion of challenges federal and non-federal officials faced in coming to agreement on the use of funds for that pilot, were previously described above.

The remaining 2 pilots are combining federal funds from some of the federal programs included in their pilots and coordinating the use of funds from other federal programs. One of them is the Eastern Kentucky pilot, and details about its use of federal funds were discussed earlier in the report and illustrated in figure 7. Officials from a pilot partner told us that combining some funds allows them to hire staff who are not tied to a specific program and therefore can work across the programs involved in the pilot. In addition, the officials added that the start-up grant the pilot received will allow the non-federal partners to establish a data collection system to share information about the participants across partners. This would allow them to use those data to identify effective strategies and support the evaluation of the pilot.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director) and Daniel Webb (Analyst-in-Charge) supervised this review and the development of the resulting report. Mitchell Cole, Karin Fangman, Mike Grogan, John Hussey, Sherrice Kerns, Ifunanya Nwokedi, Michelle Sager, Cindy Saunders, and Stephanie Shipman made significant contributions to this report. Donna Miller developed the graphics for this report. Theodore Alexander, Crystal Bernard, Kathleen Drennan, Shirley Hwang, Adam Miles, Keith O’Brien, Laurel Plume, Erik Shive, and Wes Sholtes verified the information in this report.