Performance Partnerships: Agencies Need to Better Identify Resource Contributions to Sustain Disconnected Youth Pilot Programs and Data to Assess Pilot Results
Published: 2017-04-18T00:00:00

Fast Facts:
Performance partnership initiatives allow federal agencies to give their grant recipients flexibility (such as in how they use funding) in exchange for assessing if outcomes improve. There are currently two such initiatives—one involves EPA's environmental grant programs and the other helps youth who are disconnected from school and work.
We found that both initiatives reduce administrative burdens and allow for flexibility in using grant funding. However, we recommended that the disconnected youth initiative better identify (1) resources that agencies will need to contribute to sustain the effort, and (2) data to assess pilot results.

Highlights:
section_title: Why GAO Did This Study
paragraphs: The GPRA Modernization Act of 2010 established a framework intended to increase federal agencies' use of performance information and evidence in decision making. In performance partnerships, agencies and grant recipients manage toward outcomes, which can help measure program performance and collect evidence about what works to achieve desired outcomes. OMB has encouraged the use of such partnerships by agencies that make federal grants.
GAO is required by the act to report on how its implementation is affecting federal agency performance management. This report identifies the key characteristics of 2 existing performance partnerships. It also provides an in-depth review of the design, implementation, and evaluation of 1 of the 2 initiatives—the disconnected youth pilots.
To address these objectives, GAO reviewed relevant laws, regulations, and documents and selected 8 illustrative examples from the 2 partnership initiatives (4 each), based on various criteria, such as the type and number of grants included and location. GAO also interviewed federal and non-federal officials involved in these partnerships.
section_title: What GAO Found
paragraphs: Congress has authorized 2 federal performance partnership initiatives. The Environmental Protection Agency's (EPA) Performance Partnership Grants (PPG) has been in place for 20 years and allows state agencies to consolidate funds from up to 19 environmental program grants into a single PPG. The other, Performance Partnership Pilots for Disconnected Youth (disconnected youth pilots), is a more recent initiative authorized in 2014 that allows funding from multiple programs across multiple agencies to be combined into pilot programs serving disconnected youth. GAO identified 4 key characteristics shared by the 2 federal performance partnership initiatives. Specifically:
1. Documented agreement . Federal and non-federal partners identify goals, roles, and responsibilities. EPA and state agencies accomplish this through a PPG work plan. For each disconnected youth pilot, multiple federal agencies and non-federal partners, such as local government agencies and community-based organizations, use a performance partnership agreement.
2. Flexibility in using funding. PPGs combine funding from 2 or more EPA program grants. The disconnected youth pilots can combine funding from multiple programs across the agencies involved in the initiative.
3. Additional flexibilities. PPGs reduce administrative burden for state agencies, for example, by requiring only a single application for all grants in them. Disconnected youth pilots also provide non-federal partners flexibility to serve disconnected youth, including the ability to better tailor service interventions to their target populations.
4. Accountability for results . In both initiatives, non-federal partners report to federal partners on progress towards mutually-established goals. Partners in the disconnected youth pilots are also assessing results through national and pilot-specific program evaluations.
GAO's in-depth review of the disconnected youth pilots found that agencies had taken actions consistent with leading practices for collaboration and pilot design, such as establishing a leadership model for collaboration. Although the Office of Management and Budget (OMB) is responsible for coordinating agencies' overall efforts to implement the pilots, GAO identified additional actions that OMB should take in coordination with the agencies to help ensure future success.
Resources. Agencies have not fully identified the funding and staff resources each will need to contribute to sustain their efforts over the lifetime of the pilots. This is because agencies primarily have been focused on meeting near-term needs to support design and implementation. By fully identifying specific future financial and staff resource needs, agencies can better plan for their individual contributions to ensure they are sufficient to support the pilots.
Scalability. Agencies have not developed criteria to inform determinations about whether, how, and when to implement the flexibilities tested by the pilots in a broader context (this is known as scalability). Although the agencies identified a variety of data to collect, they have not identified criteria for assessing scalability. Officials involved in the pilots told GAO it was too early in pilot implementation to determine such criteria. By not identifying these criteria during the design of the pilots, they risk not collecting needed data during their implementation.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB coordinate with federal agencies implementing the disconnected youth pilots to identify (1) agency resource contributions needed for the lifetime of the pilots and (2) criteria and related data for assessing scalability. OMB neither agreed nor disagreed with these recommendations.

section_title: Letter
paragraphs: The federal government spends more than $600 billion a year on federal  grants to state and local governments, nonprofits, and educational  institutions to fund a wide range of programs and services to address  critical national objectives, such as providing education and job training to  vulnerable populations and addressing environmental challenges. By  leveraging the funds and expertise of the various organizations involved  in administering grants, services may be delivered efficiently. However,  while the federal government and state or local grant recipients often  share common purposes, they can have different priorities. Although both  are interested in efficiency and effectiveness, the federal government  ensures accountability for the expenditure of grant funds through common  reporting and audit requirements, while grant recipients often seek  flexibility  to be responsive to state or local priorities and minimize grant- related administrative burden.
Some federal agencies and state and local governments have attempted  to balance federal interests with recipient priorities in grants  administration by using a mechanism known as a performance  partnership. Performance partnership initiatives, which are authorized by  law, allow a federal agency or multiple federal agencies to provide grant  recipients flexibility in how they use funding across 2 or more programs  along with additional flexibilities in exchange for the recipient’s  commitment to improve agreed-upon outcomes and assess progress  towards them. Within these initiatives, federal agencies can enter into  multiple individual performance partnerships tailored to the recipient. In  recent years, the Office of Management and Budget (OMB) has  encouraged agencies to explore performance partnerships and other  mechanisms, including tiered-evidence grants and pay for success, as  means by which to make federal programs more evidence-focused and  test innovations in service delivery and different approaches to improving  program outcomes.
Two federal performance partnership initiatives have been authorized by  Congress. The Environmental Protection Agency’s (EPA) Performance  Partnership Grants (PPG), under its National Environmental Performance  Partnership System (NEPPS), have been in place for 20 years. The  other and more recent initiative  is the multiple-agency Performance  Partnership Pilots for Disconnected Youth (disconnected youth pilots),  which were first authorized in January 2014. Over the past 3 years, 8  federal agencies—OMB; the Departments of Education (Education),  Health and Human Services (HHS), Housing and Urban Development  (HUD), Justice (DOJ), and Labor (DOL); the Corporation for National and  Community Service (CNCS); and the Institute for Museum and Library  Services (IMLS)—have  worked to design and implement the pilots.
The GPRA Modernization Act of 2010 (GPRAMA) put into place a  framework intended to increase the use of performance information and  other evidence in federal decision making. According to OMB, because  performance partnerships require federal agencies and grant recipients to  manage toward outcomes, these initiatives can help the 2 sides measure program performance and collect information about what works. This  focus on performance management and evidence-based decision making  can help agencies and recipients determine how federal resources can be  used more efficiently and effectively. GPRAMA also requires us to  periodically report on how its implementation is affecting performance  management at federal agencies, including whether agencies are using  performance management to improve the efficiency and effectiveness of  their programs. This report is part of our response to that requirement.  This report identifies the key characteristics of the 2 existing performance  partnership initiatives. It also provides a more in-depth review of the  design, implementation, and evaluation of 1 of the 2 initiatives—the  Performance Partnership Pilots for Disconnected Youth.
To identify the key characteristics of existing performance partnership  initiatives, we collected, reviewed, and analyzed documents about the  performance partnership initiatives, such as authorizing legislation,  regulations, and notices inviting applications, as well as from selected  individual performance partnerships within them, including applications,  performance partnership agreements, and grant work plans. To further  illustrate these key characteristics, we selected for in-depth review a  sample of states with PPGs and a sample of disconnected youth pilots.  We selected 4 states with PPGs—Alabama, California, New York, and  Utah—based on various criteria, including the number of grants they have  included in their PPGs, the length of time they have used PPGs, and the  EPA region in which they are located. We selected 4 disconnected youth  pilots—Chicago, Eastern Kentucky, Oklahoma, and the Ysleta del Sur  Pueblo Indian tribe in El Paso, Texas—based on the location of the pilot  (urban, rural, tribal) and the mix of federal programs being used in the  pilot, among other criteria.
To obtain perspectives on the key characteristics of these performance  partnership initiatives, including reported benefits and challenges they  may present, we interviewed officials involved in the initiatives. For PPGs,  we met with relevant headquarters and regional office officials at EPA,  and officials from state environmental agencies in the 4 selected states.  For the disconnected youth pilots, we interviewed officials from each of  the federal agencies currently involved in them: OMB, Education, HHS,  HUD, DOJ, DOL, CNCS, and IMLS. We also interviewed representatives  from at least 1 of the non-federal partners in the 4 selected pilots.
To assess federal agencies’ efforts to design, implement, and evaluate  the disconnected youth pilots, we obtained and reviewed key documents,  including requests for information and public comment, preliminary design  papers, interagency agreements, and evaluation plans. In addition, we  interviewed officials from each of the agencies involved in the  disconnected youth pilots about their collaboration. We then assessed  agencies’ efforts in these areas against leading practices to determine the  extent to which these efforts reflect leading practices for interagency  collaboration and effective pilot design. These leading practices were  developed in our prior work. Appendix I contains a more detailed  explanation of our objectives, scope, and methodology.
We conducted this performance audit from July 2015 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Performance partnerships, as we reported in December 2014, are a type  of hybrid approach to grant consolidation in which grant recipients can  obtain flexibility  to use funds awarded across multiple federal programs in  exchange for greater accountability for results. Figure 1 provides an  overview of the performance partnership model.
Grant consolidation can create opportunities to eliminate federal  programs that are overlapping or outdated, or for which the balance  between costs and benefits received either do not (or no longer) justify  federal spending. According to prior research by the former U.S. Advisory  Commission on Intergovernmental  Relations, grant consolidations are  generally suitable when categorical programs are too small to have much  impact or to be worth the cost of administration, or when multiple  programs exist in functional areas that have a large number of programs  (including health, education, and social services), or where there is  fragmentation (including justice, natural resources, and occupational  health and safety). Grant consolidations generally take either a block  grant or a hybrid approach. A block grant approach is usually broad in  scope, intended to increase state and local flexibility,  and generally give  recipients greater discretion to identify problems or to design programs  addressing those problems using funding from the block grant. A hybrid  approach, such as a performance partnership, can consolidate a number  of narrower categorical programs while retaining strong standards and  accountability for discrete but related federal performance goals.
Since the 1990s, the federal government has taken steps to explore and  establish performance partnerships. For example, the National  Performance Review (NPR) identified performance partnerships as a tool  for helping federal agencies reform the existing federal grant system,  which it stated, among other issues, featured too many funding  categories, an emphasis on remediating rather than preventing problems,  and no clear focus on measurable outcomes. It noted that performance  partnerships could improve federal grant making in situations in which the  federal government intends to deliver services at the state or local level,  agrees with state or local partners on goals and objectives, and progress  toward goals and objectives can be measured. In February 1995, the  President’s Budget for fiscal year 1996 proposed 6 performance  partnerships spanning 7 federal agencies—the Departments of  Agriculture, Education, HHS, HUD, DOL, and Transportation, and EPA— that it stated were aimed at combining funding streams, eliminating  overlapping authorities, and turning agencies’ focus to outcomes as the  basic measure of success. In 1996, Congress provided EPA authority to  create PPGs. More recently, the Performance Partnership Pilots for  Disconnected Youth were authorized in January 2014.
For our December 2014 report, we determined that EPA’s PPGs and the  disconnected youth pilots were the only 2 existing federal performance  partnerships, and for this report we confirmed that they remain the only  ones authorized to date.
subsections: section_title: Performance Partnership Grants within the Environmental Protection Agency’s National Environmental Performance Partnership System
paragraphs: According to EPA, the relationship between EPA and the states has long  been complex, due in part to the division of roles and responsibilities  under federal environmental statutes. Prior to EPA’s creation in 1970,  states provided the majority of environmental management controls, such  as establishing standards for the amount of pollutants that can be  released into air or water and developing public health and natural  resources regulations. Subsequently, EPA became a partner with states  and localities in environmental management. Most major federal  environmental statutes, including the Clean Water Act, permit EPA to  allow states under certain circumstances to implement key programs and  to enforce their requirements.
Several efforts to explore and improve relationships between EPA and  the states led to the creation of NEPPS and PPGs. In 1993, EPA and the  states convened the Task Force to Enhance State Capacity to generate  ideas for improving their partnership. The task force reported that new  federal environmental statutes had increased the environmental  management responsibilities being borne by states at a time when they  were facing declining resources. In addition, the task force reported that  EPA and the states faced difficulty in working together on issues of day- to-day program management, which strained their relationship. The task  force made a number of recommendations, including that EPA and the  states establish a new framework and policy for their relations and a joint  process for strategic planning and the integration of both sides’ priorities.  In May 1995, EPA and the states established NEPPS to address the task  force’s recommendations. As we reported in December 2014, NEPPS is a  performance-based system designed to direct scarce public resources  toward improving environmental results, allow states greater flexibility to  achieve those results, and enhance accountability to the public and  taxpayers. A key element of NEPPS, upon its establishment, was EPA’s  commitment to give states with strong environmental performance greater  flexibility  and autonomy in running their environmental programs. percent of the $1.08 billion in environmental program grants EPA  awarded in fiscal year 2016.
section_title: Performance Partnership Pilots for Disconnected Youth
paragraphs: As we reported in February 2008, while most youth successfully transition  to adulthood, some become disconnected from school and work, and  experience challenges in making this transition. Their disconnection  may result from incarceration, aging out of foster care, dropping out of  school, or homelessness. Some of these youth are more likely than  others to remain low-income and lose jobs during economic downturns,  and to engage in criminal activity, antisocial behavior, and teenage  parenting. Direct services intended to assist youth in transitioning to  adulthood are provided at the local level with the support of federal, state,  and local governments, and private funding sources. A range of local  entities, such as community-based organizations—which are generally  non-profit entities that provide social services—and charter schools, in  urban and rural communities nationwide, help to provide such services.
Multiple  federal agencies play a role in providing funding and assistance  to local programs that serve disconnected youth, which can create  challenges for local service providers. In February 2008, we reported that  the White House Task Force for Disadvantaged Youth identified 12  federal agencies that funded over 300 programs to assist local  communities in serving disadvantaged youth in fiscal year 2003. In  conducting that work, we also interviewed the directors of 39 local  programs serving disconnected youth, and those whose programs  received multiple federal grants from multiple federal agencies told us  they experienced difficulties in working across varying reporting  requirements, funding cycles, and eligibility  requirements. The directors  also reported experiencing challenges working across varying program  goals and sharing information about their clients that participate in  multiple federal grants.
The Performance Partnership Pilots for Disconnected Youth seek to  identify cost-effective strategies for providing services that can address  these types of challenges and achieve better results through making  better use of budgetary resources. Although implementation of the first  round of pilots began in 2015, their genesis dates back to early 2011.  Figure 2 identifies key events in the development and implementation of  the disconnected youth pilots.
According to federal officials involved in the pilots, the concept for the  disconnected youth pilots came in response to a February 2011  Presidential memorandum. It directed federal agencies to work with state,  local, and tribal governments to identify and develop strategies for  eliminating administrative, regulatory, and legislative barriers to achieving  results in federally funded programs and increase access to flexibilities  needed to produce the same or better outcomes at lower cost. Officials  from Education, HHS, and DOL told us that following the memorandum’s  issuance, representatives from their agencies met with representatives from state, local, and tribal governments to discuss policy areas in which  they thought additional flexibilities could improve outcomes. They  identified programs for disconnected youth as an area that would benefit  from such flexibilities.
In 2012, the Administration and agencies took several steps aimed at  better coordinating and integrating programs focused on disconnected  youth. In February 2012, the President’s Budget for fiscal year 2013  requested authority for a new performance partnership pilot initiative to  test approaches to improve outcomes for disconnected youth. Shortly  thereafter, in March 2012, OMB, Education, HHS, HUD, DOJ, and DOL  established the Interagency Forum for Disconnected Youth with the goal  of improving outcomes for disconnected youth through enhanced  interagency and intergovernmental collaboration. In June 2012, Education  published a request for information in the Federal Register seeking ideas  and information on effective approaches for improving outcomes for  disconnected youth. The Interagency Forum on Disconnected Youth used  the responses to the request for information to develop initial design  considerations for the disconnected youth pilots, according to  documentation of the design considerations.
The first round of the Performance Partnership Pilots for Disconnected  Youth was authorized in the Departments of Labor, Health and Human  Services and Education and Related Agencies Appropriations Act for  fiscal year 2014. Enacted in January 2014, it authorized federal  agencies appropriated funds thereunder to select and implement a round  of up to 10 pilots designed to improve outcomes for disconnected youth  that may run for 5 fiscal years (through fiscal year 2018, which ends  September 30, 2018). The act defines disconnected youth as individuals  between the ages of 14 and 24 who are low-income and either homeless,  in foster care, involved in the juvenile justice system, unemployed, or not  enrolled in or at risk of dropping out of an educational institution. The  pilots are to involve 2 or more federal programs administered by 1 or  more federal agencies. The act provides agencies authority to use  discretionary funding made available in the act and waive statutory,  regulatory, or administrative requirements related to the use of that  funding. The agencies involved in the first round of pilots—Education,  HHS, DOL, CNCS, and IMLS—issued a request for public comment on  the pilot application process in July 2014 that sought feedback on  information applicants should include in their applications, criteria  agencies should use in evaluating applications, and technical assistance  for entities preparing applications. Officials from Education told us that  the agencies incorporated public comments from the request in a notice  inviting applications, which they subsequently issued in November  2014. The federal agencies designated 9 pilot locations as finalists for  the first round in September 2015 and publicly announced the locations in  October 2015 (see fig. 3).
Subsequent appropriations laws authorized 2 additional rounds of  disconnected youth pilots and broadened the scope of the effort.
Second Round. In December 2014, a second round of up to 10  locations for disconnected youth pilots was authorized, again with a 5- year timeframe for implementation (through fiscal year 2019), and  agencies were authorized to use funds made available in the fiscal  year 2015 appropriations to participate in previously authorized  pilots. Education published a notice of proposed priorities,  requirements, definitions, and selection criteria for the second and  future rounds of pilots in the Federal Register in October 2015. The  notice proposed additional priorities for projects serving specific high- need subpopulations of disconnected youth, changed application  requirements to reduce burden on applicants, and asked for  comments on how federal agencies could improve future pilot  competitions. In April 2016, the agencies participating in the pilots  published final priorities, requirements, definitions, and selection  criteria. That same month, they published a notice inviting  applications for the second round of pilots. The notice established a  June 2016 deadline for application submissions, but in July 2016 the  agencies published a notice reopening the application process.  Agencies took this action to allow applicants additional time to prepare  and submit their applications. According to Education officials, in  September 2016 federal agencies designated 1 second round  applicant as a pilot finalist. The agencies expect to announce the pilot  publicly once the agencies and the finalist have signed a performance  partnership agreement.
Third Round. The third round of disconnected youth pilots, again  consisting of up to 10 locations with a 5-year timeframe (through fiscal  year 2020) was authorized in December 2015, and agencies were  authorized to use funds made available in the fiscal year 2016  appropriations act to participate in previously authorized pilots. DOJ  and HUD were authorized to participate in this round of pilots. In  addition, this authorization established that new pilots selected for the  second round using fiscal year 2015 funds and the subsequent third  round must include communities that have recently experienced civil  unrest. In August 2016, the agencies published a notice inviting  applications for the third round of pilots. The application period  closed in October 2016, and in January 2017 the agencies designated  6 applicants as third round pilot finalists.
section_title: Performance Partnership Initiatives Exhibit Four Key Characteristics Somewhat Differently
paragraphs: We identified 4 key characteristics that PPGs and the disconnected youth  pilots share:  1.  documented agreement outlining goals, roles, and responsibilities;  2.  flexibility  in the use of funds across multiple federal programs;  3.  additional flexibilities, such as expanded program participant eligibility  or streamlined reporting requirements; and  4.  accountability for results.
The following sections describe each of the shared key characteristics,  providing illustrative examples from selected states with PPGs and  selected pilot locations and any benefits or challenges associated with  these key characteristics as described by participants. More detailed  information about how the key characteristics are exhibited in the 2  initiatives and additional illustrative examples are contained in appendix II  (PPGs) and appendix III  (disconnected youth pilots).
subsections: section_title: Performance Partnership Goals, Roles, and Responsibilities Are Generally Documented in Agreements
paragraphs: Federal agencies and non-federal grant recipients generally document in  an agreement what is entailed by their partnership. The document  establishes the various goals the partners seek to achieve through their  partnership. It also lays out the roles and responsibilities of each partner.
subsections: section_title: PPG Work Plans Document Goals, Roles, and Responsibilities
paragraphs: A PPG generally involves an EPA regional office and a state agency,  such as a state environmental, health, or agricultural agency. Figure 4  provides the general structure of this partnership.
EPA and state agencies define the scope of their partnership in a PPG  work plan. For programs authorized under federal environmental statutes,  EPA generally is responsible for establishing program policy and  guidance and oversight, and states generally are responsible for carrying  out day-to-day program operations. Therefore, PPG work plans, like other  EPA program grant work plans, identify an agreed-upon set of planned  work activities the state agency will undertake and their timeframes for  completion, as well as information about the EPA strategic goals and  objectives that the activities are expected to help meet. EPA guidance  states that a PPG work plan should result from negotiations between EPA  and state program managers and staff and reflect joint planning, priority  setting, and mutual agreement between the 2 sides.
For example, the work plan for state fiscal year 2015 for EPA’s and the  California Department of Pesticide Regulation’s (DPR) PPG defined their  partnership for a PPG that spanned state fiscal years 2013 to 2016. The  document identified program areas in which DPR would undertake work.  These included areas such as enforcing pesticide laws and ensuring  worker safety from pesticides. DPR linked each of these program areas to  the specific EPA strategic plan goals and objectives they supported. For  instance, DPR’s work in the area of enforcing pesticide laws was linked to  EPA’s strategic plan goal of protecting human health and the environment  by enforcing laws and ensuring compliance, as well as its related  strategic objective of enforcing environmental laws to achieve  compliance. The work plan also identified specific work activities DPR  planned to complete by the end of the state fiscal year. For example, in  the program area of enforcing pesticide laws, DPR agreed to conduct 182  oversight inspections of the use of pesticides in agricultural operations.
Officials from EPA and state agencies involved in the PPGs in our review  described as a benefit how the 2 sides work together to develop PPG  work plans, noting that the partnerships have strengthened their  collaborative relationships. For example, officials from EPA Region 2 and  the New York State Department of Environmental Conservation (DEC)  said that EPA’s National Program Manager Guidance—biennial guidance  from EPA program offices that establishes priorities and key actions to  accomplish—serves as a framework for the activities that DEC will  conduct through the PPG. Officials from Region 2 and DEC annually  discuss how DEC priorities can be addressed within the framework and  what appropriate goals and targets are for identified priorities. Officials  from the New York DEC told us that the good working relationship they  have with EPA Region 2 officials allows them to effectively work together  to adjust or reconcile competing priorities when unexpected challenges  arise.
section_title: Disconnected Youth Pilots Use Performance Partnership Agreements to Document Goals, Roles, and Responsibilities
paragraphs: In contrast with the 1-on-1 partnerships in PPGs, first round disconnected  youth pilots involve 2 or more federal partners and, in most cases,  multiple grant recipients whose joint application was selected to  participate in the pilot. The general structure for this partnership is shown  in figure 5.
The federal and non-federal organizations involved in each first round  disconnected youth pilot defined the scope of their partnership in a  performance partnership agreement. These agreements establish the  terms and conditions under which the federal and non-federal partners  will participate in the pilot and identify the specific outcomes the 2 sides  will seek to achieve.
The roles and responsibilities that the partners assume in developing and  implementing the agreements are specific to the initiative.  The federal  agencies involved in developing the initiative, through a separate  interagency agreement, established the following roles for federal  partners in individual  pilots:
Lead agency: The lead agency is responsible for managing the  performance partnership agreement. OMB designated Education as  the lead agency for all 9 first round pilots. During the negotiations of  the first round partnership agreements, Education coordinated the  negotiations on behalf of, and in partnership with, all federal agencies  involved, and worked with non-federal partners to finalize planned  pilot goals and related performance measures. In addition, Education,  as the lead agency, provides and oversees start-up grants to pilots.
Consulting agency: The consulting agency leads pilot monitoring on  behalf of the involved federal agencies. It does so by providing  feedback on pilot performance reporting and facilitating  communication among federal agencies and non-federal partners.
Participating agency: The participating agency provides support to  the lead and consulting agency by, as appropriate, providing feedback  on pilot performance reporting and assistance to the other federal  agencies and non-federal partners to address any implementation  issues.
Non-federal partners assume roles and responsibilities established in the  performance partnership agreements. The partnership agreements for  each of the first round pilots designate a state, local, or tribal government  entity as the pilot lead. The pilot lead is responsible for ensuring that the  pilot is carried out in accordance with applicable federal requirements and  oversees the proper use of all federal funds. The agreement also  identifies any additional non-federal partners, such as another  government entity or a non-profit community organization, involved in the  pilot and their roles and responsibilities.
For example, in the Chicago pilot, the federal partners are Education as  the lead agency, HHS as the consulting agency, and DOL as a  participating agency. The non-federal partners, and their roles, are:
The Chicago Department of Family and Support Services, the pilot  lead, works to connect Chicago residents and families to resources  that build stability, support their well-being, and empower them to  thrive.
The Chicago Cook Workforce Partnership, a pilot partner, will consult  with the Department of Family and Support Services in  implementation and oversight of the pilot. The organization is a  collaborative effort between Cook County, Illinois and the City of  Chicago designed to align the 2 entities’ efforts in delivering services  under the Workforce Innovation  and Opportunity Act (WIOA).
The federal and non-federal partners in each first round pilot worked  during their development of a performance partnership agreement to  identify the pilot’s intended outcomes. In the partnership agreements,  these intended outcomes take the form of quantitative goals and  measures. Federal agencies require that at least 1 set of goals and  measures address educational outcomes and a second set address  employment outcomes. The 4 pilots in our review established education  and employment goals and measures tied to their particular service  interventions. The Oklahoma pilot, for example, is structured to help youth  with foster care experience in the Oklahoma City Public Schools complete  high school, attend college, and enter the workforce. Its partners  established an education goal for 80 percent of youth who complete at  least 6 months in the program to attain a high school diploma or its  equivalent. The partners established an interim measure—that 85 percent  of participants will be absent from school for 15 days or fewer during the  school year—to track progress toward the goal. According to the pilot’s  application, increased school attendance for participants is likely to lead  to an improved high school graduation rate for them.
Similar to PPGs, officials from federal and non-federal partners involved  in disconnected youth pilots told us in our interviews that their  partnerships have strengthened collaborative relationships with each  other. For example, officials from the pilot lead for the Chicago pilot told  us that the non-federal partners worked closely with HHS, the pilot’s  consulting agency, and DOL, a participating agency, to gain a better  understanding of the types of flexibilities they could use to implement the  pilot. In addition, partners told us that the partnerships have strengthened  collaboration among multiple non-federal partners working together at the  state, tribal, or local level to implement disconnected youth pilots. For  instance, officials from the pilot lead for the Oklahoma pilot told us that  they have used the pilot development process to convene a wide range of  organizations involved in addressing the needs of youth in foster care in  Oklahoma City, the pilot’s target population, to establish a network  broader than just those officially in the pilot. This convening enabled the  pilot lead to identify organizations with which it previously had not worked  that could contribute to improved outcomes for foster youth and bring  them into the new network.
However, federal and non-federal partners in disconnected youth pilots  also told us that at times the multiple-partner structure of the pilots, along  with their new and unique nature, has caused complications and delays in  the pilots’ design and implementation. Officials we spoke with from  several of the federal agencies involved in developing the initiative after  its 2014 authorization said that, given their lack of familiarity with a legal  provision like this, they had to spend time and effort to reach a common  interpretation of the provision and how it could be implemented. This  resulted in a longer than usual process for the agencies to develop and  issue the notice inviting applications for the first pilot round. The first pilot  round was authorized in January 2014 and Education released the notice  inviting applications in November 2014. Federal and non-federal partners  involved in pilots in our review told us that there were additional  challenges related to negotiating and finalizing the partnership  agreements, given the numerous parties involved. This led to additional  time being spent on finalizing the agreements. The agreements for 6 of  the 9 pilots were signed by their non-federal partners in December 2015  and January 2016, with the other 3 being signed by their non-federal  partners between February and April 2016.
Non-federal partners in the selected pilots also identified challenges in  developing and coming to agreement on the goals and measures.  Officials from the Eastern Kentucky pilot told us that it took time to work  with federal agencies to agree on the goals and measures that were  included in the final partnership agreement. The non-federal partners  wanted to set goals for pilot participants to improve their academic  performance to the average level of students in the Kentucky Highlands  Promise Zone. Federal partners, however, requested that those goals  be set at the average level of students in all of Kentucky—which is a  higher average than of those just in the Promise Zone. The partners had  discussions around the issue and agreed to establish goals at the higher  level of students across the state, with recognition that these were stretch  goals. Despite these challenges, several non-federal partners also told us  that they see benefit in being able to establish goals and measures  tailored to interventions, and not having to use standard federal  performance measures, which may not always be useful in determining  outcomes among their target populations. For example, officials from the  Ysleta del Sur Pueblo pilot told us that they established performance  measures for their pilot that will allow them to better determine the  educational and employment outcomes of the tribal youth they plan to  serve.
section_title: Performance Partnerships Are Characterized by Funding and Other Flexibilities
paragraphs: Performance partnerships provide non-federal partners with flexibility in  how they use funds from multiple federal programs. Partnerships can also  provide non-federal partners with additional flexibilities,  such as expanded  program participant eligibility  or streamlined reporting requirements. The  non-federal partners can use these flexibilities hand-in-hand to tailor  efforts to more effectively achieve their goals, as well as reduce their  administrative burden.
subsections: section_title: PPGs Provide Funding and Other Flexibilities to Ease Administrative Burden for States
paragraphs: PPGs permit state agencies to request that funding they receive from 2 or  more EPA program grants be combined into a single award. This is  intended to enable state agencies to, among other things, consider trade- offs across the breadth of their environmental program funding and  exercise flexibility to direct resources to their most pressing priorities.  Specifically, once a state agency has requested and received selected  EPA program grants in a PPG, it can choose to use the funds to support  any activity that is eligible under at least 1 of the grants included in the  PPG. PPGs also streamline administrative requirements so that state agencies can realize cost savings through reduced administrative burden  in areas such as grant applications, cost sharing, and financial reporting.
PPGs we examined in selected states exhibit variation in how state  agencies chose to receive combined EPA program grant funding and  exercise additional flexibilities. We selected states in which environmental  agencies received either a small or a large number of grants combined in  their PPGs. On the small end, we selected New York, whose DEC  received funding from 3 grants, and California, whose DPR received  funding from 4 grants in 1 environmental area (water and pesticides,  respectively) in their PPGs in fiscal year 2016. On the large end, we  selected Alabama, whose Department of Environmental Management  (DEM) received funding from 9 grants, and Utah, whose Department of  Environmental Quality (DEQ) received funding from 10 grants across  multiple environmental program areas in their PPGs in fiscal year 2016.  Figure 6 shows the grants in DEQ’s PPG in fiscal year 2016. Details   on the PPGs in Alabama, California, and New York are included in  appendix II.
According to EPA guidance on PPGs, states can take different  approaches to exercising their flexibility to direct resources to their most  pressing priorities covered by any activity that is eligible under at least 1  of the grants included in the PPG. A state can propose using this  funding to pool resources from multiple programs consolidated into the  PPG to implement projects or initiatives that cross traditional program  barriers. For example, a state can propose to conduct inspections to  assess compliance across air, water, and hazardous waste management  requirements if it has included grant programs in those areas in its PPG.  A state can also, based on its environmental priorities, propose increasing  resources and effort in 1 program area while decreasing resources and  effort in a second program area. For example, if a state has identified that  its needs in addressing water pollution are greater than its needs in  addressing air pollution, and it has included relevant water and air grant  programs in its PPG, it can propose to strategically increase resources  and effort for water pollution activities while decreasing them for air  pollution activities. States that intend to exercise programmatic flexibilities  must explain the reasons for and expected benefits of the flexibilities in  their PPG application.
State agencies involved in PPGs also see great benefit in their ability to  make use of flexibilities that reduce administrative burden related to grant  applications, cost sharing, and financial reporting, according to the state  agency officials whom we interviewed.
Grant applications. A state agency can submit a single application  covering all of the grants it is seeking to consolidate in its PPG rather  than a separate application for each. Officials from the Alabama DEM  told us that the streamlined PPG grant application requirement has  allowed the DEM to submit a single PPG application rather than 9  individual grant applications, which it had to do prior to adopting the  PPG. This has reduced the amount of administrative work that DEM  staff must complete, thereby allowing them to focus on other activities.
Cost-sharing. Certain EPA program grants require state agencies to  provide a portion of program costs in order to receive the grant. Some  grants require states to provide a certain percentage of total  expenditures under the grant, known as a match requirement, while  some others require states to spend non-federal funds for work  conducted under the grant in an amount at least equal to those spent  in a previous year, known as a maintenance of effort requirement.  When a state combines grants in a PPG, it does not have to meet the  individual cost-share requirements of the grants included in the PPG;  instead, the state’s cost-share for a PPG is not less than the sum of  the minimum required under each of the underlying grants included in  it. According to EPA guidance on PPGs, the ability to meet cost- sharing requirements in the aggregate can be valuable when a state  has more than adequate resources to meet the match required of 1  program included in the PPG but not enough for a second included  program. The state can use excess match resources to cover the  program that cannot meet its match requirement. Officials from the  Utah DEQ said that the ability to meet match requirements in the  aggregate is one of the most useful aspects of a PPG. They explained  that multiple grants they include in their PPG have match  requirements. Because of the DEQ’s line-item budget structure, the  officials stated it would be challenging for them to meet the match  requirements of specific program grants if they were not included in a  PPG. However, because the PPG allows match requirements to be  met at an aggregate level, DEQ’s expenditures in particular program  areas can be added together to meet the overall match requirement.
Financial reporting. A state agency can report on expenditures  within the PPG in the aggregate, covering all grants consolidated in  the PPG, rather than for each grant individually.  Officials from each of  the states in our review reported that they benefit from streamlined  financial reporting. For example, officials from the Alabama DEM said  that they provide 1 annual financial report for their PPG to EPA rather  than the 9 that they were required to provide for individual grants  before they adopted the PPG. They stated that this change has  significantly reduced DEM’s administrative burden.
Officials from EPA regional offices and state agencies we interviewed told  us that existing organizational silos within EPA and state agencies can  limit a state’s willingness and ability either to include EPA program grants  across multiple environmental areas in a PPG or, in cases in which states  have done so, to take full advantage of available funding flexibilities. For  example, although officials from the New York DEC told us that their  agency has been able to use Water Pollution Control and Water Non- Point Source Management funds to create a more integrated,  comprehensive clean water program by including them in a PPG, they  said that their agency’s structure makes it difficult for the agency to  include programs from other environmental areas in a PPG and thereby  take advantage of additional programmatic flexibility. They explained that  their agency organizes its operating divisions by environmental areas,  such as water and air. The management of a cross-area PPG would  require coordination across divisions, which would require the divisions to  make changes in the way they operate. The costs associated with making  the changes necessary to administer such a PPG, the officials stated,  would likely negate the benefits of the potential additional programmatic  flexibility.  Furthermore, officials from the Alabama DEM, which receives a  PPG combining program grants across environmental areas, told us that  once the agency receives a PPG from EPA it generally distributes the  funding associated with each of the underlying grants to the program  offices responsible for implementing them. In essence, the agency  reverses the combining of the funding at the state level and uses it in  much the same way it would if it received the funding from the underlying  grants outside of a PPG. The officials explained that their agency uses  PPG funding for the individual program-specific activities for which they  were originally approved because officials within the relevant program  offices at both EPA Region 4 and DEM have wanted to maintain control  over the program funds they are responsible for managing and  overseeing.
section_title: Disconnected Youth Pilots Use Flexibilities to Tailor Services to Target Populations
paragraphs: Similar to PPGs, the disconnected youth pilots enable non-federal  partners to combine funds from federal agencies’ programs and obtain  additional flexibilities, but we found that for the disconnected youth pilots  we reviewed, these flexibilities were generally used to tailor service  interventions to the specific needs of their target populations rather than  to reduce administrative burden. Specifically, the authorization for the first  round pilots allows for the combining of discretionary funding that  Education, HHS, DOL, CNCS, and IMLS received through the fiscal year  2014 appropriations act, and the waiver of statutory, regulatory, or  administrative requirements affecting target populations, as proposed by  the grantees, to carry out the pilots. This authorization included 2  safeguards on the use of these flexibilities, both which require written  determinations by the head of an agency. First, an agency can participate  in a pilot and combine funds only after its head provides a written  determination that the agency’s participation will not result in denying or  restricting the eligibility  of individuals for any of the services that are  funded by the agency’s programs or funds being used in the pilot, and  that vulnerable populations who receive such services will not be  otherwise adversely affected by the agency’s participation. In the notice  inviting applications for first round pilots, applicants were advised that  where a program’s funds are not suitable for combining (referred to as  “blending”) in a pilot given these constraints, the applicant may  nevertheless consider how to coordinate (referred to as “braiding”) such  funding in a pilot to promote more effective and efficient outcomes even  though the funds would maintain a separate identity and remain subject to  the program requirements for which the funds were appropriated.  Second, an agency also can waive program requirements associated with  funds being used in a pilot, but only after its agency head issues a written  determination that the granting of such waivers (1) are consistent with the  statutory purposes of the underlying federal program and other provisions  of the pilot authority, including that individuals will not be denied or  restricted eligibility  for services, (2) are necessary to achieve the  outcomes of the pilot and no broader in scope than is necessary to do so,  and (3) will result in either realizing efficiencies (by simplifying reporting or  reducing administrative barriers) or increasing the ability of individuals to  obtain access to services.
In addition, for the first pilot round federal agencies awarded separate  start-up grants to provide funding of up to $700,000 to each pilot to  finance evaluations, capacity building, technical assistance and other  related activities to support the pilot. According to officials from OMB,  funds available for these purposes from CNCS, DOL, and Education were  used for these start-up grants. Education officials told us that start-up  grants also were in part intended to provide an incentive to non-federal  partners to participate in and implement the disconnected youth pilots.  Non-federal partners stated that the start-up grants were a key incentive  to become involved with the pilots, as they represent a significant amount  of new funding for them, which could help them work across traditional  program lines, among other things.
The first round pilots vary in the extent to which they use combined  federal funds and waivers from selected program requirements in their  pilots, as noted below and further illustrated in appendix III.  Of the 9  pilots:
2, including Ysleta del Sur Pueblo, are combining all of the federal  funds from all of the federal programs they are including in their pilots,
2, including Eastern Kentucky, are combining federal funds from some  of the federal programs they are including as well as coordinating the  use of federal funds from other federal programs, and
5, including Chicago and Oklahoma, are not using any combined  funding but are instead coordinating the use of federal funds from  multiple federal programs.
Non-federal partners from the 2 pilots selected in our review that are  combining all or some federal funds—Eastern Kentucky and Ysleta del  Sur Pueblo—told us that they consider the ability to combine funds to be  a benefit, as it allows them to implement more effective programs and  services for disconnected youth. In addition, the use of combined funds  reduces non-federal partners’ financial reporting burden. To ensure  accountability for the proper use of combined funds and start-up grants in  pilots, non-federal partners provide financial reports on their use of these  funds to Education as the lead agency. In contrast, when a pilot has  coordinated funds, the relevant non-federal partner(s) will report on the  use of those funds separately, as prescribed by the originating agency  per its normal guidelines for the program. Ysleta del Sur Pueblo officials  told us that they see this reporting process as a key benefit of its pilot.  Officials said the tribe was motivated to apply to become a pilot site to  gain administrative benefits such as reducing the amount of reporting  they normally would have provided to CNCS and IMLS—the 2 agencies  with program funds being used in the pilot. Since the tribe is combining all  funding involved in the pilot, it instead reports to Education on its use of  those funds.
In addition, 8 of the 9 first round disconnected youth pilots, including the 4  pilots selected for this review, requested and were granted waivers of  selected requirements for at least 1 of the federal programs included in  their pilots. These waivers provide pilots with additional flexibilities to  tailor allowable activities, participant eligibility,  and reporting requirements  to better meet the needs of disconnected youth, according to the notice  inviting applications for the first pilot round. Furthermore, non-federal  partners in the 4 pilots in our review told us that, among other things, the  waivers enable them to change eligibility  requirements or the allowable  use of select federal funds, which will allow them to implement innovative  approaches tailored to disconnected youth. For these reasons, all of the  federal and non-federal partners with whom we spoke told us that the  flexibilities possible through the pilots—the ability to combine funds and  obtain waivers from selected federal program requirements—are one of  their biggest benefits.
The Eastern Kentucky pilot illustrates variation in the use of combined  and coordinated funding and waivers. As illustrated in figure 7, to improve  the academic performance of disconnected youth, the pilot is using  combined funding across DOL’s WIOA Title I Youth program and  Education’s Gaining Early Awareness and Readiness for Undergraduate  Programs (GEAR UP) and Promise Neighborhoods programs and  coordinating the use of funding from Education’s Full Service Community  Schools program.
The Eastern Kentucky pilot was also granted 5 waivers that allow the  non-federal partners to change eligibility requirements and the allowable  use of certain federal funds. Officials from Partners for Education at  Berea College (PFE), a pilot partner, told us that the various federal  programs serving disconnected youth in Eastern Kentucky were focused  on different aspects of the disconnected youth population. The non- federal partners had a difficult time providing comprehensive services to  disconnected youth in the region since they had to serve distinct  segments of the population with individual  federal grants. Officials said  that the waivers should help the non-federal partners expand the reach of  their services. One of these waivers is related to a requirement in  Education’s GEAR UP program, which is designed to help low-income  students prepare for and succeed in post-secondary education.  Traditionally, GEAR UP grant recipients can use the funds to provide  mentoring, outreach, and other services to students for 6 years, usually  between seventh grade and the completion of high school. They can also  provide those services for a seventh year as long as during that year the  student is enrolled in post-secondary education. The officials from PFE  told us that the waiver allows them to provide services to youth who,  during their seventh year, are not enrolled in post-secondary education, to  help them connect to post-secondary education. Officials from the pilot  told us that using some combined funds allows them to hire staff who are  not tied to a specific program and therefore can work across the  programs involved in the pilot. In addition, the officials from PFE told us  that the start-up grant is allowing the non-federal partners to establish a  data collection system to share information about the participants across  partners. This would allow them to use those data to identify effective  strategies and support the evaluation of the pilot. The waivers, the pilot  officials added, should help the non-federal partners expand the reach of  their services.
Federal and non-federal partners involved in disconnected youth pilots  told us that in some cases the 2 sides faced challenges in coming to  agreement on the use of funds in pilots. For example, the non-federal  partners in the Chicago pilot initially proposed combining Head Start  funds from HHS with WIOA Title I Youth funds from DOL. According to  documentation of HHS’s review of this request, HHS denied the request  because officials were concerned that combining Head Start funds could  adversely affect vulnerable populations—one of the restrictions for  combining funds in the pilot authorization—by potentially diverting funds  away from services for children in Head Start toward activities primarily  targeted at teenage and young adult mothers, who are primarily served  by the WIOA program. HHS agreed that the pilot could coordinate Head  Start funds, as is illustrated in figure 8, since this would ensure that the  funds retain their cost allocation requirements and therefore allow HHS  officials to ensure accountability for the funds’ use to support services for  children in Head Start.
section_title: Performance Partnerships Use Performance Reporting to Ensure Accountability for Results
paragraphs: Performance partnerships use performance reporting to ensure  accountability for results. Non-federal partners periodically report to  federal partners on their progress towards the goals established in the  partnership document. As was described in the prior section, non-federal  partners also ensure accountability for the use of funds through financial  reporting processes.
subsections: section_title: PPGs Use Performance Reports and Data to Ensure Accountability for Results
paragraphs: To monitor progress toward the work activities and goals and measures  established during performance planning and included in the work plans  discussed above, federal and non-federal partners in PPGs engage in  performance reporting. Related to PPGs, we reported in July 2016 that,  according to EPA policies and officials, after EPA approves a work plan  for an EPA grant, grantees generally submit information on their progress  and results to EPA in 2 ways: (1) performance reports, which are  generally written and describe the grantees’ progress toward the planned  grant results in their work plans and (2) program-specific data, which is  generally numeric and which grantees electronically submit on certain  program measures that EPA tracks in various program databases.
Performance reports. These reports describe the grantees’ progress  toward the planned grant results in their work plans, such as using grant  funds to provide technical assistance to local officials. EPA grantees are  to submit these reports at least annually. EPA policies include general  guidelines about what performance reports should include, such as a  comparison between planned and actual grant results, but allow the  frequency, content, and format of performance reports to vary by program  and grant. According to EPA officials, EPA project officers monitor these  reports to review grantee progress toward agreed-upon program results.  PPGs enable state agencies to submit a single performance report for all  the programs included in their PPGs, according to EPA and state agency  officials with whom we spoke. We reviewed the most recently available  end-of-year performance reports that the Utah DEQ submitted to EPA for  federal fiscal year 2014 and the California DPR submitted to EPA for  state fiscal year 2014-2015. For its report, DPR provided information  about how it addressed each planned activity in its work plan. For  example, the report stated that DPR conducted 253 oversight inspections  of the use of pesticides in agricultural operations, exceeding its target of  182 oversight inspections. According to information in the report, EPA,  after it received the report from the department, reviewed the material and  provided comments, as needed.
Program-specific data. Grantees electronically submit data on certain  program measures, such as the number of hazardous waste violations  issued or the acres of brownfield properties made ready for reuse, which  EPA tracks in various program databases. According to EPA policy and  program officials, program officials monitor these data to track and report  program accomplishments, at the regional and agency levels, and, as  applicable, to assess the agency’s progress in meeting its performance  targets in support of agency strategic goals. According to EPA officials,  generally grantees or EPA program officials—depending on the  database—are to enter grant results, such as the number of enforcement  actions, into EPA’s program-specific data systems at agreed-upon  intervals, such as quarterly. These requirements may be part of a grant’s  terms and conditions.
section_title: Disconnected Youth Pilots Rely on Reports and Evaluations to Ensure Accountability
paragraphs: For disconnected youth pilots, non-federal partners are to submit  quarterly reports on progress made toward the goals and measures  established in the pilots’ performance partnership agreements to  Education in its role as the lead agency. This performance reporting  covers the entire pilot and the programs included in it, regardless if the  funds involved are combined or coordinated, according to Education  officials. Education will then share the reports with the relevant consulting  and participating agencies for their review. If the quarterly report shows  that the pilot is facing challenges in making progress towards its goals,  federal officials we met with told us that the agencies can work with the  non-federal partners to address the challenges, which could include  amending waivers, providing technical assistance, or requiring the pilot to  develop a corrective action plan.
Federal and non-federal partners in the disconnected youth pilots are also  taking steps to conduct evaluations of pilot outcomes. DOL has  responsibility for leading a national evaluation of the pilots and contracted  with Mathematica Policy Research (Mathematica) to conduct it. Officials  from DOL involved in the national evaluation told us that they have sought  input, and involved individuals, from the other federal agencies in the  ongoing design of the national evaluation, and that they will also have a  chance to review the final evaluation plan. Officials from DOL and  Mathematica told us that the national evaluation will focus on 4 major  areas: technical assistance, implementation, outcomes, and impacts.  According to officials from DOL and Mathematica, they have made, and  will continue to make, adjustments to the design of the national evaluation  to better align with the data collection efforts for the individual pilot  evaluations and avoid duplicating efforts.
In addition, the federal agencies established a competitive preference  priority for first round pilot applicants whereby the agencies awarded extra  points to applicants who proposed conducting a site-specific evaluation  using a randomized control trial or quasi-experimental approach of at  least one of the pilot’s components. Each of the pilots for the first round  is conducting a site-specific program evaluation, according to DOL  officials. The Ysleta del Sur Pueblo pilot, for instance, is using an impact  evaluation to test the effectiveness of its Tigua Leadership Curriculum on  1 of the 2 cohorts of youth it plans to serve in the pilot. The evaluation includes treatment and comparison groups, though the pilot will provide  the Tigua Leadership Curriculum to both groups. It will provide the  services to the comparison group after the final data collection. Ysleta del  Sur Pueblo will collect data on both groups, which will be evaluated to  determine whether the new services had a positive effect on improving  youth attitudes towards staying in school, completing high school, and  understanding the connection between education and career  development opportunities. This evaluation will be conducted in addition  to the pilot’s collection and reporting of performance against specific  performance measures for participants’ educational and employment  outcomes.
Federal and non-federal officials with whom we spoke identified key  benefits of the national-level  and site-specific evaluations. Education  officials said that both the site-specific and the national-level  evaluations  will help federal agencies to determine if there is a need for broader  legislative authority to allow more grant recipients in different locations to  propose waivers similar to those received by the first round pilots.  Officials from the Chicago pilot lead told us that the evaluations will help  both the federal and non-federal partners learn more about what works  within their program, which could help them improve it in the future.  However, officials from the Oklahoma pilot lead told us that while  evaluations are important for tracking if pilots are meeting their intended  results, they were unsure whether the site-specific and national-level  evaluations would be able to determine what, if any, impact the ability to  consolidate funds and use waivers would have on improved outcomes for  disconnected youth. They said that the pilot’s 3-year timeframe may be  too short to allow federal or non-federal partners to clearly determine  whether the pilot has improved outcomes for disconnected youth or  created meaningful systems change. For example, they said that the  partners will likely be unable to determine whether the pilot has improved  youth involvement in post-secondary education or employment outcomes  before the pilot’s timeframe, and the funding to support the evaluations,  expires.
section_title: In Designing, Implementing, and Evaluating the Disconnected Youth Pilots, Federal Agencies Have Taken Steps Intended to Ensure Success, but Have Opportunities for Improvement
subsections: section_title: Federal Agencies Generally Took Actions Consistent with Leading Practices for Interagency Collaboration
paragraphs: The federal agencies involved in the disconnected youth pilots have taken  a number of actions consistent with leading practices for interagency  collaboration identified in our prior reports. In September 2012, we  reported that federal agencies have used a variety of mechanisms to  implement interagency collaborative efforts, which can be used to  address a range of purposes including policy development; program  implementation; oversight and monitoring; information sharing and  communication; and building organizational capacity, such as staffing and  training. We noted that although collaborative mechanisms differ in  complexity and scope, they all benefit from certain leading practices,  which raise issues to consider when implementing these mechanisms.  Table 1 provides examples of how actions taken by federal agencies in  designing and implementing the disconnected youth pilots generally were  consistent with the 7 leading practices identified in our September 2012  report.
section_title: Opportunities Exist to Improve Planning for Financial and Staff Resource Contributions
paragraphs: Although federal agencies generally have taken actions consistent with  leading practices for interagency collaboration, we identified additional  actions they could take in relation to several leading practices to better  support the success of the individual  disconnected youth pilots as well as  the overall initiative. In response to our findings, the agencies already  have taken steps to address issues we identified in the areas of written  guidance and agreements and participants during the course of our audit.  However, their planning for and management of financial and staff  resources are not yet in line with leading practices.
Written Guidance and Agreements. We previously found that written  agreements for interagency groups are most effective when updated and  monitored regularly to reflect the roles and responsibilities of current  participants. Such agreements can help strengthen agency commitment  to working collaboratively and provide a clear delineation of activities to  be undertaken by individual  agencies. Additionally,  updated written  agreements can serve as a source of current information in the case of  staff transitions. In assessing the federal agencies’ efforts related to  written guidance and agreements, we found that the 2015 interagency  agreement covered roles and responsibilities for the federal agencies for  the first 2 pilot rounds, but as of July 2016, it had not been revised to  reflect the third pilot round or the authorization for HUD to participate in  the initiative.  In discussing these findings with Education officials in July  2016, they stated several reasons why this had not happened. First, they  did not expect established roles and responsibilities to change in future  pilot rounds. Second, they stated that federal agencies had been focused  on implementing the first round pilots and preparing for the second and  third round pilots. Finally, they told us that the process for updating the  agreement was cumbersome, as it required the approval of the heads of  all agencies involved in the initiative.  Subsequently, in December 2016,  Education officials told us that the agencies, in response to the issues we  identified, were taking steps to modify the interagency agreement to cover  the third pilot round and include HUD’s roles and responsibilities.  Moreover, they agreed to streamline the process for making future  changes to the agreement. In January 2017, Education shared a draft  update to the interagency agreement reflecting these changes, which are  pending final approval by the relevant agencies.
Participants. We previously found that it is important to ensure that all  relevant participants have been included in a collaborative effort. For all  agencies authorized to participate in a particular initiative, such as the  disconnected youth pilots, their collective involvement helps ensure that  someone can commit resources and make decisions on behalf of their  agency, and contribute to the outcomes of the collaborative effort through  their individual knowledge, skills, and abilities. We identified a few  instances in which officials from federal agencies with program funds  being used in individual  pilots were not notified of their funds’ planned  use. For example, the non-federal partners in both the Oklahoma and  Seattle pilots plan to use AmeriCorps funding. However, in a June 2016  meeting with CNCS officials about their involvement in the pilots, the  agency’s key point of contact for the disconnected youth pilots told us that she was not aware of those pilots’ planned use of the funding. When we  raised this issue in a subsequent meeting with Education officials in July  2016, they told us that in these instances they did not notify CNCS  because their process only involved notifying an agency in cases in which  the non-federal partners proposed combining an agency’s funds or  requested a waiver of requirements related to that agency’s programs in a  pilot.
The non-federal partners in the Seattle and Oklahoma pilots did not  propose combining CNCS funds or request waivers of AmeriCorps  program requirements. Therefore, according to Education officials, the  established process did not require them to notify CNCS about the use of  AmeriCorps funds in those pilots because CNCS did not need to approve  anything. Because the funds are being coordinated, the recipients are still  required to adhere to the requirements of the program for which the funds  were appropriated, including reporting on the funds in accordance with  the program’s underlying requirements. As such, CNCS officials told us  that the use of AmeriCorps program funds in those 2 pilots would be  covered by the program’s usual grant oversight processes. In response to  our observations and to address any potential future issues about  involving relevant agency officials in individual  pilots, Education officials  told us in December 2016 that they have revised their processes. Moving  forward, they will notify relevant agency officials of instances in which  their programs and funds are proposed for inclusion in a pilot, regardless  of whether funds are combined or coordinated or waivers are sought.
Resources. We previously found that collaborating agencies should  identify the various resources, including financial and staff resources,  needed to initiate and sustain their collaborative effort. Collaboration  can take time and resources in order to accomplish activities such as  building trust among the participants, setting up the ground rules for the  process, attending meetings, conducting project work, and monitoring and  evaluating the results of work performed. Moreover, relying on agencies  to participate can present challenges for collaborative mechanisms. Our  past work has also found that, in cases where staff participation was  insufficient, collaboration often failed to meet key objectives and achieve  intended outcomes. Consequently, it is important for groups to ensure  that they identify and leverage sufficient funding and staffing to  accomplish the objectives.
Agencies do not have a full understanding of their future resource  needs—in terms of individual agency funds and staffing contributions—to  maintain the pilot initiative through September 2020, when the third round  is currently scheduled to end. Federal and non-federal partners in the  disconnected youth pilots told us that these resources are important for its  success. Agency officials told us that, because much of their early focus  was on designing and implementing the initiative in short time frames,  decisions related to resource contributions generally were made as  needed to ensure near-term progress. Now that agencies have  designated finalists for the second and third rounds, they can better  identify and plan for the resources they will need to contribute to support  the overall initiative through its lifetime (September 2020). However, the  agencies—including OMB which has responsibility for coordinating  agencies’ overall efforts to implement the disconnected youth pilots— have not yet fully identified how those resources will be provided. Key  aspects of the pilot initiative  rely on funding and staff contributed by  individual agencies. As such, it is important for each agency to  understand what resources it is expected to provide so that it can plan  accordingly. By fully identifying and planning for the specific financial and  staff resource contributions described below, the agencies will have  greater assurance that those contributions will sustain success in their  collaborative efforts and the overall pilot initiative.
Financial Resources. As was previously mentioned, the funding for  federal grants included in individual pilots is provided through  appropriations. According to agency officials involved in the pilots,  they identified additional activities that they considered key to the  success of the initiative. As highlighted in Table 1, these include the  start-up grants, general technical assistance, and the national and  site-specific evaluations. Education and DOL leveraged various  mechanisms, including contracts and interagency transfers, and other  agencies contributed funds, to the extent possible, to support these  activities. According to agency officials, their agencies consider  potential contributions on a year-to-year basis, depending on the  available resources in eligible programs. They told us that their  agencies have needed to make trade-off decisions among competing  priorities to contribute funds to these activities each year.
As a result, funding for some of these activities has decreased over  time. For example, although pilots could receive up to $700,000 in  start-up grants for the first round, they may receive up to $350,000 for  the second round and up to $250,000 for the third round due to  decreased contributions of funding for such purposes from among the  federal agencies. As was noted earlier, the offer of start-up grants was  a key incentive for non-federal partners from the 4 selected pilots as  they considered participating in the initiative. Agencies already have  made their contributions for the start-up grants for the 3 rounds  currently authorized, so this may not be an issue moving forward  unless further rounds are authorized. However, the contract for  general technical assistance has the potential, through annual  extensions which require additional funding, to run through 2020,  when the third round pilots are due to end. Moreover, although the  contract for the national evaluation and related evaluation technical  assistance runs through 2020, work related to completing the  evaluation—which is necessitated by the legislation authorizing the  pilots—will need to continue after this date if the national evaluation is  expanded to include the pilots from the second and third rounds. To  do so, DOL officials told us that they may be able to extend the prior  performance to continue the work under the existing contract or  otherwise award a new contract following federal acquisition  procedures. Regardless of what approach is taken, continued  performance would require additional funding.
Staff Resources. Incomplete information about existing staff  investments in the disconnected youth pilots to date, and uncertainty  in the composition of future pilot rounds has limited agencies’ abilities  to identify long-term staffing needs. Most agencies are not tracking  their current staff support to the first round pilots, which could help  them understand potential staff investments for future rounds. When  we asked agency officials for estimates of staff time spent supporting  the pilot initiative,  they could not provide overall estimates for their  agencies. Similarly, when we asked the key points of contact at each  agency—those generally most involved in the disconnected youth  pilots—about their own time spent on the initiative, they could only  provide rough estimates. Officials explained that several factors made  it difficult to track staff investment. The number of staff involved and  their related time demands varied depending on the stage of the  process and the activities that were being undertaken. Moreover, at  different points in the past, the agencies were designing and  implementing multiple rounds concurrently. In addition, in interviews  with officials from each of the agencies involved in the pilots during  summer 2016, officials told us that their agencies face difficulties  anticipating future staffing needs since the exact number and  structure of the second and third round pilots were unknown at the  time. Officials said their agencies could not sufficiently plan for staff  needs beyond what had been established in the first round of pilots  without knowing the number of pilots that would be selected for each  round, and which federal agencies and programs would be involved.
However, the selection processes for both the second and third  rounds are now complete. As was mentioned previously, the agencies  designated 1 applicant as a second round pilot finalist in September  2016, and 6 applicants as third round pilot finalists in January 2017.  Therefore, agencies should have a better sense of their involvement  in all 3 rounds. Moreover, because all 3 rounds will soon be in the  implementation phase, it should be easier for agencies to track  current staff investments to better project needed staffing  contributions moving forward.
section_title: Federal Agencies Generally Took Actions Consistent with Leading Practices for Pilot Design, but Could Improve Plans for Assessing Scalability of the Disconnected Youth Pilots
paragraphs: Federal agency efforts for the disconnected youth pilots were also  generally consistent with practices for effective pilot design. A well- developed and documented pilot program can help ensure that agency  assessments produce information needed to make effective program and  policy decisions. Such a process enhances the quality, credibility, and  usefulness of evaluations in addition to helping to ensure that time and  resources are used effectively. In April 2016, we identified 5 leading  practices that, taken together, form a framework for effective pilot  design. By following these leading practices, agencies can promote a  consistent and effective pilot design process. Examples of actions federal  agencies took in line with these practices are illustrated in table 2.
Although federal agencies generally took actions consistent with leading  practices for effective pilot design, we identified an additional action they  could take in relation to assessing scalability. We previously found that,  as part of their design, agencies should have criteria or standards for  identifying lessons about the pilot to inform decisions about scalability and  whether, how, and when to integrate pilot activities into overall efforts.  To assess scalability, criteria should relate to the similarity or  comparability of the pilot to the range of circumstances and population  expected in full implementation. The criteria or standards can be based  on lessons from past experiences or other related efforts known to  influence implementation and performance as well as on literature  reviews and stakeholder input, among other sources. The criteria and  standards should be observable and measureable events, actions, or  characteristics that provide evidence that the pilot objectives have been  met. Choosing well-regarded criteria against which to make comparisons  can lead to strong, defensible conclusions.
Although the federal agencies have identified a variety of data to collect  through performance reporting and the national and pilot-level program  evaluations, they did not identify criteria or standards for assessing  scalability of the flexibilities being tested by the pilots as part of the pilot or  evaluation design processes. The agencies conducted outreach to  stakeholders to learn about the needs of organizations assisting  disconnected youth, established key goals and objectives for the  organizations implementing the pilots and included them in partnership  agreements, which also detail data collection methodology, reporting  requirements, and interim measures. They also developed evaluation  plan reporting templates for the pilots. However, according to DOL  officials, federal agencies decided not to require any new data collection  by the pilots for several reasons, including the potential costs to pilots.
Therefore, the pilots are only reporting performance data that they would  routinely collect from their program participants, with the possibility of  reporting any new data they may be collecting for their local evaluations.  DOL officials also told us that they and their contractor for the national  evaluation are not yet thinking about the potential scalability of the various  flexibilities being tested by the pilots because it is too early in the  implementation period—in the first year of the 5-year pilots. They said  that the national evaluation is intended to review the flexibilities that each  pilot uses and determine what recommendations can be made to decision  makers, such as Congress, for broader application. Officials then plan to  combine the information on flexibilities with final outcome data and  evidence from the site-specific impact evaluations to assess and  synthesize, as appropriate, the individual pilots. DOL officials also said  that they plan to examine the partnership structures for the pilots to  determine the effectiveness of the structures. While the plans DOL  officials have laid out could, and likely will, provide useful information and  insights into the success of the individual pilots and overall initiative,  they  may not be collecting the data needed to inform conclusions about  scaling the flexibilities tested by the pilots without first determining the  criteria or standards for such an assessment. Going forward, such data  would better position Congress to decide whether and to what extent the  flexibilities tested by the pilots should be integrated into broader efforts.
section_title: Conclusions
paragraphs: Performance partnerships are part of a broader federal effort to align  federal grantmaking priorities with state and local government needs in  addressing key national objectives. One of the 2 existing partnership  initiatives—the multi-agency disconnected youth pilots—allows partners  to collaborate across organizational lines primarily to leverage  programmatic flexibilities that enable them to combine funding from  across several federal grant programs (and agencies) for interventions  aimed at improving outcomes among disconnected youth. In establishing  the disconnected youth pilots, federal agencies generally took actions  consistent with leading practices for interagency collaboration and pilot  design. For example, agencies established and documented key roles  and responsibilities, including those related to leadership, in an  interagency agreement. In addition, they established goals and objectives  for the individual pilots and overall initiative, and have plans to monitor  performance and evaluate results. However, agencies—including OMB,  which has responsibility for coordinating agencies’ overall efforts to  implement the disconnected youth pilots—have not fully identified the key  financial and staff resources each agency will need to contribute over the  lifetime of the initiative.  Doing so would help ensure they are able to  provide the support needed for successful implementation of the pilots  and to sustain their collaborative efforts. In addition, the agencies have  not identified criteria or standards to assess the scalability of the  flexibilities being tested by the pilots. Without them, agencies may not  collect the data they—and ultimately Congress—will need to determine  whether and how to implement successful approaches more broadly.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that the pilot programs for disconnected youth can be  effectively implemented over the lifetime of the initiative, the Director of  OMB should coordinate with relevant federal agencies to identify and  estimate expected annual financial and staff resource contributions from  each agency, including during the implementation and evaluation phases  of the pilots.
To ensure that federal agencies involved in the disconnected youth pilots  are able to evaluate pilot outcomes and ultimately communicate to  Congress whether and to what extent the flexibilities tested by the pilots  should be integrated into broader efforts, the Director of OMB should  coordinate with relevant federal agencies to identify criteria or standards  for assessing scalability, and collect data needed to address those criteria  or standards.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of the Office of  Management and Budget, Secretary of the Department of Education,  Secretary of the Department of Health and Human Services, Acting  Secretary of the Department of Housing and Urban Development,  Attorney General of the Department of Justice, Acting Secretary of the  Department of Labor, Chief Operating Officer of the Corporation for  National and Community Service, Administrator of the Environmental  Protection Agency, and Acting Director of the Institute of Museum and  Library Services for review and comment.
In comments provided by email, OMB’s Liaison to GAO stated that OMB  neither agreed nor disagreed with the recommendations in this report.  OMB staff also provided oral comments in which they asked us to clarify  that (1) OMB’s role is to coordinate agencies’ overall efforts to implement  the disconnected youth pilots, (2) the resource issues identified in our  report involve agencies better identifying and planning for their individual  contributions to the pilot initiative, and (3) our discussion of scalability is  focused on the flexibilities being tested by the pilots. We revised the  report accordingly to provide these clarifications.
DOL, Education, EPA, HHS, and IMLS provided technical comments,  which we incorporated as appropriate. CNCS, DOJ, and HUD informed  us that they had no comments.
We are sending copies of this report to the appropriate congressional  committees, the heads of each of the federal agencies included in this  review, and other interested parties. In addition, the report is available at  no charge on the GAO website at https://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or by email at bawdena@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) put into place a  framework intended to increase the use of performance information and  other evidence in federal decision making. GPRAMA  also requires us to  periodically report on how its implementation is affecting performance  management at federal agencies, including whether agencies are using  performance management to improve the efficiency and effectiveness of  their programs. According to the Office of Management and Budget  (OMB), because performance partnerships require federal agencies and  their grant recipients to manage toward agreed upon outcomes, they can  help the 2 sides to collect information and evidence about what works and  therefore how to employ federal resources more efficiently. We therefore  conducted this review under our GPRAMA  reporting requirement. There  are currently 2 sets of federal performance partnerships that have been  authorized. The Environmental Protection Agency’s (EPA) Performance  Partnership Grants (PPG) under its National Environmental Performance  Partnership System have been in place for 20 years. The multiple-agency  Performance Partnership Pilots for Disconnected Youth (disconnected  youth pilots) were authorized in January 2014. Over the past 3 years, 8  federal agencies—OMB; the Departments of Education (Education),  Health and Human Services (HHS), Housing and Urban Development  (HUD), Justice (DOJ), and Labor (DOL); the Corporation for National and  Community Service (CNCS); and the Institute for Museum and Library  Services (IMLS)—have  worked to implement the pilots. This report  identifies the key characteristics of those 2 existing performance  partnership initiatives. It also provides a more in-depth review of the  design, implementation, and evaluation of 1 of the 2—the Performance  Partnership Pilots for Disconnected Youth.
To address these objectives, we identified current federal performance  partnerships by reviewing relevant literature, including of our past work on  grants management and material produced by public sector and nonprofit  organizations; searching public laws for references to “performance  partnerships”; and interviewing former federal officials knowledgeable about the federal government’s efforts to develop and implement  performance partnerships. Based on this work, we determined that EPA’s  PPGs and the disconnected youth pilots were the only 2 federal  performance partnerships currently authorized.
To identify key characteristics of performance partnership initiatives and  how these key characteristics are exhibited, we collected, reviewed, and  analyzed documents about the overall performance partnership initiatives,  such as authorizing legislation, regulations, and notices inviting  applications, as well as from selected individual  performance partnerships  within them, including applications, performance partnership agreements,  and grant work plans.
To further illustrate how these key characteristics are exhibited by the 2  performance partnerships, we selected a non-generalizable sample of 4  states with EPA PPGs and 4 disconnected youth pilots for in-depth  review. To select a sample of states with PPGs, we used data that EPA  provided to us on PPG use in fiscal year 2014. We selected states based  on the number of grants they included in their PPG, the number of years  they had a PPG, and the EPA region in which they are located. We chose  1 state with a high total number of years in a PPG and a high total  number of grants in a PPG; a second with a high total number of years in  a PPG and a low total number of grants in a PPG; a third with a low total  number of years in a PPG and a high total number of grants in a PPG;  and a fourth with a low total number of years in a PPG with a low total  number of grants in a PPG. We defined high and low as being within the  top or bottom quartile of states in each category. We ensured that each of  our selections was from a different EPA region since variations in  management practices and general environmental needs across EPA  regions may impact how states use PPGs. The states we selected based  on these criteria were Utah, New York, Alabama, and California.
To select a sample of disconnected youth pilots, we considered pilots with  various locations (urban, rural, tribal), consulting agencies, and federal  programs used. First, we grouped pilots by their urban, rural, and tribal  designations to ensure that we could examine pilots in different  environments. We selected the only 2 pilots designated as rural and  tribal—Eastern Kentucky and Ysleta del Sur Pueblo. Next, we grouped  pilots by the consulting agencies responsible for overseeing them to  ensure that we could examine whether pilot oversight differs among  federal agencies. Four agencies serve as consulting agencies— Education, DOL, HHS, and CNCS. Education and CNCS serve as the  consulting agencies for Eastern Kentucky and Ysleta del Sur Pueblo,  respectively, so we then selected Chicago from the group of urban pilots  left for selection since it is the only pilot for which HHS serves as the  consulting agency. Finally, we looked to ensure that each federal  agency’s programs were being used in at least 1 of our selected pilots.  Since all of the agencies had at least 1 program being used in our first 3  selections, we selected Oklahoma as our final pilot, as DOL is its  consulting federal agency and it is using programs from 4 of the 5  agencies.
To obtain perspectives on the key characteristics of these performance  partnership initiatives, including reporting benefits and challenges they  may present, we conducted semi-structured interviews with officials  involved in them. For PPGs, we met with relevant officials at EPA  headquarters and regional offices, as well as officials from state  environmental agencies in the 4 selected states. These included officials  from EPA’s Office of Congressional and Intergovernmental Affairs; EPA  Regions 2, 4, 8, and 9; and the Alabama Department of Environmental  Management, the California Department of Pesticide Regulation, the New  York State Department of Environmental Conservation, and the Utah  Department of Environmental Quality. For the disconnected youth pilots,  we conducted semi-structured interviews with officials from each of the  federal agencies currently involved in them—OMB, Education, HHS,  HUD, DOJ, DOL, CNCS, and IMLS. We also conducted semi-structured  interviews with representatives from a federal and non-federal partner for  each of the individual partnerships. For the disconnected youth pilots,  these included Education and Partners for Education at Berea College  (Eastern Kentucky); CNCS and the Ysleta del Sur Pueblo tribal  government (Ysleta del Sur Pueblo); HHS and the Chicago Department of  Family Support Services (Chicago); and DOL and the Oklahoma  Department of Human Services (Oklahoma).
To assess federal agencies’ efforts to design, implement, and evaluate  the disconnected youth pilots, we obtained and reviewed key documents,  including requests for information and public comment, preliminary design  papers, interagency agreements, and evaluation plans. In addition, we  interviewed officials from OMB, Education, HHS, HUD, DOJ, DOL,  CNCS, and IMLS  about their collaboration and design of the pilot  initiative. We then assessed agencies’ efforts in these areas against  leading practices to determine the extent to which these efforts reflect  leading practices for interagency collaboration and effective pilot design.  These leading practices were developed in our prior work.
We conducted this performance audit from July 2015 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Additional Information on the Environmental Protection Agency’s Performance Partnership Grants
paragraphs: The Environmental Protection Agency’s (EPA) performance partnership  grants (PPG) permit state agencies to request that funding they receive  from 2 or more EPA program grants be combined into a single award.  This is intended to enable state agencies to, among other things, consider  trade-offs across the breadth of their environmental program funding and  exercise flexibility to direct resources to their most pressing priorities.  Specifically, once a state agency has requested and received selected  EPA program grants in a PPG, it can choose to use the funds to support  any activity that is eligible under at least 1 of the grants included in the  PPG. PPGs also streamline administrative requirements so that states  agencies can realize cost savings through reduced administrative burden  in areas such as grant applications, cost sharing, and financial reporting.
The number of grants that states request to include in PPGs varies. For  example, in fiscal year 2016, the 39 states that had PPGs managed by  their environmental or health agency included between 2 to 16 grants in  their PPGs, according to EPA data. Approximately half of these agencies  included 6 or fewer grants in their PPGs, with the other half including 6 or  more.
To illustrate variation in how state agencies chose to receive combined  EPA program grant funding in PPGs, we selected to review 4 in greater  depth that combined either a small or a large number of grants in their  PPGs. On the small end, we selected New York, whose Department of  Environmental Conservation received funding from 3 grants (illustrated in  figure 9), and California, whose Department of Pesticide Regulation  received funding from 4 grants in 1 environmental area (water and  pesticides, respectively, illustrated in figure 10) in their PPGs in fiscal year  2016.
On the large end, we selected Alabama, whose Department of  Environmental Management received funding from 9 grants (illustrated in  figure 11 below), and Utah, whose Department of Environmental Quality  received funding from 10 grants across multiple environmental program  areas in their PPGs in fiscal year 2016 (previously illustrated in figure 6).
section_title: Appendix III: Additional Information on the Multi-Agency Performance Partnership Pilots for Disconnected Youth
paragraphs: The multi-agency disconnected youth pilots enable non-federal partners  to combine funds from federal agencies’ programs and obtain additional  flexibilities to tailor service interventions to the specific needs of their  target populations. Three rounds of pilots have been authorized, and the  first round of 9 pilots approved is underway. Specifically, the  authorization for the first round pilots allows for the combining of  discretionary funding received by the Departments of Education  (Education), Health and Human Services (HHS), and Labor (DOL), as  well as the Corporation for National and Community Service (CNCS) and  the Institute of Museum and Library Services (IMLS)  through the fiscal  year 2014 appropriations act. In the notice inviting applications for first  round pilots, applicants were advised that where a program’s funds are  not suitable for combining (referred to as “blending”) in a pilot given the  constraints under the pilot authorization, the applicant may nevertheless  consider how to coordinate (referred to as “braiding”) such funding in a  pilot to promote more effective and efficient outcomes even though the  funds would maintain a separate identity and remain subject to the  program requirements for which the funds were appropriated.
In addition, for the first pilot round federal agencies awarded separate  start-up grants to provide funding of up to $700,000 to each pilot to  finance evaluations, capacity building, technical assistance and other  related activities to support the pilot. According to officials from OMB,  funds available for these purposes from CNCS, DOL and Education, were  used for these start-up grants.
The first round pilots vary in the extent to which they are using combined  federal funds in their pilots. Our selection of case study pilots also reflects  this variation, as noted below. Two of the 9 pilots are combining all  federal funds from all of the federal programs they are including in their  pilots. For example, the Ysleta del Sur Pueblo pilot is consolidating  funding from CNCS’s AmeriCorps program and IMLS’s Native American  Library Services Enhancement Grants program, as illustrated in figure 12.
According to Ysleta del Sur Pueblo’s pilot application, while the tribe has  had success in using the CNCS and IMLS programs individually  to meet  short-term outcomes for tribal youth, such as cultural and traditional  engagement and enhanced knowledge of the services and programs the  tribe offers, it has been unable to use the programs to provide a  continuum of services to help tribal youth meet longer-term goals, such as  staying in school, enrolling in post-secondary education, and obtaining  gainful employment. The application states that the tribe’s ability to  consolidate the 2 programs’ funding will allow its Empowerment and  Economic Development Departments to better collaborate and more  effectively provide youth services in a single, wraparound program. In  addition, officials from Ysleta del Sur Pueblo told us that the start-up grant  the tribe received will further enhance its ability to implement its pilot. The  tribe’s application stated that, among other things, the funds will help the  tribe to hire staff and support the evaluation.
Five pilots are not using any combined funding but are instead  coordinating the use of federal funds from multiple federal programs.  These include the Oklahoma and Chicago pilots we selected to review in- depth. For example, the Oklahoma pilot is coordinating funds from 3  federal programs: DOL’s WIOA Title I Youth program, HHS’s Now Is the  Time: Healthy Transitions program, and CNCS’s AmeriCorps program.  Figure 13 illustrates the structure of the Oklahoma pilot.
Officials from Oklahoma DHS told us that the non-federal partners  involved in the pilot expressed a preference to manage and report on the  use of their federal funds independently. They told us that the  organizations determined that they would be able to operate the pilot as  intended by coordinating their use of funds. They had no prior experience  combining funds across organizations, and Oklahoma DHS, as the pilot  lead, did not want to track other organizations’ funds. Oklahoma DHS  officials also told us that start-up grants have been beneficial in allowing  the non-federal partners to hire a staff member to oversee pilot  operations, support collaboration among partners, and prepare for and  conduct an evaluation of the pilot.
The structure of the Chicago pilot, illustrated in figure 8, along with a  discussion of challenges federal and non-federal officials faced in coming  to agreement on the use of funds for that pilot, were previously described  above.
The remaining 2 pilots are combining federal funds from some of the  federal programs included in their pilots and coordinating the use of funds  from other federal programs. One of them is the Eastern Kentucky pilot,  and details about its use of federal funds were discussed earlier in the  report and illustrated in figure 7. Officials from a pilot partner told us that  combining some funds allows them to hire staff who are not tied to a  specific program and therefore can work across the programs involved in  the pilot. In addition, the officials added that the start-up grant the pilot  received will allow the non-federal partners to establish a data collection  system to share information about the participants across partners. This  would allow them to use those data to identify effective strategies and  support the evaluation of the pilot.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above contact, Benjamin T. Licht (Assistant Director)  and Daniel Webb (Analyst-in-Charge) supervised this review and the  development of the resulting report. Mitchell Cole, Karin Fangman, Mike  Grogan, John Hussey, Sherrice Kerns, Ifunanya  Nwokedi, Michelle  Sager, Cindy Saunders, and Stephanie Shipman made significant  contributions to this report. Donna Miller developed the graphics for this  report. Theodore Alexander,  Crystal Bernard, Kathleen Drennan, Shirley  Hwang, Adam Miles, Keith O’Brien, Laurel Plume, Erik Shive, and Wes  Sholtes verified the information in this report.