text: Background Program Fragmentation, Overlap, and Duplication Over the years, we have issued several reports on fragmentation, overlap, and potential for duplication among federally funded employment and training (E&T) programs and identified areas where inefficiencies might result. This report, like our prior work, uses the following definitions: Fragmentation refers to circumstances in which more than one federal agency (or more than one organization within an agency) is involved in the same broad area of national need and opportunities exist to improve service delivery. Overlap occurs when multiple agencies or programs have similar goals, engage in similar activities or strategies to achieve their goals, or target similar beneficiaries. Duplication occurs when two or more agencies or programs are engaged in the same activities or provide the same services to the same beneficiaries. During the 1990s, we issued a series of reports that documented program overlap among federally funded E&T programs and identified areas where inefficiencies might result. For example, we found that program overlap might hinder people from seeking assistance and frustrate employers and program administrators. In 2000 and 2003, we identified federally funded E&T programs for which a key program goal was providing E&T assistance. In our most recent report in 2011, we identified 47 E&T programs and found that 44 of them overlapped with at least one other program in that they provided similar services to a similar population. We also found that due to the American Recovery and Reinvestment Act of 2009 (Recovery Act), both the number of—and funding for—federal E&T programs had increased since our 2003 report, but little was known about the effectiveness of most programs because only five programs had conducted impact evaluations. Our guide on identifying and reducing fragmentation, overlap, and duplication notes that determining whether fragmentation and overlap exist among programs is a key step in identifying opportunities to improve efficiency and effectiveness of programs. In some cases, it may be appropriate or beneficial for multiple agencies and programs to be involved in the same programmatic or policy area due to the complex nature or magnitude of the federal effort. However, our guide states that it is also important to use the results of existing or new evaluations of identified programs to assess options to reduce or better manage negative effects of fragmentation, overlap, and duplication, such as inefficient use of program funds. Key Changes Since Our 2011 Review of Employment and Training Programs Workforce Innovation and Opportunity Act (WIOA) Enacted in July 2014, WIOA repealed and replaced the Workforce Investment Act of 1998 (WIA). WIOA placed greater emphasis on aligning and integrating workforce programs, which are administered primarily by the Departments of Labor (DOL) and Education (Education), with support from the Department of Health and Human Services (HHS) and other agencies. For example, under WIOA, DOL and Education review and approve 4-year strategic plans for states’ workforce development systems. WIOA also requires certain programs and encourages other programs to be available through centralized service delivery points referred to as American Job Centers. In addition, WIOA requires that DOL and Education collaborate to implement a common performance accountability system for six core programs, which presents agencies with an opportunity to align definitions, streamline performance indicators, and integrate reporting across these programs. Economic Conditions Since our 2011 inventory of federal E&T programs, which focused on fiscal year 2009, both the Great Recession and one-time funding made available under the Recovery Act have ended. Recovery Act funds were provided to help preserve and create jobs and promote economic recovery, among other purposes. With the end of the recession, the unemployment rate has substantially declined. The rate increased from 4.6 in 2007 to a peak of 9.6 in 2010 before declining to 4.4 in 2017 (see fig.1). Evaluation Plans WIOA encourages DOL, Education, HHS, and other relevant federal agencies to conduct program research and evaluation. For example, WIOA requires DOL to publish a plan every 2 years that describes the research, studies, and multistate project priorities of DOL concerning employment and training for the following 5-year period. This includes a provision that the plan be consistent with certain purposes, including the purpose of aligning and coordinating core programs with other partner programs provided through American Job Centers. In addition to WIOA requirements, we have also previously reported that each federal agency should require its major program components to prepare annual and multiyear evaluation plans and to update these plans annually. The planning should take into account the need for evaluation results to inform program budgeting, reauthorization, agency strategic plans, program management, and responses to critical issues concerning program effectiveness. These plans should include an appropriate mix of short- and long-term studies to produce results for short- or long-term policy or management decisions. To the extent practical, the plans should be developed in consultation with program stakeholders. Furthermore, leading organizations, including the American Evaluation Association and the National Academy of Sciences, emphasize the need for research programs to establish specific policies and procedures to guide research activities. In addition to planning for formal evaluation, Standards for Internal Control in the Federal Government emphasize the importance of managers routinely assessing the results of their actions, for which evaluation is a potential tool. Since 2011, Employment and Training Services Are Delivered through Fewer Federal Programs and with Reduced Obligations Number of Employment and Training Programs Declined, Due in Part to Eliminations The number of federal E&T programs has decreased since our last report on them in 2011. For fiscal year 2017, we identified 43 programs, four fewer than we reported in 2011. The number decreased because more programs were eliminated or defunded (6) than added (2). For example, in 2014, the Workforce Innovation and Opportunity Act (WIOA) eliminated at least four of our identified E&T programs. This included 1) DOL’s Veterans’ Workforce Investment Program, 2) Education’s Grants to States for Workplace and Community Transition Training for Incarcerated Individuals, 3) Education’s Migrant and Seasonal Farmworkers Program, and 4) Education’s Projects with Industry program. In addition, Congress did not appropriate funds for Education’s Tech Prep Education State Grants in fiscal year 2011 and DOL’s Community Based Job Training Grants programs in fiscal year 2010, according to agencies’ budget documents. We also identified two additional E&T programs through interviews with agency officials and a related GAO report: 1) Department of Veterans Affairs’ (VA) Compensated Work Therapy, and 2) Department of Defense’s (DOD) Job Training, Employment Skills Training, Apprenticeships, and Internships. For changes in the program list from our 2011 review to our current review, see appendix II. The 43 programs we identified in fiscal year 2017 are fragmented across nine federal agencies, as programs were in 2011 (see fig. 2). Federal Obligations for Employment and Training Programs Decreased, Due in Part to the End of Recovery Act Funding Our survey results showed that the federal government obligated nearly $14 billion to the E&T components of its programs in fiscal year 2017, a decrease of about $5.4 billion or 30 percent, adjusting for inflation, from the amount in our 2011 review (which reported fiscal year 2009 obligations). According to our analysis of survey data, much of the decrease in E&T obligations can be explained by the expiration of Recovery Act funding. For example, two-thirds of the Recovery Act funding designated for E&T programs went to four DOL programs that received a combined $3.8 billion in Recovery Act appropriations. From fiscal year 2009 to fiscal year 2017, the combined E&T obligations for these four programs decreased by $4.7 billion, or 58 percent. Of the 31 E&T programs that reported E&T obligations in our survey, eight programs were responsible for more than $11 billion, or 82 percent of the total in fiscal year 2017. Their shares of 2017 E&T obligations ranged from 5 percent for DOL’s Wagner-Peyser Act Employment Service to 21 percent for Education’s State Vocational Rehabilitation Services Program (see fig. 3). Among these eight programs responsible for the vast majority of E&T obligations, all must be included in state plans required under WIOA, except for DOL’s Job Corps, VA’s Vocational Rehabilitation and Employment, and HHS’ Temporary Assistance for Needy Families (TANF). In addition, all but DOL’s Job Corps and VA’s Vocational Rehabilitation and Employment are state-administered. For complete data on reported changes in E&T obligations between fiscal years 2009 and 2017, for the 29 programs that provided estimates in both years, see appendix III for numbers adjusted for inflation and appendix IV for unadjusted numbers. The number of people served by E&T programs also declined, from 24 million to 11 million individuals in the most recent year for which data were available, or a 56 percent decrease from the number reported in the 2011 report. Two of DOL’s E&T programs—the Wagner-Peyser Act Employment Service and the WIOA Adult Program—accounted for the majority of this decrease, dropping by 8 million and 4 million, respectively. Participation in certain programs, for example, Wagner-Peyser Act Employment Service and WIOA Adult Program, changed markedly as the economy improved, suggesting that enrollment is highly sensitive to economic conditions. Since we last reviewed these programs in 2011, the U.S. economy has improved and the unemployment rate dropped by 53 percent (see fig. 4). DOL officials said these factors could have reduced the demand for certain E&T services. Unemployment is an important driver of demand for some, but not all, E&T programs. For example, demand for certain employment and training services, such as vocational rehabilitation, may be relatively insensitive to economic conditions. In addition, technology has the potential to change workforce needs in certain industries, leading to workers who need retraining. In addition, DOL officials told us that under WIOA a new definition of program participant, effective in 2016, that primarily impacted the number of participants reported for Wagner- Peyser Act Employment Service, WIOA Adult Program, and WIOA National Dislocated Worker Grants. Employment and Training Programs Administered by Various Agencies Generally Overlap, but Effects of Overlap May Vary The 43 E&T programs generally overlap in that they provide similar services to similar populations, according to our survey analysis (see table 1). In our survey, almost all of the 43 programs reported providing employment counseling and assessment services as well as job search or job placement activities (39), job readiness training (38), and job referrals (37). The least commonly provided service selected from our list of service categories–high school completion or equivalency assistance–was provided by over half (26) of the programs. Through our survey, eight of the 43 programs reported serving the general population (that is, a relatively broad target) and the remaining 35 reported serving a narrower target population, such as Native Americans (8), veterans and transitioning servicemembers (7), or youth (5)., Our survey analysis shows overlap in services exists among programs serving the general population as well as among those serving each specific target population. Specifically, a majority of programs targeting the general population, Native Americans, and youth reported providing many of the same services. For example, all of the five youth programs reported providing similar E&T services, such as employment counseling and assessment and job readiness training (see fig. 5). For more information on services provided by programs serving selected target populations, see appendix VI. Many of the E&T programs targeting specific populations are fragmented across multiple agencies. For example, four agencies administer the eight Native American E&T programs and three administer the seven programs for veterans (see table 2). Other includes older workers, women, and unemployed and underemployed residents of solid and hazardous waste-impacted neighborhoods. According to VA officials, VA’s Vocational Rehabilitation and Employment program serves individuals with a service connected disability and VA’s Compensated Work Therapy program serves individuals enrolled in Veterans Health Care and does not require a service connected disability. Overlap among program services may have benefits, but it may also suggest opportunities for coordination or efficiencies in service delivery. Overlap may be beneficial in 1) helping program participants with specific needs better access E&T services, 2) providing more tailored or intensive support services, or 3) achieving higher quality outcomes for specific populations than would be achievable from their use of a more broadly targeted program. For example: A 2015 study funded by DOL on services provided to veterans through the public workforce system in Texas found that veterans who received intensive services from DOL’s Disabled Veterans’ Outreach Program Specialist or Local Veterans’ Employment Representative staff subsequently had higher earnings than veterans who did not, although these same veterans may have been eligible for similar services provided by other programs to the general population. A 2017 study funded by the U.S. Department of Agriculture (USDA) on its Supplemental Nutrition Assistance Program (SNAP) E&T— which helps participants who are eligible to receive nutrition assistance from the federal government better access E&T services —found that program participants also received support services, such as child care vouchers and transportation assistance. Participants said these services were important to their participation in the E&T program and helped those with specific needs better access E&T services. However, when multiple programs overlap or are fragmented, there is also a risk that program administrators may not make efficient use of available resources if they do not coordinate their efforts. Without careful coordination, programs may not fully leverage mutual benefits or participants may find administrative requirements burdensome or redundant. For example: A 2018 GAO report on USDA’s SNAP E&T program found that 20 states’ SNAP E&T programs did not partner with workforce agencies to provide E&T services. States that do not fully leverage resources available through the workforce development system may miss opportunities to serve a greater number of SNAP E&T participants and provide a wider variety of services. GAO recommended the administrator of the Food and Nutrition Service take additional steps to assist states in leveraging available workforce development system resources. A 2017 study funded by DOL on American Job Centers found that customers became frustrated filling out applications in what they viewed as redundant paperwork requirements for multiple programs with varying eligibility criteria. Almost All Agencies Reported Actions to Address Program Fragmentation and Overlap, but Effectiveness of these Actions Remains Uncertain Employment and Training Program Officials Reported Taking Actions to Address Fragmentation and Overlap In response to our survey of agency officials for the 43 E&T programs, almost all (38) reported taking at least one action to manage fragmentation, overlap, and/or potential duplication. Common actions included providing program guidance and technical assistance, coordinating participant services (e.g., co-locating services or co-enrolling participants), and effectively managing grants (see table 3). Our survey analysis showed that of 43 E&T programs, 31 across eight agencies reported taking at least one action to manage fragmentation. In addition, 38 programs across all nine agencies reported taking at least one action to manage overlap. For example, to address fragmentation and overlap, officials representing seven programs within DOL and Education reported in our survey that they participated in interagency workgroups to share information and to facilitate cross-agency communication to coordinate services. Likewise, VA reported that the agency and DOL updated their interagency technical assistance guide to better align the agencies’ veteran E&T programs. (See table 4.) Program officials reported that their actions were motivated by a variety of factors, including their own assessments, legal requirements such as those in WIOA, and audit recommendations. They attributed some of their actions to their assessment of the potential for duplicative services, or to promote streamlined administration. For example: In 2014, DOL released updated guidance to administrators of its Disabled Veterans’ Outreach Program to encourage coordination with its Wagner-Peyser Act Employment Service program to help ensure that the two programs were not providing similar services to veterans. Education reported that its data collection and reporting system integrates data from the State Vocational Rehabilitation Services Program and State Supported Employment Services Programs. Likewise, Education reported that its monitoring and technical assistance guide addresses both the State Vocational Rehabilitation Services Program and the State Supported Employment Services Program. In addition, DOL and other agencies reported taking actions that are either required or encouraged by federal law in order to manage fragmentation and overlap. For example: DOL officials reported that since WIOA was enacted in 2014, DOL, Education, and HHS have jointly issued directives and guidance to help states implement and administer WIOA, such as guidance on developing their required state strategic plans. Also under WIOA, DOL and Education have issued joint regulations and established common data definitions and joint data collection instruments to align performance reporting for WIOA six core programs. Agencies with E&T programs targeted toward Native Americans reported that tribes’ use of authorized plans to integrate employment, training, and related services programs can help manage fragmentation and overlap. The potential scope of such plans (referred to as 477 plans), which had been originally authorized in 1992, was increased via legislation in 2017 to include programs with more purposes. With an authorized plan in place, tribes can integrate certain federal funds received by the tribe and coordinate employment, training, and related services across multiple programs that serve the tribe. In December 2018, 12 agencies signed a memorandum of agreement intended to set forth the basic functions and relationships of those agencies in the funding and oversight of tribal 477 plans and to facilitate coordination and collaboration between the agencies. Agencies have also taken actions to improve collaboration across multiple E&T programs based on our recommendations or on internal audits. For example: In 2011, we recommended that the Secretaries of DOL and HHS work together to develop and disseminate information that could facilitate further progress by states and localities in increasing administrative efficiencies in E&T programs, such as state initiatives to consolidate program administrative structures and state and local efforts to co- locate E&T programs at one-stop centers. In response, DOL and HHS took a number of steps, including issuing a January 2015 study focused on identifying and documenting potentially promising practices in coordinating Temporary Assistance for Needy Families (TANF) and WIA services at the state and local levels. In 2012, we found that the interagency handbook used by DOL and VA to coordinate E&T services for veterans did not include, for example, incorporating labor market information into rehabilitation plans. In 2015, as GAO recommended, these agencies revised the interagency handbook by outlining how VA and DOL staff should coordinate efforts to provide veterans with labor market information when developing employment and training objectives and assist them in selecting training and credentialing opportunities as a part of their rehabilitation plans. In 2012, EPA’s Office of Inspector General conducted an audit of its Environmental Workforce Development and Job Training Cooperative Agreements program which concluded that, absent internal controls, the program was at risk for duplication with other E&T programs. To mitigate that risk, the lead program administrator now provides other federal agencies a list of program applicants to ensure that no applicant is receiving funds for the same purposes outlined in the Environmental Workforce Development and Job Training program application. While most programs reported taking action to manage fragmentation or overlap, officials from five programs reported in our survey that they had taken no action. Officials from four of these programs reported that no action was necessary because their program offered a unique service or served a specialized population. While we did not further review the need for coordination among these programs and others, they nonetheless reported one or more services in common with others serving the same population. In addition, while unique aspects may be protective to some extent against the risk of duplication, unique features may not necessarily reduce the risk of overlap or need for coordination. For example, DOD officials stated that apart from its Job Training, Employment Skills Training, Apprenticeships, and Internships program, they were not aware of any other federal program that allows servicemembers to participate in job training, including apprenticeships and internships, beginning up to 6 months before their service obligation is completed. DOL officials confirmed that its Transition Assistance Program does not offer job training to service members, but it does, like the DOD program, offer pre-separation employment services and counseling. VA also noted in its technical comments that servicemembers who meet Vocational Rehabilitation and Employment eligibility criteria may, with DOD permission, receive these job training services as part of their rehabilitative program and that it partners with DOD to train transitioning servicemembers as veterans’ services representatives. We did not further review the need for coordination among these or other programs that reported no action, but absent a more complete evaluation, it is not possible to assess whether these programs have taken sufficient steps to address overlap. Regarding duplication, 14 programs reported no action either to detect it or to prevent it. Agencies Did Not Consistently Assess the Effectiveness of their Actions to Manage Overlap and Fragmentation Agencies administering E&T programs did not consistently have information on results to know how well their actions to manage program fragmentation and overlap were working. DOL officials told us that they generally had not assessed the actions they reported in our survey to manage overlap, fragmentation, and potential for duplication, but noted that the agency has begun an implementation study of WIOA that will include examining state and local efforts to increase program coordination and collaboration. DOL expects the final report will be completed in fall 2019, and agency officials said it is coordinating with other agency partners. Asked about efforts made by specific programs to manage overlap and fragmentation, other agency officials said they had assessed results of these efforts in some cases, but not others. For example, VA officials told us that in 2016 they started tracking referrals between its Vocational Rehabilitation and Employment Program and DOL’s programs targeted to veterans to help ensure participants were obtaining labor market information from DOL programs. In contrast, in the case of integrating multiple E&T programs targeted toward Native Americans, HHS officials reported that the agency has not made specific efforts to assess the effectiveness of plans first provided for in 1992 which might reduce administrative burden by allowing tribes more flexibility to combine E&T services funded by multiple federal agencies. GAO’s guide on fragmentation and overlap states it is important to use the results of existing or new evaluations of identified programs to assess options to reduce or better manage negative effects of fragmentation, overlap, and duplication, such as inefficient use of program funds. For example, evaluation and other periodic reviews could help identify ways to address (1) gaps in information on how multiple programs are serving the employment and training needs of specific populations, such as Native Americans, youth, and refugees, or (2) the extent to which they have implemented practices to manage unwanted effects of fragmentation and overlap and improve coordination and efficiency. Agencies reported completing additional impact studies since our 2011 review, but evaluations examining their programs’ effects have generally been confined to a single program and/or specific target populations. Four of the nine agencies in our review reported that they had completed at least 13 impact studies since 2011 of individual programs that measured effectiveness in terms of outputs and outcomes. (See appendix VII for a list of these studies.) DOL officials told us that programs tend to be evaluated individually for their effectiveness in achieving individual goals and objectives rather than for collective effects or performance. DOL officials said that they perform some research covering multiple programs in preparation for conducting program impact or effectiveness studies, but that the related findings tend to be more descriptive in nature. They also cited plans to use common measures developed under WIOA to look at outcomes across the core programs. Some agencies have sponsored studies that focus on populations served by multiple programs, including customer experience with receiving services from multiple programs, and an early snapshot of the extent of state-level coordination in implementing WIOA. Specific examples of studies that reviewed issues related to implementing multiple programs include: A 2015 Mathematica study funded by HHS of WIOA-funded programs that included numerous efforts state level administrators could undertake to improve coordination among the programs, including exchanging more information on strategies and methods used by each program to address obstacles that impede coordination. A 2015 Rand Corporation study funded by DOD that examined employment support programs for reservists and recommended assessing the costs and benefits of streamlining the current program line-up to reduce any redundancies. A 2017 study by IMPAQ International funded by DOL that identified areas where customer service in WIOA job centers could be improved, such as streamlining enrollment and registration procedures and providing more information about the full array of services at the centers. However, of the six completed studies we identified that examined more than one E&T program, only one study assessed how any coordinated or integrated activities benefited the population served. We found no similar studies conducted on the effects of multiple programs targeted toward other populations, such as Native Americans, youth, or refugees. VA officials told us that it is important that reviews of E&T programs for specific population take into account the complex needs of that population to understand when there is a need for involvement of multiple programs. For example, officials said that special populations such as homeless veterans require a breadth of unique services that may not available through a single program or by programs serving the general population. Further, as programs more commonly work together, learning about the programs’ collective impact may be as important as studying the programs’ individual results. DOL officials told us that DOL, HHS, and Education tend to independently create their evaluation plans for employment and training services. After WIOA was enacted, these agencies formed the WIOA Evaluation Workgroup with the intent of establishing greater collaboration among federal agencies on E&T program evaluation. DOL E&T programs make up over a third of all federal E&T programs, and some of these programs under WIOA coordinate or align their services with programs administered by other agencies. DOL officials told us WIOA Evaluation Workgroup members interacted with staff from other agencies, such as USDA, who administered E&T programs to encourage their participation. The workgroup met for the first time in September 2017. After the initial meeting, according to DOL officials, the agencies dissolved the group because they concluded that the topic of WIOA-related evaluation could be covered through existing periodic interagency meetings. However, DOL officials told us that these efforts do not focus on evaluation across programs. In addition, the DOL agency-wide evaluation plan for fiscal year 2018—issued in September 2018—does not list evaluations focused primarily on cross-program coordination or collaboration, nor does it address potential overlap and fragmentation among its E&T services. Since 2013, DOL has not published a 5-year strategic research plan for E&T programs. In our 2011 review of DOL’s research and evaluation program for its E&T programs, we recommended that DOL develop a mechanism to enhance the transparency and accountability of its E&T research by consulting other key federal agencies and involving advisory bodies or other entities outside DOL. In 2010, the Employment and Training Administration (ETA), the division with lead responsibility for DOL’s E&T programs, began a series of meetings with a panel of outside experts to develop a 5-year research plan. This strategic research plan set the research agenda for E&T programs by identifying and prioritizing what research and evaluations would be initiated over the following 5 years. Before finalizing its research agenda, DOL obtained broad input from federal officials at Education and HHS and a range of other key stakeholders, such as officials in local and state government and academics from the workforce community. In May 2013, DOL submitted to Congress and posted on its website a 5-year strategic research plan for its E&T programs which covered program years 2012 to 2017. In contrast to the broad consultation and public exposure that characterized past strategic planning for E&T research, in recent years DOL has instead relied on an internal process to set its research and evaluation priorities for its E&T programs and publishes only an agency- wide evaluation plan that is shorter-term and developed for a different purpose. Specifically, ETA develops an annual learning agenda that officials indicated highlights its research priorities, ideas, and proposed studies. Officials stated that the E&T learning agenda is provided for consideration with other agency-wide agendas in developing an annual evaluation plan for all of DOL. While DOL’s annual evaluation plan and the results of its evaluations are posted publicly through its website and submitted to the relevant congressional committees, the learning agendas, including those for E&T programs, are internal documents, and DOL does not release them to the public. The DOL-wide evaluation plan that is published presents neither a strategy for E&T evaluation nor plans for any evaluation to be initiated more than a year in the future. The fiscal year 2018 DOL-wide evaluation plan discusses only research to be initiated during the next year (fiscal year 2019) and lists studies that remain in progress from previous years. Rather than project longer-term research needs, the plan’s main purpose, according to DOL officials, is to comply with specific appropriations language. DOL officials told us that the list of proposed studies in the learning agendas may not ultimately appear in the annual evaluation plan because they are not near-term priorities for the agency-wide plan. DOL’s fiscal year 2018 agency-wide plan describes initiation of four studies—two on apprenticeship, one on strategies to prevent improper unemployment insurance payments, and another on potential effects of application fees for certain ETA programs. WIOA requires that DOL publish a plan every 2 years that describes “the research, studies, and multistate project priorities of the Department of Labor concerning employment and training for the 5-year period following the submission of the plan.” DOL officials told us that it is complying with this requirement by providing ETA’s annual learning agendas to be included in DOL’s overall evaluation plan. However, the resulting agency- wide plan falls short of meeting best practices for robust strategic planning. As we have previously reported, these practices include: Preparing annual and multiyear evaluation plans and updating these plans annually to take into account the need for evaluation results to inform program budgeting, reauthorization, agency strategic plans, program management, and responses to critical issues concerning program effectiveness. Including an appropriate mix of short- and long-term studies to produce results for short- or long-term policy or management decisions. Developing plans in consultation with program stakeholders to help agencies ensure that their efforts and resources are targeted at the highest priorities and to create a basic understanding among the stakeholders of the competing demands that confront most agencies. A 2010 internal DOL memo stated that such a plan can guide the development of research and evaluation projects and be a valuable tool for the broader workforce research community. Furthermore, leading organizations, including the American Evaluation Association and the National Academy of Sciences, emphasize the need for research programs to establish specific policies and procedures to guide research activities. For example, a 2016 American Evaluation Association guide stated that having annual and multi-year evaluation plans is useful in guiding program decision-making in such areas as program management and budgeting, and responding to issues concerning program effectiveness. Finally, Standards for Internal Control in the Federal Government state more broadly that program managers may need to conduct periodic assessments to evaluate the effectiveness of their actions. These may include but are not limited to formal evaluations. However, without a long-term evaluation plan developed in consultation with key stakeholders, DOL may not learn whether its actions to improve E&T program coordination and integration are working, and thus may continue undertaking activities that are not leading to desired results. Conclusions With the enactment of WIOA in 2014, steps were taken toward aligning employment and training programs and ensuring greater cross-agency coordination. Since then, agencies and programs have reported taking a range of actions to increase coordination among E&T programs and manage fragmentation and overlap. However, without knowing whether these actions are working to improve program coordination and integration, agencies may persist in activities that are ineffective, fail to expand those that work, or ignore unintended consequences. Further, the lack of evaluation focused on program coordination has resulted in a void of information on programs’ collective impact. Without strategically planning the use of evaluation resources, DOL and other agencies will not learn efficiently about whether their efforts to coordinate the programs have been successful and what impact the newly coordinated programs are having, collectively, on their shared objectives. Recommendation for Executive Action We are making the following recommendation to DOL: The Secretary of DOL should develop and publish a multi-year strategic research plan for evaluation of its employment and training programs that includes assessing the completeness and results of efforts to coordinate among E&T programs to address overlap and fragmentation. In developing this plan, DOL should also consult with other federal agencies and key stakeholders on ways to address gaps in information on how multiple programs are serving the employment and training needs of specific populations, such as Native Americans, youth, and refugees. (Recommendation 1) Agency Comments and Our Evaluation We provided a draft of this report for review and comment to the Departments of Agriculture, Defense, Education, Health and Human Services, Interior, Justice, Labor, and Veterans Affairs, and to the Environmental Protection Agency. We received formal written comments from DOL and VA that are reproduced in appendix VIII and IX. In addition, DOL, Education, HHS, Interior, USDA and VA provided technical comments which we incorporated into the report as appropriate. EPA, DOD, and DOJ did not have any comments. DOL agreed with our recommendation that it develop and publish a multi- year strategic research plan for evaluation of its E&T programs consistent with the purpose of aligning and coordinating these programs. DOL stated that it actively plans and makes public the research and evaluation topics for these evaluations, but it did not identify a timeline or measures it would take to augment these basic steps. We recommended that DOL consult with other federal agencies and key stakeholders in developing a strategic research plan that assesses the completeness and results of efforts to coordinate among E&T programs to address overlap and fragmentation. Consultation should include ways to address gaps in information on how multiple programs are serving the employment and training needs of specific populations, such as Native Americans, youth, and refugees. DOL stated that it will consult with stakeholders regarding the employment and training needs of specific populations. VA commented that such reviews of E&T programs for specific populations should take into account the complex needs of the population being served and the breadth of needed services. We agree that any such reviews should address how the collection of programs is serving each population’s needs. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix X. Appendix I: Objectives, Scope, and Methodology This appendix discusses our scope and methodology for our three research objectives examining (1) how participation in and obligations for federal employment and training programs have changed since our 2011 report, (2) the extent to which employment and training programs continue to provide similar services to similar populations, and examples of potential effects, and (3) the extent to which agencies have taken actions to address previously identified fragmentation and overlap among the programs and lessons learned. The sections below discuss the methods we used to address each of the three objectives. In addition to these methods, we reviewed relevant federal guidance and other program documents; and interviewed federal agency officials at headquarters offices. The focus of this review was how employment and training services are coordinated among programs specifically designed to deliver such services. As such, our scope excluded some programs that offer or finance employment and training services, but for which this is not a program objective (for example, student loan programs, which focus primarily on enhancing access to postsecondary education). Similarly, we focused on programs that deliver direct service rather than tax expenditures, which may finance or incentivize similar services through tax benefits. Program Selection To address all of our objectives, we compiled a list of employment and training programs by starting with the 47 programs administered by nine federal agencies that were identified in our prior work. We updated the original list by asking federal agency officials to provide the current status of previously identified programs and identify any new ones that might meet our criteria. As in our 2011 review, we included programs for which objectives cited in the Catalog of Federal Domestic Assistance (CFDA) covered: enhancing the specific job skills of individuals in order to increase their identifying job opportunities, and/or helping job seekers obtain employment. We also searched the CFDA electronically in February 2018 to identify any additional programs that met our inclusion criteria. To conduct an electronic text search of the CFDA database, we used 12 search terms used in GAO-11-92. These included: We excluded any programs that met one or more of the following criteria: Program objectives do not explicitly include helping job seekers enhance their job skills, find job opportunities, or obtain employment. Program does not provide employment and training services itself (e.g., it provides financial support to other employment and training programs, or subsidizes the cost of employment through tax credits). Program is small or is a component of a larger employment and training program, such as a pilot or demonstration program. Programs that are economic development programs that aim to increase job opportunities but do not provide services to individuals to enhance their job skills, identify job opportunities, or find employment. Programs that aim to achieve broad workforce-related goals, such as increasing educational opportunities for minority individuals in particular fields or improving the status of and working conditions for wage-earning women, but do not provide employment or training services themselves. Education programs that fund student loans for educational expenses, initiatives for student recruitment and retention, or other student support services. Programs that support training for training providers, such as vocational rehabilitation specialists, or other programs that support job-specific training for individuals who are already employed. Two analysts independently reviewed the list of 211 programs identified in the list generated from the 2018 CFDA search against the inclusion and exclusion criteria described above. To reach concurrence on the programs list, the analysts compared their lists and reached agreement on which to include. If the analysts were undecided about including a program, another analyst was consulted. We also reviewed other GAO reports published since 2011 that provided a more in-depth review of employment and training programs to identify any additional programs that met our three inclusion criteria. As a result of that process, we identified three programs that met our criteria and added them to our list. It is important to note that the number of programs identified will vary with the definition used, and applying any definition can require subjective judgment. After evaluating all identified potential programs, we determined that 46 employment and training (E&T) programs met all criteria to be included in our audit. Once our determinations were made, we sent emails to agency liaisons asking them to confirm the list of programs to be included in and excluded from our review, and to provide the names and contact information for the officials who would be responsible for completing our planned survey. Agencies confirmed our final inclusion and exclusion decisions. After administering our survey, we excluded DOD’s Troops to Teachers Program because the program generally focused on teacher quality rather than E&T services. We also excluded DOD’s Hiring Heroes Program because DOD officials told us the program does not receive a specific appropriation and is a small program that is part of DOD’s larger effort to encourage the employment of servicemembers and veterans. After we administered our survey, DOL officials clarified that the Women in Apprenticeship and Nontraditional Occupations (WANTO) program was not a sub-program under the Registered Apprenticeship Program, but rather a discrete program. We sent a survey to WANTO program officials. At the end of this process, we confirmed that 43 programs met our definition and should be included in our review. We generally maintained consistency with decisions made in our 2011 review. Survey To address all of our objectives, we administered a survey to program officials that included questions about services provided, budgetary information, and participants served. In addition, we included questions asking agency officials to confirm or correct program objectives and eligibility and beneficiary requirements listed in the CFDA. We also included questions about agencies’ actions to manage overlap and fragmentation. We conducted two pretests with VA to ensure (1) our questions were clear and unambiguous, (2) terminology was used correctly, (3) the survey did not place an undue burden on agency officials, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. To assess the reliability of the data provided by agencies, we asked officials to identify the databases and information sources they used to respond to our survey questions and any limitations of the data they provided. We then discussed with agency officials any identified data limitations and, if unresolved issues remained, annotated the data, as appropriate. We also identified responses that appeared to be inconsistent or outliers, such as instances in which participants increased as funds declined, and submitted them to agencies for verification. From April to August 2018, we emailed the surveys to agency officials as an attached Microsoft Excel form that they could return electronically. All of the 45 surveys were completed and returned. Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the surveys, collecting the data, and analyzing them to minimize such nonsampling error. For example, to minimize difficulties interpreting a particular survey question, we incorporated the suggestions from an independent reviewer to add explicit instructions for how to use the pull-down menus and consistently phrased requests for information. We reviewed the completed surveys and clarified information with agency officials, as needed. We further reviewed the survey to ensure the ordering of survey sections was appropriate and that the questions within each section were clearly stated and easy to comprehend. To reduce nonresponse, another source of nonsampling error, we sent out email reminder messages to encourage officials to complete the survey. In reviewing the survey data, we performed automated checks to identify inappropriate answers. We further reviewed the data for missing or ambiguous responses and followed up with agency officials when necessary to clarify their responses. On the basis of our application of recognized survey design practices and follow-up procedures, we determined that the data were of sufficient quality for our purposes. In terms of agency actions to manage overlap and fragmentation and to detect/prevent duplication, we followed up with select agencies to better understand what prompted the actions they took and the lessons they learned from evaluating those efforts. We did not conduct a legal analysis to confirm the various characterizations of the programs in this report, such as information on their budgetary obligations, services provided, target population, eligibility criteria, or program goals. Instead, all such program information in this report is based on our survey results, as confirmed by agency officials. Further, we did not review agencies’ financial reporting systems or audit the figures provided to us. We reviewed fiscal year 2019 budget documents to determine if they could be used to verify data provided by the agencies, but they did not consistently contain the program-level details needed. Instead, to help mitigate reliability limitations that might have accompanied agency reports, we asked agencies to identify the data source of reported budgetary information and to list any data limitations. Overlap To address our second objective to identify areas of overlap among E&T programs, we reviewed information reported by federal agency officials in our survey. We used the definition of overlap established in GAO’s prior work: overlap occurs when two or more programs provide at least one similar service to a similar population. After reviewing survey responses regarding the primary population groups served by the 43 programs and the services they provided, we categorized programs according to the primary population group served and identified programs within each category that provided similar services. We did not focus on the effects of potential duplication, which occurs when two or more agencies or programs are engaged in the same activities or provide the same services to the same beneficiaries. GAO has not previously identified duplication in federal E&T programs, and our objectives in this engagement focused on overlap and fragmentation previously identified in these programs. We categorized programs based on the type of program participant served according to program objectives and program eligibility criteria listed in the CFDA. Then, we verified these categorizations with agency officials. In categorizing programs by target population, we used the following categories: 1) general population, 2) dislocated workers or trade-impacted workers, 3) migrant and seasonal farm workers, 4) Native Americans (in this report, the term Native Americans refers to American Indians and Native Hawaiians), 5) people with physical or mental disabilities, 6) prisoners or ex-offenders, 7) refugees, 8) veterans or transitioning servicemembers, 9) youth, and 10) older workers, women, and unemployed and underemployed residents of solid and hazardous waste-impacted neighborhoods (collectively, other). We also categorized the VA’s Vocational Rehabilitation and Employment program with other programs that target veterans, but noted that the program serves veterans with a service-connected disability. Review of Prior GAO Reports and Agency Funded Research To address our second and third objectives, we also reviewed GAO reports and agency funded research published since 2011. We used these sources, in part, to illustrate effects of overlap and fragmentation among E&T programs and provide examples of actions agencies have taken to address our prior findings or recommendations. To address our second research objective, we reviewed this literature to identify examples of documented effects of overlap and fragmentation among these programs, including positive effects (e.g., to fill a gap or complement an existing program) and negative effects (e.g., inefficient use of resources or confusion among individuals). To address our third research objective, we conducted a literature search of agency- sponsored research on E&T programs and ultimately determined that six of these studies were sufficiently rigorous and appropriately scoped to include in our review. To identify studies on coordination and collaboration of federally-funded programs, we conducted a literature search through ProQuest. Our initial search terms included “federal employment and training” and “coordination” or “collaboration,” “overlap,” and “fragmentation”. We also reviewed these studies to assess the extent to which agencies had evaluated actions to manage overlap and fragmentation. In addition, our survey asked program officials about whether an impact study had been completed since 2011 to evaluate program performance with regard to E&T activities and, if so, to provide a citation for at least one of these studies. An impact study assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. This type of study is conducted when external factors are known to influence the program outcomes, in order to isolate the program’s contribution to the achievement of its objectives. Program officials provided 16 citations of what they believed to be impact studies. Of the 16 cited studies, we determined that 13 can accurately be described as impact studies. To make this assessment, we reviewed the methodology section of each study. We conducted this performance audit from September 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Update on the List of Federal Employment and Training Programs since GAO 2011 Review The following table is a list of federal employment and training programs using as a baseline programs identified in our most recent prior report (GAO, Multiple Employment and Training Programs: Providing Information on Colocating Services and Consolidating Administrative Structures Could Promote Efficiencies, GAO-11-92 (Washington, D.C.: Jan. 13, 2011)). We also reviewed the Catalog of Federal Domestic Assistance (CFDA) to ensure that programs met our selection criteria and to identify new programs. We did not conduct an independent legal analysis to verify the information provided about the programs described in this appendix, such as information on their status. For a description of our methodology, see appendix I. Appendix III: Change in Federal Employment and Training Program Obligations, Adjusted for Inflation in 2017 Dollars Appendix IV: Change in Federal Employment and Training Obligations in Nominal Values Appendix IV: Change in Federal Employment and Training Obligations in Nominal Values *Program name from 2011 review was updated based on information confirmed by agency officials. Appendix V: Estimated Number of Program Participants Who Received Federal Employment and Training Services Appendix V: Estimated Number of Program Participants Who Received Federal Employment and Training Services *Program name from 2011 review was updated based on information confirmed by agency officials. Appendix VI: Employment and Training Services Provided by Federal Programs Serving Selected Target Populations Appendix VII: Agency-Funded Studies Examining Employment and Training Programs Appendix VIII: Comments from the Department of Labor Appendix IX: Comments from the Department of Veterans Affairs Appendix X: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Betty Ward-Zukerman (Assistant Director), Sheranda Campbell (Analyst-in-Charge), Camille Henley, Joel Marus, David Perkins, and Jill Yost made key contributions to this report. Also contributing to this report were Amy Anderson, Stephen Betsock, Caitlin Croake, Alex Galuten, Kristen Jones, Benjamin Licht, Mimi Nguyen, James Bennett, David Blanding, Elizabeth Mixon, Steven Putansu, Monica Savoy, Paul Schearf, Ardith Spence, Almeta Spencer, Kathleen van Gelder, and John Yee.
target: Federally funded employment and training (E&T) programs help job seekers enhance their job skills, identify job opportunities, and obtain employment. In 2011, GAO identified overlap and fragmentation among E&T programs administered by nine federal agencies. The Workforce Innovation and Opportunity Act (WIOA) was enacted in 2014, in part, to improve coordination and integration among these programs. This report examines (1) how the number of and obligations for federal E&T programs have changed since GAO's 2011 review, (2) the extent to which E&T programs continue to provide similar services to similar populations and examples of potential effects, and (3) the extent to which agencies have taken actions to address previously identified fragmentation and overlap among E&T programs and what agencies have learned about the results. To address these objectives, GAO surveyed E&T program administrators, reviewed relevant reports and studies, and interviewed federal agency officials. The number of federal employment and training (E&T) programs and program obligations have declined since GAO's 2011 report. In that review, GAO identified 47 E&T programs and found that 44 had overlap with at least one other program in that they provided similar services to a similar population. In fiscal year 2017, the most recent year data are available, GAO identified 43 E&T programs, or 4 fewer than in 2011 (see figure). From fiscal year 2009 to 2017, federal agencies' annual obligations for E&T programs decreased from about $20 billion to $14 billion. GAO analysis of survey data found the decrease in obligations was largely due to the expiration of funding from the American Recovery and Reinvestment Act of 2009, which had provided additional funding for selected E&T programs during and after the Great Recession. Survey results from federal administrators of the 43 E&T programs show that the programs continue to span nine agencies and generally overlap by providing similar services, such as employment counseling and assessment services (39 of 43) and job readiness training (38 of 43). Further, programs targeting a specific population, such as Native Americans, veterans, or youth, also provided similar services. In some cases, such overlap may be appropriate or beneficial, but it may also suggest opportunities for greater efficiency. Almost all (38 of 43) E&T programs reported at least one action to manage fragmentation or overlap, such as co-locating services and sharing information. However, the agencies were not able to consistently provide information on the results of these actions and few evaluations encompassed multiple programs. Among studies GAO identified, six examined more than one E&T program, but only one assessed how any coordinated activities benefited the population served. None of the six studies focused on Native Americans, youth, or refugees. The Workforce Innovation and Opportunity Act (WIOA) encourages agencies to conduct evaluations, and specifically requires the Department of Labor (DOL) to publish a 5-year plan describing certain E&T priorities, consistent with the purpose of aligning and coordinating certain programs. While DOL reported it took some steps, it continues to lack a strategic plan for E&T evaluations over a multi-year period. As a result, DOL does not know whether actions to manage overlap are successful.