Veterans' Employment and Training: Better Targeting, Coordinating, and Reporting Needed to Enhance Program Effectiveness
Released: 2013-01-10T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: In fiscal year 2011, the federal government spent an estimated $1.2 billion on six veterans' employment and training programs, serving about 880,000 participants. Labor administers five of these programs and VA administers one. Despite these efforts, the unemployment rate for veterans who have recently separated from the military is higher than that for the civilian population. The number of service members transitioning to the civilian workforce is expected to increase. In response to a request, this report examines (1) the extent to which federal veterans' employment and training programs vary in services they deliver and veterans who receive them; (2) the extent to which federal agencies coordinate programs; and (3) what is known about the performance of these programs. To address these objectives, GAO reviewed agency data, policy documents, and relevant federal laws and regulations, reports, and studies, and interviewed federal and regional officials and state officials in six states selected to achieve geographic and demographic diversity. In examining coordination, GAO included in its review employment assistance DOD provides to Guard and Reserve members.

section_title: What GAO Found
paragraphs: The six federal veterans' employment and training programs offer similar employment services, but largely target different groups. Among these programs, the Department of Labor's (Labor) Disabled Veterans' Outreach Program has the greatest potential for overlap with other veterans' programs and Labor's employment programs for the general population. Federal law governing the Disabled Veterans' Outreach Program makes all veterans who meet the broad definition of "eligible veteran" eligible for its services, but gives disabled veterans and economically and educationally disadvantaged veterans the highest priority for services. However, Labor's guidance does not provide states--who administer the program using federal funds--criteria for prioritizing services. The law also generally requires that program staff provide participants with intensive services (e.g., individual employment plans), but Labor's data indicate that nationally 28 percent of participants received such services in 2011. In explaining this statistic, Labor officials said one possible explanation was that staff are enrolling people who do not need intensive services. Labor said it plans to develop guidance on prioritizing services, and it also has a six-state pilot to improve monitoring, but neither of these efforts has been completed.

In 2008, Labor and the Department of Veterans Affairs (VA) compiled a handbook intended to guide the roles of their respective staff in coordinating services to disabled veterans; however, they have not updated the handbook nor included related Department of Defense (DOD) employment initiatives in their interagency agreements. GAO's interviews with VA and Labor officials identified certain challenges with meeting desired program outcomes resulting, in part, from sections of the handbook that provide insufficient guidance or are subject to misunderstanding. For example, the handbook says Labor and VA are to coordinate to achieve "suitable employment"--employment that follows the veteran's rehabilitation plan and does not aggravate the disability. However, it does not explicitly say how staff should navigate situations where a veteran's financial need or preferences do not align with this goal. In such instances, program staff may work at cross purposes and veterans may accept jobs that do not count as suitable employment. Further, DOD is expanding its employment assistance, but does not have an interagency agreement to coordinate with Labor and VA efforts. Absent an updated handbook and integration of DOD into the coordination framework, there is increased risk for poor coordination and program overlap.

While available performance information shows that most programs' outcomes are below pre-2007 levels, the information Labor reports and the research it has conducted make it difficult to know the extent to which each program is achieving its annual performance goals. Veterans' employment outcomes for programs administered by both Labor and VA have generally not regained levels seen before the recession that began in 2007, which is similar to employment programs for the general population. In reporting performance, Labor does not relate employment outcomes to individual program goals. In contrast, Labor reports outcomes and goals for its other workforce programs aimed at the general population. Moreover, while both agencies have studies completed or under way, neither has conducted impact evaluations that assess program effectiveness to determine whether outcomes are attributable to program participation and not other factors. As a result, Congress and other key stakeholders lack essential information needed to assess each program's performance. GAO is making four recommendations aimed at improving the guidance provided to staff in the coordination handbook, integrating DOD into the interagency coordination framework, improving agency reporting on achievement of program performance goals, and assessing program effectiveness.

section_title: What GAO Recommends
paragraphs: GAO is making four recommendations aimed at improving the guidance provided to staff in the coordination handbook, integrating DOD into the interagency coordination framework, improving agency reporting on achievement of program performance goals, and assessing program effectiveness.

section_title: Letter
paragraphs: In fiscal year 2011, the Department of Labor (Labor) and the Department  of Veterans Affairs (VA) spent an estimated $1.2 billion on veterans’  employment and training programs and served about 880,000  participants. In addition, the Department of Defense (DOD) has begun  providing additional employment assistance to Guard and Reserve  members. In 2011, Congress enacted a new law, and the Administration  launched additional initiatives to improve employment opportunities for  veterans. While the overall unemployment rate for veterans is comparable  to that of non-veterans, the unemployment rate for veterans who have  more recently separated from the military is higher than that of civilians  and other veterans. Moreover, more than 1 million service members are  projected to separate from the military and transition to civilian life from  2011 to 2016. In addition, the number of veterans with service-connected  disabilities is on the rise.

Over the last 20 years, we have periodically reported on individual  employment and training programs as well as on specific populations of  veterans who use them. In January 2011, as a part of a larger review of all federal employment and training programs, we identified six  employment and training programs administered by Labor and VA that  are targeted toward veterans. Labor oversees five of these programs: (1)  the Disabled Veterans’ Outreach Program, (2) the Homeless Veterans’  Reintegration Program, (3) the Local Veterans’ Employment  Representative Program (Employment Representative Program), (4) the  Transition Assistance Program (TAP), and (5) the Veterans’ Workforce  Investment Program (VWIP). VA oversees the sixth program: the  Vocational Rehabilitation & Employment Program (Vocational  Rehabilitation Program). More specifically, our report identified services,  eligibility requirements, and outcome measures that these programs had  in common.

For this report, we focus in more detail on the six programs identified in  our January 2011 report and examine (1) the extent to which federal  veterans’ employment and training programs vary in terms of the services  they deliver and the veterans who receive them; (2) the extent to which  federal agencies coordinate these programs; and (3) what is known about  the performance of these programs.

To address our first objective, we analyzed Labor and VA data on  services provided and veteran participants, as well as agency annual  reports, budget justifications, guidance, and relevant federal laws and  regulations for the six programs listed above. To address our second  objective, we reviewed key agency agreements and guidance, and  interviewed agency officials and associations representing the interests of  veterans. In responding to this objective, we included not only the six  programs indicated above but also three Labor programs available to the  general population: the Workforce Investment Act (WIA) Adult and  Dislocated Worker programs, and the Employment Service (ES)  Program. We also included two DOD programs which have recently  begun providing employment services: (1) the Yellow Ribbon  Reintegration Program (Yellow Ribbon), which includes the Employment  Initiative Program, and (2) the Employer Support of the Guard and  Reserve (ESGR). As part of answering our first two objectives, we also  conducted case studies in six selected states. For all 50 states, we first  determined whether each was high, medium, or low on the following  characteristics: the percentage of veteran population, amount of program  expenditures, program performance, and veterans’ unemployment rate.  We then selected one state from each of Labor’s six regions to achieve  variation on the above characteristics, as well as diversity in terms of  geography and state size. These states were Florida, Massachusetts,  Ohio, Oregon, Texas, and Virginia. Within these states we interviewed  federal officials assigned to the state and/or region and the Director or  designee of the state workforce agency. State workforce agencies  operate employment programs using federal funds. Finally, within three  states, we interviewed the state government’s Director of Veterans  Affairs. For each state, we also reviewed state plans that specify how the  state will enact federal policies. To address our third objective, we  analyzed agency annual reports, budget justifications, and other agency  documents for the six programs indicated above. We also synthesized  findings from relevant past GAO reports and agency-sponsored program  evaluations. We used data from the Labor Exchange Reporting System,  Veterans’ Employment and Training Service (VETS) Operations and  Programs Activity Report data system, the VA Corporate Case  Management System, and Defense Manpower Data Center. We found  the data from these systems sufficiently reliable for our reporting  purposes. Additional details regarding our methodology can be found in  appendix I.

We conducted this performance audit from September 2011 through  December 2012 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe the information and data obtained, and the analysis conducted,  provide a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Background
subsections: section_title: Labor and VA Employment and Training Programs Serving Veterans
paragraphs: Labor and VA oversee six employment and training programs targeted to  veterans (see table 1). Labor administers its programs through state  workforce agencies in each state. Within Labor, VETS administers five  employment programs targeted to veterans.  VETS provides grants to  states to support state workforce agency staff who serve veterans  through the Disabled Veterans’ Outreach Program, Employment  Representative Program, and TAP. Through the Homeless Veterans’  Reintegration Program and VWIP, VETS also provides funding to  organizations that serve eligible veterans, including nonprofits. Labor  oversees these programs through federal officials stationed in each  region, as well as a Director of Veterans’ Employment and Training  located in each state. Within VA, the Vocational Rehabilitation Program  provides employment services to certain veterans with disabilities. VA  offers the program in 56 regional offices and 169 satellite offices. The  program has about 1,000 staff, including vocational rehabilitation  counselors, employment coordinators, support staff, and managers.  Rehabilitation counselors determine entitlement to services.

In addition to its programs administered by VETS, Labor offers  employment and training services to the general population—including  veterans. These services are administered by the Employment and  Training Administration (ETA). First, ETA administers the ES Program,  which provides a national system of public employment services to all  individuals seeking employment. ES provides services such as job  search, labor market information, and job referrals to the public, including  job seekers and employers. ETA carries out its ES Program through state  workforce agencies. ETA also administers the WIA Adult and Dislocated  Worker programs, which provide a broad range of services including job  search assistance, skill assessment, and training for eligible individuals.  When funds are limited, the WIA Adult Program is to give priority for  intensive and training services to low income adults or those on public  assistance. In program year 2010, 94,629 veterans exited from the WIA  Adult Program. WIA’s Dislocated Worker Program generally targets  adults who have been terminated or laid off from employment and meet  other criteria. In program year 2010, 58,350 veterans exited from the WIA  Dislocated Worker Program.

Federal law requires VETS, ES, and WIA Adult and Dislocated Worker  programs to offer their services through the one-stop system—which  includes centers through which job seekers can access a range of  employment and training programs. Two of VETS’ programs—Disabled  Veterans’ Outreach and Employment Representative programs—have  about 2,100 staff who work primarily in local one-stop centers. Federal  law also requires other Labor-funded programs—including ES and WIA  Adult and Dislocated Worker programs—to give veterans priority over the  general population when they seek services (referred to as priority of  service). VETS and ETA jointly monitor compliance with this  requirement.

Most ETA and VETS programs report the same performance measures,  known as the common measures. They include percentage of program exiters who have obtained employment  (entered employment rate), percentage retaining employment for 6 months after exiting the  program (employment retention rate), and

6-month average earnings of program exiters (average earnings).

For each of these, Labor establishes annual performance goals.

VA reports an employment rehabilitation rate as a measure of  performance for the Vocational Rehabilitation Program. A “rehabilitated”  veteran is one who successfully completes a rehabilitation plan and is  equipped with the required skills and tools needed to obtain and maintain  suitable employment (i.e., employment that is consistent with the  veteran’s skills, aptitudes, and interests).

section_title: DOD Programs
paragraphs: DOD works with Labor and VA to provide transition assistance workshops  as a part of TAP.  In addition, DOD helps Guard and Reserve members  obtain civilian employment though its operation of several programs,  including the Yellow Ribbon Program and ESGR. The Yellow Ribbon  Program serves National Guard and Reserve members and their families  by hosting events that provide information on employment opportunities,  health care, education/training opportunities, finances, and legal  benefits.

The ESGR is a nationwide network of volunteers who address  unemployment and underemployment of Guard and Reserve members  through participation in employment-related events.

section_title: Veterans’ Programs Provide Similar Services and Largely Serve Different Populations, but It Is Unclear If the Disabled Veterans’ Outreach Program Is Appropriately Serving Its Targeted Population Six Federal Programs Provide Similar Types of Employment Services but Largely Serve Different Populations
paragraphs: As shown in figure 1, the six federal employment and training programs  targeted to veterans offer similar types of employment and training  services. assessment, job search or job placement activities, and job readiness  skills training. Other services available from more than one of these  programs include the development of job opportunities, job referrals, and  occupational and vocational training, among others.

section_title: Labor and VA Have an Established Framework for Coordination, but It Does Not Include New Employment Initiatives DOD Is Pursuing
subsections: section_title: Labor and VA Have an Established Coordination Framework and Are Taking Steps to Periodically Review It
paragraphs: Labor and VA have established a framework to coordinate their  employment and training programs. In 2005, Labor and VA signed an  interagency memorandum of agreement that outlines how the agencies  plan to coordinate the Vocational Rehabilitation and Disabled Veterans’  Outreach and Employment Representative programs to serve disabled  veterans, respectively. The agencies have also collaboratively created an  interagency handbook that delineates roles and responsibilities and  establishes a referral process between the Disabled Veterans’ Outreach  and the Vocational Rehabilitation programs. To assist field staff, the  interagency handbook also provides standard language and guidance for  agreements between local Labor and VA offices. As a result, local offices  from both agencies can tailor the standard agreement language to meet  local situations. The handbook has not been updated since 2008.

Labor and VA have provided staff with training on the handbook and  formed a group to monitor coordination. Labor and VA conducted a  national training webinar based on the interagency handbook for both  agencies’ staff after it was published, have made virtual trainings  available since 2009, and provided technical assistance to staff. To  monitor the coordination activities outlined in the interagency handbook,  Labor and VA created a Joint Work Group. According to Labor and VA  officials, this group recently discussed and agreed on a plan to review  one-third of local agreements made between Labor and VA field locations  annually.

Labor and VA have collected information that could be useful in updating  the handbook. The Joint Work Group recently conducted its first in-depth  review of states’ implementation of the handbook since it was established  in 2008. According to Labor and VA officials, the Joint Work Group  electronically surveyed the VA employment coordinators in all 56 VA  Regional Offices and the 52 state-level directors of the Disabled   While  Veterans’ Outreach and Employment Representative programs. VA officials stated that they are currently reviewing the survey results to  determine if the handbook needs to be updated, Labor officials told us  told us they believe the handbook needs to be updated. We have  reported previously that agencies need to create means to monitor  agreements and related guidance periodically to keep them current and to  identify areas for improvement.

section_title: Some Coordination Issues Have Arisen across Veterans’ Employment and Training Programs
paragraphs: Our interviews with Labor and VA officials identified certain challenges  with meeting desired program outcomes resulting, in part, from sections  of the handbook that are subject to misunderstanding or provide  insufficient guidance. They pertain to incorporating labor market  information into rehabilitation plans and finding “suitable employment” for  participants.

The first challenge with referrals as outlined in the handbook involved  ensuring that participants’ rehabilitation plans prepared them for jobs that  existed in their local area. According to the referral process outlined in the  interagency handbook and by agency officials (see fig. 3), there are two  main referral points from the Vocational Rehabilitation Program to Labor’s  Disabled Veterans’ Outreach Program staff: (1) before the participant’s  rehabilitation plan is completed and (2) after the participant has  completed a rehabilitation plan and been deemed job-ready, or ready for  employment, by VA staff. Disabled Veterans’ Outreach Program staff may  provide participants with labor market information or other employment  assistance at the first referral point and are required to at the second  referral point. While VA officials in four of the six states we reviewed  reported that they connected participants with Labor staff to receive labor  market information and other employment consulting, only three of these  states reported that they did this early in the process before the  rehabilitation plan is completed. In two other states, VA officials reported  they understood that they were supposed to refer participants to Labor  only after they had completed rehabilitation plans and were job-ready,  essentially skipping the first step where labor market information may  have been useful. VA officials reported that labor market information may  be provided to participants through small group presentations with  Disabled Veterans’ Outreach staff. For their part, state-level Labor  officials noted that job placement was more challenging for Disabled  Veterans’ Outreach Program staff when participants’ rehabilitation plans  were developed without labor market information. In such cases,  according to Labor officials, Disabled Veterans’ Outreach Program staff  were sometimes working with plans focused on training in occupations  not available in the local labor market—in effect using programs’ funds to  prepare participants for jobs that do not exist in their local area. According  to Labor officials, this made it more difficult for participants to have  successful employment outcomes.

The second challenge with referrals as outlined in the handbook involved  ensuring that job-ready participants are directed to “suitable employment.”  When veterans are referred to the Disabled Veterans’ Outreach Program  at the job-ready stage, Disabled Veterans’ Outreach Program and VA  staff are supposed to coordinate to find “suitable employment,” or  employment that will not aggravate the participant’s disability and follows  the participant’s rehabilitation plan. State-level Labor officials noted that,  in some cases, veterans may choose to accept jobs they want or need  but that do not fit in their employment or rehabilitation plan. Such jobs do  not count as “suitable employment” for VA because the job may, in the  long run, aggravate the veteran’s disability. While the handbook says  agencies are to coordinate to achieve “suitable employment,” it does not  explicitly say how Disabled Veterans’ Outreach Program and VA staff  should deal with situations where a veteran’s financial need or  preferences do not align with the goal of suitable employment. Absent  guidance about how to navigate such situations, program staff may be  working at cross purposes and program participants may be taking  employment they cannot retain in the long run. This employment, in turn,  may make a veteran’s disability worse and may make finding future  employment more difficult. One official stressed that having labor market  information incorporated into rehabilitation plans early may help veterans  avoid taking a job that does not match their plans.

section_title: DOD Is Expanding Its Employment Initiatives, Which Are Not Included in the Labor-VA Coordination Framework
paragraphs: DOD is expanding its employment assistance to National Guard and  Reserve members, but does not have employment service agreements  with Labor or VA beyond an agreement for TAP. In fiscal year 2011,  DOD launched an employment assistance initiative under the Yellow  Ribbon program, known as the Employment Initiative Program, that  provides job workshops and job fairs to connect Guard and Reserve  members to employers. The Employment Initiative Program under the  Yellow Ribbon Program has hired and placed 56 “Employment Transition  Coordinators” covering all 50 states, territories, and the District of  Columbia who provide service members employment assistance among  other services.  The Yellow Ribbon Program has also held 27 job fairs  since the beginning of fiscal year 2012. To support the job fairs, the  Yellow Ribbon Employment Initiative Program leverages the network of  4,900 volunteers who are affiliated with ESGR. These volunteers also  provide resume-building workshops, mock interviews, and career  counseling. DOD also recently testified that it is leading, per a White  House directive, a new Credentialing and Licensing Task Force to  address gaps between military occupational specialties and civilian  licensing requirements.

DOD reported that this additional employment assistance is needed to  support Guard and Reserve members who may not meet veteran status  requirements necessary for participating in Labor or VA programs. Most  of the ESGR representatives we spoke with also anticipated the program  would continue to provide employment-related services.

Although DOD has established these employment assistance services,  no agreement or formal mechanism has been established for coordinating   Specifically,  them with Labor’s and VA’s veterans’ employment efforts. there is no interagency agreement for coordinating employment services  beyond DOD’s and Labor’s work on the Uniformed Services Employment  and Reemployment Rights Act of 1994 (USERRA) and TAP. Although  DOD and VA have an agreement, it focuses on connecting service  members who are leaving the military for civilian life with vocational  rehabilitation services. Further, ESGR has no formal mechanism for  identifying and referring eligible veterans to the Disabled Veterans’  Outreach and Employment Representative programs. We have previously  reported that agencies with common goals and programs can enhance  and sustain collaboration by creating mutually agreed-upon strategies to  help align agencies’ activities and leverage resources to meet their  common goals.

Currently, ESGR in the states we reviewed reported informal  coordination—such as meetings and co-participation in job fairs—with  Labor-funded programs. For example, a DOD official noted that the  Washington ESGR used a grant to hire 13 employment transition  counselors in areas that needed service not provided by the state  workforce agencies. According to this official, this ultimately increased  impact while saving funds.

However, this informal coordination may be affecting Labor resources and  confusing employers. According to Labor officials, Disabled Veterans’  Outreach Program staff participation at DOD job fairs reduces the amount  of time available for their primary duties, such as providing intensive  services to program participants. A variety of officials from the states we  reviewed also said that some employers were confused regarding which  agency was leading the initiatives to employ veterans.

section_title: While Most Programs’ Outcomes Are Below Pre-Recession Levels, Labor Has Not Reported the Extent to Which Programs Are Achieving Their Performance Goals
subsections: section_title: Most Programs’ Outcomes Are Below Pre-Recession Levels, and Outcomes for Veterans Are Lower Than They Are for the General Population
paragraphs: Employment outcomes for veterans’ programs have generally not  regained levels attained prior to the recent recession. (See appendix II  for performance outcomes for each veterans’ program as well as veteran  participants served by WIA Adult and ES programs over a 5-year period.)  From program years 2007 to 2009—which spanned July 2007 to June  2010—most Labor veterans’ programs that have outcome measures  saw a decline in their entered employment rate and a slight decline in  their 6-month job retention rates. In program year 2010,  all programs  except VWIP had lower entered employment and employment retention  rates than in program year 2006, prior to the recession. The number of  VA Vocational Rehabilitation Program participants who were rehabilitated  to employment has also declined from 9,225 participants in fiscal year  2006, to 7,975 participants in fiscal year 2011.

Officials at both VA and Labor attributed the declines to various causes.  For example, VA officials attribute some of the decline in the number of  participants rehabilitated to the establishment of the Post-9/11 GI Bill  Program. The Post-9/11 GI Bill Program is an education benefit  administered by VA for individuals who served on active duty after  September 10, 2001. According to VA officials, VA’s Vocational  Rehabilitation Program lost some participants who had begun  rehabilitation efforts but switched to the Post-9/11 GI Bill Program. They  switched, according to VA officials, because the GI Bill Program provided  a more generous living stipend than the Vocational Rehabilitation  Program. At the same time, Labor officials identified national economic  conditions as the primary reason for the drop in performance of its  programs.

In addition to the decline in outcomes for veterans’ programs, veterans  participating in broader workforce programs also achieved somewhat  lesser outcomes than those in the general population. (See app. II, figs. 7  and 8.) From program years 2007 to 2009, the WIA Adult and ES  programs saw declines in measures for the percentage of participants  who entered employment and the percentage who retained their  employment for 6 months. These measures have generally rebounded  slightly in 2010, although they have generally not regained levels attained  prior to the recent recession. Since 2006, veterans have had slightly  lower entered employment outcomes than those for all participants using  the ES Program. In the WIA Adult Program, veterans’ employment and  retention outcomes have been slightly lower than outcomes for all  participants since 2009. Further, between 2006 and 2010, employment  and retention outcomes were similar but slightly lower for veterans who  worked with Disabled Veterans’ Outreach and Employment  Representative program staff, in comparison with outcomes for veterans  in the WIA Adult Program.

According to Labor officials, some of these differences in outcomes may  be explained by differences in characteristics of the populations served.  They noted that veteran participants in the WIA Adult Program are more  likely to be over the age of 55 than nonveteran participants, and  historically, older workers have achieved lower outcomes in both the WIA  Adult and ES programs. In addition, Labor officials stated that because  the Disabled Veterans’ Outreach and Employment Representative  programs serve veterans who face barriers to employment, their  outcomes are likely to be lower than outcomes for veterans who are job- ready and nonveterans served by the WIA Adult or ES programs.

section_title: Labor Has Not Consistently Reported the Extent to Which Veterans’ Programs Are Meeting Performance Goals
paragraphs: While Labor reports some data on veterans’ program outcomes, it does  not report the extent to which each of these programs is achieving its  established performance goals. Labor provides Congress an annual  veterans’ program report that provides certain performance information,  such as the number of disabled and recently separated veterans who  received intensive services. For this annual report, however, Labor is not  required to report program outcomes in relation to performance goals.  Labor sets annual performance goals for its veterans’ programs, but it is  not reporting the results relative to those goals. In previous fiscal years,  Labor included some of this information for the Disabled Veterans’  Outreach Program, Employment Representative Program, Homeless  Veterans’ Reintegration Program, and VWIP in its agencywide  performance report. However, in fiscal year 2011, it only reported  aggregate goals for three programs, rather than the separate outcomes  and goals it maintains for each of these veterans’ programs. In contrast,  Labor’s website on general employment programs—WIA Adult and ES— includes both performance goals and outcomes. This information includes  a national average for each measure comparing goals against  performance, as well as each state’s negotiated goals and performance  against those goals. Further, VA reports both an employment outcome  and associated goal for the Vocational Rehabilitation Program. We have  previously reported that relevant performance information should be  reported both internally and externally in order to maintain accountability  and transparency for achieving results. Without information on how the  outcomes for each veterans’ program compare against their annual  performance goals, Congress and other key stakeholders lack essential  information needed to assess the performance of the program.

section_title: Labor and VA Are Working to Collect Data for New Performance Measures
paragraphs: Labor is working to implement new performance measures which have  been mandated by the VOW to Hire Heroes Act of 2011 (VOW Act).Specifically, the act requires Labor to measure participants’ median  earnings 90 and 180 days after a participant stops using a veterans’  program. Prior to the VOW Act, Labor only measured participants’  average earnings over 6 months after participants stop using a veterans’  program, for those who retained employment. The VOW Act also  requires Labor to track the percentage of participants obtaining a  certificate, degree, diploma, licensure, or industry-recognized credential  after participating in its veterans’ programs.

VA also plans to collect additional information about its programs’  outcomes. VA officials said that they decided to track the number, in  addition to the rate, of veterans rehabilitated to employment, because the  employment rehabilitation rate can fluctuate based on a number of  factors. employment in fiscal year 2012. VA officials said that this is the first  fiscal year this goal has been used. Consequently, VA has not yet  reported its performance against this goal.

VA has set a national goal to rehabilitate 9,000 veterans to  In addition, the Vocational Rehabilitation Program has established a  working group to develop new national performance measures. According  to VA officials, the new measures will focus on the middle of the  rehabilitation process, because a veteran can be in the program from 1 to  6 years, with an average of 4 years. The measures that already exist  focus on the front-end (e.g., timeliness of services) and back-end (e.g.,  outcomes). Although the new measures have not been finalized, VA  plans to implement them in fiscal year 2014, contingent on resources to  make changes to the program’s database structure to capture data and  report on new measures.

section_title: Agencies Have Not Conducted Research That Links Veterans’ Outcomes to the Programs That Serve Them
paragraphs: For example, the rehabilitation rate can be negatively affected by veterans who choose  to stop participating before completing a rehabilitation plan. participant outcomes, these studies can be difficult and potentially  expensive to conduct. Impact evaluations can be designed in several  ways, but fall into two basic design categories: experimental, using  random assignment, and quasiexperimental. Quasiexperimental  designs use a comparison group that is not created with random  assignment. While Labor has not conducted impact evaluations, it has  conducted research that examines veterans’ outcomes in relation to their  characteristics and has other studies planned or under way (see table 2).  These studies, though, have limitations. For example, Labor’s 2007 study  of veterans’ outcomes covered five states, and its findings cannot be  generalized to all states. In addition, in the study conducted on the  Homeless Veterans’ Reintegration Program, researchers lacked access  to participant-level data and consequently could not determine whether  certain veterans’ characteristics were associated with positive or negative  employment outcomes for the program as a whole. Labor is funding an  evaluation of the pilot of the redesigned TAP, but has not conducted any  studies or evaluations of VWIP in the last 10 years.

While Labor has not conducted impact evaluations of its veterans’  employment and training programs, it is funding an impact evaluation of  the WIA Adult and Dislocated Worker programs, which is planned to be  completed in 2015. This study will include a supplemental study of  veterans using the public workforce system, but this portion of the study,  as described in the draft research plan for the study, is not an impact  evaluation and cannot determine the extent to which veterans’ outcomes  are due to the services they receive in the public workforce system.

Similar to Labor, VA has not conducted evaluations that allow it to  determine if veterans’ employment outcomes result from program  services or if they are the result of other factors. As shown in table 3, VA  has funded research that examines data related to the completion of  veterans’ rehabilitation plans and participant outcomes. For example,  VA is funding a longitudinal study of Vocational Rehabilitation Program  participants and has issued two reports on the study. The most recent  report begins to analyze VA administrative data to determine  characteristics associated with completing rehabilitation or discontinuing  the program within the first 2 years. However, the report states that its  findings thus far are only descriptive and may have little or no predictive  value. VA plans further study of emerging trends. VA also plans additional  follow-up of program participants in its case management process.  Specifically, the agency plans to send a questionnaire to collect  information on whether former participants are employed and whether  they need additional services.

section_title: Conclusions
paragraphs: Given that the number of service members transitioning to civilian  employment is expected to increase and the number of veterans with  service-connected disabilities is on the rise, Labor’s Disabled Veterans’  Outreach Program is likely to see an increased demand for its services.  Labor attempts to maximize the employment services for those veterans  who need them most. However, we found that there is a need for clearer  guidance to states on how to prioritize services and additional monitoring  of their implementation of such guidance. Labor said it is developing such  guidance but has not completed it, and has tested new monitoring  protocols in six states but has not finalized them. It is encouraging that  Labor has these efforts under way, and it will be important for the  department to complete both efforts.

Labor and VA both provide employment and training programs targeted to  veterans. Although Labor and VA have a handbook governing their  coordination with respect to employment and training for veterans, it has  not been updated since 2008. Our work identified sections of the  handbook that provided insufficient guidance, resulting in situations where  the practices of one department presented difficulties for the other in  meeting desired program outcomes. At the same time, DOD has begun  expanding employment assistance initiatives to segments of the veteran  population, such as National Guard and Reserve members, some of  whom may also meet Labor and VA veterans’ programs eligibility  requirements. However, Labor and VA’s agreement does not govern their  coordination with DOD’s programs. Without an agreement that includes  all three departments, efforts to help veterans find employment are at  greater risk of being fragmented or overlapping, and may not leverage  federal resources.

Finally, the federal investment in veterans’ employment and training  programs warrants greater transparency with regard to the extent to  which these programs are meeting their performance goals and whether  outcomes are attributable to program participation and not other factors.  Labor reports substantial information on outcomes for these programs.  However, Labor is not consistently reporting the extent to which outcomes  for each of its veterans’ programs are achieving the specific performance  goals that were established for these programs. This stands in contrast to  the level of performance reporting by Labor for its WIA Adult and ES  programs, which identifies the extent to which outcomes in these  programs are achieving performance goals. In addition, while the federal  government makes a substantial investment in Labor and VA programs to  achieve employment outcomes for veterans, neither agency has  conducted studies to see if these outcomes can be attributed to the  programs’ services, instead of other factors. As a result, Congress and  other stakeholders lack essential information to assess how well these  programs are performing and hold federal agencies accountable for  achieving results.

section_title: Recommendations for Executive Action
paragraphs: We are making the following four recommendations based on our review:

To increase the effectiveness of coordination efforts, the Secretaries  of Labor and VA should incorporate additional guidance to address  the two problem areas we identified into any update to the interagency  handbook that governs their coordination for veterans’ employment  and training programs.

To ensure government resources are used efficiently, the Secretaries  of Labor, VA, and DOD should incorporate DOD’s employment  assistance initiatives into the agreements that guide interagency  coordination.

To enhance transparency and accountability for achieving results, the  Secretary of Labor should consistently report both performance goals  and associated performance outcomes for each of its veterans’  employment and training programs.

To assess veterans’ employment programs’ effectiveness, Secretaries  of Labor and VA should, to the extent possible, determine the extent  to which veterans’ employment outcomes result from program  participation or are the result of other factors.

section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Labor, the  Department of Veterans Affairs, and the Department of Defense for  review and comment. Written comments from Labor, VA, and DOD  appear in appendixes III, IV, and V, respectively.  In addition to the  comments discussed below, Labor, VA, and DOD provided technical  comments that we incorporated where appropriate.

All three agencies generally concurred with our recommendations. Both  Labor and VA said they would work to enhance coordination with each  other with respect to the guidance in their interagency handbook. All three  agencies said they would work to ensure interagency coordination  included DOD. In response to our recommendation on reporting program  performance, Labor said it will explore ways to increase consistency and  transparency of the information it reports.  In response to our  recommendation to Labor and VA regarding assessing program  effectiveness, VA concurred and Labor did not specify whether or not it  agreed.  Labor said that it is committed to robust program evaluation and  that each agency, including VETS, develops an annual evaluation agenda  and sets priorities.  Labor said it has a multi-component agenda for  evaluating services to veterans and cited some current studies, such as   a study of the TAP program and a statistical analysis of services received  by veterans and their outcomes using the public workforce system.  We  think obtaining information about the effectiveness of veterans' programs  is important because such information can assist Congress in assessing  program results and identifying areas where adjustments may be needed.   As Labor and VA conduct research on program outcomes, it will be  important for them to consider approaches that would enable them to  separate the impact of their programs from other factors that might  influence participants’ outcomes.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to appropriate  congressional committees, the Secretary of Labor, the Secretary of  Veterans Affairs, the Secretary of Defense, and other interested parties.  In addition, this report will be available at no charge on GAO’s website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix VI.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In a January 2011 report, we identified six employment and training  programs administered by the Department of Labor (Labor) and the  Department of Veterans Affairs (VA) targeted toward veterans as a part of  a larger review of all federal employment and training programs. We  defined an employment and training program as one specifically designed  to enhance the specific job skills of individuals in order to increase their  employability, identify job opportunities, or help job seekers obtain  employment. Labor oversees five of these programs for veterans: (1) the  Disabled Veterans’ Outreach Program, (2) the Homeless Veterans’  Reintegration Program, (3) the Local Veterans’ Employment  Representative Program (Employment Representative Program), (4) the  Transition Assistance Program (TAP), and (5) the Veterans’ Workforce  Investment Program (VWIP). VA oversees the sixth program called the  Vocational Rehabilitation & Employment Program (Vocational  Rehabilitation Program).requirements, and outcome measures that these programs had in  common.

Our 2011 report identified services, eligibility  For this report, we focused on the six programs identified in our January  2011 report in more detail and examined (1) the extent to which federal  veterans’ employment and training programs vary in terms of the services  they deliver and the veterans who receive them; (2) the extent to which  federal agencies coordinate these programs; and (3) what is known about  the performance of these programs.

Our approach overall involved reviewing relevant literature, relevant  federal laws and regulations, and analyzing Labor and VA data on  veteran participants, services provided, and performance. We also  interviewed federal Labor, VA, and Department of Defense (DOD) agency  officials who govern agency policy at the national level and key  stakeholder associations.

To determine the extent to which these programs vary in terms of the  services they deliver and the veterans receive them, we analyzed Labor  and VA data on services provided and veteran participants, agency  annual reports, budget justifications, and other agency policy documents.  To assess the reliability of Labor’s data on services to veterans in one- stops, we reviewed available information about the data and conducted  interviews with officials knowledgeable about the data. We determined  that these data were sufficiently reliable for our purposes. We reviewed  relevant federal laws and regulations to determine program eligibility  requirements. We interviewed agency officials, veterans’ service  organizations, and workforce associations to better understand why  programs may serve similar populations with similar services. We  reviewed state plans and interviewed state-level Labor, VA, and DOD  staff for our case study states.

In addition, we conducted six case studies at the state level. In each  state, we reviewed state plans and interviewed Labor and VA officials  assigned to the state or the region. We also interviewed the Employer  Support of the Guard and Reserve (ESGR) chairperson operating in the  state. These are federal representatives of DOD, but not federal  employees, as they are volunteers. In addition, we interviewed the  directors of state workforce agencies, which carry out veterans’  employment and training programs using federal funds. Finally, within  three states, we interviewed the Director of Veterans Affairs in each state,  a state government official responsible for veterans’ programs and  services. To select states, for all 50 states, we determined whether each  was high, medium, or low on the following characteristics: the percentage  of veteran population, amount of program expenditures, program  performance (veterans’ entered employment rate), and veterans’  unemployment rate. We selected one state from each of Labor’s six  regions to achieve variation on the above characteristics, as well as  diversity in terms of geography and state size. These states were Florida,  Massachusetts, Ohio, Oregon, Texas, and Virginia.

To determine the extent to which federal agencies coordinate these  programs, we reviewed key agency agreements and guidance, and used  the same six case studies at the state level, and interviewed federal and  state agency officials and associations representing the interests of  veterans. In examining coordination we included not only the five  programs indicated above but also three Labor programs available to the  general population: the Workforce Investment Act (WIA) Adult and  Dislocated Worker and the Employment Service (ES) programs. We also  included programs recently begun by DOD: the Yellow Ribbon and ESGR  programs. We reviewed memoranda of understanding, agency guidance,  and other policy documents related to collaborative efforts among federal  agencies. We also interviewed National Veterans’ Training Institute  officials to discuss the extent to which required training for outreach  specialists and employment representatives includes instruction on how  to foster inter- and intra-agency coordination. In our case studies in six  states, we interviewed state-level officials from the Veterans’ Employment  and Training Services, as well as VA officials. We also interviewed ESGR  officials and Directors of State Offices of Veterans Affairs. To understand  stakeholders’ views on coordination, we interviewed officials from  workforce associations and veterans’ service organizations. We also used  data from the Defense Manpower Data Center to determine the number  of Guard and Reserve members that may meet the eligibility  requirements for Labor veterans’ programs and VA’s Vocational  Rehabilitation Program. We assessed the reliability of information on  Guard and Reserve members’ length of service and disability status, and  determined that these data were sufficiently reliable for the purposes of  this report.

To determine what is known about program performance, we analyzed  relevant federal laws and regulations, and agency documents, and  interviewed agency officials and stakeholders. We reviewed agency  reports on veterans’ programs containing information on program  outcomes and agency goals established for these programs, such as  Labor’s annual report to Congress on veterans’ programs and  agencywide performance reports. We assessed Labor and VA data on  participant employment outcomes by reviewing available information  about the data and conducting interviews with officials knowledgeable  about the data. We determined that these data were sufficiently reliable  for our purposes. We reviewed the design and methodology of relevant  agency-sponsored program evaluations using GAO criteria on program  evaluation design. We also interviewed Labor and VA national and  regional officials.

section_title: Appendix II: Summary of Veterans’ Program Performance

section_title: Appendix III: Comments from the Department of Labor

section_title: Appendix IV: Comments from the Department of Veterans Affairs

section_title: Appendix V: Comments from the Department of Defense

section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Patrick Dibattista (Assistant  Director), Sheranda Campbell, Maria Gaona, and Dana Hopings made  key contributions to this report. In addition, key support provided by  James Bennett, David Chrisinger, Holly Dye, Rachel Frisk, David  Forgosh, Alexander Galuten, Kathy Leslie, Ashley McCall, and Walter  Vance.


Question-summary annotations: questions: question: How do the six federal veterans' programs compare?
answer: The six federal veterans' employment and training programs offer similar employment services, but largely target different groups.
child_questions: question: Which of these programs has the greatest potential for overlap with other similar programs?
answer: Among these programs, the Department of Labor's (Labor) Disabled Veterans' Outreach Program has the greatest potential for overlap with other veterans' programs and Labor's employment programs for the general population.
child_questions: question: What characterizes the Disabled Veterans' Outreach Program?
answer: Federal law governing the Disabled Veterans' Outreach Program makes all veterans who meet the broad definition of "eligible veteran" eligible for its services, but gives disabled veterans and economically and educationally disadvantaged veterans the highest priority for services.
child_questions: question: What are the problems with this method of prioritization?
answer: However, Labor's guidance does not provide states--who administer the program using federal funds--criteria for prioritizing services. The law also generally requires that program staff provide participants with intensive services (e.g., individual employment plans), but Labor's data indicate that nationally 28 percent of participants received such services in 2011.
child_questions: question: What might explain this statistic?
answer: In explaining this statistic, Labor officials said one possible explanation was that staff are enrolling people who do not need intensive services.

question: How have Labor's efforts to address this problem progressed?
answer: Labor said it plans to develop guidance on prioritizing services, and it also has a six-state pilot to improve monitoring, but neither of these efforts has been completed.

questions: question: What is the purpose of the handbook compiled by Labor and VA?
answer: In 2008, Labor and the Department of Veterans Affairs (VA) compiled a handbook intended to guide the roles of their respective staff in coordinating services to disabled veterans; however, they have not updated the handbook nor included related Department of Defense (DOD) employment initiatives in their interagency agreements.
child_questions: question: What information did GAO's interviews with Labor and VA provide about the handbook?
answer: GAO's interviews with VA and Labor officials identified certain challenges with meeting desired program outcomes resulting, in part, from sections of the handbook that provide insufficient guidance or are subject to misunderstanding.

question: How specifically can the handbook lead to misunderstanding?
answer: For example, the handbook says Labor and VA are to coordinate to achieve "suitable employment"--employment that follows the veteran's rehabilitation plan and does not aggravate the disability. However, it does not explicitly say how staff should navigate situations where a veteran's financial need or preferences do not align with this goal. In such instances, program staff may work at cross purposes and veterans may accept jobs that do not count as suitable employment.

question: Why does this lead to increased risk for poor coordination and program overlap?
answer: Further, DOD is expanding its employment assistance, but does not have an interagency agreement to coordinate with Labor and VA efforts. Absent an updated handbook and integration of DOD into the coordination framework, there is increased risk for poor coordination and program overlap.

questions: question: What does available performance information suggest?
answer: While available performance information shows that most programs' outcomes are below pre-2007 levels, the information Labor reports and the research it has conducted make it difficult to know the extent to which each program is achieving its annual performance goals.

question: How do current Veterans' employment outcomes compare to those before 2007 recession?
answer: Veterans' employment outcomes for programs administered by both Labor and VA have generally not regained levels seen before the recession that began in 2007, which is similar to employment programs for the general population.

question: How does Labor report outcomes and goals?
answer: In reporting performance, Labor does not relate employment outcomes to individual program goals. In contrast, Labor reports outcomes and goals for its other workforce programs aimed at the general population.

question: Why do Congress and stakeholders lack information to assess each program's performance?
answer: Moreover, while both agencies have studies completed or under way, neither has conducted impact evaluations that assess program effectiveness to determine whether outcomes are attributable to program participation and not other factors. As a result, Congress and other key stakeholders lack essential information needed to assess each program's performance.
child_questions: question: What recommendations does GAO have to fix this problem?
answer: GAO is making four recommendations aimed at improving the guidance provided to staff in the coordination handbook, integrating DOD into the interagency coordination framework, improving agency reporting on achievement of program performance goals, and assessing program effectiveness.

questions: question: What kind of resources did the federal government provide to veterans in fiscal year 2011?
answer: In fiscal year 2011, the federal government spent an estimated $1.2 billion on six veterans' employment and training programs, serving about 880,000 participants.
child_questions: question: Which federal agencies manage these programs?
answer: Labor administers five of these programs and VA administers one.

question: What problem needs to be addressed despite high spending for veterans' programs?
answer: Despite these efforts, the unemployment rate for veterans who have recently separated from the military is higher than that for the civilian population.
child_questions: question: How is the number of service members transitioning to the civilian workforce expected to change?
answer: The number of service members transitioning to the civilian workforce is expected to increase.

question: What does this report examine?
answer: In response to a request, this report examines (1) the extent to which federal veterans' employment and training programs vary in services they deliver and veterans who receive them; (2) the extent to which federal agencies coordinate programs; and (3) what is known about the performance of these programs.
child_questions: question: How did GAO get information for the report?
answer: To address these objectives, GAO reviewed agency data, policy documents, and relevant federal laws and regulations, reports, and studies, and interviewed federal and regional officials and state officials in six states selected to achieve geographic and demographic diversity. In examining coordination, GAO included in its review employment assistance DOD provides to Guard and Reserve members.

question: How did GAO examine coordination between federal agencies?
answer: In examining coordination, GAO included in its review employment assistance DOD provides to Guard and Reserve members.