Veterans' Employment: Need for Further Workshops Should Be Considered before Making Decisions on Their Future
Released: 2015-07-16T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The federal government has long offered programs that assist veterans with finding employment. In 2013, the Dignified Burial and Other Veterans' Benefits Improvement Act of 2012 was enacted, which required DOL to provide employment workshops to veterans and their spouses at locations other than military facilities through a 2-year pilot that ended in January 2015. The act also included a provision for GAO to report on the training and possible expansion of the pilot.

This report addresses: (1) how DOL implemented the pilot, (2) what state officials reported regarding the benefits and challenges of the pilot, and (3) how the pilot informs decisions about its possible expansion.

GAO reviewed relevant federal laws and regulations; identified leading practices on pilot design from federal agencies, subject matter experts, and GAO's standards for internal control; and interviewed officials from DOL, the Department of Veterans Affairs, state workforce agencies in each of the three pilot states, and veteran service organizations. GAO also obtained information on the pilot from DOL data and a DOL survey of workshop participants.

section_title: What GAO Found
paragraphs: The Department of Labor (DOL) was required by law to provide employment workshops to veterans and their spouses in a pilot program. In response, DOL used the same 3-day employment workshops for the pilot that it provides to servicemembers on military bases as part of the Transition Assistance Program (TAP) in order to implement the pilot within time and resource constraints, according to DOL officials. DOL selected three states for the pilot—Georgia, Washington, and West Virginia—based on a number of factors, including two states with a high veteran unemployment rate, as required by law. DOL instructed each of the states to conduct five workshops—West Virginia held an additional seven and Washington canceled one—and delegated the responsibility for choosing locations and marketing the pilot to state workforce agencies. States held the workshops at locations other than military facilities and employed different marketing approaches to publicize the workshops, including flyers and e-mail. DOL officials said that time and resource constraints, such as implementing the pilot within its existing TAP budget, influenced the department's pilot implementation, including its decision to use the same TAP workshops, conducted over 3 consecutive days, and to offer five workshops per state.

Officials in all three pilot states reported that the workshops benefitted veterans by enhancing their job search capabilities—including resume writing and interviewing—but states had difficulty attracting participants. The workshops generally fell short of DOL's attendance goals: a minimum of 10 participants and a preferred class size of 30-35 participants. A total of 250 participants attended the workshops and fewer than half of the workshops had 10 or more participants. Several state officials noted that it was difficult for veterans to schedule 3 consecutive business days to attend the workshop, and some suggested that shortening the course or offering night or weekend alternatives could have increased attendance.

DOL's design of the pilot limits the ability to inform Congress about the feasibility and advisability of expanding the pilot. DOL's two annual reports to Congress on the pilot provided information on topics such as workshop attendance, participant demographics and satisfaction with the workshop, and noteworthy state practices and challenges. While such information is useful, DOL's pilot design leaves unanswered key questions about the need for the program, the pilot's role amid other federal programs, and the goals and objectives for measuring its progress. For example, sound pilot design practices call for agencies to conduct a needs assessment, which could have helped DOL identify the population best targeted by the pilot, given veterans' varied employment experience and limited federal resources. DOL officials said that they did not have the time and resources to do such an assessment. Additionally, DOL did not assess the extent to which such a program might fill gaps in existing federal employment programs available to veterans, as sound pilot practices suggest. As a result, it remains unclear whether, as DOL officials contend, this pilot unnecessarily duplicates other programs. Moreover, this type of information could assist congressional deliberation about the need for future employment workshops and leverage the federal investment that has already been made in implementing the now-completed pilot program.

section_title: What GAO Recommends
paragraphs: GAO recommends that DOL assess and report to Congress the extent to which further delivery of the employment workshop to veterans and their spouses can fill a niche not fully served by existing federal programs. DOL agreed with GAO's recommendation and noted several actions it plans to take to address the recommendation.

section_title: Letter
paragraphs: As the United States continues to draw down its military following an  official end to the wars in Afghanistan and Iraq, hundreds of thousands of  military servicemembers are expected to transition to civilian life, some of  whom may face challenges finding employment. In 2014, over 7 percent  of post-9/11 veterans were unemployed, according to the Bureau of Labor  Statistics. Through a number of agencies, the federal government  provides programs that help ensure the readiness of servicemembers  leaving the military and help recently discharged veterans to transition to  civilian life. For example, under the Transition Assistance Program (TAP),  the Department of Labor (DOL) provides employment workshops at  military facilities to help servicemembers prepare for civilian employment.  These now-mandatory 3-day workshops teach servicemembers how to  search for jobs and provide information on labor market conditions.

In 2013, the Dignified Burial and Other Veterans’ Benefits Improvement  Act of 2012 (Dignified Burial Act) was enacted, which required DOL to  provide TAP to veterans and their spouses at locations other than military  facilities through a 2-year pilot that ended in January 2015. The act required DOL to conduct the pilot to assess the feasibility and advisability  of providing such a program. The Dignified Burial Act also included a  provision for GAO to report on the training provided under the act,  including the feasibility and advisability of expanding that training  nationally. We examined the following questions:  1.  How did DOL implement the veterans’ employment workshop pilot?  2.  What did state officials report regarding the benefits and challenges of  the workshops in the pilot states?  3.  How does the pilot inform decisions about its possible expansion?

To address our objectives, we reviewed relevant federal laws and  regulations and agency documentation. We also identified sound  practices on pilot design from federal agencies, subject matter experts,  and GAO’s guide that identifies elements of effective internal control,  including management’s responsibility for defining objectives in specific  and measurable terms, communicating necessary information to achieve  objectives, and establishing and operating monitoring activities. In  addition, we reviewed our past reports on pilot design, pilot  implementation, and high performance organizations to help identify  sound design practice criteria. GAO subject matter experts in program  design and evaluation assessed the extent to which those criteria were  relevant to our review of the veterans’ employment workshop pilot by  comparing the criteria with the steps of evaluation design listed in our  guide on program evaluations.Employment and Training Service (VETS) and DOL Chief Evaluation  Office officials to identify how the pilot was designed and how it could be  evaluated. Because of their expertise and involvement in addressing the  needs of veterans, we interviewed officials from the Department of  Veterans Affairs (VA) and several veteran service organizations about  how the pilot could have been designed and the extent to which DOL  collaborated with them in developing and implementing the pilot. We also  interviewed DOL and state workforce agency officials from each of the  three pilot states: Georgia, Washington, and West Virginia. During our  interviews we asked officials about how they implemented the pilot,   We interviewed DOL Veterans’  benefits of the employment workshops for pilot participants, and any  challenges the states faced. We obtained additional information on the  pilot from DOL participant data. Through interviews with knowledgeable  agency officials and reviewing agency documentation, we determined the  data were sufficiently reliable to describe the populations that took the  workshops and were enrolled in DOL’s American Job Centers (AJC).  DOL collected survey responses from veterans who attended the  workshops; however, we determined that we could not present these  survey data as representative of pilot participants generally because of  the relatively low survey response rate. Nonetheless, through interviews  with knowledgeable agency officials and reviewing agency  documentation, we determined that some use of survey data was  sufficiently reliable for our purposes, such as presenting qualitative  response information. Appendix I provides a more detailed discussion of  our scope and methodology.

We conducted this performance audit from July 2014 to July 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis from our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: DOL and VA Employment and Training Programs Serving Veterans
paragraphs: DOL and VA oversee four employment and training programs targeted to  veterans (see table 1). DOL administers its programs through state  workforce agencies in each state. DOL oversees these programs through  federal officials stationed in each region, including a Director of Veterans’  Employment and Training located in each state. DOL’s VETS administers  three of the employment programs targeted to veterans. VETS also funds  its portion of the TAP, which is a joint program with the Department of  Defense and VA. VA funds the Vocational Rehabilitation & Employment  Program. In December 2012, we examined the extent to which certain  federal veterans’ employment and training programs vary in terms of the  services they deliver and the veterans who receive them. We reported  that some federal veterans’ programs provide similar services (e.g., job  placement) but largely serve different populations.

In addition to its programs administered by VETS, DOL offers  employment and training services to the general population—including  veterans. These programs include a national system of public  employment services available to all individuals seeking employment.  Those services include job search and labor market information and, as  with its veterans programs, DOL administers its general programs  through state workforce agencies that provide services at AJCs. Formerly  known as One-Stop Career Centers, AJCs unify service locations for  multiple federally-funded employment and training programs in a single  system. AJCs serve two types of customers—job seekers and  employers—and provide access to a full range of services pertaining to  employment, training and education, employer assistance, and guidance  for obtaining other assistance. Federal law requires these programs to  give veterans priority over the general population, meaning that veterans  can access services ahead of the general population or, if funds are  limited, that veterans access services instead of the general population.  From April 1, 2012, to March 30, 2013, AJCs served about 1.6 million  veterans nationally through DOL’s Employment and Training  Administration.

Most of DOL’s employment service programs—both for veterans and the  general public—report the same performance measures, known as  common measures:

Percentage of program exiters who have obtained employment  (entered employment rate); percentage retaining employment for 6 months after exiting the  program (employment retention rate); and

6-month average earnings of program exiters (average earnings).

section_title: The Transition Assistance Program
paragraphs: TAP was established over 20 years ago to meet the needs of separating  servicemembers during their period of transition to civilian life by offering  job-search assistance and related services (see fig. 1). TAP consists of  several significant components provided by the Department of Defense,  VA, DOL, and the Small Business Administration, among others.

One of the components, DOL’s TAP employment workshops, consists of  comprehensive 3-day workshops at selected military installations  nationwide. Workshop attendees learn about job searches, career  decision-making, current occupational and labor market conditions,  resume and cover letter preparation, and interviewing techniques.  Participants also are provided with an evaluation of their employability  relative to the job market. TAP workshops must have a minimum of 10  participants and a maximum of 50 participants, per DOL guidelines. The  Department of Defense is generally required to mandate participation,  with some exceptions. Under the VOW to Hire Heroes Act of 2011, DOL  has been required to use a contractor to conduct these employment  workshops since November 21, 2013.

The funding level for the DOL-provided employment workshop  component of TAP for fiscal year 2014 was about $14 million, according  to DOL’s 2015 Congressional Budget Justification. With these funds,  VETS provided nearly 7,000 employment workshops that served over  207,000 participants, according to DOL. DOL reported that this was an 11  percent increase in participants, compared to fiscal year 2013. DOL  attributed the increased demand for its employment workshops to the fact  that TAP became mandatory for all transitioning servicemembers in fiscal  year 2013. DOL requested $14 million for fiscal year 2015 in order to  provide about 5,400 employment workshops with a planned average  class size of 35 exiting servicemembers or spouses. This amount  included the costs for conducting the pilot workshops outlined in the  Dignified Burial Act.

section_title: Online TAP
paragraphs: All of the content from TAP is also available online. The online or “virtual”  curriculum was added to benefit servicemembers at geographically  separated units, those with a short-notice separation and those  contemplating retirement because some servicemembers may not have  access to classrooms for transition instruction. Online TAP mirrors the  traditional TAP offerings, and allows servicemembers, veterans, and  spouses to access it from anywhere in the world.

section_title: Off-Base Transition Training
paragraphs: The Dignified Burial Act required DOL to provide TAP to veterans and  their spouses at locations other than military installations for the purposes  of assessing the feasibility and advisability of providing such a program.

The act required the workshops to be piloted for 2 years starting in  January 2013:  in three to five states (at least two of which had to have high levels of  veteran unemployment); at a sufficient number of locations to meet the needs of veterans and  spouses within each pilot state; anywhere except military installations (could, however, include  National Guard or reserve facilities not on active duty military  installations); and,  in a manner that generally follows the content of TAP.

The Dignified Burial Act also required DOL to provide to Congress an  annual report on the provision of the workshops for each year of the pilot.

section_title: DOL Used the Same Employment Workshops for Veterans It Provides to Servicemembers, Citing Time and Resource Constraints for Pilot’s Design
paragraphs: In response to the Dignified Burial Act, DOL provided the same  employment workshops—including the same curriculum delivered by the  same contractor—for the veterans’ employment workshop pilot as it uses  for active servicemembers as part of on-base TAP. DOL officials said the  TAP workshops offered employment training elements that were relevant  to veterans, such as information on the federal hiring process, resume  writing, and job searching practices. DOL officials also said that its TAP  workshops satisfied the Dignified Burial Act’s requirement that the training  should generally follow the content of the TAP.DOL has published two annual reports on the pilot.

As required by the act,  DOL provided the workshops to veterans in three states that volunteered  to assist with the pilot—Georgia, Washington, and West Virginia—from  November 2013 to December 2014. DOL officials said they selected the  pilot states using a number of factors, including selecting two states with  high veteran unemployment rates—Georgia and West Virginia—as   DOL also considered states’ veteran population,  required by law. number of military installations, infrastructure, geographic dispersion, and  capacity to administer the program.West Virginia because it also presented the opportunity to test the pilot in  one rural state.

Officials said that they selected  DOL instructed each state to hold 5 workshops; West Virginia  subsequently held 7 additional workshops and Washington canceled 1,  bringing the three-state total to 21. DOL provided guidance to the pilot  states about where to hold the workshops and other requirements and  then asked the state workforce agencies to select individual workshop  locations based on DOL officials’ belief that state workforce agency  officials would have a better idea of where the workshops were most  needed. For example, DOL told pilot states the workshops could be held  anywhere except military installations, as generally required by law, and  that the classrooms needed to have internet access and be able to  accommodate a large number of people. While each state held the  workshops at locations other than military installations, the three state  workforce agencies used different approaches in selecting locations for  the workshops. Georgia, for instance, chose locations in part using  veteran population data from VA’s website, Washington chose several  locations close to military installations, and West Virginia chose locations  mostly near rural areas. While states used different methods to select the  workshop locations, Georgia and Washington held most workshops at  AJCs. West Virginia held workshops mainly at National Guard  installations and community college campuses because the state’s AJCs  did not have classrooms that could accommodate large numbers of  participants or the technology to present the course materials, according  to state officials.

DOL also instructed states to market the pilot to veterans and their  spouses; each state developed its own marketing strategy. According to  state workforce agency officials we interviewed, they worked with other  organizations to inform veterans about the workshops. For example,  officials from Washington’s state workforce agency said they collaborated  with federal, state, and non-profit organizations, such as the National  Guard, Job Corps, and local VA offices, and also sent marketing e-mails  to unemployed veterans they identified through the Department of  Defense’s Unemployment Compensation for Ex-servicemembers  program. Similarly, in West Virginia, officials said they asked their  partners, including veteran service organizations and community  colleges, to inform their constituents about the workshops. West Virginia  officials advertised the workshops through mass marketing efforts, such  as television and the state workforce agency website. A Georgia official  told us they also worked with state and local partners to inform veterans  about the workshops and that they recruited participants through a  homeless women’s veterans center. Additionally, all three states  developed flyers that they distributed through their various partners.

Finally, DOL instructed pilot states to help DOL collect information on pilot  participants by enrolling workshop participants in the AJC database to get  basic demographic data on them and eventually track their employment  outcomes. Officials said they are collecting outcome data for workshop  participants using the same measures as other DOL employment  programs, namely whether veterans found a job, if they retained the job,  and their average wage earnings. Additionally, the department asked  states to administer DOL surveys that asked participants about their  satisfaction with the workshop, including whether it accomplished its  objectives and how much participants valued the training.

Workshops were conducted during weekday business hours due to AJCs not being  open during evenings or weekends. on the veterans’ employment workshops; officials said that developing a  new survey would have been costly and time consuming because, for  example, it would have required both departmental and Office of  Management and Budget approval.customer satisfaction questions such as the extent to which participants  felt the workshop was useful rather than why participants took the  workshop. Officials noted that DOL implemented the pilot within its  existing TAP budget and spent about $52,000 on costs for the pilot.

section_title: State Officials Reported That the Workshops Enhanced Veterans’ Job Search Skills but Attracted Few Participants
subsections: section_title: Some Officials Said the Pilot Helped Veterans Write Resumes and Improve Interviewing Skills
paragraphs: Officials in all three pilot states reported that the workshops benefitted  veterans by enhancing their job search capabilities, including helping  them (1) write resumes, (2) build interviewing skills, and (3) translate their  military experience into civilian job skills. For example, officials in West  Virginia stated that one workshop helped a veteran translate experience  as an infantryman and command sergeant major into core competencies  of personnel management and logistics, noting the veteran could market  his skills as a logistics expert and apply for management positions. The  officials further noted that the workshop likely increased participants’  confidence in their civilian lives, pointing specifically to one participant  who had been so motivated by the workshop that he found work and  shelter after being homeless. A National Guard representative in West  Virginia told us that the workshops also helped veterans prepare for job  fairs. Officials in Washington noted that some participants found the  workshop useful because it incorporated all of DOL’s individual  employment services into a single 3-day session in a more intensive  learning environment.

On average, the 110 participants who completed the DOL satisfaction  survey rated the workshop highly—respondents rated the course a 9 out  of 10 overall, with 10 being “excellent”—and many echoed the specific  benefits that officials noted. For example, 95 of 110 participants  reported that they believed both that the training was relevant to their job  search needs and that the training improved their skills. A little more  than half of the participants (59 of 110) also wrote in—rather than  checking a box—that they particularly valued the workshop’s resume  writing assistance. DOL’s second annual report to Congress on the pilot  noted that the satisfaction survey results across the three pilot states  were similar. In addition, officials from two veteran service organizations  told us that the workshops could enhance veterans’ job search skills. One  veteran service organization official also told us that the workshops could  be more meaningful to veterans than the TAP workshops they took before  they left the military because some separating servicemembers may not  be able to anticipate what challenges they will face in civilian life and they  may not realize how important the skills that are taught in TAP will be to  them.

section_title: States Struggled to Attract Participants and Meet Attendance Goals
paragraphs: Despite efforts noted earlier to market the pilot and recruit participants,  states struggled to attract participants and meet DOL’s targets for class  participation. In 21 workshops across the three states, a total of 250  veterans participated in the pilot, according to DOL data. DOL instructed  states to have a minimum of 10 participants enrolled in order to schedule  a workshop and a maximum of 50 participants—the same guidelines as  for DOL’s TAP workshops—with a preferred class size of 30-35  participants. Although several state officials told us that most workshops  had at least 10 veterans enrolled initially, fewer than half of the  workshops had 10 or more veterans actually attend; dozens of enrolled  veterans did not show up. Furthermore, only 2 workshops had more than  30 participants. The five largest workshops were all located in an area of  West Virginia in which participants may have been Department of  Defense civilian contract workers facing a reduction in force. As in  Georgia and Washington, workshops in other areas of West Virginia  averaged fewer than 10 participants (see fig. 2).

State officials and others offered several potential reasons for the  generally low participation in the pilot workshops. For example, a state  official in Georgia explained that some veterans may not have registered  for the workshops because they could not spare 3 consecutive business  days if, for example, they were engaged in a job search. Similarly, an  official from a veteran service organization noted some veterans might  need the time to work at a part-time job. Likewise, DOL’s second annual  report indicated that states felt challenged by the 3-day format, which they  believed precluded many individuals from participating and also resulted  in a number of participants leaving at some point during the 3 days. Some  state officials also said that some veterans who had registered for the  workshops dropped out because of “life demands,” such as not having  child care, which kept them from participating. An official in Georgia noted  that an ice storm forced them to postpone four of the workshops and  might have decreased participation in the rescheduled workshops. One  DOL official said that participation might have been low in some locations  because some veterans used services from similar programs such as job  search services through AJCs. After noticing that registrants were  dropping out, officials in West Virginia started calling registrants before  workshops to remind them. West Virginia officials also worked with a  hotel chain to offer free lodging during workshops to try to increase  participation. However, a West Virginia official said that these efforts did  not seem to have a significant effect on participation.

Several state and veteran service organization officials suggested ideas  for increasing participation for any future iteration of the workshops,  including (1) holding shorter workshops; (2) providing options for  participants to attend only certain portions, such as resume writing; (3)  offering workshops at night or on weekends; and (4) having DOL help  with marketing. DOL’s second annual report noted that states found the  lack of funding for marketing the pilot a challenge, saying that they could  have engaged in a more comprehensive recruiting effort to ensure greater  commitment from participants if funding had been available.

section_title: DOL’s Design of the Pilot Constrains the Ability to Inform Congressional Deliberation about the Pilot’s Future
subsections: section_title: DOL’s Annual Reports Address Attendance, Participant Satisfaction, and Noteworthy State Practices
paragraphs: DOL has published information about the veterans’ employment  workshop pilot in two annual reports required by the Dignified Burial Act.  The 2014 pilot report provided an interim picture of workshop attendance  and plans for data collection, and the 2015 report provided final  attendance numbers—250 participants in three states—and certain  participant demographics. The second annual report noted that a majority  of participants were male and between the ages of 25 and 44. We  conducted some further analyses of the data DOL collected, breaking  down age ranges, employment status, and education levels (see sidebar).  For example, the majority of participants for whom DOL collected  demographic data were employed. DOL also collected data through a  participant satisfaction survey; as we mentioned earlier, many survey  respondents believed the workshop had increased their job search  skills.

Additionally, DOL interviewed state workforce agencies on best practices  and challenges states faced. For example, DOL identified as a best  practice West Virginia’s formation of a diverse working group that  included the West Virginia National Guard, local workforce investment  boards, community colleges, and local veteran service organizations.  Regarding challenges, states noted a lack of funding for state marketing  as significant, according to DOL’s 2015 report. States also indicated that  the 3 consecutive day format excluded veterans who were unable to  commit to attending for that period of time and resulted in a number of  participants leaving at some point during the workshop. Lastly, the 2015  report also noted DOL plans to collect employment outcome data for  workshop participants, which will not be available until October 2016.

section_title: Design of the Pilot Missed Opportunities to Answer Key Questions That Could Inform Sound Decisions about Its Future
paragraphs: The Dignified Burial Act tasked us with reporting to Congress on the pilot,  including the feasibility and advisability of providing veterans’ employment  workshops nationally. However, DOL’s pilot design left unanswered at  least three key questions that sound design practices suggest should be  considered. Reponses to these questions could provide important  information regarding the feasibility and advisability of expanding  veterans’ employment workshops nationally.  1.  What is the need for the program overall and for any specific, targeted  groups of veterans?  2.  What is the pilot’s role amid existing federal employment and training  programs?  3.  Relative to any needs it identifies, against what goals and objectives  can DOL assess the pilot or an expanded version of it?
subsections: section_title: Identifying the Need
paragraphs: Sound pilot design practices call for a needs assessment to better identify  the population best served by the program or services being piloted as  well as those populations that might not need the services. DOL officials  said that they intended the pilot to serve the general veteran population  and their spouses, the eligible beneficiaries identified in the Dignified  Burial Act. Furthermore, DOL officials noted that because the law had  defined a population of eligible beneficiaries, it would have been  duplicative to conduct a needs assessment. As DOL noted in its second  annual report, however, the general veteran population has widely varied  employment experience, dates of separation, and disability status; this  suggests different levels of need for the workshops. Had DOL more  closely followed sound design practices by conducting a needs  assessment, it could have tested this assumption and determined where  the greatest need, if any, lay. For example, DOL could have determined  whether to target the program to all veterans, regardless of their  experiences or, instead, to certain targeted groups that may, because of  their situations, be more likely to need and benefit from the workshops.

DOL officials also said that there was not enough time and financial  resources to conduct a needs assessment, pointing, as noted earlier, to  the fact that DOL received no additional funding for this pilot and had to  complete it within 2 years.information without expending significant time or resources. For example,  DOL could have consulted with significant external stakeholders, such as  the VA and veteran service organizations, to seek input on questions  about veterans’ needs. Sound design practices indicate the importance of  having relevant and timely communications with all stakeholders  throughout the design and delivery of the program to fully engage all  parties and obtain information in order to achieve all of the pilot’s  objectives. DOL officials did not reach out to these stakeholders with  expertise in veterans’ issues because, as they designed it, the pilot did  not include information on benefits and services offered through the VA,  and thus they viewed collaboration with them as unnecessary. As a  result, DOL missed opportunities to gather potentially valuable input to  inform the pilot’s design, such as better determining the needs of the  veteran population, what their participation rates might be, and how to  most effectively use limited resources to market the workshops to  veterans. For example, officials from two veteran service organizations  said they would have helped DOL market the workshops in an effort to  increase attendance.

However, DOL could have gathered some  Without specific guidance from DOL about the needs the pilot should  address, states determined their own target populations. For example,  West Virginia targeted individuals in rural areas, while Georgia focused  on veterans who were not required to take TAP prior to leaving the  military, according to state workforce agency officials. lacks the benefit of knowing for which veterans the program was most  useful and thus whether its approach likely helped those most in need of  the resources it devoted to the program.

As previously noted, federal law mandates that the Department of Defense require  transitioning servicemembers to participate in an employment workshop, with some  exceptions.

In addition, because the design did not include a needs assessment, DOL  did not know how many workshops would be required to meet the  potential demand and the extent to which its existing resources would  allow it to meet the identified need. DOL officials said they determined  that existing resources would cover only 15 workshops across the three  states. Officials could not point to any formal analysis conducted to  determine this number, despite the Dignified Burial Act requiring DOL to  offer the workshops at a sufficient number of locations within each state  to meet the needs of eligible individuals. Officials from West Virginia’s  state workforce agency asked DOL to authorize additional workshops  because they said that five was insufficient to reach all of the veterans  they believed would likely benefit from attending.the additional workshops would help make them more geographically  accessible across the state. The overall low attendance levels for the pilot  workshops raise questions about the extent to which existing workshops  met the demand for them. If DOL had determined the number of  workshops needed in each state as part of a needs assessment, it could  have been better positioned to learn what additional demand—if any— there might be for the workshops and what the costs could be of meeting  that demand.

section_title: Identifying Pilot’s Role amid Existing Federal Programs
paragraphs: According to sound design practices, agencies should determine the  extent to which a pilot might fill gaps in federal services or enhance  existing programs while avoiding inappropriate overlap or duplication with  those programs. DOL followed this design practice somewhat, but could  have gone further. DOL officials said, prior to the enactment of the  Dignified Burial Act and in interviews during our review, that the pilot  duplicated existing employment and training programs already available  to veterans. However, DOL did not conduct a formal assessment of the  extent to which similar programs not only exist, but are also serving  veterans (and their spouses). By more closely following sound design  practices, DOL could have used the 2-year pilot to better understand the  population the program should target. For example, in addition to  consulting with stakeholders as previously noted, DOL could have used  data underlying a DOL-funded study of how AJCs serve veterans to learn  the numbers of veterans served by AJCs in the pilot states. By not  building into its design an assessment of the extent to which existing  programs serve the identified needs of all veterans (or targeted groups), it  is difficult to draw well-informed conclusions about the advisability of  expanding the pilot while being mindful of DOL’s resources and the need  to avoid unnecessary overlap and duplication of services veterans may  already be receiving.

Officials in one pilot state told us they thought the workshops filled a gap  by providing higher quality and more updated information on how to write  resumes and prepare for interviews than anything the state could deliver  to veterans. Officials from two veteran service organizations also  suggested the workshops could enhance DOL’s employment programs.  Officials from one veteran service organization told us that veterans could  receive basic employment information through the workshops and then  focus on specific training needs when they access other DOL services. In  contrast, a DOL official told us that he had attended a workshop and  thought DOL’s other employment and training programs covered all of the  topics in the workshop. Conducting a formal assessment of the extent to  which the pilot could either fill a service gap or enhance current programs  could have better positioned DOL to serve individuals whom existing  programs overlook or could serve in the future.

section_title: Goals and Objectives
paragraphs: Sound design principles indicate that goals and objectives can help an  agency define a pilot’s need and scope, as well as help uncover any  differences in expectations and concerns program stakeholders have with  implementing the pilot. DOL officials told us that the goal of the pilot was  to test its feasibility. However, DOL did not create clear, measurable  goals—as well as objectives that link to those goals—that would define  what testing the pilot’s feasibility meant. States, in turn, administered the  pilot based on their perception of the pilot’s goals. For instance, West  Virginia officials said the purpose of the pilot was to help veterans get  “job-ready.” Washington and Georgia officials said the purpose was to  serve individuals who did not take TAP.

The lack of clearly and consistently defined goals and objectives limits  DOL’s ability to measure the pilot’s performance related to any expected  outcome or better inform decisions about its future. DOL’s annual report  provides information on participant demographics, survey data, and  upcoming employment outcomes, but DOL cannot determine how well  the pilot performed against specific goals and objectives. Therefore, the  department missed an opportunity to better establish the value of the  pilot, which could have assisted stakeholders such as the Congress in  making sound decisions about its future.

section_title: Conclusions
paragraphs: The federal government has created a number of programs that assist  veterans and servicemembers transitioning to civilian life, including the  employment workshop pilot for veterans and their spouses. In embarking  on this effort, DOL’s approach—asking states to volunteer to participate  and making no modifications to the existing TAP workshop—allowed it to  implement the pilot within 2 years and without additional funding. While  this approach was understandable given the time and resource  constraints DOL cited, the pilot design did not fully embrace certain sound  practices that would have enabled it to better inform congressional  decision-making about the advisability of expanding the pilot. Specifically,  questions about the need for offering the workshops to veterans and their  spouses, the program’s role amid other existing federal programs, and  specific goals and objectives against which program performance could  be measured remain unanswered. The pilot has ended, but these  questions raise the issue of how to leverage the federal investment that  was made in the pilot as well as the efforts the states put into delivering  the workshops.

In light of the low attendance levels of veterans in the pilot workshop,  information on the fundamental issue of whether such a workshop could  fill a niche for veterans that is not currently met by existing programs is  important. Collaboration with stakeholders such as VA and veteran  service organizations could provide useful perspective in developing this  information. Such information could assist Congress in making sound  decisions on any potential future iterations of the veterans’ employment  workshop.

section_title: Recommendation for Executive Action
paragraphs: To inform decisions on any potential future iterations of the veterans’  employment workshop, we recommend that the Secretary of Labor  assess and report to Congress the extent to which further delivery of  employment workshops to veterans and their spouses could fill a niche  not fully served by existing federal programs. Such an assessment could  involve collaboration with VA and other stakeholder organizations.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft copy of this report to DOL for review and comment.  DOL concurred with our recommendation and noted that it will conduct a  deliberate assessment of (1) the need for the services offered under the  pilot; (2) which services are most useful for veterans and their spouses;  and (3) what overlap exists with programs providing similar services to  this population. DOL added that the assessment should compare the  services the AJC system provides to veterans and their spouses with the  information contained in DOL’s veterans’ employment workshops and  online curriculum. DOL also stated that the assessment should identify if  specific services provided within the AJC system require revision or  redesign to better meet the employment needs of veterans and their  spouses. DOL’s comments are reproduced in appendix II.

We also provided relevant portions of the draft to VA, which had no  comments.

We are sending copies of this report to relevant congressional  committees, the Secretary of Labor, and other interested parties. In  addition, this report will be available at no charge on GAO’s website at  http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix VI.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to examine (1) how the Department of  Labor (DOL) implemented the veterans’ employment workshop pilot  program, (2) what state officials reported regarding the benefits and  challenges of the workshops in the pilots, and (3) how the pilot informs  decisions about its possible expansion.

To provide information on how DOL designed the pilot, how it planned to  evaluate it, and the extent to which DOL collaborated with other  organizations during the pilot’s design, we reviewed relevant federal laws  and regulations and e-mail correspondence between DOL officials on the  steps they took to design the pilot and documents describing DOL’s plan  for collecting data to determine the pilot performance. We also  interviewed officials from DOL’s Veterans’ and Employment Training  Service (VETS) about how they selected the states to test the pilot, what  steps were taken to set up the infrastructure of the workshops, such as  classroom requirements and training materials, and the extent to which  they collaborated with other federal and non-federal entities. We also  asked VETS officials about the types of performance data they planned to  collect and how they planned to report the results of the pilot. We  interviewed Department of Veterans Affairs and veteran service  organization officials about the extent to which DOL collaborated with  them during the design of the pilot and the extent to which DOL  collaborated with them in developing and implementing the pilot.

To provide information on how the pilot was implemented and what state  officials reported regarding the benefits and challenges of the workshops  in the pilots, we conducted telephonic interviews with DOL and state  workforce agency officials in each of the pilot states: Georgia,  Washington, and West Virginia. During our interviews we asked these  officials what their roles and responsibilities were, how workshop  locations were selected, what marketing efforts states employed to  publicize the workshops, and the extent to which they collaborated with  other federal and non-federal entities. We also asked about the benefits  the veterans’ employment workshops offered participants as they  searched for employment, the extent to which the workshops provided  information on veterans’ benefits not offered by DOL, and any challenges  the states faced in delivering the workshops. We reviewed DOL  participant data, the results of DOL’s survey of workshop participants, and  DOL’s two annual reports on the pilot, including verifying the reports’  workshop attendance figures with DOL and pilot states. To determine the  reliability of the data, we interviewed agency officials knowledgeable of  DOL’s data collection and reporting systems and reviewed agency  documentation. We concluded the participant data were sufficiently  reliable to describe certain demographic characteristics of the population  that took the workshops and were enrolled in DOL’s American Job  Centers (AJC). To determine the reliability of the survey data, we  interviewed agency officials who developed the survey, reviewed agency  documentation, and consulted with GAO subject matter experts in survey  design. We determined that DOL’s survey data were sufficiently reliable  for presenting qualitative response information.

To identify how the pilot informs decisions about its possible expansion,  we reviewed how practices in pilot design can affect what an agency  learns in its evaluation of a pilot. We reviewed guidance from federal  agencies on program design and evaluation found during the course of  our research and guidance identified by subject matter experts. We  reviewed design and evaluation guidance from the Centers for Disease  Control and Prevention because officials from DOL’s Chief Evaluation  Office said they use that guidance for their evaluation practices. We  reviewed design guidance recommended by GAO subject matter experts  on program design and evaluation. We also reviewed GAO’s Standards  for Internal Control, which identifies elements of effective internal controls  including management responsibility for defining objectives in specific and  measurable terms, communicating necessary information to achieve  objectives, and establishing and operating monitoring activities.  Moreover, we conducted an independent search for guidance to further  support the standards for program design noted in this report. We found,  for example, guidance supporting the standard of establishing and  maintaining clear communication channels with relevant stakeholders  from the Department of Health and Human Services’ Agency for  Healthcare Research and Quality and the National Institutes of Health.  We found additional support for conducting a needs assessment  identifying whether a new program is needed from the Department of  Education and the Department of Transportation. GAO subject matter  experts in program design and evaluation verified the leading practices  and the extent to which the practices were relevant to our review of the  veterans’ employment workshop pilot by comparing the criteria with steps  of evaluation design listed in our guide on program evaluations. See  table 2 for the list of leading practices we identified in pilot design and  pilot implementation.

section_title: Appendix II: Comments from the Department of Labor

section_title: Appendix III: Selected State Marketing Flyers for Veterans’ Employment Workshops

section_title: Appendix IV: Department of Labor Veterans’ Employment Workshop Locations and Participation Levels
subsections: section_title: Total attendance for all states

section_title: Appendix V: Department of Labor Survey of Veterans’ Employment Workshop Participants

section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Andrew Sherrill, (202) 512-7215 or sherrilla@gao.gov.

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Bill MacBlane (Assistant  Director), Drew Nelson (Analyst in Charge), David Chrisinger, David  Forgosh, Alex Galuten, Monika Gomez, Stephanie Shipman, and Walter  Vance made key contributions to this report. Also contributing to this  report were Juli Cutts, Laura Hoffrey, Kathy Leslie, Mimi Nguyen, Ronni  Schwartz, and James Whitcomb.


Question-summary annotations: questions: question: What is required of the Department of Labor?
answer: The Department of Labor (DOL) was required by law to provide employment workshops to veterans and their spouses in a pilot program.

question: How has the DOL facilitated growth of veterans in the workforce?
answer: In response, DOL used the same 3-day employment workshops for the pilot that it provides to servicemembers on military bases as part of the Transition Assistance Program (TAP) in order to implement the pilot within time and resource constraints, according to DOL officials.
child_questions: question: How has the DOL tested the effectiveness of these programs?
answer: DOL selected three states for the pilot—Georgia, Washington, and West Virginia—based on a number of factors, including two states with a high veteran unemployment rate, as required by law.
child_questions: question: What was expected of these three states?
answer: DOL instructed each of the states to conduct five workshops—West Virginia held an additional seven and Washington canceled one—and delegated the responsibility for choosing locations and marketing the pilot to state workforce agencies.

question: What were some of the outcomes of these workshops?
answer: DOL officials said that time and resource constraints, such as implementing the pilot within its existing TAP budget, influenced the department's pilot implementation, including its decision to use the same TAP workshops, conducted over 3 consecutive days, and to offer five workshops per state.

questions: question: What are some benefits and drawbacks of the pilot design?
answer: Officials in all three pilot states reported that the workshops benefitted veterans by enhancing their job search capabilities—including resume writing and interviewing—but states had difficulty attracting participants.

question: How did the workshops' outcomes fail to meet the DOL's goals?
answer: The workshops generally fell short of DOL's attendance goals: a minimum of 10 participants and a preferred class size of 30-35 participants.
child_questions: question: How many participants were there in attendance?
answer: A total of 250 participants attended the workshops and fewer than half of the workshops had 10 or more participants.

question: Why were the attendance and participation goals not met?
answer: Several state officials noted that it was difficult for veterans to schedule 3 consecutive business days to attend the workshop, and some suggested that shortening the course or offering night or weekend alternatives could have increased attendance.

questions: question: What is one consequence of the DOL's pilot design?
answer: DOL's design of the pilot limits the ability to inform Congress about the feasibility and advisability of expanding the pilot.

question: What is included in the DOL's annual reports to Congress?
answer: DOL's two annual reports to Congress on the pilot provided information on topics such as workshop attendance, participant demographics and satisfaction with the workshop, and noteworthy state practices and challenges.
child_questions: question: What topics are not fully addressed in the DOL's pilot design?
answer: While such information is useful, DOL's pilot design leaves unanswered key questions about the need for the program, the pilot's role amid other federal programs, and the goals and objectives for measuring its progress.
child_questions: question: Why were these topics not addressed completely?
answer: DOL officials said that they did not have the time and resources to do such an assessment. Additionally, DOL did not assess the extent to which such a program might fill gaps in existing federal employment programs available to veterans, as sound pilot practices suggest.

question: How does this affect the clarity of the program's effectiveness?
answer: As a result, it remains unclear whether, as DOL officials contend, this pilot unnecessarily duplicates other programs.
child_questions: question: Why is the information that is lacking from the DOL's programs valuable?
answer: Moreover, this type of information could assist congressional deliberation about the need for future employment workshops and leverage the federal investment that has already been made in implementing the now-completed pilot program.

questions: question: How does the federal government help with veterans seeking employment?
answer: The federal government has long offered programs that assist veterans with finding employment.

question: What is the purpose of the Dignified Burial and Other Veterans' Benefits Improvement Act of 2012?
answer: In 2013, the Dignified Burial and Other Veterans' Benefits Improvement Act of 2012 was enacted, which required DOL to provide employment workshops to veterans and their spouses at locations other than military facilities through a 2-year pilot that ended in January 2015.

question: Why might the GAO be involved with events relating to the act?
answer: The act also included a provision for GAO to report on the training and possible expansion of the pilot.