Veterans' Employment: Need for Further Workshops Should Be Considered before Making Decisions on Their Future
Released: 2015-07-16T00:00:00

Letter

Background

DOL and VA Employment and Training Programs Serving Veterans

DOL and VA oversee four employment and training programs targeted to veterans (see table 1). DOL administers its programs through state workforce agencies in each state. DOL oversees these programs through federal officials stationed in each region, including a Director of Veterans’ Employment and Training located in each state. DOL’s VETS administers three of the employment programs targeted to veterans. VETS also funds its portion of the TAP, which is a joint program with the Department of Defense and VA. VA funds the Vocational Rehabilitation & Employment Program. In December 2012, we examined the extent to which certain federal veterans’ employment and training programs vary in terms of the services they deliver and the veterans who receive them. We reported that some federal veterans’ programs provide similar services (e.g., job placement) but largely serve different populations.

In addition to its programs administered by VETS, DOL offers employment and training services to the general population—including veterans. These programs include a national system of public employment services available to all individuals seeking employment. Those services include job search and labor market information and, as with its veterans programs, DOL administers its general programs through state workforce agencies that provide services at AJCs. Formerly known as One-Stop Career Centers, AJCs unify service locations for multiple federally-funded employment and training programs in a single system. AJCs serve two types of customers—job seekers and employers—and provide access to a full range of services pertaining to employment, training and education, employer assistance, and guidance for obtaining other assistance. Federal law requires these programs to give veterans priority over the general population, meaning that veterans can access services ahead of the general population or, if funds are limited, that veterans access services instead of the general population. From April 1, 2012, to March 30, 2013, AJCs served about 1.6 million veterans nationally through DOL’s Employment and Training Administration.

Most of DOL’s employment service programs—both for veterans and the general public—report the same performance measures, known as common measures:

Percentage of program exiters who have obtained employment (entered employment rate); percentage retaining employment for 6 months after exiting the program (employment retention rate); and

6-month average earnings of program exiters (average earnings).

The Transition Assistance Program

TAP was established over 20 years ago to meet the needs of separating servicemembers during their period of transition to civilian life by offering job-search assistance and related services (see fig. 1). TAP consists of several significant components provided by the Department of Defense, VA, DOL, and the Small Business Administration, among others.

One of the components, DOL’s TAP employment workshops, consists of comprehensive 3-day workshops at selected military installations nationwide. Workshop attendees learn about job searches, career decision-making, current occupational and labor market conditions, resume and cover letter preparation, and interviewing techniques. Participants also are provided with an evaluation of their employability relative to the job market. TAP workshops must have a minimum of 10 participants and a maximum of 50 participants, per DOL guidelines. The Department of Defense is generally required to mandate participation, with some exceptions. Under the VOW to Hire Heroes Act of 2011, DOL has been required to use a contractor to conduct these employment workshops since November 21, 2013.

The funding level for the DOL-provided employment workshop component of TAP for fiscal year 2014 was about $14 million, according to DOL’s 2015 Congressional Budget Justification. With these funds, VETS provided nearly 7,000 employment workshops that served over 207,000 participants, according to DOL. DOL reported that this was an 11 percent increase in participants, compared to fiscal year 2013. DOL attributed the increased demand for its employment workshops to the fact that TAP became mandatory for all transitioning servicemembers in fiscal year 2013. DOL requested $14 million for fiscal year 2015 in order to provide about 5,400 employment workshops with a planned average class size of 35 exiting servicemembers or spouses. This amount included the costs for conducting the pilot workshops outlined in the Dignified Burial Act.

Online TAP

All of the content from TAP is also available online. The online or “virtual” curriculum was added to benefit servicemembers at geographically separated units, those with a short-notice separation and those contemplating retirement because some servicemembers may not have access to classrooms for transition instruction. Online TAP mirrors the traditional TAP offerings, and allows servicemembers, veterans, and spouses to access it from anywhere in the world.

Off-Base Transition Training

The Dignified Burial Act required DOL to provide TAP to veterans and their spouses at locations other than military installations for the purposes of assessing the feasibility and advisability of providing such a program.

The act required the workshops to be piloted for 2 years starting in January 2013: in three to five states (at least two of which had to have high levels of veteran unemployment); at a sufficient number of locations to meet the needs of veterans and spouses within each pilot state; anywhere except military installations (could, however, include National Guard or reserve facilities not on active duty military installations); and, in a manner that generally follows the content of TAP.

The Dignified Burial Act also required DOL to provide to Congress an annual report on the provision of the workshops for each year of the pilot.

DOL Used the Same Employment Workshops for Veterans It Provides to Servicemembers, Citing Time and Resource Constraints for Pilot’s Design

In response to the Dignified Burial Act, DOL provided the same employment workshops—including the same curriculum delivered by the same contractor—for the veterans’ employment workshop pilot as it uses for active servicemembers as part of on-base TAP. DOL officials said the TAP workshops offered employment training elements that were relevant to veterans, such as information on the federal hiring process, resume writing, and job searching practices. DOL officials also said that its TAP workshops satisfied the Dignified Burial Act’s requirement that the training should generally follow the content of the TAP.DOL has published two annual reports on the pilot.

As required by the act, DOL provided the workshops to veterans in three states that volunteered to assist with the pilot—Georgia, Washington, and West Virginia—from November 2013 to December 2014. DOL officials said they selected the pilot states using a number of factors, including selecting two states with high veteran unemployment rates—Georgia and West Virginia—as DOL also considered states’ veteran population, required by law. number of military installations, infrastructure, geographic dispersion, and capacity to administer the program.West Virginia because it also presented the opportunity to test the pilot in one rural state.

Officials said that they selected DOL instructed each state to hold 5 workshops; West Virginia subsequently held 7 additional workshops and Washington canceled 1, bringing the three-state total to 21. DOL provided guidance to the pilot states about where to hold the workshops and other requirements and then asked the state workforce agencies to select individual workshop locations based on DOL officials’ belief that state workforce agency officials would have a better idea of where the workshops were most needed. For example, DOL told pilot states the workshops could be held anywhere except military installations, as generally required by law, and that the classrooms needed to have internet access and be able to accommodate a large number of people. While each state held the workshops at locations other than military installations, the three state workforce agencies used different approaches in selecting locations for the workshops. Georgia, for instance, chose locations in part using veteran population data from VA’s website, Washington chose several locations close to military installations, and West Virginia chose locations mostly near rural areas. While states used different methods to select the workshop locations, Georgia and Washington held most workshops at AJCs. West Virginia held workshops mainly at National Guard installations and community college campuses because the state’s AJCs did not have classrooms that could accommodate large numbers of participants or the technology to present the course materials, according to state officials.

DOL also instructed states to market the pilot to veterans and their spouses; each state developed its own marketing strategy. According to state workforce agency officials we interviewed, they worked with other organizations to inform veterans about the workshops. For example, officials from Washington’s state workforce agency said they collaborated with federal, state, and non-profit organizations, such as the National Guard, Job Corps, and local VA offices, and also sent marketing e-mails to unemployed veterans they identified through the Department of Defense’s Unemployment Compensation for Ex-servicemembers program. Similarly, in West Virginia, officials said they asked their partners, including veteran service organizations and community colleges, to inform their constituents about the workshops. West Virginia officials advertised the workshops through mass marketing efforts, such as television and the state workforce agency website. A Georgia official told us they also worked with state and local partners to inform veterans about the workshops and that they recruited participants through a homeless women’s veterans center. Additionally, all three states developed flyers that they distributed through their various partners.

Finally, DOL instructed pilot states to help DOL collect information on pilot participants by enrolling workshop participants in the AJC database to get basic demographic data on them and eventually track their employment outcomes. Officials said they are collecting outcome data for workshop participants using the same measures as other DOL employment programs, namely whether veterans found a job, if they retained the job, and their average wage earnings. Additionally, the department asked states to administer DOL surveys that asked participants about their satisfaction with the workshop, including whether it accomplished its objectives and how much participants valued the training.

Workshops were conducted during weekday business hours due to AJCs not being open during evenings or weekends. on the veterans’ employment workshops; officials said that developing a new survey would have been costly and time consuming because, for example, it would have required both departmental and Office of Management and Budget approval.customer satisfaction questions such as the extent to which participants felt the workshop was useful rather than why participants took the workshop. Officials noted that DOL implemented the pilot within its existing TAP budget and spent about $52,000 on costs for the pilot.

State Officials Reported That the Workshops Enhanced Veterans’ Job Search Skills but Attracted Few Participants

Some Officials Said the Pilot Helped Veterans Write Resumes and Improve Interviewing Skills

Officials in all three pilot states reported that the workshops benefitted veterans by enhancing their job search capabilities, including helping them (1) write resumes, (2) build interviewing skills, and (3) translate their military experience into civilian job skills. For example, officials in West Virginia stated that one workshop helped a veteran translate experience as an infantryman and command sergeant major into core competencies of personnel management and logistics, noting the veteran could market his skills as a logistics expert and apply for management positions. The officials further noted that the workshop likely increased participants’ confidence in their civilian lives, pointing specifically to one participant who had been so motivated by the workshop that he found work and shelter after being homeless. A National Guard representative in West Virginia told us that the workshops also helped veterans prepare for job fairs. Officials in Washington noted that some participants found the workshop useful because it incorporated all of DOL’s individual employment services into a single 3-day session in a more intensive learning environment.

On average, the 110 participants who completed the DOL satisfaction survey rated the workshop highly—respondents rated the course a 9 out of 10 overall, with 10 being “excellent”—and many echoed the specific benefits that officials noted. For example, 95 of 110 participants reported that they believed both that the training was relevant to their job search needs and that the training improved their skills. A little more than half of the participants (59 of 110) also wrote in—rather than checking a box—that they particularly valued the workshop’s resume writing assistance. DOL’s second annual report to Congress on the pilot noted that the satisfaction survey results across the three pilot states were similar. In addition, officials from two veteran service organizations told us that the workshops could enhance veterans’ job search skills. One veteran service organization official also told us that the workshops could be more meaningful to veterans than the TAP workshops they took before they left the military because some separating servicemembers may not be able to anticipate what challenges they will face in civilian life and they may not realize how important the skills that are taught in TAP will be to them.

States Struggled to Attract Participants and Meet Attendance Goals

Despite efforts noted earlier to market the pilot and recruit participants, states struggled to attract participants and meet DOL’s targets for class participation. In 21 workshops across the three states, a total of 250 veterans participated in the pilot, according to DOL data. DOL instructed states to have a minimum of 10 participants enrolled in order to schedule a workshop and a maximum of 50 participants—the same guidelines as for DOL’s TAP workshops—with a preferred class size of 30-35 participants. Although several state officials told us that most workshops had at least 10 veterans enrolled initially, fewer than half of the workshops had 10 or more veterans actually attend; dozens of enrolled veterans did not show up. Furthermore, only 2 workshops had more than 30 participants. The five largest workshops were all located in an area of West Virginia in which participants may have been Department of Defense civilian contract workers facing a reduction in force. As in Georgia and Washington, workshops in other areas of West Virginia averaged fewer than 10 participants (see fig. 2).

State officials and others offered several potential reasons for the generally low participation in the pilot workshops. For example, a state official in Georgia explained that some veterans may not have registered for the workshops because they could not spare 3 consecutive business days if, for example, they were engaged in a job search. Similarly, an official from a veteran service organization noted some veterans might need the time to work at a part-time job. Likewise, DOL’s second annual report indicated that states felt challenged by the 3-day format, which they believed precluded many individuals from participating and also resulted in a number of participants leaving at some point during the 3 days. Some state officials also said that some veterans who had registered for the workshops dropped out because of “life demands,” such as not having child care, which kept them from participating. An official in Georgia noted that an ice storm forced them to postpone four of the workshops and might have decreased participation in the rescheduled workshops. One DOL official said that participation might have been low in some locations because some veterans used services from similar programs such as job search services through AJCs. After noticing that registrants were dropping out, officials in West Virginia started calling registrants before workshops to remind them. West Virginia officials also worked with a hotel chain to offer free lodging during workshops to try to increase participation. However, a West Virginia official said that these efforts did not seem to have a significant effect on participation.

Several state and veteran service organization officials suggested ideas for increasing participation for any future iteration of the workshops, including (1) holding shorter workshops; (2) providing options for participants to attend only certain portions, such as resume writing; (3) offering workshops at night or on weekends; and (4) having DOL help with marketing. DOL’s second annual report noted that states found the lack of funding for marketing the pilot a challenge, saying that they could have engaged in a more comprehensive recruiting effort to ensure greater commitment from participants if funding had been available.

DOL’s Design of the Pilot Constrains the Ability to Inform Congressional Deliberation about the Pilot’s Future

DOL’s Annual Reports Address Attendance, Participant Satisfaction, and Noteworthy State Practices

DOL has published information about the veterans’ employment workshop pilot in two annual reports required by the Dignified Burial Act. The 2014 pilot report provided an interim picture of workshop attendance and plans for data collection, and the 2015 report provided final attendance numbers—250 participants in three states—and certain participant demographics. The second annual report noted that a majority of participants were male and between the ages of 25 and 44. We conducted some further analyses of the data DOL collected, breaking down age ranges, employment status, and education levels (see sidebar). For example, the majority of participants for whom DOL collected demographic data were employed. DOL also collected data through a participant satisfaction survey; as we mentioned earlier, many survey respondents believed the workshop had increased their job search skills.

Additionally, DOL interviewed state workforce agencies on best practices and challenges states faced. For example, DOL identified as a best practice West Virginia’s formation of a diverse working group that included the West Virginia National Guard, local workforce investment boards, community colleges, and local veteran service organizations. Regarding challenges, states noted a lack of funding for state marketing as significant, according to DOL’s 2015 report. States also indicated that the 3 consecutive day format excluded veterans who were unable to commit to attending for that period of time and resulted in a number of participants leaving at some point during the workshop. Lastly, the 2015 report also noted DOL plans to collect employment outcome data for workshop participants, which will not be available until October 2016.

Design of the Pilot Missed Opportunities to Answer Key Questions That Could Inform Sound Decisions about Its Future

The Dignified Burial Act tasked us with reporting to Congress on the pilot, including the feasibility and advisability of providing veterans’ employment workshops nationally. However, DOL’s pilot design left unanswered at least three key questions that sound design practices suggest should be considered. Reponses to these questions could provide important information regarding the feasibility and advisability of expanding veterans’ employment workshops nationally. 1. What is the need for the program overall and for any specific, targeted groups of veterans? 2. What is the pilot’s role amid existing federal employment and training programs? 3. Relative to any needs it identifies, against what goals and objectives can DOL assess the pilot or an expanded version of it?

Identifying the Need

Sound pilot design practices call for a needs assessment to better identify the population best served by the program or services being piloted as well as those populations that might not need the services. DOL officials said that they intended the pilot to serve the general veteran population and their spouses, the eligible beneficiaries identified in the Dignified Burial Act. Furthermore, DOL officials noted that because the law had defined a population of eligible beneficiaries, it would have been duplicative to conduct a needs assessment. As DOL noted in its second annual report, however, the general veteran population has widely varied employment experience, dates of separation, and disability status; this suggests different levels of need for the workshops. Had DOL more closely followed sound design practices by conducting a needs assessment, it could have tested this assumption and determined where the greatest need, if any, lay. For example, DOL could have determined whether to target the program to all veterans, regardless of their experiences or, instead, to certain targeted groups that may, because of their situations, be more likely to need and benefit from the workshops.

DOL officials also said that there was not enough time and financial resources to conduct a needs assessment, pointing, as noted earlier, to the fact that DOL received no additional funding for this pilot and had to complete it within 2 years.information without expending significant time or resources. For example, DOL could have consulted with significant external stakeholders, such as the VA and veteran service organizations, to seek input on questions about veterans’ needs. Sound design practices indicate the importance of having relevant and timely communications with all stakeholders throughout the design and delivery of the program to fully engage all parties and obtain information in order to achieve all of the pilot’s objectives. DOL officials did not reach out to these stakeholders with expertise in veterans’ issues because, as they designed it, the pilot did not include information on benefits and services offered through the VA, and thus they viewed collaboration with them as unnecessary. As a result, DOL missed opportunities to gather potentially valuable input to inform the pilot’s design, such as better determining the needs of the veteran population, what their participation rates might be, and how to most effectively use limited resources to market the workshops to veterans. For example, officials from two veteran service organizations said they would have helped DOL market the workshops in an effort to increase attendance.

However, DOL could have gathered some Without specific guidance from DOL about the needs the pilot should address, states determined their own target populations. For example, West Virginia targeted individuals in rural areas, while Georgia focused on veterans who were not required to take TAP prior to leaving the military, according to state workforce agency officials. lacks the benefit of knowing for which veterans the program was most useful and thus whether its approach likely helped those most in need of the resources it devoted to the program.

As previously noted, federal law mandates that the Department of Defense require transitioning servicemembers to participate in an employment workshop, with some exceptions.

In addition, because the design did not include a needs assessment, DOL did not know how many workshops would be required to meet the potential demand and the extent to which its existing resources would allow it to meet the identified need. DOL officials said they determined that existing resources would cover only 15 workshops across the three states. Officials could not point to any formal analysis conducted to determine this number, despite the Dignified Burial Act requiring DOL to offer the workshops at a sufficient number of locations within each state to meet the needs of eligible individuals. Officials from West Virginia’s state workforce agency asked DOL to authorize additional workshops because they said that five was insufficient to reach all of the veterans they believed would likely benefit from attending.the additional workshops would help make them more geographically accessible across the state. The overall low attendance levels for the pilot workshops raise questions about the extent to which existing workshops met the demand for them. If DOL had determined the number of workshops needed in each state as part of a needs assessment, it could have been better positioned to learn what additional demand—if any— there might be for the workshops and what the costs could be of meeting that demand.

Identifying Pilot’s Role amid Existing Federal Programs

According to sound design practices, agencies should determine the extent to which a pilot might fill gaps in federal services or enhance existing programs while avoiding inappropriate overlap or duplication with those programs. DOL followed this design practice somewhat, but could have gone further. DOL officials said, prior to the enactment of the Dignified Burial Act and in interviews during our review, that the pilot duplicated existing employment and training programs already available to veterans. However, DOL did not conduct a formal assessment of the extent to which similar programs not only exist, but are also serving veterans (and their spouses). By more closely following sound design practices, DOL could have used the 2-year pilot to better understand the population the program should target. For example, in addition to consulting with stakeholders as previously noted, DOL could have used data underlying a DOL-funded study of how AJCs serve veterans to learn the numbers of veterans served by AJCs in the pilot states. By not building into its design an assessment of the extent to which existing programs serve the identified needs of all veterans (or targeted groups), it is difficult to draw well-informed conclusions about the advisability of expanding the pilot while being mindful of DOL’s resources and the need to avoid unnecessary overlap and duplication of services veterans may already be receiving.

Officials in one pilot state told us they thought the workshops filled a gap by providing higher quality and more updated information on how to write resumes and prepare for interviews than anything the state could deliver to veterans. Officials from two veteran service organizations also suggested the workshops could enhance DOL’s employment programs. Officials from one veteran service organization told us that veterans could receive basic employment information through the workshops and then focus on specific training needs when they access other DOL services. In contrast, a DOL official told us that he had attended a workshop and thought DOL’s other employment and training programs covered all of the topics in the workshop. Conducting a formal assessment of the extent to which the pilot could either fill a service gap or enhance current programs could have better positioned DOL to serve individuals whom existing programs overlook or could serve in the future.

Goals and Objectives

Sound design principles indicate that goals and objectives can help an agency define a pilot’s need and scope, as well as help uncover any differences in expectations and concerns program stakeholders have with implementing the pilot. DOL officials told us that the goal of the pilot was to test its feasibility. However, DOL did not create clear, measurable goals—as well as objectives that link to those goals—that would define what testing the pilot’s feasibility meant. States, in turn, administered the pilot based on their perception of the pilot’s goals. For instance, West Virginia officials said the purpose of the pilot was to help veterans get “job-ready.” Washington and Georgia officials said the purpose was to serve individuals who did not take TAP.

The lack of clearly and consistently defined goals and objectives limits DOL’s ability to measure the pilot’s performance related to any expected outcome or better inform decisions about its future. DOL’s annual report provides information on participant demographics, survey data, and upcoming employment outcomes, but DOL cannot determine how well the pilot performed against specific goals and objectives. Therefore, the department missed an opportunity to better establish the value of the pilot, which could have assisted stakeholders such as the Congress in making sound decisions about its future.

Conclusions

The federal government has created a number of programs that assist veterans and servicemembers transitioning to civilian life, including the employment workshop pilot for veterans and their spouses. In embarking on this effort, DOL’s approach—asking states to volunteer to participate and making no modifications to the existing TAP workshop—allowed it to implement the pilot within 2 years and without additional funding. While this approach was understandable given the time and resource constraints DOL cited, the pilot design did not fully embrace certain sound practices that would have enabled it to better inform congressional decision-making about the advisability of expanding the pilot. Specifically, questions about the need for offering the workshops to veterans and their spouses, the program’s role amid other existing federal programs, and specific goals and objectives against which program performance could be measured remain unanswered. The pilot has ended, but these questions raise the issue of how to leverage the federal investment that was made in the pilot as well as the efforts the states put into delivering the workshops.

In light of the low attendance levels of veterans in the pilot workshop, information on the fundamental issue of whether such a workshop could fill a niche for veterans that is not currently met by existing programs is important. Collaboration with stakeholders such as VA and veteran service organizations could provide useful perspective in developing this information. Such information could assist Congress in making sound decisions on any potential future iterations of the veterans’ employment workshop.

Recommendation for Executive Action

To inform decisions on any potential future iterations of the veterans’ employment workshop, we recommend that the Secretary of Labor assess and report to Congress the extent to which further delivery of employment workshops to veterans and their spouses could fill a niche not fully served by existing federal programs. Such an assessment could involve collaboration with VA and other stakeholder organizations.

Agency Comments and Our Evaluation

We provided a draft copy of this report to DOL for review and comment. DOL concurred with our recommendation and noted that it will conduct a deliberate assessment of (1) the need for the services offered under the pilot; (2) which services are most useful for veterans and their spouses; and (3) what overlap exists with programs providing similar services to this population. DOL added that the assessment should compare the services the AJC system provides to veterans and their spouses with the information contained in DOL’s veterans’ employment workshops and online curriculum. DOL also stated that the assessment should identify if specific services provided within the AJC system require revision or redesign to better meet the employment needs of veterans and their spouses. DOL’s comments are reproduced in appendix II.

We also provided relevant portions of the draft to VA, which had no comments.

We are sending copies of this report to relevant congressional committees, the Secretary of Labor, and other interested parties. In addition, this report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to examine (1) how the Department of Labor (DOL) implemented the veterans’ employment workshop pilot program, (2) what state officials reported regarding the benefits and challenges of the workshops in the pilots, and (3) how the pilot informs decisions about its possible expansion.

To provide information on how DOL designed the pilot, how it planned to evaluate it, and the extent to which DOL collaborated with other organizations during the pilot’s design, we reviewed relevant federal laws and regulations and e-mail correspondence between DOL officials on the steps they took to design the pilot and documents describing DOL’s plan for collecting data to determine the pilot performance. We also interviewed officials from DOL’s Veterans’ and Employment Training Service (VETS) about how they selected the states to test the pilot, what steps were taken to set up the infrastructure of the workshops, such as classroom requirements and training materials, and the extent to which they collaborated with other federal and non-federal entities. We also asked VETS officials about the types of performance data they planned to collect and how they planned to report the results of the pilot. We interviewed Department of Veterans Affairs and veteran service organization officials about the extent to which DOL collaborated with them during the design of the pilot and the extent to which DOL collaborated with them in developing and implementing the pilot.

To provide information on how the pilot was implemented and what state officials reported regarding the benefits and challenges of the workshops in the pilots, we conducted telephonic interviews with DOL and state workforce agency officials in each of the pilot states: Georgia, Washington, and West Virginia. During our interviews we asked these officials what their roles and responsibilities were, how workshop locations were selected, what marketing efforts states employed to publicize the workshops, and the extent to which they collaborated with other federal and non-federal entities. We also asked about the benefits the veterans’ employment workshops offered participants as they searched for employment, the extent to which the workshops provided information on veterans’ benefits not offered by DOL, and any challenges the states faced in delivering the workshops. We reviewed DOL participant data, the results of DOL’s survey of workshop participants, and DOL’s two annual reports on the pilot, including verifying the reports’ workshop attendance figures with DOL and pilot states. To determine the reliability of the data, we interviewed agency officials knowledgeable of DOL’s data collection and reporting systems and reviewed agency documentation. We concluded the participant data were sufficiently reliable to describe certain demographic characteristics of the population that took the workshops and were enrolled in DOL’s American Job Centers (AJC). To determine the reliability of the survey data, we interviewed agency officials who developed the survey, reviewed agency documentation, and consulted with GAO subject matter experts in survey design. We determined that DOL’s survey data were sufficiently reliable for presenting qualitative response information.

To identify how the pilot informs decisions about its possible expansion, we reviewed how practices in pilot design can affect what an agency learns in its evaluation of a pilot. We reviewed guidance from federal agencies on program design and evaluation found during the course of our research and guidance identified by subject matter experts. We reviewed design and evaluation guidance from the Centers for Disease Control and Prevention because officials from DOL’s Chief Evaluation Office said they use that guidance for their evaluation practices. We reviewed design guidance recommended by GAO subject matter experts on program design and evaluation. We also reviewed GAO’s Standards for Internal Control, which identifies elements of effective internal controls including management responsibility for defining objectives in specific and measurable terms, communicating necessary information to achieve objectives, and establishing and operating monitoring activities. Moreover, we conducted an independent search for guidance to further support the standards for program design noted in this report. We found, for example, guidance supporting the standard of establishing and maintaining clear communication channels with relevant stakeholders from the Department of Health and Human Services’ Agency for Healthcare Research and Quality and the National Institutes of Health. We found additional support for conducting a needs assessment identifying whether a new program is needed from the Department of Education and the Department of Transportation. GAO subject matter experts in program design and evaluation verified the leading practices and the extent to which the practices were relevant to our review of the veterans’ employment workshop pilot by comparing the criteria with steps of evaluation design listed in our guide on program evaluations. See table 2 for the list of leading practices we identified in pilot design and pilot implementation.

Appendix II: Comments from the Department of Labor

Appendix III: Selected State Marketing Flyers for Veterans’ Employment Workshops

Appendix IV: Department of Labor Veterans’ Employment Workshop Locations and Participation Levels

Total attendance for all states

Appendix V: Department of Labor Survey of Veterans’ Employment Workshop Participants

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Andrew Sherrill, (202) 512-7215 or sherrilla@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Bill MacBlane (Assistant Director), Drew Nelson (Analyst in Charge), David Chrisinger, David Forgosh, Alex Galuten, Monika Gomez, Stephanie Shipman, and Walter Vance made key contributions to this report. Also contributing to this report were Juli Cutts, Laura Hoffrey, Kathy Leslie, Mimi Nguyen, Ronni Schwartz, and James Whitcomb.