VA Benefits: Increasing Outreach and Measuring Outcomes Would Improve the Post-9/11 GI Bill On-the-Job Training and Apprenticeship Programs
Released: 2015-11-12T00:00:00

Letter

Background

GI Bill Overview

The Department of Veterans Affairs (VA) has been providing veterans GI Bill benefits since 1944. Among several GI Bill programs, the Post-9/11 GI Bill program has become the largest. The Post-9/11 GI Bill—which took effect in August 2009—provides benefits to veterans and servicemembers serving on active duty after September 10, 2001.

Depending on what education or training program they select, participants may be eligible for benefit payments for up to 36 months to cover tuition and fees, housing, and books and supplies. From 2009 to 2011, the program provided benefits only to participants attending a degree- granting institution of higher learning. However, in 2011 the Post 9/11 Veterans Educational Assistance Improvements Act of 2010 expanded approved programs of education to include on-the-job training and apprenticeship, as well as other types of non-degree programs, such as vocational or technical school.

VA OJT and Apprenticeship Programs

OJT and Apprenticeship Post- 9/11 GI Bill Benefits

Eligible veterans can use their benefits through the VA’s Post-9/11 GI Bill to participate in an approved OJT or apprenticeship program. Veterans can receive benefits for an OJT program lasting from 6 months to 2 years, or for an apprenticeship as long as 36 months until their Post-9/11 GI Bill benefits are exhausted. When veterans opt to apply their Post-9/11 GI Bill benefits to an eligible OJT or apprenticeship program, in addition to the wages they earn from their employer as a trainee or apprentice, they receive a tax free monthly housing payment from VA to help cover living expenses. The amount of the benefit is based on the basic allowance for housing (BAH) paid to servicemembers. Veterans receive scheduled wage increases from their employer during the training period. Every 6 months, the monthly benefit received from VA decreases. Furthermore, because OJT programs can be supplemented with classroom components, and apprenticeship programs must include related instruction, participants may receive up to $83 per month to help cover the cost of books and supplies. (See table 2.)

Program Requirements and Approval Process

Program requirements and the approval process are set out by VA and the State Approving Agencies (SAAs) in their training manual for state officials. To be approved, employers sponsoring OJT programs must complete an application indicating that they meet a number of criteria, including creating and adhering to a training plan, designating a certifying official, having “reasonable certainty” that a position will be available when the veteran completes training, and paying wages that meet program requirements. To be approved for the GI Bill, apprenticeship programs must meet the same regulatory criteria they would need to be registered with DOL or a State Apprenticeship Agency.

Specifically, GI Bill-approved OJT programs are intended to be for a job that is learned in a practical way through a structured, systematic program of supervised on-the-job training; however, there is generally no requirement for classroom training. The training period must be full-time, compensated employment that is a minimum of 6 months and a maximum of 2 years in length. Full-time training is generally considered to be not less than 30 hours per week.

Similarly, the training manual provides that a GI Bill-approved apprenticeship program must be a structured, organized, written program combining on-the-job training and related instruction in which workers learn the practical and theoretical aspects of a skilled occupation. Employers must provide incremental wage increases to participants throughout the apprenticeship. The length of an apprenticeship is generally not less than 2,000 hours of full-time work experience that is consistent with training requirements as established by industry practices and a recommended minimum of 144 hours of related instruction per year. Full-time training is generally considered to be not less than 30 hours per week or 120 hours per month. According to DOL, individuals participating in an apprenticeship program are expected to advance from a low or no skill, entry-level position to full occupational proficiency at the end of training.

GI Bill-approved OJT and apprenticeship programs for non-federal employers or apprenticeship sponsors must generally be approved by an SAA, while VA approves OJT and apprenticeship programs sponsored by federal agencies.

SAAs are state agencies designated by the state’s governor that evaluate, approve, and monitor education and training programs for the GI Bill (see fig. 1). SAAs generally contract with VA to approve both educational institutions and OJT and apprenticeship programs. In addition to approving programs, SAA staff is responsible for providing technical assistance, oversight, and program outreach.

OJT and Apprenticeship Resources through DOL

DOL OJT Services

DOL offers OJT services that are separate from the VA Post-9/11 GI Bill OJT program and follow a similar model, but there are several key structural differences between these programs and services. DOL’s OJT services were authorized under the Workforce Investment Act of 1998 (WIA) and now operate under the Workforce Innovation and Opportunity Act (WIOA) enacted in 2014. Under WIOA, eligible individuals, including veterans, can receive employment training assistance and referrals from local workforce offices known as American Job Centers (formerly called One-Stop centers). Consistent with WIOA, American Job Centers provide different levels of services, depending on an individual’s needs:

Career Services, which include assistance with job searches and job placement, career counseling, and developing an individual employment plan; and

Training Services, including OJT programs and other occupational training opportunities.

American Job Center staff help match DOL OJT participants to participating employers for employment and training opportunities. Upon establishing a contract for the provision of occupational training, participating employers receive a subsidy of up to 75 percent of the trainee’s wages to compensate for training costs. In order to maintain their eligibility, employers must be able to demonstrate a pattern of providing program participants with long-term employment and wages, benefits, and working conditions commensurate with unsubsidized employees of similar work experience.

There are several key differences between the OJT services available to veterans through DOL and the OJT programs that are part of the Post- 9/11 GI Bill (see table 3). Whereas veterans receive a regular benefit equivalent to their monthly housing allowance as part of the Post-9/11 GI Bill, the employers receive wage subsidies as part of DOL OJT services. To be eligible for DOL OJT services, a veteran must have been determined by American Job Center staff to require training in order to gain or retain employment. In contrast, Post-9/11 GI Bill OJT participants must only meet the length of military service requirements in order to be eligible for an OJT program. Unlike the VA, which officials indicated is not required to track outcome measures for OJT participants, states are required to collect and report data to DOL, using Unemployment Insurance wage records, on post-program performance measures, including employment retention rates and earnings gains.

DOL Apprenticeship Programs

Distinct from Post-9/11 GI Bill apprenticeship programs, veterans can also participate in DOL’s Registered Apprenticeship program. In concert with recognized state apprenticeship agencies, DOL’s Office of Apprenticeship determines which programs meet the standards to be registered with DOL. For the Post-9/11 GI Bill, an apprenticeship program does not need to be registered with DOL to be approved, but if a program is registered, its training plan is deemed approved for purposes of the VA program.

Registered Apprenticeship occupations are offered in such fields as electrical work, carpentry, plumbing, and roofing. In addition to monitoring compliance with program requirements, the Office of Apprenticeship partners with employers, workforce agencies, colleges, the military, and other organizations to promote Registered Apprenticeships and provide technical assistance. As with OJT programs, DOL requires Registered Apprenticeship agencies to monitor performance using completion rates, among other indicators, and enables these agencies to access participants’ wage records through the Wage Record Interchange System (WRIS) to report on performance measures such as employment retention and earnings gains. VA is not required by law to track outcomes for Post-9/11 GI Bill apprenticeship participants, according to VA officials. (See table 4.)

DOL-Funded Staff to Support Veterans’ Employment

In addition to American Job Center staff who serve all jobseekers, including veterans, DOL’s Veterans’ Employment and Training Service funds two staff positions that focus on promoting veterans’ employment. Specifically, DOL funds Local Veterans’ Employment Representatives (LVER) to promote veteran employment opportunities and coordinate training services for veterans by building relationships with employers and working with staff at American Job Centers. Additionally, Disabled Veterans’ Outreach Program (DVOP) specialists provide case- management to veterans with significant barriers to employment, such as homelessness or disabilities.

Veterans in Our Review Said the Post-9/11 OJT and Apprenticeship Programs Helped Them Transition to Civilian Life, but Relatively Few Veterans Have Participated

Veterans in Our Review Said the Program’s Supplemental Income Helped Them Cover Living Expenses while They Trained

The majority of veteran participants who replied to our survey (125 of 156) cited more than one benefit to the program. Many specifically noted that receiving supplemental income to help them cover living expenses during their transition to civilian life was a primary benefit (112 of 156). In 2015, this supplemental income ranged from $896 to $3,923 per month for the first 6 months, depending on the location of the employer or apprenticeship sponsor. Many veterans we interviewed (21 of 28) said that this supplemental income helped them offset the loss of income they experienced after leaving the military. For example, slightly more than half the veterans we interviewed (16 of 28) said that the supplemental income helped them pay critical living expenses, such as a mortgage or car payment, or support their families. (See text box.) A few veterans (4 of 28) told us that without the Post-9/11 GI Bill benefits they would have needed a second job, which would have made it difficult to focus on learning their new occupation.

In addition to providing supplemental income, about half of the OJT and apprenticeship participants who responded to our survey (80 of 156) reported that the programs allowed them to use their GI Bill benefits, to which they are entitled, when college was not a good fit for them. Several veterans we interviewed (4 of 28) said they had not expected to be able to use their GI Bill benefits because they knew they wanted to enter a specific trade or occupation that did not require a college degree, such as pipefitter or police officer. These veterans said they were pleased to be able to use the benefits they had earned while starting their chosen careers. Others (6 of 28) said that due to family obligations they needed to work full time and would not have been able to afford college. Further, some OJT and apprenticeship participants who responded to our survey—including both those who had and had not used Post-9/11 GI Bill benefits to attend college—reported one of the main benefits of the program was that it helped them learn a new occupation (86 of 156) and further develop skills they had obtained in the military (60 of 156).

VA officials told us that employers and apprenticeship sponsors do not receive any federal funds for participating in these programs, though the employers and apprenticeship sponsors who responded to our survey cited a number of benefits for participating, particularly recruitment and retention. Specifically, just over half of the employers and apprenticeship sponsors who responded to our survey (87 of 153) reported the program helped them recruit veterans to be employees. Almost all of the employers and apprenticeship sponsors we interviewed (12 of 15) praised the discipline, work ethic, or customer service skills of participating veterans, or said they made valuable employees or apprentices. In addition to recruitment benefits, half of the employers and apprenticeship sponsor respondents (76 of 153) indicated that the programs had helped them retain veterans as employees. Some employers and apprenticeship sponsors who responded (47 of 153) also indicated that the program enhanced their standing in the community. (See text box.)

Since 2011, About 27,000 Veterans Have Participated in the Post- 9/11 GI Bill OJT and Apprenticeship Programs to Train in a Variety of Occupations

Of the approximately 1.2 million veterans who used their Post-9/11 GI Bill benefits since October 2011, about 27,000—or about 2 percent— participated in the OJT or apprenticeship programs, according to VA program data. Approximately half of these veterans participated in OJT opportunities, while the other half pursued apprenticeships. The average age of participants was 33, and most were male (94 percent). Over half of all participants had served either in the Army (28 percent) or the National Guard or Reserves (27 percent). (See table 5.) The number of participating veterans varied considerably by state, for example, from a low of 14 participants in Wyoming to a high of 799 in Texas during fiscal year 2014.

During fiscal year 2014, there were approximately 2,700 employers and apprenticeship sponsors approved to train Post-9/11 GI Bill veterans. According to VA and state officials, not all of these approved OJT and apprenticeship sponsors actually had veterans receiving benefits under the program during this timeframe.

Participating veterans have used the OJT and apprenticeship programs to train in a variety of occupations. While VA officials told us the agency does not collect standardized occupational data, our analysis of VA’s administrative data on approved programs indicates that the following occupations are among those most frequently pursued by participants:

Police and sheriff officer

Correctional officer and jailor

Truck driver

Electrician

Firefighter

Air traffic controller

Veterans service representative

Carpenter

Plumber, pipefitter, and steamfitter In addition, veterans we interviewed reported being trained for a variety of other positions, including maintenance and repair workers, special agents, and cable installation and maintenance technicians.

VA and States Provide Varying Levels of Information and Outreach to Veterans and Employers

VA Provides Some Program Information That Generally Highlights School-Based Options and Lacks Sufficient Detail On OJT and Apprenticeships

While VA provides some information on Post-9/11 GI Bill OJT and apprenticeship programs, the focus of most outreach information is on higher education benefits. OJT and apprenticeship information lacks sufficient detail to inform veterans these benefits are available or how they might be used. There are a few different resources available to servicemembers—and to some extent, employers—that provide information on Post-9/11 GI Bill benefits, including information shared at mandatory pre-separation briefings as part of the Transition Assistance Program (TAP), on agency websites, and as part of VA’s education and career counseling.

Transition Assistance Program (TAP)

In general, mandatory briefings provided by VA for separating servicemembers as part of TAP cover Post-9/11 GI Bill benefits, but almost all of this information is focused on using these benefits for higher education. The purpose of TAP is to prepare servicemembers leaving the military for their transition back into civilian life. TAP includes both mandatory and optional components. For example, all servicemembers participating in TAP must take VA’s Veterans Benefits I and II courses, which discuss available benefits and services, including education benefits. Federal, state, and veterans service organization officials we interviewed identified TAP as one of the primary ways that veterans can learn about the Post-9/11 OJT and apprenticeship programs. However, out of 77 total pages in the TAP facilitator guide and briefing slides for the VA Benefits I and II courses, there is one reference to the OJT and apprenticeship programs as part of a description of the books and supplies stipend portion of the Post-9/11 GI Bill. In discussing Post-9/11 GI Bill benefits, the briefing slides generally refer to “education,” “tuition,” “school,” and “student,” and do not refer to employers, OJT, or apprenticeship opportunities, although these options are also available. Presenting this information in Veterans Benefits I and II courses would align with VA’s stated goal that these briefings be comprehensive and cover the entire spectrum of VA benefits for servicemembers and their families.

Similarly, the Veterans Benefits I and II briefing slides do not include any specific examples of veterans who have used their benefits for OJT or apprenticeship opportunities or how these benefits work. Information on the OJT and apprenticeship program of Post-9/11 GI Bill benefits is included in an optional portion of TAP, called the Career Technical Training Track. But here as well there is relatively little information provided to servicemembers on how they can use their Post-9/11 benefits to pursue OJT and apprenticeship opportunities. Several slides in this track provide information on what OJT and apprenticeship programs are, however, it is not clear that veterans can use their Post-9/11 GI Bill benefits for these programs.

As discussed later in this report, 39 of 44 SAAs reported that lack of awareness of the Post-9/11 GI Bill OJT and apprenticeship programs was one of two top challenges in facilitating veteran participation. This finding was also supported by the results of our other survey efforts. Specifically, about half the veterans who responded to our survey reported that TAP briefings do not include enough information on OJT and apprenticeship programs under Post-9/11 GI Bill benefits. Of the 156 veterans who responded to our survey and have used their Post-9/11 GI Bill benefits for OJT and apprenticeship, 81 reported that they did not think TAP adequately informs veterans about these options. Similarly, 63 of the 127 responding veterans who have used their Post-9/11 GI Bill benefits for a non-college degree, such as a trade school program, had the same response. While 5 of the 7 veterans service organization leaders we interviewed indicated that TAP should not be the only source of information for veterans, some of whom find it difficult to process the volume of information presented during TAP, additional information about the Post-9/11 GI Bill OJT and apprenticeship programs could be beneficial in informing servicemembers of their options.

VA officials told us they have not considered expanding the curriculum of the mandatory courses to include more information about the OJT and apprenticeship programs. They stated they only have a limited amount of time to cover a wide range of benefits during the TAP briefings and have prioritized content. VA officials told us that they revisit their TAP curriculum on an annual basis. During this annual process, officials update benefits information and accommodate new material. They also stated that they encourage veterans to attend the optional Career Technical Training Track, which includes information on these programs. However, without additional information about the OJT and apprenticeship program options under the Post-9/11 GI Bill, veterans may continue to lack awareness of these benefits, as discussed later in the report.

Agency Websites

Focus on Higher Education Benefits Another source of information about Post-9/11 GI Bill benefits are agency websites, and while VA’s and DOL’s websites provide some information on these benefits, VA’s website is more focused on higher education benefits. According to the Plain Writing Act of 2010, information supplied to the public regarding any federal benefit (including in paper and electronic publications, forms, notices or instructions) must be clear, concise, well-organized, and appropriate for the intended audience. The President’s guidance for implementing this act states that “agencies should communicate with the public in a way that is clear, simple, meaningful, and jargon-free,” and notes that “a lack of clarity may prevent people from becoming sufficiently aware of programs or services” for which they are eligible. VA’s website has resources for individuals to learn about GI Bill benefits, including a web page for the GI Bill, a web page dedicated to the Post-9/11 GI Bill, and a GI Bill Comparison Tool. While there is also a web page for the OJT and apprenticeship programs, the language used on the other GI Bill-focused web pages, and resources generally, refers to higher education and does not reference OJT or apprenticeships. For example, the main heading of the GI Bill web page is “Education and Training.” However, the rest of the page uses language and visual cues that reference higher education, such as “choose a school” and “verify school attendance.” Moreover, the logo for the Post- 9/11 GI Bill—a graduation cap with dog tag tassels—further emphasizes higher education to the exclusion of other types of GI Bill benefits.

VA’s GI Bill Comparison Tool also uses language that emphasizes higher education, even though the tool can be used to access information on employers. The main heading, for example, is “Choosing a School,” and the options below are “About Your School” and “Compare This School” even when an employer is chosen—it may not be apparent to veterans that they can use this tool to explore OJT and apprenticeship benefits.

Lack of Sufficient Detail on OJT and Apprenticeship Programs In addition, the information provided on VA’s Post-9/11 GI Bill web page and on its OJT and apprenticeship web page lacks enough detail for users to reasonably understand how to use their GI Bill benefits for the OJT and apprenticeship programs. For example, the page content does not articulate how OJT programs are different from apprenticeship programs—a difference that could have meaningful career implications for veterans. Participants in an apprentice program, for example, earn a license or national industry certification and generally train longer, while OJT participants generally receive a certificate of completion and train for a shorter period of time. In addition, the web page lists three occupations that might be available through OJT or apprenticeship, union plumber, hotel manager, and firefighter, but does not provide additional details, such as descriptions of what OJT or apprenticeship programs associated with these occupations might involve, or testimonials on how actual veterans used and benefited from these programs. In addition to the requirements for clear and meaningful communication described in the Plain Writing Act of 2010, Standards for Internal Control in the Federal Government state that management should ensure there are adequate means of communicating with external stakeholders in which there could be a significant impact on the agency achieving its goals. Further, VA’s strategic goals include empowering veterans to improve their well-being and managing and improving VA operations to deliver seamless and integrated support. Without providing sufficient detail regarding the OJT and apprenticeship programs, VA may not provide veterans complete information to understand these benefits, and may not fully meet its goals of serving this population.

While VA’s OJT and apprenticeship web page links to a fact sheet on these Post-9/11 GI Bill programs, this fact sheet also lacks sufficient detail to help veterans and employers reasonably understand how they can participate. For example, the fact sheet describes the amount of Post-9/11 GI Bill benefit payments, but does not describe the difference between OJT and apprenticeship programs, or provide examples of how veterans used benefits from these programs.

In contrast, DOL has a comprehensive web page for its Registered Apprenticeship program that specifically targets veterans. It also includes information about potential benefits of participating in an apprenticeship, and profiles several veterans who participated in the program while receiving GI Bill benefits. These profiles provide concrete examples of how veterans have used the GI Bill benefits during apprenticeships in a variety of fields and how their apprenticeships helped them after leaving the military. VA does not link to this page from its OJT and apprenticeship web page, missing an opportunity to leverage this resource and potentially making it more difficult for veterans to learn about these benefits through its website.

VA’s GI Bill web pages also provide little information for employers who may be interested in participating. The OJT and apprenticeship web page includes two sentences directed to employers stating that they can participate in the program and directing them to their SAA office. It does not, however, provide information about requirements for employers or why employers might want to participate, which could help attract employers or apprenticeship sponsors to help VA achieve its goal of improving veterans’ well-being.

VA’s Education and Vocational Counseling

Finally, VA’s education and vocational counseling includes some information on GI Bill benefits, but according to VA, counselors rely on the GI Bill website for information regarding the OJT and apprenticeship programs. In addition to TAP and agency websites, servicemembers may participate in VA’s education and vocational counseling to learn more about their benefits. VA officials told us that benefits advisors are permanently staffed at almost 300 military installations and other sites worldwide and provide transition and outreach briefings, as well as individual assistance, guidance, and counseling on all VA benefits and services. According to VA officials, the counselors who deliver these benefits are trained and have knowledge of OJT or apprenticeship programs. When asked to provide training materials related to the OJT and apprenticeship programs, VA officials told us that counselors find information about OJT and apprenticeships on the GI Bill website.

State Officials We Surveyed Reported Providing Outreach to Veterans and Employers, Which Varied By State

Most state officials we surveyed reported that they reach out to veterans using direct methods, such as attending job fairs and providing briefings and presentations to veterans’ groups (see table 6). Fewer state officials reported using broader outreach methods, such as radio or television advertisements or social media, to raise awareness of the programs. Some state officials said that they devote considerable staff time and resources to outreach efforts. For example, an SAA director in one state told us his agency created various informational materials to raise awareness of these programs, including a video titled “OJT: It’s not Just for College” to distribute to stakeholders, a monthly newsletter on veterans’ employment issues, and program fact sheets and brochures to be displayed in American Job Centers and higher education veterans’ offices. An SAA official in another state told us his agency arranged for OJT and apprenticeship program information to be printed on the back of state unemployment insurance (UI) benefit checks and works with DOL workforce staff to leverage employer contacts. However, our survey results also indicate that the level of outreach conducted varies by state. Specifically, officials in 5 states we surveyed indicated that due to resource constraints, they were unable to engage in more intensive outreach efforts for veterans or employers.

Without adequate program outreach, veterans who could benefit may remain unaware of these programs and miss opportunities to improve their long-term employment prospects, consistent with agency goals.

VA and DOL Have Coordinated to Build Awareness of the Post- 9/11 GI Bill Apprenticeship Programs

VA has coordinated some aspects of its Post-9/11 GI Bill apprenticeship program with DOL’s Registered Apprenticeship program to help make apprenticeship sponsors and veterans aware of the GI Bill program. By statute, the same regulatory requirements apply to apprenticeship sponsors seeking GI Bill approval for their program as those seeking approval through DOL’s Registered Apprenticeship program. Prior to 2014, this process was not streamlined, and SAAs independently assessed whether a Registered Apprenticeship program sponsor seeking approval for GI Bill eligibility met these regulatory requirements. In September 2014, the Secretaries of Veterans Affairs and Labor announced that any DOL or State Apprenticeship Agency-approved Registered Apprenticeship program was qualified to be approved under the Post-9/11 GI Bill. The joint letter making this announcement was sent to approximately 10,000 Registered Apprenticeship programs to alert program officials that participating veterans could be eligible to receive benefits under the Post-9/11 GI Bill, according to VA officials. The letter included detailed information on how Registered Apprenticeship programs could obtain GI Bill approval and also contained two outreach flyers. One flyer was designed for veterans. The other, for Registered Apprenticeship sponsors, provided detailed information on the GI Bill program and its benefits.

DOL also conducted two additional outreach efforts, one targeted at apprenticeship sponsors and the other at veterans, to raise awareness that veterans can receive GI Bill benefits while participating in a DOL Registered Apprenticeship program. Regarding the first effort, DOL officials told us they created a pop-up window that appears when apprenticeship sponsors enter information into the Registered Apprenticeship database about a new apprentice. The pop-up window reminds sponsors that their apprenticeship program is qualified to be approved under the GI Bill, and that veteran apprentices may be entitled to receive GI Bill benefits while participating in the program. The pop-up window also links to several different resources created by both DOL and VA that apprenticeship sponsors can use to learn more about VA’s program. Regarding the second outreach effort to veterans, DOL created a Registered Apprenticeship website targeted specifically to veterans that highlights the availability of GI Bill benefits for apprenticeships. VA requires SAA officials to meet biannually with DOL’s Local Veterans Employment Representatives (LVER). According to VA’s contract with SAA staff, the purpose of these meetings is to ensure that students who graduate in the SAA’s state are made aware of employment resources and opportunities. SAA directors we interviewed in 10 of 13 states told us these meetings were generally helpful, noting that they increased program awareness or helped SAA staff connect with employers. One official said LVERs are “wonderful ambassadors” for the Post-9/11 GI Bill program. In contrast, SAA directors we interviewed in 3 of the 13 states indicated that while their staff meet with LVER staff as required, these interactions rarely result in increased program awareness about the GI Bill OJT or apprenticeship programs.

VA and DOL have more coordination for apprenticeship programs than for OJT, possibly due to structural differences in how OJT services are provided. For example, the administration of OJT services funded by DOL, which occurs at the local level, is more decentralized than for Registered Apprenticeship programs, which are overseen at either the federal or state level.

Lack of Program Awareness and Administrative Burdens Have Challenged Veterans and Employers, According to State Officials and Program Participants

Veterans and Employers Often Lack Awareness of OJT and Apprenticeships as Options under the Post- 9/11 GI Bill, According to State Officials and Program Participants

Veterans and employers generally lack awareness of OJT and apprenticeship options, and they most often heard of these programs from word-of-mouth sources, according to state officials and program participants we surveyed. Specifically, most state officials we surveyed cited lack of awareness as a top challenge in facilitating veteran (39 of 44) and employer (39 of 43) use of the Post-9/11 GI Bill OJT and apprenticeship programs (see table 7). In addition, most state officials (39 of 44) cited a lack of understanding among veterans as to how they can use these benefits to pursue career goals as a top challenge. Similarly, the most common challenge experienced by responding state officials trying to facilitate employer participation in OJT and apprenticeship programs is a lack of knowledge of this option among employers (39 of 43).

State officials we surveyed also cited some of the same challenges that we identified in our review of VA’s information resources, and some confusion about the OJT and apprenticeship programs that may contribute to a lack of knowledge about these programs (see table 7). Specifically, 36 of 44 responding state officials pointed to the emphasis on Post-9/11 GI Bill materials on education rather than OJT or apprenticeship as a challenge in facilitating veteran participation. Additionally, 30 of 44 responding state officials indicated that veterans have difficulty distinguishing between benefit options and programs available to them. Twenty-eight of 43 responding state officials said that employers lacked knowledge of the SAA and its functions and responsibilities. In addition, 25 of 43 responding state officials indicated that employers have difficulty distinguishing OJT and apprenticeship GI Bill programs from other government efforts to promote veterans’ employment. “This is a hidden program that should be more prominent in availability. I would not have known I could use the OJT program if not for word-of-mouth from another employee.”

Similarly, participating employers and apprenticeship sponsors who responded to our survey most frequently said they heard about the program from a veteran employee (63 of 153) or an SAA official (42 of 153).

Of veterans we surveyed who used GI Bill benefits for a non-college degree, such as at a vocational school, about half of those who responded (65 of 127) said they did not know about the OJT and apprenticeship program options when deciding how to use their benefits. About half of the responding veterans (37 of 65) who had not heard about the programs reported they would have considered using their benefits to pursue OJT or apprenticeships rather than an education program had they known this was an option. Further, about half (63 of 127) of the responding veterans who used GI Bill benefits for a non-college degree, such as at a vocational school, stated that TAP did not adequately inform them that they can use their GI Bill benefits for these programs.

To build awareness of these programs, VA officials said they send multiple letters to servicemembers prior to their separation to let them know they may be entitled to education benefits. Two of these letters include a statement that says Post-9/11 GI Bill benefits are available for trade schools, apprenticeships, and OJT in addition to colleges and universities. In addition, in 2011 and 2013 VA completed two targeted outreach campaigns. VA officials told us these outreach campaigns were online and targeted the five states that had the highest rates of veteran unemployment in 2011, and the five states with the next-highest rates of veteran unemployment in 2013. According to VA, the first campaign resulted in nearly 354,000 unique individuals visiting the web page that explained the expansion of the GI Bill to OJT and apprenticeship, among other new options. The second campaign resulted in an additional 76,000 visits to this web page. VA officials told us they have since consolidated their web pages and in doing so eliminated this GI Bill expansion web page. They said no additional VA outreach campaigns have been completed or are planned, absent additional funding, among other reasons. While word-of-mouth discussion of the Post-9/11 GI Bill OJT and apprenticeship program benefits can increase program awareness, this is not a systematic method of outreach, and VA officials acknowledged there is room for improvement in their outreach approach.

State Officials and Employers We Surveyed Reported VA’s Paper- Based Payment System, which VA Plans to Replace by 2017, Is Burdensome, and Payments Are Sometimes Delayed

“There is no way to verify and track paperwork that has been submitted via fax or mail and it is ‘lost’ 1 out of 3 submissions.” “The biggest time drain is ‘faxing’ in the monthly hour logs. This is an antiquated way of doing this and the option for scanning and emailing should be incorporated.” “The GI/Bill OJT program is beneficial to me, but only when I received my benefits. In the past year, only 3 to 4 months have been ‘on time.’ I’m currently waiting on 4 months’ worth of benefits. I personally am struggling with bills and I know many others are too… That is the biggest concern… that our paperwork gets ‘lost’ for months on end.” “It takes too long for the VA to process the paperwork, and there really is no easy way to correct mistakes if they happen. This delays payment and causes bills to pile up.” “There is no communication between the VA and us. If there is a problem with the application, the VA does not let us know if there are any problems, and the veteran has to wait for his or her application to be processed.” “It was extremely difficult to get everything to and from everybody that needed paperwork. It took over a year after I started the program to get all of the things needed to the right people…. It was a very frustrating process, if I was not persistent, I could have given up.”

VA officials told us delays sometimes occur because a veteran’s benefit application is received before the employer has been approved or because not all required paperwork has been submitted. VA officials also told us that they are in the process of developing a new data system called Veterans Approval, Certification, Enrollment, Reporting and Tracking System (VA-CERTS), which will combine and replace the program certification for educational institutions, and monthly certification and claims processing functions currently housed in two separate systems. Officials indicated that VA-CERTS will include the program certification and monthly hours certification, as well as claim processing for the OJT and apprenticeship programs. According to officials, VA- CERTS is in the early stage of development. Specifically, the business requirements document to build the new system was completed in July 2015 and delivered to VA’s Office of Information and Technology. It includes the requirements for the OJT and apprenticeship elements mentioned above. Officials told us that funding to develop VA-CERTS has been identified, though final implementation of the new system may not occur until at least 2017. In the interim, absent efforts to ease the administrative challenges associated with submitting monthly paperwork, some employers and veterans are likely to continue to experience difficulties and delays. Such efforts to ease the administrative challenges could include allowing employers and apprenticeship sponsors to submit monthly certification forms by email, or providing confirmation that monthly forms have been received.

Little Is Known about the Performance of VA’s OJT and Apprenticeship Programs

VA Does Not Measure Outcomes for Participants in the Post-9/11 GI Bill OJT and Apprenticeship Programs

VA has not established outcome measures to report on the performance of Post-9/11 GI Bill OJT and apprenticeship participants. Specifically, VA does not measure such outcomes as whether participants retained employment or experienced earnings gains after completing the program. As previously mentioned, VA officials indicated that there is no requirement for VA to measure OJT and apprenticeship performance. However, standards for internal control call for establishing and reviewing performance measures to allow an agency to evaluate relevant data and take appropriate actions. Moreover, a recent review by the Vice President concluded that an integral element of successful job training programs is measuring outcomes for all participants and informing participants and employers of the results. The Vice President’s report noted the importance of tracking outcomes such as employment retention and earnings gains so participants can choose training options wisely— and so programs can improve performance. Additionally, VA officials told us that having access to data on participant outcomes could help them promote the programs to employers and veterans.

According to VA officials, the agency lacks a data mechanism and resources for measuring employment outcomes once veterans have exhausted their Post-9/11 GI Bill benefits. States’ Unemployment Insurance wage records, which DOL uses to measure outcomes for its OJT and apprenticeship programs, are one possible data source VA could explore to measure outcomes for participating veterans. According to DOL officials, VA could consult with DOL to determine if it could access this data source through the Wage Records Interchange System 2 (WRIS 2). VA officials told us they had not previously considered participating in WRIS 2 to access wage records, and consequently have not considered if they need to seek legislative authority to do so. In addition to this shared resource, other mechanisms for measuring outcomes could potentially be developed, such as collecting outcome data directly from employers and apprenticeship sponsors.

Although VA does not measure outcomes for OJT and apprenticeship programs, it does collect and report data for some outcome measures for the higher education portions of the Post-9/11 GI Bill. Specifically, VA collects and publicly reports data for GI Bill participants on such measures as graduation rates and loan default rates. VA officials also indicated that they plan to report on transfer rates and employment retention rates, among other measures, for graduates of higher education programs. This effort was a product of a 2012 Executive Order that called for the Secretaries of Defense, Veterans Affairs, and Education to develop such outcome measures to help veterans make informed decisions about how to use their GI Bill benefits. VA officials told us that even though they currently have no plans to develop or report on similar post-program performance measures for OJT and apprenticeship participants, they recognize the importance of such an effort.

State officials we surveyed and employers and apprenticeship sponsors who have participated in the OJT and apprenticeship programs and responded to our nongeneralizable survey generally said the programs promote employment retention and effectively serve veterans. Of the 44 SAAs we surveyed, officials from 38 said they believed the programs were generally either very or somewhat effective in improving veterans’ employment prospects (see table 8). Additionally, of the 153 participating employers and apprenticeship sponsors who responded to our survey, 128 said that, in their experience, veteran participants typically continued to work in the same area for which they received training while in the programs. Nevertheless, without establishing and reporting data on outcome measures for these programs, VA will not be able to verify whether these perceptions are accurate and know how well these programs are performing.

Data on DOL Programs Indicate Potential for Positive Employment Outcomes for OJT and Apprenticeship Models

DOL collects performance data for its OJT services and Registered Apprenticeship program, and these data suggest that OJT and apprenticeship training models can be associated with positive outcomes for measures such as employment retention and earnings gains. According to 2013 DOL program data, 85 percent of Post-9/11-era veterans who received OJT services entered employment within 3 months of completing their training, and 86 percent of these veterans retained employment over the subsequent 6 months. Additionally, Post- 9/11-era veterans who used OJT services earned 25 percent more in the second and third quarters after finishing their OJT services than they had earned prior to receiving services.

While these data indicate the potential for positive employment outcomes for an OJT approach to training, they do not necessarily reflect outcomes for veterans in VA’s programs. As previously mentioned, DOL’s OJT services and the Post-9/11 GI Bill OJT program differ in several key ways. First, while all veterans who served for at least 90 days after Sept. 11, 2001, are entitled to Post-9/11 GI Bill benefits, American Job Center staff must determine that DOL OJT participants—civilians and veterans alike— need additional services to obtain employment. Second, whereas veterans receive Post-9/11 GI Bill benefits in the form of a monthly benefit payment, DOL OJT benefits are paid to employers in the form of a wage subsidy. The subsidy covers the costs of training services needed to help participants gain or retain employment. Third, the Post-9/11 GI Bill OJT programs are limited in duration to 6 to 24 months, while the duration of subsidized DOL OJT services varies according to the needs of the position and the availability of local workforce funds.

Outcome data and available research on DOL’s Registered Apprenticeship program also point to the potential for employment benefits of an apprenticeship training model. DOL reports outcome measures for all Registered Apprenticeship participants, without distinguishing participants by veteran status. Nevertheless, 2014 program data show that 91 percent of participants who completed the DOL program found employment within 3 months compared to 76 percent of those who exited the program without completing it. Of those individuals who completed the program and found employment within 3 months, 91 percent retained their employment over the subsequent 6 months. DOL data also indicate that the average wages of apprentices who completed the program increased about 65 percent from $13.78 per hour to $22.69 per hour during the course of the program. Moreover, a study commissioned by DOL estimated that those individuals who completed a Registered Apprenticeship earned substantially more over a career than similar individuals who did not complete a Registered Apprenticeship. It is important to note that some of these earnings gains are likely due to unobserved differences between those individuals who completed a Registered Apprenticeship and those individuals who did not.

As with OJT services, outcomes associated with DOL’s Registered Apprenticeship do not necessarily reflect the outcomes for veterans in Post-9/11 GI Bill apprenticeship programs. First, DOL does not differentiate veterans from nonveterans in its Registered Apprenticeship outcomes; the reported data refer to the entire population of those who either complete or exit a Registered Apprenticeship. It is therefore not possible to use these data to determine how effectively Registered Apprenticeships are serving Post-9/11-era veterans. Second, these data reflect outcomes in the 25 states in which DOL’s Office of Apprenticeship administers the Registered Apprenticeship program. Therefore, Registered Apprenticeship outcome data from DOL do not represent nationwide participant outcomes. Absent measures for VA’s OJT and apprenticeship programs, the performance of these programs will remain unknown.

Conclusions

While many veterans benefit from higher education after their military service ends, for some veterans it may not be the ideal career path to successfully attain civilian jobs. Given the wide range of education and training options available under the Post-9/11 GI Bill, it is important that veterans be fully aware of their options so they can make an informed decision about how to use their benefits to effectively transition to civilian life. Most Post-9/11 GI Bill outreach materials emphasize higher education programs, and most state officials we surveyed told us that awareness of OJT and apprenticeship opportunities is low. While VA provides some information about OJT and apprenticeship programs, many veterans and employers learn about the program through word-of- mouth rather than through systematic outreach efforts. Without improved program outreach, veterans who could benefit from these programs may remain unaware of them and could miss opportunities to improve their long-term employment prospects, consistent with agency goals.

Overall, veterans, employers, and state officials we contacted were generally positive about VA’s OJT and apprenticeship programs. However, many suggested that VA’s current, paper-based system for paying monthly benefits is inefficient, leading to administrative burdens for employers and delayed benefit payments for veterans—adding economic stress for veterans who are counting on this assistance. VA has taken a first step towards easing these administrative challenges by deciding to incorporate the OJT and apprenticeship programs into its development of the new VA-CERTS data system to process program approvals and benefit payments. However, this system may not be implemented until at least 2017, and increasing numbers of veterans will become eligible for Post-9/11 OJT and apprenticeship benefits as the military continues to draw down its forces. In the interim, and in the absence of efforts to ease administrative challenges for employers who experience difficulty in submitting monthly certifications and for veterans who experience delayed payments, some veterans will likely continue to experience delayed payments. Such delays can undermine one of the primary benefits of the program cited by participants: helping veterans cover living expenses while transitioning to the civilian workforce.

With regard to performance, outcome data collected by DOL on the OJT and apprenticeship programs it administers indicate the potential for positive employment outcomes for these training models. However, VA is limited in its ability to assess the impact of its Post-9/11 GI Bill OJT and apprenticeship programs on veterans because VA does not measure performance outcomes for participants. While VA has undertaken efforts to better measure and report outcomes in its higher education programs, the agency currently has no plans to do so for its OJT and apprenticeship programs. Without measuring and reporting on outcomes for these programs, VA will not be able to assess their effectiveness, and veterans may not be well-positioned to determine which programs are most suitable for them when choosing how to use their benefits.

Recommendations for Executive Action

We are making the following three recommendations: 1. To help ensure that veterans are aware of the Post-9/11 OJT and apprenticeship programs so they can make informed decisions about how they use their benefits, the Secretary of VA should identify and implement appropriate, cost-effective actions to increase awareness of OJT and apprenticeship benefits under the Post-9/11 GI Bill. 2. To help address challenges veterans and employers reported facing in using the OJT and apprenticeship programs, the Secretary of VA should identify and implement cost-effective steps to ease administrative challenges in submitting paperwork or receiving payments as the new automated VA-CERTS system to process program approvals and benefit payments is being developed and implemented. 3. To increase accountability for program performance, the Secretary of VA should establish measures to report on program outcomes for Post-9/11 GI Bill OJT and apprenticeship programs, including considering relevant data sources and seeking legislative authority to gain access to data, if necessary.

Agency Comments

We provided a draft of this report to VA, DOL, and DOD for comment. VA provided written comments that are reproduced in appendix II. Neither DOD nor DOL provided a formal response. In addition, VA and DOL provided technical comments that we incorporated as appropriate.

VA agreed with each of our three recommendations. In response to our recommendation to increase awareness of OJT and apprenticeship benefits under the Post-9/11 GI Bill, VA said it will develop a guide for employers and apprenticeship sponsors about Post-9/11 GI Bill OJT and apprenticeship benefits. VA also said it will publicize this guide and make it available on VA’s GI Bill website. Further, VA said it will send veterans and stakeholders information on OJT and apprenticeship benefits via an email blast and by posting information to the Employment Center section of VA’s eBenefits website. In response to our recommendation to ease administrative challenges in submitting paperwork or receiving payments while the new automated VA-CERTS system is being developed and implemented, VA said it will explore the feasibility of cost-effective options and will develop a plan to ease challenges. Finally, in response to our recommendation to establish measures to report on program outcomes, VA agreed that program performance metrics should be developed to report on program outcomes. VA said it will develop a plan to determine the feasibility of collecting and publishing program outcome data for the OJT and apprenticeship programs. VA set a target completion date of June 2016 for all actions set forth in its response to our recommendations.

We are sending copies of this report to the appropriate congressional committees and the Secretaries of Veterans Affairs, Labor, and Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff has any questions about this report please contact me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix III.

Appendix I: Objectives, Scope, and Methodology

In conducting our review of the Department of Veterans Affairs (VA) Post- 9/11 GI Bill on-the-job training (OJT) and apprenticeship programs, our objectives were to examine (1) how selected veterans and employers used the programs and how widely they have been used, (2) to what extent VA and states have taken steps to inform veterans and employers about these programs, (3) what challenges, if any, veterans and employers have faced in using them, and (4) to what extent VA has assessed the performance and effectiveness of its programs. This appendix provides a detailed account of the data sources used to answer these questions, the analyses we conducted, and any limitations we encountered.

We conducted this performance audit from August 2014 to November 2015 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Analysis of VA Administrative Data

To describe how widely the OJT and apprenticeship programs have been used, we requested and analyzed program data from VA. VA provided information from two different databases—the Long Term Solution (LTS) database and the Web Enabled Approval Management System (WEAMS). We assessed the reliability of these data by reviewing existing information about the data and the systems that produced them and by interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. The summary statistics that VA provided from its LTS data included the number of veterans who have participated in the OJT and apprenticeship programs since these benefits first became available on Oct. 1, 2011, and the number of veterans participating in other Post-9/11 GI Bill programs through March 23, 2015—the most recent data available at the time VA analyzed them. In addition, VA provided summary information from the WEAMS dataset as of March 23, 2015 on the number of facilities (employers or apprenticeship sponsors) that have been approved to provide OJT or apprenticeship training opportunities under the Post-9/11 GI Bill. VA also provided selected participant-level information from the LTS dataset that we analyzed to describe key demographic characteristics of the participant population, including average age and the percentage of participants by program type, gender, and branch of service. We also used this data source to identify occupations for which veterans frequently train while participating in the Post-9/11 OJT and apprenticeship programs.

Analysis of DOL Administrative Data

To provide information on what is known about the effectiveness of OJT services and Registered Apprenticeship programs in serving Post-9/11- era veterans, we reviewed available data sources from DOL, as VA does not have performance data on GI Bill OJT and apprenticeship programs. First, we examined a publicly-available performance data guide for the Workforce Investment Act Standardized Record Data (WIASRD) from program year 2012 to determine what data elements we would request for Post-9/11-era veteran participants in DOL’s OJT services. We then requested and analyzed DOL aggregate outcome data from the fourth quarter of program year 2013, which included program exiters from prior quarters of that year, on: 1) the rate at which participants enter employment during the first quarter after their OJT services; 2) the rate at which participants retain employment during the second and third quarters after their OJT services; 3) average quarterly earnings for participants during the quarter(s) before their OJT services; and 4) average quarterly earnings for participants across the second and third quarters after their OJT services. Per our request, these data were presented by veteran status and included outcomes for Post-9/11-era veterans specifically. In order to verify the reliability of the data, we interviewed data specialists at DOL and requested and evaluated supporting documentation regarding the data source. We determined that the data were sufficiently reliable for the purposes of this report.

We also reviewed available DOL data on the Registered Apprenticeship program in order to provide some information on the effectiveness of apprenticeships in serving Post-9/11-era veterans. Specifically, we examined the Registered Apprenticeship Partners Information Management Data System (RAPIDS), which contains Registered Apprenticeship participant information from states whose programs are administered by DOL’s Office of Apprenticeship within the Employment and Training Administration (ETA), as well as individual states that have agreed to report data nationally.

Through a data sharing agreement with participating states, DOL’s Office of Apprenticeship assesses outcome measures quarterly, according to DOL officials, using states’ Unemployment Insurance wage records. We analyzed the same aggregate performance data from RAPIDS as we requested data for OJT services from 2014, the most recent year from which DOL could provide data. However, DOL officials noted that Registered Apprenticeship outcomes data are not categorized by veteran status and do not include pre-program earnings. We also interviewed officials from the Office of Apprenticeship to verify data reliability and requested and evaluated supporting documentation regarding the data source. We determined that the data were sufficiently reliable for the purposes of this report.

Participating Veteran Survey

To learn about how selected veterans have used the Post-9/11 GI Bill OJT or apprenticeship programs and what challenges they may have faced, we conducted a web-based survey of veterans who have participated in these programs. We limited our in-scope population to those veterans for which VA was able to provide email addresses. Specifically, while VA provided detailed participant level information on 26,375 unique participating veterans, we drew our sample from the 12,011 veterans for which email addresses were provided. As a result, the results of this survey are not generalizable to the population of participating veterans. To generate the original sample, participants were selected in a probability sample stratified by gender. The sample size was calculated to allow us to produce generalizable estimates having 95 percent confidence intervals no larger than plus or minus 10 percent for the overall sample size of 200, assuming a response rate of 50 percent. Of this original sample of 200 participants, 20 participants were women and 180 were male or no gender identified. Due to lower than expected response rates, we decided to select an additional sample so that we could receive more responses, even though the results of the survey would not be generalizable. Specifically, the survey was sent to an additional sample of 5,000 participating veterans, 500 of whom were women and 4,500 of whom were male or no gender identified. Consequently, the survey’s final sample size was 5,200 participating veterans. Prior to fielding the survey, we conducted 4 pretests to ensure that the questions were clear, comprehensive, did not introduce bias, and were not burdensome to complete. We made minor revisions to the survey instrument based upon these pretests. We launched the web- based survey on June 11, 2015, and sent two follow-up emails to sampled veterans on June 17 and June 24, 2015. We made the web- based sample available to the additional 5,000 sampled veterans on July 9, 2015 and sent a follow-up email to all nonrespondents on July 15, 2015. We closed the survey on July 31, 2015. Overall, GAO received responses from 156 participating veterans—a 3 percent response rate.

Non-participating Veteran Survey

To learn to what extent selected veterans who have used Post-9/11 GI Bill benefits to pursue a non-college degree program, such as vocational school, but who have not participated in the apprenticeship or OJT programs, were aware of these programs, we conducted a web-based survey of such veterans. We limited our in-scope population to those veterans for which VA was able to provide email addresses. Specifically, while VA provided detailed participant level information on 145,525 unique veterans who had used their Post-9/11 GI Bill benefits for a non- college degree program, we drew our sample from the 70,695 veterans for whom email addresses were provided. To generate the original sample, 150 veterans who did not participate in OJT or apprenticeship were drawn from the population of veterans who had used their Post-9/11 GI Bill benefits for a non-college degree program and for which email addresses were available. We assumed a 20 percent response rate for this population. Due to lower than expected response rates, we decided to select an additional sample so that we could receive more responses, even though the results of the survey would not be generalizable. Specifically, the survey was sent to an additional sample of 5,000 non- participating veterans. Consequently, the survey’s final sample size was 5,150 veterans. Prior to fielding the survey, we conducted 2 pretests to ensure that the questions were clear, comprehensive, did not introduce bias, and were not burdensome to complete. We made minor revisions to the survey instrument based upon these pretests. We launched the web- based survey on June 10, 2015 and sent follow-up emails to sampled veterans on June 17 and June 24, 2015. We made the web-based sample available to the additional 5,000 sampled veterans on July 8, 2015 and sent a follow-up email to all non-respondents on July 16, 2015. We closed the survey on Aug. 3, 2015. Overall, we received responses from 127 non-participating veterans—a 2.5 percent response rate. Given the low response rate, the findings of this survey are not generalizable.

Employer and Apprenticeship Sponsor Survey

To learn about how selected employers and apprenticeship sponsors have used the Post-9/11 GI Bill OJT or apprenticeship programs and what challenges they may have faced, we conducted a web-based survey of employers and apprenticeship sponsors who have participated in these programs. We limited our in scope population to those employers and apprenticeship sponsors for which VA was able to provide email addresses. Specifically, while VA provided detailed information on 5,099 unique participating employers and apprenticeship sponsors, we drew our sample from the 2,691 employers and apprenticeship sponsors for which email addresses were provided. To generate the original sample, 600 employers and apprenticeship sponsors were selected in a simple random sample. The sample size was calculated assuming a response rate of 5 percent, and included 7 employers and apprenticeship sponsors who had completed questionnaires during our site visit to North Carolina. Prior to fielding the survey, we conducted 3 pretests to ensure that the questions were clear, comprehensive, did not introduce bias, and were not burdensome to complete. We made minor revisions to the survey instrument based upon these pretests. We launched the web-based survey on June 11, 2015 and sent follow-up emails to sampled employers and apprenticeship sponsors on June 17 and July 9, 2015. GAO closed the survey on July 24, 2015. Overall, GAO received responses from 153 participating employers and apprenticeship sponsors—a 25 percent response rate. The findings of this survey are not generalizable.

State Approving Agency (SAA) Survey

To learn about what states are doing to help veterans and employers use the Post-9/11 OJT and apprenticeship programs, as well as gain insight on veterans’ and employers’ use of these programs and challenges states faced, we emailed a MS Word questionnaire to SAA officials in the 44 states in which these officials oversee implementation of these programs. After developing our questionnaire, we conducted two pretests with state officials to ensure that the questions were clear, comprehensive, did not introduce bias, and were not burdensome to complete. We made minor revisions to the survey instrument based upon these pretests.

The survey included questions about methods of outreach to veterans and employers, assistance to employers, challenges to veteran and employer outreach and participation, and challenges with SAA implementation of the program. It also included questions about the value or interest in key improvements, such as automating the payment system.

We sent the survey by email in March and April 2015 using email addresses from the National Association of State Approving Agencies website. We followed-up with unresponsive states via email until we reached 100 percent participation in July 2015. Survey responses were collected using a Microsoft Word form and imported into Microsoft Excel for summary and analysis. In some cases a state official sent their response in another type of file, and we transposed responses into the Microsoft Word form and verified that the responses matched the state’s supplied answers. While we had survey responses for all 44 states, in several instances a state did not supply an answer for a single question. For summary and analysis performed on those questions we excluded those states from the denominator. All analyses of responses were reviewed and verified to be accurate.

Review of Relevant Laws, Regulations, and Other Documents

To learn about VA’s implementation of the OJT and apprenticeship programs under the Post-9/11 GI Bill, we reviewed relevant provisions of the Post-9/11 GI Bill and related regulations. We also reviewed the Plain Writing Act of 2010, a law related to the public dissemination of information on benefits. In addition, we reviewed VA’s Strategic Plan for 2014 to 2020 to understand greater agency objectives, and other federal best practice documents such as the Job Driven Checklist in the Vice President’s Ready to Work: Job-Driven Training and American Opportunity report, and Standards for Internal Control in the Federal Government.

Analysis of VA and DOL Outreach Materials

To understand VA and DOL outreach to veterans and employers, we analyzed key sources of information that are readily available regarding Post-9/11 OJT and apprenticeship programs. Specifically, we analyzed curriculum documents for VA’s mandatory Transition Assistance Program (TAP), as well as information publically available to veterans and employers on VA and DOL websites. To understand information presented in VA’s TAP program, we reviewed the facilitator’s and participant’s guides for the mandatory TAP courses called Veterans Benefits I and II, as well as a relevant optional training course called the Career Technical Training Track, and compared it to VA’s goals for TAP after the VOW to Hire Heroes Act of 2011 was implemented. We tallied the number of pages in the facilitator’s guides that mentioned OJT and apprenticeship as options under the Post-9/11 GI Bill. Similarly, we reviewed the online version of the Veterans Benefits I and II training to see if additional information about OJT and apprenticeship options was presented. We also reviewed relevant pages of VA and DOL websites. We reviewed and documented Post-9/11 GI Bill and related web pages using screen shots of each relevant page, and noted key portions of content. We assessed the information available from these sources to determine if a veteran or employer could reasonably understand these programs, as described in the Plain Writing Act of 2010 and Standards for Internal Control in the Federal Government. In both the TAP and website analyses, we reviewed included examples and exercises, visual symbols such as the Post-9/11 GI Bill logo, and the language used to describe Post-9/11 GI Bill benefits to determine how, if at all, OJT and apprenticeship options were included. For the website analyses we also looked to see how, if at all, the agencies created links to each other’s content and resources. The calculations and content analysis of the TAP and the website analyses were independently reviewed and verified by another GAO team member.

Interviews with Officials and Stakeholders

To gain perspectives on how Post-9/11 GI Bill OJT and apprenticeship programs are serving veterans, we conducted interviews with a variety of federal and state officials. We spoke with officials at the VA and DOL concerning program administration, outreach efforts, performance measurement, and coordination of their respective OJT and apprenticeship programs. We also interviewed officials at DOD to better understand transition services that are available for service members upon exiting the military. For perspectives on Post-9/11 GI Bill program usage and outreach, we also interviewed State Approving Agency (SAA) officials from 13 states. We chose these 13 states in order to get a range in the number of active OJT and apprenticeship programs and geographic location. We also chose these states based on the institutional location of the SAAs, so as to gain greater perspective on program use and what, if any, challenges SAAs observe in administering the Post-9/11 GI Bill.

We also interviewed stakeholders and experts to learn how OJT and apprenticeship programs are used and, from these stakeholders’ perspective, how effectively they serve veterans. Specifically, we interviewed representatives from a variety of veteran service organizations (VSOs) that serve Post-9/11 veterans and were recommended by internal and external experts on veterans issues. Additionally, we spoke with workforce experts with job training expertise to understand what, if any, academic literature could shed light on the question of OJT and apprenticeship program effectiveness, though their views do not represent the entire research field. Subsequently, we reviewed a 2012 Mathematica Policy Research study, commissioned by DOL, which was cited by experts we interviewed. This study conducted a cost-benefit analysis that included an assessment of the net effects of the Registered Apprenticeship program on participant outcomes. As noted, we found the study to be generally reliable with some methodological limitations.

Site Visits

Among our 13 case illustration states, we selected two—Missouri and North Carolina—for site visits because they have high numbers of Post- 9/11 GI Bill-approved OJT and apprenticeship programs, and different institutional settings for their SAAs. During these site visits, we interviewed SAA officials, OJT and apprenticeship employers and sponsors, veteran participants, and local workforce officials so that we could get a wide range of stakeholder perspectives of Post-9/11 GI Bill OJT and apprenticeship programs, though these perspectives do not necessarily reflect those of states more generally. We also observed training programs on site in such professions as communications technology maintenance and police work. For additional context, we also conducted interviews at two local site visits in the Washington, D.C. area that involved federal programs that were registered through the Post-9/11 GI Bill—the Department of State’s Bureau of Diplomatic Security OJT program and the Department of Defense’s Defense Logistics Agency Energy OJT program.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making key contributions to this report were Laura Heald, Assistant Director, Amy Buck, Devin Braun, Matthew Lowney, Rachel Pittenger, and Michelle Loutoo Wilson. In addition, key support was provided by James Bennett, Rachel Beers, Mindy Bowman, David Chrisinger, Michael Hoffman, Kathy Leslie, Sheila McCoy, Jean McSween, Sara Pelton, Almeta Spencer, Sonya Vartivarian, and Craig Winslow.