VA Benefits: Increasing Outreach and Measuring Outcomes Would Improve the Post-9/11 GI Bill On-the-Job Training and Apprenticeship Programs
Published: 2015-11-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As the military draws down its forces, many veterans will enter the civilian workforce and may seek educational and training opportunities to further their transition into civilian jobs. Because pursuing a higher education degree may not be the best path for some veterans, the Post-9/11 GI Bill OJT and apprenticeship programs provide alternative opportunities. GAO was asked to review these programs.
This report examines (1) how selected veterans and employers used the programs and how widely they have been used; (2) to what extent VA and states have taken steps to inform veterans and employers about these programs; (3) what challenges, if any, veterans and employers have faced in using them; and (4) to what extent VA has assessed the performance of its programs. GAO analyzed VA program data as of March 2015 and DOL program data from 2013 and 2014, and assessed outreach materials. GAO also surveyed officials in all 44 states overseeing VA's programs; conducted nongeneralizable surveys of randomly selected veterans and employers; and interviewed veterans and employers in two states selected for variation in veteran population and type of state agency.
section_title: What GAO Found
paragraphs: Veterans surveyed and interviewed by GAO said the on-the-job training (OJT) and apprenticeship programs offered under the Post-9/11 GI Bill—the largest education benefit program overseen by the Department of Veterans Affairs (VA)—have helped them transition to civilian life, though program data show relatively few veterans have participated. Post-9/11 GI Bill benefits were initially available only for higher education, but in 2011 provisions were enacted that expanded benefits to cover OJT and apprenticeship. Many veterans GAO interviewed (21 of 28) said that the supplemental income the programs provided helped them offset income losses they experienced when leaving the military. About half of the veterans responding to GAO's survey (80 of 156) reported that the program allowed them to use their GI Bill benefits even though college was not a good fit for them. Since OJT and apprenticeship benefits became available in 2011, about 27,000 of the 1.2 million veterans who have received Post-9/11 GI Bill benefits have participated in these programs.
VA primarily provides information about the OJT and apprenticeship programs through mandatory briefings for transitioning servicemembers and on its website. While VA's outreach efforts include some information on these programs, VA's mandatory briefings and web resources generally emphasize higher education and lack sufficient detail for veterans to reasonably understand how to use their GI Bill benefits for OJT and apprenticeships. State officials GAO surveyed reported conducting outreach in a variety of ways, such as attending job fairs and speaking to veterans groups. Without more outreach, veterans who could benefit from these programs may not learn about them.
Key challenges faced by veterans and employers using these programs include lack of awareness and administrative burdens, according to state officials, veterans, and employers GAO surveyed. Most state officials surveyed (39 of 44) reported that lack of awareness about the programs is a primary challenge they face in facilitating veteran and employer participation. Further, over half of state officials surveyed (24 of 42) cited challenges related to VA's current paper-based payment processing system, which requires employers to fax or mail monthly forms to VA in order for a veteran to receive benefits. In addition, 11 of the 15 employers and apprenticeship sponsors GAO interviewed said the process is burdensome or inefficient, and 6 of the 28 veterans GAO interviewed said their benefits have sometimes been delayed. VA is developing a new data system, but it may not be implemented until 2017 at the earliest, according to VA officials, and administrative challenges in the interim could hinder program participation.
Little is known about the performance of VA's Post-9/11 GI Bill OJT and apprenticeship programs because VA does not measure program outcomes, such as whether participants retain employment after completing the program. Absent such information, GAO examined Department of Labor (DOL) outcome data for its related OJT and apprenticeship programs, which indicate the potential for positive outcomes for these training models. Standards for internal control call for establishing and reviewing performance measures to allow an agency to evaluate relevant data and take appropriate actions. Without such measures, VA is limited in its ability to assess its programs.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA improve outreach, ease administrative challenges, and establish outcome measures for its OJT and apprenticeship programs. VA agreed with GAO's conclusions and concurred with all three recommendations.

section_title: Letter
paragraphs: As the military draws down its forces over the next few years, tens of  thousands of servicemembers are expected to transition into civilian life.  To better prepare themselves for meaningful employment, many will seek  educational and training opportunities, which include on-the-job training  (OJT) and apprenticeships. In 2011, provisions were enacted that  extended benefits under the Post-9/11 Veterans Educational Assistance  Act of 2008 (Post-9/11 GI Bill) for eligible veterans interested in such  opportunities. Before those provisions were enacted, Post-9/11 GI Bill  benefits were available to attend a college or university, which may not be  the best path for every veteran. For those who may not be interested in  higher education, OJT and apprenticeships can offer interested veterans  an opportunity for employment and to improve their economic  opportunities.
Given the importance of veterans’ employment to their long-term financial  well-being, you asked us to examine the Post-9/11 GI Bill OJT and  apprenticeship programs administered by the U.S. Department of  Veterans Affairs (VA). This report provides information on the following  questions:  (1) How have selected veterans and employers used the Post-9/11  OJT and apprenticeship programs, and how widely have these  programs been used?  (2) To what extent have VA and states taken steps to inform veterans  and employers about these programs?  (3) What challenges, if any, do veterans and employers report facing  in using the Post-9/11 OJT and apprenticeship programs?  (4) To what extent has VA assessed the performance and  effectiveness of its Post-9/11 OJT and apprenticeship programs?
Our approach for reviewing the Post-9/11 OJT and apprenticeship  programs was comprised of various methodologies. For example, we  conducted four surveys to gather various perspectives on aspects of the  programs that spanned several of our research questions, including  program use, outreach, and challenges veterans and employers reported  in using the programs. (See table 1.)
For our first objective, we analyzed VA program data and conducted two  nongeneralizable surveys—one of veterans who have participated in the  Post-9/11 GI Bill OJT or apprenticeship programs and one of employers  and apprenticeship sponsors who have participated in these programs.  Specifically, we requested and analyzed data from two VA datasets, the  Long Term Solution database, which includes information on the number  of veterans who have received Post-9/11 GI Bill benefits under the OJT  and apprenticeship programs, and the Web Enabled Approval  Management System, which includes information on the number of  approved OJT and apprenticeship facilities. For both sources, the data  VA provided covered participants, employers, and apprenticeship  sponsors who participated in the program between October 1, 2011,  when OJT and apprenticeship benefits first became available, and March  23, 2015, the most current data available at the time of our review. We  assessed the reliability of these data by reviewing existing information  about the data and the system that produced them and by interviewing  agency officials knowledgeable about the data. We determined that the  data were sufficiently reliable for the purposes of this report. For our  survey of participating veterans, we selected participants in a probability  sample stratified by gender. We sent a web-based survey to 5,200  veterans and received responses from 156. Similarly, for our survey of  participating employers and apprenticeship sponsors, we randomly  sampled 600 participating employers and apprenticeship sponsors with  email addresses, and received responses from 153. While not  generalizable, the findings from these surveys allowed us to provide a  greater range of participant perspectives than we could have gathered  through site visits alone.
For our second objective, we conducted a third survey of officials in all 44  states that oversee the OJT and apprenticeship programs. We also  conducted site visits in Missouri and North Carolina and interviewed state  officials from an additional 11 states (Arizona, Illinois, Iowa, Louisiana,  Maine, New York, Oregon, South Carolina, Tennessee, Texas, and  Virginia.) We chose these 13 states to reflect a range in the number of  active OJT and apprenticeship programs, the number of Post-9/11  veterans within the population, and geographic location. While not  generalizable, the perspectives of officials in these 13 states provided  context and specific examples to supplement our survey findings. We  also analyzed VA and Department of Labor (DOL) website information,  and VA and DOL outreach materials. DOL separately administers an  apprenticeship program and OJT services in its programs serving adult  and dislocated workers. Because all servicemembers participating in the  Transition Assistance Program (TAP) must take VA Benefits I and II  courses, we analyzed curriculum materials from these courses. We  selected these portions of the TAP curriculum because they are intended  to be comprehensive and cover the entire spectrum of VA benefits for  servicemembers and their families. In analyzing what steps VA has  taken to inform veterans of these programs, we examined to what extent  VA materials include information on the Post-9/11 OJT and  apprenticeship programs. We compared these informational materials to  criteria established in the Plain Writing Act of 2010 and Standards for  Internal Control in the Federal Government regarding agencies’  communications with external stakeholders.
For the third objective, we conducted a survey to learn whether a lack of  awareness among veterans could be a barrier to participation.  Specifically, we randomly sampled 5,150 veterans who used their GI Bill  benefits to attend a non-college degree program, such as a vocational  school, but did not participate in the OJT or apprenticeship programs, and  received responses from 127 veterans. Even though these results are not  generalizable, the survey provides information on the extent to which  survey respondents were aware of the OJT and apprenticeship programs.  We also analyzed results from our survey of state officials and our other  two non-generalizable surveys of participating veterans and of employers  and apprenticeship sponsors to identify their reported challenges. In  addition, we interviewed 28 participating veterans and 15 participating  employers and apprenticeship sponsors during our site visits to obtain  specific examples. Lastly, we compared information on VA’s process for  submitting and processing paperwork for benefits with standards for  internal control in the federal government.
For our fourth objective, we requested that VA provide outcome data for  its OJT and apprenticeship programs. In assessing VA’s capability to  provide such data on program performance, we used as criteria  standards for internal control in the federal government and leading  practices for federally funded employment and training programs, as set  out in the Vice President’s Job Driven Training Report. To provide  broader information on the performance of OJT and apprenticeship  training models, we obtained outcome data on DOL’s programs. More  specifically, we requested and analyzed DOL outcome data on OJT  services in its programs serving adult and dislocated workers from the  Workforce Investment Act Standardized Record Data (WIASRD) data on  Post-9/11 veterans in the fourth quarter of 2013. We also examined DOL  outcome data from 2014 pertaining to participants in Registered  Apprenticeship programs from the Registered Apprenticeship Partners  Information Management Data System (RAPIDS).We selected these  timeframes because they were the most current data available at the time  of our review. We assessed the reliability of these data by reviewing  existing information about the data and the system that produced them  and by interviewing agency officials knowledgeable about the data. We  determined that the data were sufficiently reliable for the purposes of this  report. In addition, we interviewed experts on employment and training  programs to better understand the academic literature on the  effectiveness of DOL’s OJT services and apprenticeship programs. While  the views of these experts are not generalizable, they helped provide  perspectives on the literature related to the effectiveness of OJT and  apprenticeship training models. Finally, for all objectives we reviewed  relevant federal laws and regulations. See appendix I for additional  information on our objectives, scope, and methodology.
We conducted this performance audit from August 2014 to November  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: GI Bill Overview
paragraphs: The Department of Veterans Affairs (VA) has been providing veterans GI  Bill benefits since 1944. Among several GI Bill programs, the Post-9/11  GI Bill program has become the largest. The Post-9/11 GI Bill—which  took effect in August 2009—provides benefits to veterans and  servicemembers serving on active duty after September 10, 2001.
Depending on what education or training program they select, participants  may be eligible for benefit payments for up to 36 months to cover tuition  and fees, housing, and books and supplies. From 2009 to 2011, the  program provided benefits only to participants attending a degree- granting institution of higher learning. However, in 2011 the Post 9/11  Veterans Educational Assistance Improvements Act of 2010 expanded  approved programs of education to include on-the-job training and  apprenticeship, as well as other types of non-degree programs, such as  vocational or technical school.
section_title: VA OJT and Apprenticeship Programs
subsections: section_title: OJT and Apprenticeship Post- 9/11 GI Bill Benefits
paragraphs: Eligible veterans can use their benefits through the VA’s Post-9/11 GI Bill  to participate in an approved OJT or apprenticeship program. Veterans  can receive benefits for an OJT program lasting from 6 months to 2 years,  or for an apprenticeship as long as 36 months until their Post-9/11 GI  Bill benefits are exhausted. When veterans opt to apply their Post-9/11 GI  Bill benefits to an eligible OJT or apprenticeship program, in addition to  the wages they earn from their employer as a trainee or apprentice, they  receive a tax free monthly housing payment from VA to help cover living  expenses. The amount of the benefit is based on the basic allowance for  housing (BAH) paid to servicemembers. Veterans receive scheduled  wage increases from their employer during the training period. Every 6  months, the monthly benefit received from VA decreases. Furthermore,  because OJT programs can be supplemented with classroom  components, and apprenticeship programs must include related  instruction, participants may receive up to $83 per month to help cover  the cost of books and supplies. (See table 2.)
section_title: Program Requirements and Approval Process
paragraphs: Program requirements and the approval process are set out by VA and  the State Approving Agencies (SAAs) in their training manual for state  officials. To be approved, employers sponsoring OJT programs must  complete an application indicating that they meet a number of criteria,  including creating and adhering to a training plan, designating a certifying  official, having “reasonable certainty” that a position will be available  when the veteran completes training, and paying wages that meet  program requirements. To be approved for the GI Bill, apprenticeship  programs must meet the same regulatory criteria they would need to be  registered with DOL or a State Apprenticeship Agency.
Specifically, GI Bill-approved OJT programs are intended to be for a job  that is learned in a practical way through a structured, systematic  program of supervised on-the-job training; however, there is generally no  requirement for classroom training. The training period must be full-time,  compensated employment that is a minimum of 6 months and a  maximum of 2 years in length. Full-time training is generally considered to  be not less than 30 hours per week.
Similarly, the training manual provides that a GI Bill-approved  apprenticeship program must be a structured, organized, written program  combining on-the-job training and related instruction in which workers  learn the practical and theoretical aspects of a skilled occupation.  Employers must provide incremental wage increases to participants  throughout the apprenticeship. The length of an apprenticeship is  generally not less than 2,000 hours of full-time work experience that is  consistent with training requirements as established by industry practices  and a recommended minimum of 144 hours of related instruction per  year. Full-time training is generally considered to be not less than 30  hours per week or 120 hours per month. According to DOL, individuals  participating in an apprenticeship program are expected to advance from  a low or no skill, entry-level position to full occupational proficiency at the  end of training.
GI Bill-approved OJT and apprenticeship programs for non-federal  employers or apprenticeship sponsors must generally be approved by an  SAA, while VA approves OJT and apprenticeship programs sponsored by  federal agencies.
SAAs are state agencies designated by the state’s governor that  evaluate, approve, and monitor education and training programs for the  GI Bill (see fig. 1). SAAs generally contract with VA to approve both  educational institutions and OJT and apprenticeship programs. In  addition to approving programs, SAA staff is responsible for providing  technical assistance, oversight, and program outreach.
section_title: OJT and Apprenticeship Resources through DOL
subsections: section_title: DOL OJT Services
paragraphs: DOL offers OJT services that are separate from the VA Post-9/11 GI Bill  OJT program and follow a similar model, but there are several key  structural differences between these programs and services. DOL’s OJT  services were authorized under the Workforce Investment Act of 1998  (WIA) and now operate under the Workforce Innovation and Opportunity  Act (WIOA) enacted in 2014. Under WIOA, eligible individuals, including  veterans, can receive employment training assistance and referrals from  local workforce offices known as American Job Centers (formerly called  One-Stop centers). Consistent with WIOA, American Job Centers provide  different levels of services, depending on an individual’s needs:
Career Services, which include assistance with job searches and job  placement, career counseling, and developing an individual  employment plan; and
Training Services, including OJT programs and other occupational  training opportunities.
American Job Center staff help match DOL OJT participants to  participating employers for employment and training opportunities. Upon  establishing a contract for the provision of occupational training,  participating employers receive a subsidy of up to 75 percent of the  trainee’s wages to compensate for training costs. In order to maintain  their eligibility, employers must be able to demonstrate a pattern of  providing program participants with long-term employment and wages,  benefits, and working conditions commensurate with unsubsidized  employees of similar work experience.
There are several key differences between the OJT services available to  veterans through DOL and the OJT programs that are part of the Post- 9/11 GI Bill (see table 3). Whereas veterans receive a regular benefit  equivalent to their monthly housing allowance as part of the Post-9/11 GI  Bill, the employers receive wage subsidies as part of DOL OJT services.  To be eligible for DOL OJT services, a veteran must have been  determined by American Job Center staff to require training in order to  gain or retain employment. In contrast, Post-9/11 GI Bill OJT  participants must only meet the length of military service requirements in  order to be eligible for an OJT program. Unlike the VA, which officials  indicated is not required to track outcome measures for OJT participants,  states are required to collect and report data to DOL, using  Unemployment Insurance wage records, on post-program performance  measures, including employment retention rates and earnings gains.
section_title: DOL Apprenticeship Programs
paragraphs: Distinct from Post-9/11 GI Bill apprenticeship programs, veterans can  also participate in DOL’s Registered Apprenticeship program. In concert  with recognized state apprenticeship agencies, DOL’s Office of  Apprenticeship determines which programs meet the standards to be  registered with DOL. For the Post-9/11 GI Bill, an apprenticeship  program does not need to be registered with DOL to be approved, but if a  program is registered, its training plan is deemed approved for purposes  of the VA program.
Registered Apprenticeship occupations are offered in such fields as  electrical work, carpentry, plumbing, and roofing. In addition to  monitoring compliance with program requirements, the Office of  Apprenticeship partners with employers, workforce agencies, colleges,  the military, and other organizations to promote Registered  Apprenticeships and provide technical assistance. As with OJT programs,  DOL requires Registered Apprenticeship agencies to monitor  performance using completion rates, among other indicators, and  enables these agencies to access participants’ wage records through the  Wage Record Interchange System (WRIS) to report on performance  measures such as employment retention and earnings gains. VA is not  required by law to track outcomes for Post-9/11 GI Bill apprenticeship  participants, according to VA officials. (See table 4.)
section_title: DOL-Funded Staff to Support Veterans’ Employment
paragraphs: In addition to American Job Center staff who serve all jobseekers,  including veterans, DOL’s Veterans’ Employment and Training Service  funds two staff positions that focus on promoting veterans’ employment.  Specifically, DOL funds Local Veterans’ Employment Representatives  (LVER) to promote veteran employment opportunities and coordinate  training services for veterans by building relationships with employers and  working with staff at American Job Centers. Additionally, Disabled  Veterans’ Outreach Program (DVOP) specialists provide case- management to veterans with significant barriers to employment, such as  homelessness or disabilities.
section_title: Veterans in Our Review Said the Post-9/11 OJT and Apprenticeship Programs Helped Them Transition to Civilian Life, but Relatively Few Veterans Have Participated
subsections: section_title: Veterans in Our Review Said the Program’s Supplemental Income Helped Them Cover Living Expenses while They Trained
paragraphs: The majority of veteran participants who replied to our survey (125 of  156) cited more than one benefit to the program. Many specifically noted  that receiving supplemental income to help them cover living expenses  during their transition to civilian life was a primary benefit (112 of 156). In  2015, this supplemental income ranged from $896 to $3,923 per month  for the first 6 months, depending on the location of the employer or  apprenticeship sponsor. Many veterans we interviewed (21 of 28) said  that this supplemental income helped them offset the loss of income they  experienced after leaving the military. For example, slightly more than half  the veterans we interviewed (16 of 28) said that the supplemental income  helped them pay critical living expenses, such as a mortgage or car  payment, or support their families. (See text box.) A few veterans (4 of  28) told us that without the Post-9/11 GI Bill benefits they would have  needed a second job, which would have made it difficult to focus on  learning their new occupation.
In addition to providing supplemental income, about half of the OJT and  apprenticeship participants who responded to our survey (80 of 156)  reported that the programs allowed them to use their GI Bill benefits, to  which they are entitled, when college was not a good fit for them. Several  veterans we interviewed (4 of 28) said they had not expected to be able  to use their GI Bill benefits because they knew they wanted to enter a  specific trade or occupation that did not require a college degree, such as  pipefitter or police officer. These veterans said they were pleased to be  able to use the benefits they had earned while starting their chosen  careers. Others (6 of 28) said that due to family obligations they needed  to work full time and would not have been able to afford college. Further,  some OJT and apprenticeship participants who responded to our  survey—including both those who had and had not used Post-9/11 GI Bill  benefits to attend college—reported one of the main benefits of the  program was that it helped them learn a new occupation (86 of 156) and  further develop skills they had obtained in the military (60 of 156).
VA officials told us that employers and apprenticeship sponsors do not  receive any federal funds for participating in these programs, though the  employers and apprenticeship sponsors who responded to our survey  cited a number of benefits for participating, particularly recruitment and  retention. Specifically, just over half of the employers and apprenticeship  sponsors who responded to our survey (87 of 153) reported the program  helped them recruit veterans to be employees. Almost all of the  employers and apprenticeship sponsors we interviewed (12 of 15) praised  the discipline, work ethic, or customer service skills of participating  veterans, or said they made valuable employees or apprentices. In  addition to recruitment benefits, half of the employers and apprenticeship  sponsor respondents (76 of 153) indicated that the programs had helped  them retain veterans as employees. Some employers and apprenticeship  sponsors who responded (47 of 153) also indicated that the program  enhanced their standing in the community. (See text box.)
section_title: Since 2011, About 27,000 Veterans Have Participated in the Post- 9/11 GI Bill OJT and Apprenticeship Programs to Train in a Variety of Occupations
paragraphs: Of the approximately 1.2 million veterans who used their Post-9/11 GI Bill  benefits since October 2011, about 27,000—or about 2 percent— participated in the OJT or apprenticeship programs, according to VA  program data. Approximately half of these veterans participated in OJT  opportunities, while the other half pursued apprenticeships. The average  age of participants was 33, and most were male (94 percent). Over half of  all participants had served either in the Army (28 percent) or the National  Guard or Reserves (27 percent). (See table 5.) The number of  participating veterans varied considerably by state, for example, from a  low of 14 participants in Wyoming to a high of 799 in Texas during fiscal  year 2014.
During fiscal year 2014, there were approximately 2,700 employers and  apprenticeship sponsors approved to train Post-9/11 GI Bill veterans.  According to VA and state officials, not all of these approved OJT and  apprenticeship sponsors actually had veterans receiving benefits under  the program during this timeframe.
Participating veterans have used the OJT and apprenticeship programs to  train in a variety of occupations. While VA officials told us the agency  does not collect standardized occupational data, our analysis of VA’s  administrative data on approved programs indicates that the following  occupations are among those most frequently pursued by participants:
Police and sheriff officer
Correctional officer and jailor
Truck driver
Electrician
Firefighter
Air traffic controller
Veterans service representative
Carpenter
Plumber, pipefitter, and steamfitter  In addition, veterans we interviewed reported being trained for a variety of  other positions, including maintenance and repair workers, special  agents, and cable installation and maintenance technicians.
section_title: VA and States Provide Varying Levels of Information and Outreach to Veterans and Employers
subsections: section_title: VA Provides Some Program Information That Generally Highlights School-Based Options and Lacks Sufficient Detail On OJT and Apprenticeships
paragraphs: While VA provides some information on Post-9/11 GI Bill OJT and  apprenticeship programs, the focus of most outreach information is on  higher education benefits. OJT and apprenticeship information lacks  sufficient detail to inform veterans these benefits are available or how  they might be used. There are a few different resources available to  servicemembers—and to some extent, employers—that provide  information on Post-9/11 GI Bill benefits, including information shared at  mandatory pre-separation briefings as part of the Transition Assistance  Program (TAP), on agency websites, and as part of VA’s education and  career counseling.
subsections: section_title: Transition Assistance Program (TAP)
paragraphs: In general, mandatory briefings provided by VA for separating  servicemembers as part of TAP cover Post-9/11 GI Bill benefits, but  almost all of this information is focused on using these benefits for higher  education. The purpose of TAP is to prepare servicemembers leaving the  military for their transition back into civilian life. TAP includes both  mandatory and optional components. For example, all servicemembers  participating in TAP must take VA’s Veterans Benefits I and II courses,  which discuss available benefits and services, including education  benefits. Federal, state, and veterans service organization officials we  interviewed identified TAP as one of the primary ways that veterans can  learn about the Post-9/11 OJT and apprenticeship programs. However,  out of 77 total pages in the TAP facilitator guide and briefing slides for the  VA Benefits I and II courses, there is one reference to the OJT and  apprenticeship programs as part of a description of the books and  supplies stipend portion of the Post-9/11 GI Bill. In discussing Post-9/11  GI Bill benefits, the briefing slides generally refer to “education,” “tuition,”  “school,” and “student,” and do not refer to employers, OJT, or  apprenticeship opportunities, although these options are also available.  Presenting this information in Veterans Benefits I and II courses would  align with VA’s stated goal that these briefings be comprehensive and  cover the entire spectrum of VA benefits for servicemembers and their  families.
Similarly, the Veterans Benefits I and II briefing slides do not include any  specific examples of veterans who have used their benefits for OJT or  apprenticeship opportunities or how these benefits work. Information on  the OJT and apprenticeship program of Post-9/11 GI Bill benefits is  included in an optional portion of TAP, called the Career Technical  Training Track. But here as well there is relatively little information  provided to servicemembers on how they can use their Post-9/11 benefits  to pursue OJT and apprenticeship opportunities. Several slides in this  track provide information on what OJT and apprenticeship programs are,  however, it is not clear that veterans can use their Post-9/11 GI Bill  benefits for these programs.
As discussed later in this report, 39 of 44 SAAs reported that lack of  awareness of the Post-9/11 GI Bill OJT and apprenticeship programs was  one of two top challenges in facilitating veteran participation. This finding  was also supported by the results of our other survey efforts. Specifically,  about half the veterans who responded to our survey reported that TAP  briefings do not include enough information on OJT and apprenticeship  programs under Post-9/11 GI Bill benefits. Of the 156 veterans who  responded to our survey and have used their Post-9/11 GI Bill benefits for  OJT and apprenticeship, 81 reported that they did not think TAP  adequately informs veterans about these options. Similarly, 63 of the 127  responding veterans who have used their Post-9/11 GI Bill benefits for a  non-college degree, such as a trade school program, had the same  response. While 5 of the 7 veterans service organization leaders we  interviewed indicated that TAP should not be the only source of  information for veterans, some of whom find it difficult to process the  volume of information presented during TAP, additional information about  the Post-9/11 GI Bill OJT and apprenticeship programs could be  beneficial in informing servicemembers of their options.
VA officials told us they have not considered expanding the curriculum of  the mandatory courses to include more information about the OJT and  apprenticeship programs. They stated they only have a limited amount of  time to cover a wide range of benefits during the TAP briefings and have  prioritized content. VA officials told us that they revisit their TAP  curriculum on an annual basis. During this annual process, officials  update benefits information and accommodate new material. They also  stated that they encourage veterans to attend the optional Career  Technical Training Track, which includes information on these programs.  However, without additional information about the OJT and  apprenticeship program options under the Post-9/11 GI Bill, veterans may  continue to lack awareness of these benefits, as discussed later in the  report.
section_title: Agency Websites
paragraphs: Focus on Higher Education Benefits  Another source of information about Post-9/11 GI Bill benefits are agency  websites, and while VA’s and DOL’s websites provide some information  on these benefits, VA’s website is more focused on higher education  benefits. According to the Plain Writing Act of 2010, information supplied  to the public regarding any federal benefit (including in paper and  electronic publications, forms, notices or instructions) must be clear,  concise, well-organized, and appropriate for the intended audience. The  President’s guidance for implementing this act states that “agencies  should communicate with the public in a way that is clear, simple,  meaningful, and jargon-free,” and notes that “a lack of clarity may prevent  people from becoming sufficiently aware of programs or services” for  which they are eligible. VA’s website has resources for individuals to learn  about GI Bill benefits, including a web page for the GI Bill, a web page  dedicated to the Post-9/11 GI Bill, and a GI Bill Comparison Tool. While  there is also a web page for the OJT and apprenticeship programs, the  language used on the other GI Bill-focused web pages, and resources  generally, refers to higher education and does not reference OJT or  apprenticeships. For example, the main heading of the GI Bill web page is  “Education and Training.” However, the rest of the page uses language  and visual cues that reference higher education, such as “choose a  school” and “verify school attendance.” Moreover, the logo for the Post- 9/11 GI Bill—a graduation cap with dog tag tassels—further emphasizes  higher education to the exclusion of other types of GI Bill benefits.
VA’s GI Bill Comparison Tool also uses language that emphasizes higher  education, even though the tool can be used to access information on  employers. The main heading, for example, is “Choosing a School,” and  the options below are “About Your School” and “Compare This School”  even when an employer is chosen—it may not be apparent to veterans  that they can use this tool to explore OJT and apprenticeship benefits.
Lack of Sufficient Detail on OJT and Apprenticeship Programs  In addition, the information provided on VA’s Post-9/11 GI Bill web page  and on its OJT and apprenticeship web page lacks enough detail for  users to reasonably understand how to use their GI Bill benefits for the  OJT and apprenticeship programs. For example, the page content does  not articulate how OJT programs are different from apprenticeship  programs—a difference that could have meaningful career implications  for veterans. Participants in an apprentice program, for example, earn a  license or national industry certification and generally train longer, while  OJT participants generally receive a certificate of completion and train for  a shorter period of time. In addition, the web page lists three occupations  that might be available through OJT or apprenticeship, union plumber,  hotel manager, and firefighter, but does not provide additional details,  such as descriptions of what OJT or apprenticeship programs associated  with these occupations might involve, or testimonials on how actual  veterans used and benefited from these programs. In addition to the  requirements for clear and meaningful communication described in the  Plain Writing Act of 2010, Standards for Internal Control in the Federal  Government state that management should ensure there are adequate  means of communicating with external stakeholders in which there could  be a significant impact on the agency achieving its goals. Further, VA’s  strategic goals include empowering veterans to improve their well-being  and managing and improving VA operations to deliver seamless and  integrated support. Without providing sufficient detail regarding the OJT  and apprenticeship programs, VA may not provide veterans complete  information to understand these benefits, and may not fully meet its goals  of serving this population.
While VA’s OJT and apprenticeship web page links to a fact sheet on  these Post-9/11 GI Bill programs, this fact sheet also lacks sufficient  detail to help veterans and employers reasonably understand how they  can participate. For example, the fact sheet describes the amount of  Post-9/11 GI Bill benefit payments, but does not describe the difference  between OJT and apprenticeship programs, or provide examples of how  veterans used benefits from these programs.
In contrast, DOL has a comprehensive web page for its Registered  Apprenticeship program that specifically targets veterans. It also includes  information about potential benefits of participating in an apprenticeship,  and profiles several veterans who participated in the program while  receiving GI Bill benefits. These profiles provide concrete examples of  how veterans have used the GI Bill benefits during apprenticeships in a  variety of fields and how their apprenticeships helped them after leaving  the military. VA does not link to this page from its OJT and apprenticeship  web page, missing an opportunity to leverage this resource and  potentially making it more difficult for veterans to learn about these  benefits through its website.
VA’s GI Bill web pages also provide little information for employers who  may be interested in participating. The OJT and apprenticeship web page  includes two sentences directed to employers stating that they can  participate in the program and directing them to their SAA office. It does  not, however, provide information about requirements for employers or  why employers might want to participate, which could help attract  employers or apprenticeship sponsors to help VA achieve its goal of  improving veterans’ well-being.
section_title: VA’s Education and Vocational Counseling
paragraphs: Finally, VA’s education and vocational counseling includes some  information on GI Bill benefits, but according to VA, counselors rely on the  GI Bill website for information regarding the OJT and apprenticeship  programs. In addition to TAP and agency websites, servicemembers may  participate in VA’s education and vocational counseling to learn more  about their benefits. VA officials told us that benefits advisors are  permanently staffed at almost 300 military installations and other sites  worldwide and provide transition and outreach briefings, as well as  individual assistance, guidance, and counseling on all VA benefits and  services. According to VA officials, the counselors who deliver these  benefits are trained and have knowledge of OJT or apprenticeship  programs. When asked to provide training materials related to the OJT  and apprenticeship programs, VA officials told us that counselors find  information about OJT and apprenticeships on the GI Bill website.
section_title: State Officials We Surveyed Reported Providing Outreach to Veterans and Employers, Which Varied By State
paragraphs: Most state officials we surveyed reported that they reach out to veterans  using direct methods, such as attending job fairs and providing briefings  and presentations to veterans’ groups (see table 6). Fewer state officials  reported using broader outreach methods, such as radio or television  advertisements or social media, to raise awareness of the programs.  Some state officials said that they devote considerable staff time and  resources to outreach efforts. For example, an SAA director in one state  told us his agency created various informational materials to raise  awareness of these programs, including a video titled “OJT: It’s not Just  for College” to distribute to stakeholders, a monthly newsletter on  veterans’ employment issues, and program fact sheets and brochures to  be displayed in American Job Centers and higher education veterans’  offices. An SAA official in another state told us his agency arranged for  OJT and apprenticeship program information to be printed on the back of  state unemployment insurance (UI) benefit checks and works with DOL  workforce staff to leverage employer contacts. However, our survey  results also indicate that the level of outreach conducted varies by state.  Specifically, officials in 5 states we surveyed indicated that due to  resource constraints, they were unable to engage in more intensive  outreach efforts for veterans or employers.
Without adequate program outreach, veterans who could benefit may  remain unaware of these programs and miss opportunities to improve  their long-term employment prospects, consistent with agency goals.
section_title: VA and DOL Have Coordinated to Build Awareness of the Post- 9/11 GI Bill Apprenticeship Programs
paragraphs: VA has coordinated some aspects of its Post-9/11 GI Bill apprenticeship  program with DOL’s Registered Apprenticeship program to help make  apprenticeship sponsors and veterans aware of the GI Bill program. By  statute, the same regulatory requirements apply to apprenticeship  sponsors seeking GI Bill approval for their program as those seeking  approval through DOL’s Registered Apprenticeship program. Prior to  2014, this process was not streamlined, and SAAs independently  assessed whether a Registered Apprenticeship program sponsor seeking  approval for GI Bill eligibility met these regulatory requirements. In  September 2014, the Secretaries of Veterans Affairs and Labor  announced that any DOL or State Apprenticeship Agency-approved  Registered Apprenticeship program was qualified to be approved under  the Post-9/11 GI Bill. The joint letter making this announcement was sent  to approximately 10,000 Registered Apprenticeship programs to alert  program officials that participating veterans could be eligible to receive  benefits under the Post-9/11 GI Bill, according to VA officials. The letter  included detailed information on how Registered Apprenticeship  programs could obtain GI Bill approval and also contained two outreach  flyers. One flyer was designed for veterans. The other, for Registered  Apprenticeship sponsors, provided detailed information on the GI Bill  program and its benefits.
DOL also conducted two additional outreach efforts, one targeted at  apprenticeship sponsors and the other at veterans, to raise awareness  that veterans can receive GI Bill benefits while participating in a DOL  Registered Apprenticeship program. Regarding the first effort, DOL  officials told us they created a pop-up window that appears when  apprenticeship sponsors enter information into the Registered  Apprenticeship database about a new apprentice. The pop-up window  reminds sponsors that their apprenticeship program is qualified to be  approved under the GI Bill, and that veteran apprentices may be entitled  to receive GI Bill benefits while participating in the program. The pop-up  window also links to several different resources created by both DOL and  VA that apprenticeship sponsors can use to learn more about VA’s  program. Regarding the second outreach effort to veterans, DOL created  a Registered Apprenticeship website targeted specifically to veterans that  highlights the availability of GI Bill benefits for apprenticeships. VA  requires SAA officials to meet biannually with DOL’s Local Veterans  Employment Representatives (LVER). According to VA’s contract with  SAA staff, the purpose of these meetings is to ensure that students who  graduate in the SAA’s state are made aware of employment resources  and opportunities. SAA directors we interviewed in 10 of 13 states told us  these meetings were generally helpful, noting that they increased  program awareness or helped SAA staff connect with employers. One  official said LVERs are “wonderful ambassadors” for the Post-9/11 GI Bill  program. In contrast, SAA directors we interviewed in 3 of the 13 states  indicated that while their staff meet with LVER staff as required, these  interactions rarely result in increased program awareness about the GI  Bill OJT or apprenticeship programs.
VA and DOL have more coordination for apprenticeship programs than  for OJT, possibly due to structural differences in how OJT services are  provided. For example, the administration of OJT services funded by  DOL, which occurs at the local level, is more decentralized than for  Registered Apprenticeship programs, which are overseen at either the  federal or state level.
section_title: Lack of Program Awareness and Administrative Burdens Have Challenged Veterans and Employers, According to State Officials and Program Participants
subsections: section_title: Veterans and Employers Often Lack Awareness of OJT and Apprenticeships as Options under the Post- 9/11 GI Bill, According to State Officials and Program Participants
paragraphs: Veterans and employers generally lack awareness of OJT and  apprenticeship options, and they most often heard of these programs  from word-of-mouth sources, according to state officials and program  participants we surveyed. Specifically, most state officials we surveyed  cited lack of awareness as a top challenge in facilitating veteran (39 of  44) and employer (39 of 43) use of the Post-9/11 GI Bill OJT and  apprenticeship programs (see table 7). In addition, most state officials (39  of 44) cited a lack of understanding among veterans as to how they can  use these benefits to pursue career goals as a top challenge. Similarly,  the most common challenge experienced by responding state officials  trying to facilitate employer participation in OJT and apprenticeship  programs is a lack of knowledge of this option among employers (39 of  43).
State officials we surveyed also cited some of the same challenges that  we identified in our review of VA’s information resources, and some  confusion about the OJT and apprenticeship programs that may  contribute to a lack of knowledge about these programs (see table 7).  Specifically, 36 of 44 responding state officials pointed to the emphasis  on Post-9/11 GI Bill materials on education rather than OJT or  apprenticeship as a challenge in facilitating veteran participation.  Additionally, 30 of 44 responding state officials indicated that veterans  have difficulty distinguishing between benefit options and programs  available to them. Twenty-eight of 43 responding state officials said that  employers lacked knowledge of the SAA and its functions and  responsibilities. In addition, 25 of 43 responding state officials indicated  that employers have difficulty distinguishing OJT and apprenticeship GI  Bill programs from other government efforts to promote veterans’  employment. “This is a hidden program that should be more prominent in availability. I would not have  known I could use the OJT program if not for word-of-mouth from another employee.”
Similarly, participating employers and apprenticeship sponsors who  responded to our survey most frequently said they heard about the  program from a veteran employee (63 of 153) or an SAA official (42 of  153).
Of veterans we surveyed who used GI Bill benefits for a non-college  degree, such as at a vocational school, about half of those who  responded (65 of 127) said they did not know about the OJT and  apprenticeship program options when deciding how to use their benefits.  About half of the responding veterans (37 of 65) who had not heard about  the programs reported they would have considered using their benefits to  pursue OJT or apprenticeships rather than an education program had  they known this was an option. Further, about half (63 of 127) of the  responding veterans who used GI Bill benefits for a non-college degree,  such as at a vocational school, stated that TAP did not adequately inform  them that they can use their GI Bill benefits for these programs.
To build awareness of these programs, VA officials said they send  multiple letters to servicemembers prior to their separation to let them  know they may be entitled to education benefits. Two of these letters  include a statement that says Post-9/11 GI Bill benefits are available for  trade schools, apprenticeships, and OJT in addition to colleges and  universities. In addition, in 2011 and 2013 VA completed two targeted  outreach campaigns. VA officials told us these outreach campaigns were  online and targeted the five states that had the highest rates of veteran  unemployment in 2011, and the five states with the next-highest rates of  veteran unemployment in 2013. According to VA, the first campaign  resulted in nearly 354,000 unique individuals visiting the web page that  explained the expansion of the GI Bill to OJT and apprenticeship, among  other new options. The second campaign resulted in an additional 76,000  visits to this web page. VA officials told us they have since consolidated  their web pages and in doing so eliminated this GI Bill expansion web  page. They said no additional VA outreach campaigns have been  completed or are planned, absent additional funding, among other  reasons. While word-of-mouth discussion of the Post-9/11 GI Bill OJT and  apprenticeship program benefits can increase program awareness, this is  not a systematic method of outreach, and VA officials acknowledged  there is room for improvement in their outreach approach.
section_title: State Officials and Employers We Surveyed Reported VA’s Paper- Based Payment System, which VA Plans to Replace by 2017, Is Burdensome, and Payments Are Sometimes Delayed
paragraphs: “There is no way to verify and track paperwork that has been submitted via fax or mail and  it is ‘lost’  1 out of 3 submissions.” “The biggest time drain is ‘faxing’ in the monthly hour logs. This is an antiquated way of  doing this and the option for scanning and emailing should be incorporated.”  “The GI/Bill OJT program is beneficial to me, but only when I received my benefits. In the  past year, only 3 to 4 months  have been ‘on time.’ I’m currently waiting on 4  months’ worth of benefits. I personally am struggling with bills and I know many others are  too… That is the biggest concern… that our paperwork gets ‘lost’ for months on end.” “It takes too long for the VA to process the paperwork, and there really is no easy way to  correct mistakes if they happen. This delays payment and causes bills to pile up.”  “There is no communication between the VA and us. If there is a problem with the  application, the VA does not let us know if there are any problems, and the veteran has to  wait for his or her application to be processed.” “It was extremely difficult to get everything to and from everybody that needed paperwork.  It took over a year after I started the program to get all of the things needed to the right  people…. It was a very frustrating process,  if I was not persistent, I could have given  up.”
VA officials told us delays sometimes occur because a veteran’s benefit  application is received before the employer has been approved or  because not all required paperwork has been submitted. VA officials also  told us that they are in the process of developing a new data system  called Veterans Approval, Certification, Enrollment, Reporting and  Tracking System (VA-CERTS), which will combine and replace the  program certification for educational institutions, and monthly certification  and claims processing functions currently housed in two separate  systems. Officials indicated that VA-CERTS will include the program  certification and monthly hours certification, as well as claim processing  for the OJT and apprenticeship programs. According to officials, VA- CERTS is in the early stage of development. Specifically, the business  requirements document to build the new system was completed in July  2015 and delivered to VA’s Office of Information and Technology. It  includes the requirements for the OJT and apprenticeship elements  mentioned above. Officials told us that funding to develop VA-CERTS has  been identified, though final implementation of the new system may not  occur until at least 2017. In the interim, absent efforts to ease the  administrative challenges associated with submitting monthly paperwork,  some employers and veterans are likely to continue to experience  difficulties and delays. Such efforts to ease the administrative challenges  could include allowing employers and apprenticeship sponsors to submit  monthly certification forms by email, or providing confirmation that  monthly forms have been received.
section_title: Little Is Known about the Performance of VA’s OJT and Apprenticeship Programs
subsections: section_title: VA Does Not Measure Outcomes for Participants in the Post-9/11 GI Bill OJT and Apprenticeship Programs
paragraphs: VA has not established outcome measures to report on the performance  of Post-9/11 GI Bill OJT and apprenticeship participants. Specifically, VA  does not measure such outcomes as whether participants retained  employment or experienced earnings gains after completing the program.  As previously mentioned, VA officials indicated that there is no  requirement for VA to measure OJT and apprenticeship performance.  However, standards for internal control call for establishing and reviewing  performance measures to allow an agency to evaluate relevant data and  take appropriate actions. Moreover, a recent review by the Vice  President concluded that an integral element of successful job training  programs is measuring outcomes for all participants and informing  participants and employers of the results. The Vice President’s report  noted the importance of tracking outcomes such as employment retention  and earnings gains so participants can choose training options wisely— and so programs can improve performance. Additionally, VA officials  told us that having access to data on participant outcomes could help  them promote the programs to employers and veterans.
According to VA officials, the agency lacks a data mechanism and  resources for measuring employment outcomes once veterans have  exhausted their Post-9/11 GI Bill benefits. States’ Unemployment  Insurance wage records, which DOL uses to measure outcomes for its  OJT and apprenticeship programs, are one possible data source VA  could explore to measure outcomes for participating veterans. According  to DOL officials, VA could consult with DOL to determine if it could access  this data source through the Wage Records Interchange System 2 (WRIS  2). VA officials told us they had not previously considered participating  in WRIS 2 to access wage records, and consequently have not  considered if they need to seek legislative authority to do so. In addition  to this shared resource, other mechanisms for measuring outcomes could  potentially be developed, such as collecting outcome data directly from  employers and apprenticeship sponsors.
Although VA does not measure outcomes for OJT and apprenticeship  programs, it does collect and report data for some outcome measures for  the higher education portions of the Post-9/11 GI Bill. Specifically, VA  collects and publicly reports data for GI Bill participants on such  measures as graduation rates and loan default rates. VA officials also  indicated that they plan to report on transfer rates and employment  retention rates, among other measures, for graduates of higher education  programs. This effort was a product of a 2012 Executive Order that called  for the Secretaries of Defense, Veterans Affairs, and Education to  develop such outcome measures to help veterans make informed  decisions about how to use their GI Bill benefits. VA officials told us that  even though they currently have no plans to develop or report on similar  post-program performance measures for OJT and apprenticeship  participants, they recognize the importance of such an effort.
State officials we surveyed and employers and apprenticeship sponsors  who have participated in the OJT and apprenticeship programs and  responded to our nongeneralizable survey generally said the programs  promote employment retention and effectively serve veterans. Of the 44  SAAs we surveyed, officials from 38 said they believed the programs  were generally either very or somewhat effective in improving veterans’  employment prospects (see table 8). Additionally, of the 153 participating  employers and apprenticeship sponsors who responded to our survey,  128 said that, in their experience, veteran participants typically continued  to work in the same area for which they received training while in the  programs. Nevertheless, without establishing and reporting data on  outcome measures for these programs, VA will not be able to verify  whether these perceptions are accurate and know how well these  programs are performing.
section_title: Data on DOL Programs Indicate Potential for Positive Employment Outcomes for OJT and Apprenticeship Models
paragraphs: DOL collects performance data for its OJT services and Registered  Apprenticeship program, and these data suggest that OJT and  apprenticeship training models can be associated with positive outcomes  for measures such as employment retention and earnings gains.  According to 2013 DOL program data, 85 percent of Post-9/11-era  veterans who received OJT services entered employment within 3  months of completing their training, and 86 percent of these veterans  retained employment over the subsequent 6 months. Additionally, Post- 9/11-era veterans who used OJT services earned 25 percent more in the  second and third quarters after finishing their OJT services than they had  earned prior to receiving services.
While these data indicate the potential for positive employment outcomes  for an OJT approach to training, they do not necessarily reflect outcomes  for veterans in VA’s programs. As previously mentioned, DOL’s OJT  services and the Post-9/11 GI Bill OJT program differ in several key ways.  First, while all veterans who served for at least 90 days after Sept. 11,  2001, are entitled to Post-9/11 GI Bill benefits, American Job Center staff  must determine that DOL OJT participants—civilians and veterans alike— need additional services to obtain employment. Second, whereas  veterans receive Post-9/11 GI Bill benefits in the form of a monthly benefit  payment, DOL OJT benefits are paid to employers in the form of a wage  subsidy. The subsidy covers the costs of training services needed to help  participants gain or retain employment. Third, the Post-9/11 GI Bill OJT  programs are limited in duration to 6 to 24 months, while the duration of  subsidized DOL OJT services varies according to the needs of the  position and the availability of local workforce funds.
Outcome data and available research on DOL’s Registered  Apprenticeship program also point to the potential for employment  benefits of an apprenticeship training model. DOL reports outcome  measures for all Registered Apprenticeship participants, without  distinguishing participants by veteran status. Nevertheless, 2014 program  data show that 91 percent of participants who completed the DOL  program found employment within 3 months compared to 76 percent of  those who exited the program without completing it. Of those individuals  who completed the program and found employment within 3 months, 91  percent retained their employment over the subsequent 6 months. DOL  data also indicate that the average wages of apprentices who completed  the program increased about 65 percent from $13.78 per hour to $22.69  per hour during the course of the program. Moreover, a study  commissioned by DOL estimated that those individuals who completed a  Registered Apprenticeship earned substantially more over a career than  similar individuals who did not complete a Registered Apprenticeship. It  is important to note that some of these earnings gains are likely due to  unobserved differences between those individuals who completed a  Registered Apprenticeship and those individuals who did not.
As with OJT services, outcomes associated with DOL’s Registered  Apprenticeship do not necessarily reflect the outcomes for veterans in  Post-9/11 GI Bill apprenticeship programs. First, DOL does not  differentiate veterans from nonveterans in its Registered Apprenticeship  outcomes; the reported data refer to the entire population of those who  either complete or exit a Registered Apprenticeship. It is therefore not  possible to use these data to determine how effectively Registered  Apprenticeships are serving Post-9/11-era veterans. Second, these data  reflect outcomes in the 25 states in which DOL’s Office of Apprenticeship  administers the Registered Apprenticeship program. Therefore,  Registered Apprenticeship outcome data from DOL do not represent  nationwide participant outcomes. Absent measures for VA’s OJT and  apprenticeship programs, the performance of these programs will remain  unknown.
section_title: Conclusions
paragraphs: While many veterans benefit from higher education after their military  service ends, for some veterans it may not be the ideal career path to  successfully attain civilian jobs. Given the wide range of education and  training options available under the Post-9/11 GI Bill, it is important that  veterans be fully aware of their options so they can make an informed  decision about how to use their benefits to effectively transition to civilian  life. Most Post-9/11 GI Bill outreach materials emphasize higher  education programs, and most state officials we surveyed told us that  awareness of OJT and apprenticeship opportunities is low. While VA  provides some information about OJT and apprenticeship programs,  many veterans and employers learn about the program through word-of- mouth rather than through systematic outreach efforts. Without improved  program outreach, veterans who could benefit from these programs may  remain unaware of them and could miss opportunities to improve their  long-term employment prospects, consistent with agency goals.
Overall, veterans, employers, and state officials we contacted were  generally positive about VA’s OJT and apprenticeship programs.  However, many suggested that VA’s current, paper-based system for  paying monthly benefits is inefficient, leading to administrative burdens for  employers and delayed benefit payments for veterans—adding economic  stress for veterans who are counting on this assistance. VA has taken a  first step towards easing these administrative challenges by deciding to  incorporate the OJT and apprenticeship programs into its development of  the new VA-CERTS data system to process program approvals and  benefit payments. However, this system may not be implemented until at  least 2017, and increasing numbers of veterans will become eligible for  Post-9/11 OJT and apprenticeship benefits as the military continues to  draw down its forces. In the interim, and in the absence of efforts to ease  administrative challenges for employers who experience difficulty in  submitting monthly certifications and for veterans who experience  delayed payments, some veterans will likely continue to experience  delayed payments. Such delays can undermine one of the primary  benefits of the program cited by participants: helping veterans cover living  expenses while transitioning to the civilian workforce.
With regard to performance, outcome data collected by DOL on the OJT  and apprenticeship programs it administers indicate the potential for  positive employment outcomes for these training models. However, VA is  limited in its ability to assess the impact of its Post-9/11 GI Bill OJT and  apprenticeship programs on veterans because VA does not measure  performance outcomes for participants. While VA has undertaken efforts  to better measure and report outcomes in its higher education programs,  the agency currently has no plans to do so for its OJT and apprenticeship  programs. Without measuring and reporting on outcomes for these  programs, VA will not be able to assess their effectiveness, and veterans  may not be well-positioned to determine which programs are most  suitable for them when choosing how to use their benefits.
section_title: Recommendations for Executive Action
paragraphs: We are making the following three recommendations:  1.  To help ensure that veterans are aware of the Post-9/11 OJT and  apprenticeship programs so they can make informed decisions about  how they use their benefits, the Secretary of VA should identify and  implement appropriate, cost-effective actions to increase awareness  of OJT and apprenticeship benefits under the Post-9/11 GI Bill.  2.  To help address challenges veterans and employers reported facing  in using the OJT and apprenticeship programs, the Secretary of VA  should identify and implement cost-effective steps to ease  administrative challenges in submitting paperwork or receiving  payments as the new automated VA-CERTS system to process  program approvals and benefit payments is being developed and  implemented.  3.  To increase accountability for program performance, the Secretary of  VA should establish measures to report on program outcomes for  Post-9/11 GI Bill OJT and apprenticeship programs, including  considering relevant data sources and seeking legislative authority to  gain access to data, if necessary.
section_title: Agency Comments
paragraphs: We provided a draft of this report to VA, DOL, and DOD for comment. VA  provided written comments that are reproduced in appendix II. Neither  DOD nor DOL provided a formal response. In addition, VA and DOL  provided technical comments that we incorporated as appropriate.
VA agreed with each of our three recommendations. In response to our  recommendation to increase awareness of OJT and apprenticeship  benefits under the Post-9/11 GI Bill, VA said it will develop a guide for  employers and apprenticeship sponsors about Post-9/11 GI Bill OJT and  apprenticeship benefits. VA also said it will publicize this guide and make  it available on VA’s GI Bill website. Further, VA said it will send veterans  and stakeholders information on OJT and apprenticeship benefits via an  email blast and by posting information to the Employment Center section  of VA’s eBenefits website. In response to our recommendation to ease  administrative challenges in submitting paperwork or receiving payments  while the new automated VA-CERTS system is being developed and  implemented, VA said it will explore the feasibility of cost-effective options  and will develop a plan to ease challenges. Finally, in response to our  recommendation to establish measures to report on program outcomes,  VA agreed that program performance metrics should be developed to  report on program outcomes. VA said it will develop a plan to determine  the feasibility of collecting and publishing program outcome data for the  OJT and apprenticeship programs. VA set a target completion date of  June 2016 for all actions set forth in its response to our  recommendations.
We are sending copies of this report to the appropriate congressional  committees and the Secretaries of Veterans Affairs, Labor, and Defense.  In addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff has any questions about this report please contact me  at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in Appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In conducting our review of the Department of Veterans Affairs (VA) Post- 9/11 GI Bill on-the-job training (OJT) and apprenticeship programs, our  objectives were to examine (1) how selected veterans and employers  used the programs and how widely they have been used, (2) to what  extent VA and states have taken steps to inform veterans and employers  about these programs, (3) what challenges, if any, veterans and  employers have faced in using them, and (4) to what extent VA has  assessed the performance and effectiveness of its programs. This  appendix provides a detailed account of the data sources used to answer  these questions, the analyses we conducted, and any limitations we  encountered.
We conducted this performance audit from August 2014 to November  2015 in accordance with generally accepted government auditing  standards. These standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Analysis of VA Administrative Data
paragraphs: To describe how widely the OJT and apprenticeship programs have been  used, we requested and analyzed program data from VA. VA provided  information from two different databases—the Long Term Solution (LTS)  database and the Web Enabled Approval Management System  (WEAMS). We assessed the reliability of these data by reviewing existing  information about the data and the systems that produced them and by  interviewing agency officials knowledgeable about the data. We  determined that the data were sufficiently reliable for the purposes of this  report. The summary statistics that VA provided from its LTS data  included the number of veterans who have participated in the OJT and  apprenticeship programs since these benefits first became available on  Oct. 1, 2011, and the number of veterans participating in other Post-9/11  GI Bill programs through March 23, 2015—the most recent data available  at the time VA analyzed them. In addition, VA provided summary  information from the WEAMS dataset as of March 23, 2015 on the  number of facilities (employers or apprenticeship sponsors) that have  been approved to provide OJT or apprenticeship training opportunities  under the Post-9/11 GI Bill. VA also provided selected participant-level  information from the LTS dataset that we analyzed to describe key  demographic characteristics of the participant population, including  average age and the percentage of participants by program type, gender,  and branch of service. We also used this data source to identify  occupations for which veterans frequently train while participating in the  Post-9/11 OJT and apprenticeship programs.
section_title: Analysis of DOL Administrative Data
paragraphs: To provide information on what is known about the effectiveness of OJT  services and Registered Apprenticeship programs in serving Post-9/11- era veterans, we reviewed available data sources from DOL, as VA does  not have performance data on GI Bill OJT and apprenticeship programs.  First, we examined a publicly-available performance data guide for the  Workforce Investment Act Standardized Record Data (WIASRD) from  program year 2012 to determine what data elements we would request  for Post-9/11-era veteran participants in DOL’s OJT services. We then  requested and analyzed DOL aggregate outcome data from the fourth  quarter of program year 2013, which included program exiters from prior  quarters of that year, on: 1) the rate at which participants enter  employment during the first quarter after their OJT services; 2) the rate at  which participants retain employment during the second and third  quarters after their OJT services; 3) average quarterly earnings for  participants during the quarter(s) before their OJT services; and 4)  average quarterly earnings for participants across the second and third  quarters after their OJT services. Per our request, these data were  presented by veteran status and included outcomes for Post-9/11-era  veterans specifically. In order to verify the reliability of the data, we  interviewed data specialists at DOL and requested and evaluated  supporting documentation regarding the data source. We determined that  the data were sufficiently reliable for the purposes of this report.
We also reviewed available DOL data on the Registered Apprenticeship  program in order to provide some information on the effectiveness of  apprenticeships in serving Post-9/11-era veterans. Specifically, we  examined the Registered Apprenticeship Partners Information  Management Data System (RAPIDS), which contains Registered  Apprenticeship participant information from states whose programs are  administered by DOL’s Office of Apprenticeship within the Employment  and Training Administration (ETA), as well as individual states that have  agreed to report data nationally.
Through a data sharing agreement with participating states, DOL’s Office  of Apprenticeship assesses outcome measures quarterly, according to  DOL officials, using states’ Unemployment Insurance wage records. We  analyzed the same aggregate performance data from RAPIDS as we  requested data for OJT services from 2014, the most recent year from  which DOL could provide data. However, DOL officials noted that  Registered Apprenticeship outcomes data are not categorized by veteran  status and do not include pre-program earnings. We also interviewed  officials from the Office of Apprenticeship to verify data reliability and  requested and evaluated supporting documentation regarding the data  source. We determined that the data were sufficiently reliable for the  purposes of this report.
section_title: Participating Veteran Survey
paragraphs: To learn about how selected veterans have used the Post-9/11 GI Bill  OJT or apprenticeship programs and what challenges they may have  faced, we conducted a web-based survey of veterans who have  participated in these programs. We limited our in-scope population to  those veterans for which VA was able to provide email addresses.  Specifically, while VA provided detailed participant level information on  26,375 unique participating veterans, we drew our sample from the  12,011 veterans for which email addresses were provided. As a result,  the results of this survey are not generalizable to the population of  participating veterans. To generate the original sample, participants were  selected in a probability sample stratified by gender. The sample size was  calculated to allow us to produce generalizable estimates having 95  percent confidence intervals no larger than plus or minus 10 percent for  the overall sample size of 200, assuming a response rate of 50 percent.  Of this original sample of 200 participants, 20 participants were women  and 180 were male or no gender identified. Due to lower than expected  response rates, we decided to select an additional sample so that we  could receive more responses, even though the results of the survey  would not be generalizable. Specifically, the survey was sent to an  additional sample of 5,000 participating veterans, 500 of whom were  women and 4,500 of whom were male or no gender identified.  Consequently, the survey’s final sample size was 5,200 participating  veterans. Prior to fielding the survey, we conducted 4 pretests to ensure  that the questions were clear, comprehensive, did not introduce bias, and  were not burdensome to complete. We made minor revisions to the  survey instrument based upon these pretests. We launched the web- based survey on June 11, 2015, and sent two follow-up emails to  sampled veterans on June 17 and June 24, 2015. We made the web- based sample available to the additional 5,000 sampled veterans on July  9, 2015 and sent a follow-up email to all nonrespondents on July 15,  2015. We closed the survey on July 31, 2015. Overall, GAO received  responses from 156 participating veterans—a 3 percent response rate.
section_title: Non-participating Veteran Survey
paragraphs: To learn to what extent selected veterans who have used Post-9/11 GI  Bill benefits to pursue a non-college degree program, such as vocational  school, but who have not participated in the apprenticeship or OJT  programs, were aware of these programs, we conducted a web-based  survey of such veterans. We limited our in-scope population to those  veterans for which VA was able to provide email addresses. Specifically,  while VA provided detailed participant level information on 145,525  unique veterans who had used their Post-9/11 GI Bill benefits for a non- college degree program, we drew our sample from the 70,695 veterans  for whom email addresses were provided. To generate the original  sample, 150 veterans who did not participate in OJT or apprenticeship  were drawn from the population of veterans who had used their Post-9/11  GI Bill benefits for a non-college degree program and for which email  addresses were available. We assumed a 20 percent response rate for  this population. Due to lower than expected response rates, we decided  to select an additional sample so that we could receive more responses,  even though the results of the survey would not be generalizable.  Specifically, the survey was sent to an additional sample of 5,000 non- participating veterans. Consequently, the survey’s final sample size was  5,150 veterans. Prior to fielding the survey, we conducted 2 pretests to  ensure that the questions were clear, comprehensive, did not introduce  bias, and were not burdensome to complete. We made minor revisions to  the survey instrument based upon these pretests. We launched the web- based survey on June 10, 2015 and sent follow-up emails to sampled  veterans on June 17 and June 24, 2015. We made the web-based  sample available to the additional 5,000 sampled veterans on July 8,  2015 and sent a follow-up email to all non-respondents on July 16, 2015.  We closed the survey on Aug. 3, 2015. Overall, we received responses  from 127 non-participating veterans—a 2.5 percent response rate. Given  the low response rate, the findings of this survey are not generalizable.
section_title: Employer and Apprenticeship Sponsor Survey
paragraphs: To learn about how selected employers and apprenticeship sponsors  have used the Post-9/11 GI Bill OJT or apprenticeship programs and  what challenges they may have faced, we conducted a web-based survey  of employers and apprenticeship sponsors who have participated in these  programs. We limited our in scope population to those employers and  apprenticeship sponsors for which VA was able to provide email  addresses. Specifically, while VA provided detailed information on 5,099  unique participating employers and apprenticeship sponsors, we drew our  sample from the 2,691 employers and apprenticeship sponsors for which  email addresses were provided. To generate the original sample, 600  employers and apprenticeship sponsors were selected in a simple  random sample. The sample size was calculated assuming a response  rate of 5 percent, and included 7 employers and apprenticeship sponsors  who had completed questionnaires during our site visit to North Carolina.  Prior to fielding the survey, we conducted 3 pretests to ensure that the  questions were clear, comprehensive, did not introduce bias, and were  not burdensome to complete. We made minor revisions to the survey  instrument based upon these pretests. We launched the web-based  survey on June 11, 2015 and sent follow-up emails to sampled employers  and apprenticeship sponsors on June 17 and July 9, 2015. GAO closed  the survey on July 24, 2015. Overall, GAO received responses from 153  participating employers and apprenticeship sponsors—a 25 percent  response rate. The findings of this survey are not generalizable.
section_title: State Approving Agency (SAA) Survey
paragraphs: To learn about what states are doing to help veterans and employers use  the Post-9/11 OJT and apprenticeship programs, as well as gain insight  on veterans’ and employers’ use of these programs and challenges states  faced, we emailed a MS Word questionnaire to SAA officials in the 44  states in which these officials oversee implementation of these  programs. After developing our questionnaire, we conducted two  pretests with state officials to ensure that the questions were clear,  comprehensive, did not introduce bias, and were not burdensome to  complete. We made minor revisions to the survey instrument based upon  these pretests.
The survey included questions about methods of outreach to veterans  and employers, assistance to employers, challenges to veteran and  employer outreach and participation, and challenges with SAA  implementation of the program. It also included questions about the value  or interest in key improvements, such as automating the payment system.
We sent the survey by email in March and April 2015 using email  addresses from the National Association of State Approving Agencies  website. We followed-up with unresponsive states via email until we  reached 100 percent participation in July 2015. Survey responses were  collected using a Microsoft Word form and imported into Microsoft Excel  for summary and analysis. In some cases a state official sent their  response in another type of file, and we transposed responses into the  Microsoft Word form and verified that the responses matched the state’s  supplied answers. While we had survey responses for all 44 states, in  several instances a state did not supply an answer for a single question.  For summary and analysis performed on those questions we excluded  those states from the denominator. All analyses of responses were  reviewed and verified to be accurate.
section_title: Review of Relevant Laws, Regulations, and Other Documents
paragraphs: To learn about VA’s implementation of the OJT and apprenticeship  programs under the Post-9/11 GI Bill, we reviewed relevant provisions of  the Post-9/11 GI Bill and related regulations. We also reviewed the Plain  Writing Act of 2010, a law related to the public dissemination of  information on benefits. In addition, we reviewed VA’s Strategic Plan for  2014 to 2020 to understand greater agency objectives, and other federal  best practice documents such as the Job Driven Checklist in the Vice  President’s Ready to Work: Job-Driven Training and American  Opportunity report, and Standards for Internal Control in the Federal  Government.
section_title: Analysis of VA and DOL Outreach Materials
paragraphs: To understand VA and DOL outreach to veterans and employers, we  analyzed key sources of information that are readily available regarding  Post-9/11 OJT and apprenticeship programs. Specifically, we analyzed  curriculum documents for VA’s mandatory Transition Assistance Program  (TAP), as well as information publically available to veterans and  employers on VA and DOL websites. To understand information  presented in VA’s TAP program, we reviewed the facilitator’s and  participant’s guides for the mandatory TAP courses called Veterans  Benefits I and II, as well as a relevant optional training course called the  Career Technical Training Track, and compared it to VA’s goals for TAP  after the VOW to Hire Heroes Act of 2011 was implemented. We tallied  the number of pages in the facilitator’s guides that mentioned OJT and  apprenticeship as options under the Post-9/11 GI Bill. Similarly, we  reviewed the online version of the Veterans Benefits I and II training to  see if additional information about OJT and apprenticeship options was  presented. We also reviewed relevant pages of VA and DOL websites.  We reviewed and documented Post-9/11 GI Bill and related web pages  using screen shots of each relevant page, and noted key portions of  content. We assessed the information available from these sources to  determine if a veteran or employer could reasonably understand these  programs, as described in the Plain Writing Act of 2010 and Standards for  Internal Control in the Federal Government. In both the TAP and website  analyses, we reviewed included examples and exercises, visual symbols  such as the Post-9/11 GI Bill logo, and the language used to describe  Post-9/11 GI Bill benefits to determine how, if at all, OJT and  apprenticeship options were included. For the website analyses we also  looked to see how, if at all, the agencies created links to each other’s  content and resources. The calculations and content analysis of the TAP  and the website analyses were independently reviewed and verified by  another GAO team member.
section_title: Interviews with Officials and Stakeholders
paragraphs: To gain perspectives on how Post-9/11 GI Bill OJT and apprenticeship  programs are serving veterans, we conducted interviews with a variety of  federal and state officials. We spoke with officials at the VA and DOL  concerning program administration, outreach efforts, performance  measurement, and coordination of their respective OJT and  apprenticeship programs. We also interviewed officials at DOD to better  understand transition services that are available for service members  upon exiting the military. For perspectives on Post-9/11 GI Bill program  usage and outreach, we also interviewed State Approving Agency (SAA)  officials from 13 states. We chose these 13 states in order to get a range  in the number of active OJT and apprenticeship programs and geographic  location. We also chose these states based on the institutional location of  the SAAs, so as to gain greater perspective on program use and what, if  any, challenges SAAs observe in administering the Post-9/11 GI Bill.
We also interviewed stakeholders and experts to learn how OJT and  apprenticeship programs are used and, from these stakeholders’  perspective, how effectively they serve veterans. Specifically, we  interviewed representatives from a variety of veteran service  organizations (VSOs) that serve Post-9/11 veterans and were  recommended by internal and external experts on veterans issues.  Additionally, we spoke with workforce experts with job training expertise  to understand what, if any, academic literature could shed light on the  question of OJT and apprenticeship program effectiveness, though their  views do not represent the entire research field. Subsequently, we  reviewed a 2012 Mathematica Policy Research study, commissioned by  DOL, which was cited by experts we interviewed. This study conducted a  cost-benefit analysis that included an assessment of the net effects of the  Registered Apprenticeship program on participant outcomes. As noted,  we found the study to be generally reliable with some methodological  limitations.
section_title: Site Visits
paragraphs: Among our 13 case illustration states, we selected two—Missouri and  North Carolina—for site visits because they have high numbers of Post- 9/11 GI Bill-approved OJT and apprenticeship programs, and different  institutional settings for their SAAs. During these site visits, we  interviewed SAA officials, OJT and apprenticeship employers and  sponsors, veteran participants, and local workforce officials so that we  could get a wide range of stakeholder perspectives of Post-9/11 GI Bill  OJT and apprenticeship programs, though these perspectives do not  necessarily reflect those of states more generally. We also observed  training programs on site in such professions as communications  technology maintenance and police work. For additional context, we also  conducted interviews at two local site visits in the Washington, D.C. area  that involved federal programs that were registered through the Post-9/11  GI Bill—the Department of State’s Bureau of Diplomatic Security OJT  program and the Department of Defense’s Defense Logistics Agency  Energy OJT program.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making key  contributions to this report were Laura Heald, Assistant Director, Amy  Buck, Devin Braun, Matthew Lowney, Rachel Pittenger, and Michelle  Loutoo Wilson. In addition, key support was provided by James Bennett,  Rachel Beers, Mindy Bowman, David Chrisinger, Michael Hoffman, Kathy  Leslie, Sheila McCoy, Jean McSween, Sara Pelton, Almeta Spencer,  Sonya Vartivarian, and Craig Winslow.