VA VOCATIONAL 

REHABILITATION 

AND EMPLOYMENT 

Further Performance 

and Workload 

Management 

Improvements Are 

Needed 

Report to Congressional Committees

January 2014 

GAO-14-61 

United States Government Accountability Office 
United States Government Accountability Office

Highlights of GAO-14-61, a report to 

congressional committees

January 2014

VA VOCATIONAL REHABILITATION AND 

EMPLOYMENT

Further Performance and Workload Management 

Improvements Are Needed

Why GAO Did This Study

Veterans with disabilities face special

challenges to finding employment—

resulting, for example, from the 

veterans’ physical or mental health, or 

negative attitudes or stereotypes on 

the part of some employers. VA’s 

VR&E program aims to help veterans 

with disabilities obtain and maintain 

suitable employment—compatible with 

their disabilities—through services 

such as training and job search 

assistance. The VOW to Hire Heroes 

Act of 2011 directed GAO to review the 

program. GAO examined (1) the 

outcomes for veterans seeking 

employment through the program, and 

(2) the progress VA has made in 

addressing critical management 

issues. GAO reviewed relevant laws, 

regulations, and guidance as well as

recent studies; reviewed various 

program management criteria; 

analyzed VA administrative data on 

veterans who entered the program 

between fiscal years 2003 and 2012; 

interviewed staff at the VA central 

office and 8 regional offices;

interviewed a random but nongeneralizable sample of 17 program 

participants; and analyzed data from a 

VA survey of participants.

What GAO Recommends

GAO recommends that VA reflect 

success rates in revised performance 

measures, ensure the reliability of its 

customer satisfaction survey results, 

re-visit its staff allocation formula, 

study staff assignments, and close 

certain gaps in its training for staff. In 

its comments, VA generally concurred 

with these recommendations and 

noted steps it plans to take to address 

them. 

What GAO Found

About half of the almost 17,000 veterans who entered the Department of 

Veterans Affairs’ (VA) Vocational Rehabilitation and Employment (VR&E) 

program in fiscal year 2003 and received employment-related services were 

placed in suitable jobs, one-third left the program, and most of the others are still 

participating. It often took veterans 6 years or more to achieve success (see 

figure), due in part to veterans often leaving the program temporarily. Interviews 

with VR&E staff and participants and administrative data GAO reviewed suggest 

veterans face numerous challenges that affect their ability to obtain employment,

especially related to mental health conditions, working with multiple VR&E 

counselors over time, and civilian employers’ limited understanding of military 

work experience.

Time to Achieve Suitable Employment for Fiscal Year 2003 Applicants

VA has taken steps to improve VR&E performance management, workload 

management, and staff training, but weaknesses remain. With regard to 

performance management, VA has an ongoing initiative to revise its approach for 

measuring rehabilitation success at the individual employee, regional, and 

national levels. However, the new approach VA is considering for employees

reflects only the number, not the rate of successful outcomes, and therefore 

would not provide sufficient context for understanding program success. VA has 

not yet developed its new approaches for assessing rehabilitation success 

regionally and nationally. Also, VA began surveying participants’ satisfaction with 

the program and plans to use the results to manage performance; while VA has 

generally followed good survey design practices, the agency has not fully 

assessed the reliability of early customer satisfaction results. In terms of 

workload management, VA has taken steps to reduce paperwork burdens on 

regional offices. However, several offices still reported heavy workloads and 

noted that VA’s formula for allocating staff among offices does not consider other 

staff duties affecting workloads, such as education counseling. In addition, VA 

has not studied the relative effectiveness and efficiency of regional offices’ 

approaches for assigning staff to manage workloads. Finally, with respect to 

training, VA has addressed redundancy and most gaps in training for VR&E staff, 

but gaps remain in the areas of job placement assistance and workplace 

accommodations.

View GAO-14-61. For more information, 

contact Daniel Bertoni (202) 512-7215 or 

bertonid@gao.gov 
Page i GAO-14-61 VA Vocational Rehabilitation and Employment

Letter 1

Background 3

About Half of Veterans Seeking Employment Are Successful, 

Although Program Participants Face Many Challenges 11

VA Has Made Limited Progress in Improving VR&E Performance 

Management, Workload Management, and Training 21

Conclusions 34

Recommendations for Executive Action 35

Agency Comments and Our Evaluation 36

Appendix I Objectives, Scope, and Methodology 38

Appendix II Statistical Analysis of Administrative Data on Vocational 

Rehabilitation and Employment Outcomes 46

Appendix III Vocational Rehabilitation Counselor Training Gaps Identified by 

VA in 2010 and the Agency’s Efforts to Address 55

Appendix IV Comments from the Department of Veterans Affairs 56

Appendix V GAO Contact and Staff Acknowledgments 60

Related GAO Products 61

Tables

Table 1: VA’s Vocational Rehabilitation and Employment 

Performance Measures, with Fiscal Year 2013 Goals 9

Table 2: Description of Vocational Rehabilitation and 

Employment’s Voice of the Veteran Pilot Surveys 43

Table 3: Sample Outcomes for Vocational Rehabilitation and 

Employment’s Voice of the Veteran Pilot Surveys 44

Contents
Page ii GAO-14-61 VA Vocational Rehabilitation and Employment

Table 4: Sample Size by Fiscal Year of Program Entry and 

Outcomes 47

Table 5: Estimated Hazard Ratios of Time to Vocational 

Rehabilitation and Employment Program Completion 53

Figures

Figure 1: Total Vocational Rehabilitation and Employment 

Caseload and Counselors, and Average Caseload, Fiscal 

Years 2008 through 2012, and as of March 2013 6

Figure 2: The Vocational Rehabilitation and Employment Process 

from Application to Outcome 8

Figure 3: Outcomes to Date for Fiscal Year 2003 Applicants Who 

Received Employment-Related Services 12

Figure 4: Time to Achieve Suitable Employment for Fiscal Year 

2003 Applicants 13

Figure 5: Likelihood of Achieving Suitable Employment for 

Veterans with Selected Disabilities Compared to Those 

with Other Disabilities 16

Figure 6: Likelihood of Achieving Suitable Employment for 

Veterans Working with Multiple Counselors 18

Figure 7: Regional Office Caseloads for Vocational Rehabilitation 

Counselors—Lowest, Highest and Overall Average—

Fiscal Years 2008 through 2012 and March 2013 28

Figure 8: Regional Offices Contacted and Their Characteristics 41
Page iii GAO-14-61 VA Vocational Rehabilitation and Employment

Abbreviations

CWINRS Corporate WINRS 

EC Employment coordinator

FTE Full-time equivalent

Labor Department of Labor

MRG maximum rehabilitation gain

OMB Office of Management and Budget

PTSD Post-traumatic stress disorder

TBI Traumatic brain injury

VA Department of Veterans Affairs

VBA Veterans Benefits Administration

VRC Vocational rehabilitation counselor 

VR&E Vocational Rehabilitation and Employment

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-14-61 VA Vocational Rehabilitation and Employment

441 G St. N.W.

Washington, DC 20548

January 14, 2014

The Honorable Bernard Sanders 

Chairman

The Honorable Richard Burr 

Ranking Member

Committee on Veterans’ Affairs

United States Senate

The Honorable Jeff Miller 

Chairman

The Honorable Michael Michaud 

Ranking Member

Committee on Veterans’ Affairs

House of Representatives

About 3 million veterans—including more than 700,000 from the Iraq and 

Afghanistan era—have a disability connected to their military service,1

To improve the transition of veterans to civilian employment, Congress 

passed the VOW to Hire Heroes Act of 2011 (VOW Act).

and many face special challenges obtaining employment. Veterans with 

disabilities may face employment challenges related to physical 

impairments, less visible mental health conditions, or negative attitudes 

and stereotypes among some employers. The Department of Veterans 

Affairs (VA) operates the Vocational Rehabilitation and Employment 

(VR&E) program to help veterans with service-connected disabilities 

obtain and maintain suitable employment, which it considers to be 

employment consistent with their abilities, aptitudes, and interests. The 

program provides services including vocational assessment, education 

and training, and job placement. In fiscal year 2012, about 121,000 

veterans participated in the program at a cost of almost $1 billion.

2

 1 These data are based on a survey of veterans conducted in August 2012 by the U.S. 

Census Bureau.

In addition to 

provisions intended to promote veterans’ employment, including 

expanded access to VR&E benefits for certain unemployed veterans, the 

2 Pub. L. No. 112-56, tit II, 125 Stat. 711, 712. 
Page 2 GAO-14-61 VA Vocational Rehabilitation and Employment

VOW Act mandated that GAO review the VR&E program.3

We conducted this performance audit from October 2012 to January 2014 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives. Additional information on our 

scope and methodology is provided in appendix I.

We examined 

(1) the outcomes for veterans seeking employment through the VR&E 

program, and (2) the progress VA has made in addressing critical 

management issues. To address these objectives, we reviewed relevant 

federal laws, regulations, and guidance as well as documentation related 

to recent management initiatives. We analyzed VA administrative data on 

VR&E participants who applied to the program between fiscal years 2003 

and 2012, including their characteristics, services received, and outcomes 

obtained. We assessed the reliability of these data through electronic 

testing and interviews with VA staff, and found them sufficiently reliable 

for our reporting purposes. We interviewed VA central office staff 

responsible for the VR&E program, as well as VR&E managers and staff 

in 8 of VA’s 57 regional offices. We selected offices to achieve diversity in 

VR&E caseload, performance in job placement and case processing 

timeliness, state unemployment rate, and geographic location. We 

conducted phone interviews with a random but non-generalizable sample 

of 17 veterans who had participated in the VR&E program and exited with 

either a successful or unsuccessful outcome. Finally, we obtained 

selected data resulting from a pilot VA customer satisfaction survey that 

includes responses from VR&E participants, after reviewing the survey’s 

methodology and finding it to be sufficiently reliable for our purposes

 3 See id. at § 233(d), 125 Stat. 711, 720.
Page 3 GAO-14-61 VA Vocational Rehabilitation and Employment

To be entitled to VR&E services and related benefits, veterans generally 

must (1) have at least a 20 percent disability rating from the VA4 and (2) 

be in need of rehabilitation because of an employment handicap caused 

in substantial part by a service-connected disability.5,6 If needed to 

achieve employment, entitled veterans may receive a maximum of 48 

months of vocational rehabilitation services and an additional 18 months 

of employment services, which include counseling, placement, and 

postplacement services.7

 4 VA rates veterans’ disabilities that are incurred or aggravated as a result of their military 

service, which are referred to by VA as service-connected disabilities, from 0 percent to 

100 percent in increments of 10 percentage points. A veteran must also have a military 

discharge that is not dishonorable to be eligible for the VR&E program. 

Veterans are typically eligible to receive these 

services within a 12-year period beginning on either: (1) the date of 

separation from military service, or (2) the date the veteran received a 

5 38 U.S.C. § 3102(a)(1). The law defines an employment handicap as an impairment of a 

veteran’s ability to prepare for, obtain, or retain employment consistent with the veteran’s 

abilities, aptitudes, and interests. 38 U.S.C. § 3101(1). Veterans with a 10 percent serviceconnected disability may also be eligible to receive VR&E services if they have a serious 

employment handicap. 38 U.S.C. § 3102(a)(2). A serious employment handicap is defined 

as a significant impairment of a veteran’s ability to prepare for, obtain, or retain 

employment consistent with his or her abilities, aptitudes and interests. 38 U.S.C. § 

3101(7). The definitions of employment handicap and serious employment handicap 

provide that the handicap must result in substantial part from a service-connected 

disability. 

6 Active duty servicemembers may also be eligible for VR&E services under certain 

circumstances. For example, servicemembers may qualify if, while awaiting discharge 

from the military, they are hospitalized or receiving outpatient treatment at a VA facility or 

a facility that is under contract with the VA, have a disability that VA will likely find to be 

service-connected with a 20 percent or higher rating, and are determined to be in need of 

rehabilitation because of an employment handicap. 38 U.S.C. § 3102(a)(1)(A)(ii) and (B). 

Until December 31, 2014, servicemembers are entitled to certain VR&E services if they 

are seriously ill or injured. Pub. L. No. 110-181, §1631(b)(2), 122 Stat. 3, 458 (2008) (10 

U.S.C 1071 note) as amended by Pub. L. No. 112-56, § 231, 125 Stat. 711, 719 (2011). 

7 38 U.S.C. § 3105(b)(1). Eligible veterans for whom a vocational goal is not currently 

considered reasonably feasible may receive assistance intended to help them achieve 

independent living. 38 U.S.C. § 3109.

Background

VR&E Entitlement and 

Eligibility
Page 4 GAO-14-61 VA Vocational Rehabilitation and Employment

disability rating from VA, whichever is later. In certain circumstances, the 

48-month limit and the 12-year period of eligibility may be extended.8

Under certain circumstances, veterans who have successfully completed 

a VR&E program and achieved rehabilitation may qualify for additional 

VR&E benefits and services. Prior to the VOW Act, veterans who 

successfully completed a VR&E program could receive additional 

services if (1) their service-connected condition worsened to the extent 

that they could no longer perform the duties of the occupation for which 

they were previously rehabilitated, or (2) their occupation was no longer 

suitable because of their current capabilities and employment handicap.9

The VOW Act created another pathway for previously rehabilitated 

veterans to receive additional services. Under section 233 of the VOW 

Act, veterans who have exhausted their right to regular unemployment 

compensation under state law may receive up to 12 additional months of 

VR&E services and benefits following a successful rehabilitation, as long 

as they are otherwise eligible and they begin the additional program

within 6 months after they exhausted unemployment compensation.10

This provision took effect on June 1, 2012, and veterans must apply no 

later than March 31, 2014 to qualify.

Within VA’s Veterans Benefits Administration’s (VBA) Office of Economic 

Opportunity in Washington D.C., the central VR&E Office is responsible 

for overseeing the VR&E program. VR&E services are provided by 

managers and staff at 57 regional offices and 179 satellite offices, under 

 8 The 48-month limit on training services can be extended if, for example, the veteran has 

a serious employment handicap and the extension is necessary to accomplish the 

veteran’s vocational or independent living goal. 38 U.S.C. § 3105(c)(2) and 38 C.F.R. § 

21.78(c). The 12-year limit on the eligibility period can also be extended for veterans who 

have a serious employment handicap and have not previously been rehabilitated to the 

point of employability. 38 U.S.C. § 3103(c)(1) and 38 C.F.R. § 21.44(a)(1). 

9 38 C.F.R. § 21.284. Veterans who meet these conditions may receive such additional 

services even if they have previously exhausted their 48-month entitlement to VR&E 

services. 38 U.S.C. § 3105(c)(1). Veterans who meet these conditions and have a serious 

employment handicap may receive such additional services even if they are no longer 

within their 12-year period of eligibility. 38 U.S.C. § 3103(c)(2). 

10 Codified at 38 U.S.C. § 3102(b). Such veterans may receive these additional services 

even if they have previously exhausted their 48 months of services or are no longer within 

their 12-year eligibility period. 38 U.S.C. §§ 3105(b)(2) and 3103(e). 

Availability of Additional 

Benefits pre- and postVOW Act

Administrative Structure
Page 5 GAO-14-61 VA Vocational Rehabilitation and Employment

the management of VBA’s central Office of Field Operations, which

manages workload and performance to achieve targets set in 

coordination with the central VR&E Office. VR&E positions typically 

include (1) a VR&E officer who oversees the VR&E field office and its 

staff; (2) vocational rehabilitation counselors (VRC) who work directly with 

veterans to assess their entitlement, develop their rehabilitation plans, 

and manage their progress; (3) employment coordinators (EC) who focus 

on assisting veterans with job search and placement, which also includes 

employer outreach to facilitate contact between veterans and employers; 

and (4) other staff to support the administration of the program. VR&E 

field staff are also located at select college campuses to help veterans 

successfully complete their training and become employed, and at military 

sites to help servicemembers with disabilities as they begin their transition 

to veteran status and the civilian workforce.

As of March 2013, 1,281 field staff were administering the VR&E 

program, of which about 70 percent (890) were VRCs. VR&E’s total 

caseload has generally increased every year from 98,015 at the end of 

fiscal year 2008 to 124,100 on March 31, 2013.11

 11 These caseload totals include veterans who are pursuing a rehabilitation plan to 

achieve either employment or independent living.

Over the same period, 

the number of VRCs has also generally been growing. As shown in figure 

1, the increase in staffing has helped to bring down the average caseload 

per VRC, which grew from 137 cases to 152 cases between fiscal years

2008 and 2009, then steadily decreased to 139 cases as of March 31, 

2013.
Page 6 GAO-14-61 VA Vocational Rehabilitation and Employment

Figure 1: Total Vocational Rehabilitation and Employment Caseload and 

Counselors, and Average Caseload, Fiscal Years 2008 through 2012, and as of 

March 2013

The provision of services under the VR&E program begins with the 

veteran’s application for services, a determination of entitlement and 

eligibility,12 and a visit to a regional or satellite office for further 

assessment.13

 12 For purposes of veterans’ benefits, including VR&E benefits and services, a veteran is 

defined as a person who was an active member of the U.S. military and was discharged or 

released from service under conditions other than dishonorable. 38 U.S.C. § 101(2). 

If a veteran is entitled and eligible to receive VR&E 

13 For veterans found not entitled to VR&E program services, VRCs will inform them of 

their appeal rights and use the information collected during the application process to 

direct the veteran to other services, such as those offered by state workforce systems. 

VR&E Process
Page 7 GAO-14-61 VA Vocational Rehabilitation and Employment

services and found to be employable,14 the case manager, VRC and 

veteran will work together to select one of the four VR&E employment 

tracks best suited for the veteran, to identify a suitable employment goal, 

and to incorporate that goal and the needed services and benefits to 

achieve it into a rehabilitation plan.15

Once a plan is finalized, the veteran begins receiving training or 

employment services in accordance with the chosen employment track. 

At any point in this process, the veteran in consultation with the VRC may 

agree on a redirection, an additional assessment, or transition to another 

employment track or job goal. A change in medical or personal situation 

could lead the veteran to interrupt or discontinue the program, as well. 

According to VA, VR&E uses the interrupted status as a tool to allow 

veterans time to resolve unexpected issues without using up time-limited 

employment services.16

 14 If a veteran is entitled to VR&E services, but his or her handicap is such that 

employment is deemed not currently feasible, and the veteran has a serious employment 

handicap resulting in substantial part from a service-connected disability rated at 20 

percent or more, the veteran is entitled to independent living services and assistance and 

will receive assistance to build and pursue a plan to achieve the maximum independence 

in daily living possible. 38 U.S.C. § 3109. Independent living services may include 

counseling, diagnostic, medical, social, psychological, and educational services. With 

these services and benefits, some veteran may initially achieve the outcome of maximum 

independence in daily living, and, if appropriate, may work with VR&E to develop an 

employment plan and pursue suitable employment. For information on the independent 

living track, see GAO, VA Vocational Rehabilitation and Employment Program: Improved 

Oversight of Independent Living Services and Supports Is Needed, 

At the end of the process, one of two program 

closure outcomes are possible for veterans with a plan for employment: 

(1) the veteran is rehabilitated (i.e., successfully achieves suitable 

employment for 60 days or 1 year of self-employment) or (2) is 

discontinued from the program. The discontinuance can be with either (1) 

a maximum rehabilitation gain (MRG) (e.g., the veteran is (a) employed in 

a job that is not suitable, or (b) has chosen to not pursue any job or a 

suitable job, but planned services resulted in meaningful skills or 

increased independence which aided the veteran) or (2) without an MRG 

(e.g., the veteran has not maximized employment potential and no longer 

GAO-13-474 

(Washington, D.C.: June 7, 2013).

15 VA considers suitable employment to be work consistent with the veteran’s abilities, 

aptitudes, and interests. 38 C.F.R. § 21.283(b).

16 VA’s regulations state that the purpose of assignment to interrupted status is to assure 

that all appropriate actions have been taken to help a veteran continue in his or her 

program before discontinuing benefits and services. 38 C.F.R. § 21.197(c)(4). 
Page 8 GAO-14-61 VA Vocational Rehabilitation and Employment

wants to participate in the program). Veterans who have discontinued the 

program can re-enter after VR&E has determined that they are still 

entitled to services.17

Figure 2: The Vocational Rehabilitation and Employment Process from Application to Outcome

(See fig. 2 for a summary of the VR&E process from 

application to case closure.)

 17Before the veteran’s reentry to the program, VR&E must also determine that the reason 

for the discontinuance has been removed. 38 C.F.R. 21.198(c). 
Page 9 GAO-14-61 VA Vocational Rehabilitation and Employment

a

If a veteran does not pursue a comprehensive evaluation or complete the development of a 

rehabilitation plan, VR&E can discontinue the veteran from the program without negatively affecting 

the program’s performance outcomes.

b

If the veteran decides not to pursue the plan, VR&E may discontinue the veteran from the program’s 

current rolls and will include the veteran’s discontinuance in its performance outcomes.

c

Possible reasons for discontinuing the plan of a veteran who has not achieved an MRG from the 

VR&E program include the veteran: (1) has decided to not pursue the plan; (2) has serious medical 

problems that interfere with pursuing rehabilitation and are expected to last for an extended, but 

uncertain, period; or (3) has decided to pursue training through another VA education program, such 

as the Post- 9/11 GI Bill.

VA measures the performance of its VR&E program across several 

dimensions, and uses performance goals and data to help manage the 

program. VA provides transparency on its performance through (1) its 

annual performance and accountability report; (2) its annual budget 

submission to the Congress; and (3) a database on VA’s website.18

Table 1: VA’s Vocational Rehabilitation and Employment Performance Measures, with Fiscal Year 2013 Goals

VA’s 

VR&E measures cover the dimensions of program outcomes, timeliness, 

and accuracy (see table 1).

Measure Description FY 2013 goal

Outcomes: Rehabilitation rates: Percentage of veterans who were rehabilitated, out of veterans rehabilitated and 

discontinued without rehabilitation.

General (VA key measure)

a

Includes veterans in all five VR&E tracks, including the Independent 

Living track.

77%

Employment Includes veterans in the four employment-related tracks; excludes 

the Independent Living track.

77%

Serious employment handicap Includes veterans found by VA to have serious employment 

handicaps, in all five VR&E tracks, including the Independent Living 

track.

77%

Independent Living Includes only veterans in the Independent Living track. n/ab

Timeliness: Average days required to complete specific parts of the VR&E process.

Entitlement determination Days from VR&E application to VA’s notification to the veteran of 

entitlement decision.

45 days

Evaluation and planning phase Days from VR&E application to (1) completion of the veteran’s 

rehabilitation plan or (2) determination that the veteran is not entitled 

to VR&E services.

105 days

 18 Included in the Veterans Benefits Administration’s on-line ASPIRE Dashboard, 

http://www.vba.va.gov/reports/aspiremap.asp.

VR&E Performance 

Management
Page 10 GAO-14-61 VA Vocational Rehabilitation and Employment

Measure Description FY 2013 goal

Accuracy: Percentage of actions found to be accurate by VR&E quality reviewers.

Entitlement determination Accuracy of decisions on veterans’ entitlement to VR&E services. 96%

Services Accuracy of VR&E evaluation, planning, and rehabilitative services 

provided.

85%

Program completion Accuracy of determinations that participants were rehabilitated by, or 

discontinued from, the VR&E program.

97%

Fiscal Accuracy of financial transactions, such as payments for VR&E 

services; for example, to educational institutions and vendors of 

goods and services for participating veterans. 

92%

Source: VA.

Note: Includes measures VA reported in its fiscal year 2012 Performance and Accountability Report; 

fiscal year 2014 Budget Submission; or its web-based ASPIRE Dashboard.

a

Does not include veterans who were not rehabilitated but obtained maximum rehabilitation gain from 

the VR&E program, or veterans who left the VR&E program to use Post-9/11 GI Bill benefits.

b

VA did not provide a fiscal year 2013 goal for this measure. Its fiscal year 2012 goal was 94 percent.

VA also holds the regional offices that administer the VR&E program 

accountable through performance goals, both for regional VR&E 

management and for individual VRC and EC staff members. To 

accomplish this, VA uses the above goals, as well as additional measures 

and goals. For example, individual VRCs are evaluated on case 

production and participants’ outcomes based on the general and serious 

employment handicap rehabilitation rates; the number of successful 

rehabilitations they complete; and the percentage of their caseloads in 

interrupted status.
Page 11 GAO-14-61 VA Vocational Rehabilitation and Employment

About half of the veterans who entered the VR&E program in fiscal year 

2003 and received employment-related services have obtained suitable 

jobs.19 Of the 45,520 veterans who applied in fiscal year 2003, 16,726 

received some type of employment-related services.20

 19 We focus on veterans who applied in fiscal year 2003 because that is the earliest year 

for which reliable data are available. It is possible that tracking veterans’ outcomes over a 

longer period of time would show a higher percentage of veterans placed in suitable jobs. 

Some veterans who are still participating in services 10 years after application may 

ultimately achieve success, and even some who have discontinued may re-apply and 

achieve success.

Among veterans in 

this group, 7,686 had achieved the program goal of suitable employment 

by the end of fiscal year 2012. Many others—4,841—had discontinued 

(stopped participating) and not yet re-applied. Although these veterans 

have not achieved suitable employment, a substantial number are 

considered by VA to have attained MRG and therefore some benefits 

from participating. In fact, according to VA data, some of these veterans 

were ultimately placed in a job, but the job was not seen as suitable given 

their disability. Most of the remaining veterans who started in fiscal year 

2003 and received employment-related services were still participating in 

20 In our analysis of VA data, we found that many fiscal year 2003 applicants did not 

receive employment-related services because, for example, they entered the independent 

living track instead or received an evaluation but never started a specific service track.

About Half of 

Veterans Seeking 

Employment Are 

Successful, Although 

Program Participants 

Face Many 

Challenges

About Half of Veterans 

Find Suitable Jobs, but 

Time Frames Can Be 

Lengthy
Page 12 GAO-14-61 VA Vocational Rehabilitation and Employment

VR&E services, while a few had transferred to the independent living 

program within VR&E and achieved rehabilitation.21

Figure 3: Outcomes to Date for Fiscal Year 2003 Applicants Who Received 

Employment-Related Services

(See fig. 3.)

Note: The percentages in the four outcome categories do not add up to 100 percent due to rounding. 

It is possible that some veterans who discontinued in fiscal year 2010 or later are included as MRG 

cases in this figure even though they actually discontinued for a different reason. This is because of a 

change in fiscal year 2010 in the meaning assigned to certain discontinuation reason codes in VA 

data.

While the amount of time veterans spend in the VR&E program varies 

substantially among participants, it can be lengthy for many veterans. 

 21 While a success rate of about 50 percent appears considerably lower than the 74 

percent employment rehabilitation rate VA reported for fiscal year 2012, this discrepancy 

is due at least in part to how VA calculates its rate. VA divides the number of veterans 

placed in suitable jobs during a year by the total number placed in suitable jobs plus those 

discontinued from an employment-related rehabilitation plan. However, VA excludes from 

this calculation those still participating in services and those who discontinued but 

achieved certain types of MRG. If we used the same formula to calculate the success rate 

for veterans who entered the program in fiscal year 2003, then our rate would be 67 

percent—less than but much closer to what VA reports. 
Page 13 GAO-14-61 VA Vocational Rehabilitation and Employment

Among those who started in fiscal year 2003, the average time from 

application to successful placement in a suitable job was 4 years 7 

months. While about half of these veterans completed the program within 

3 to 6 years, many took considerably longer (see fig. 4). Furthermore, it 

appears that time frames for completing the program have not changed 

substantially for more recent applicants. For example, looking at VR&E 

participants who successfully attained suitable employment during fiscal 

year 2012, we found their average time from application to completion 

was also 4 years 7 months, with a range similar to that for fiscal year 

2003 applicants. Meanwhile, among veterans who entered in fiscal year 

2003 and have discontinued without yet returning, the average time from 

application to the final discontinuation was very similar—5 years.

Figure 4: Time to Achieve Suitable Employment for Fiscal Year 2003 Applicants

Note: Percentages of veterans placed in each year do not add up to 100 due to rounding.

a

Indicates the percentage of successfully rehabilitated fiscal year 2003 applicants placed in suitable 

employment in each year following application.

b

Indicates the cumulative percentage of successfully rehabilitated fiscal year 2003 applicants placed 

in suitable employment by each year following application.\

Factors contributing to varied but generally lengthy time frames include 

the services veterans receive, as well as the fact that many extend their 

overall time in the program by interrupting or discontinuing along the way. 

Of those veterans who started since fiscal year 2003 and received some 
Page 14 GAO-14-61 VA Vocational Rehabilitation and Employment

employment services, most (about 70 percent) pursued a 4-year college 

degree, and these veterans typically took longer to complete the program 

than those who pursued other types of training such as community 

college degrees or vocational training. In addition, we estimate that 

among veterans who started since fiscal year 2003 and achieved suitable 

employment, about one-quarter interrupted their programs at least once.22

Once veterans attain suitable employment, they rarely come back for 

additional services, even though they may be entitled to such services if, 

for example, their disability worsens or they qualify under new expanded 

criteria established by the VOW Act. Of veterans who entered the 

program in fiscal year 2003 and were placed in suitable jobs, 4 percent 

have applied for and 2 percent have actually received additional 

employment-related VR&E services, with 3 years on average between 

completing the program and re-applying. The data on applying again 

following rehabilitation are similar for veterans who entered in fiscal years 

2004 and 2005. Moreover, the expanded eligibility criteria authorized by 

the VOW Act have not so far had a substantial impact on the number of 

veterans returning for additional services. According to data provided by 

VA, between May 2012 and April 2013 only 10 veterans nationwide 

received additional VR&E services under these new VOW Act criteria. 

Managers or staff in a number of regional offices told us the new criteria 

would not significantly expand the population of veterans eligible for 

additional VR&E services: several noted that almost all the veterans who 

would qualify under the new criteria are already eligible under the old 

criteria. Some also said it is preferable to obtain additional services under 

the old criteria, which provide more months of additional services.

Almost the same proportion discontinued and returned to the program at 

least once before ultimately achieving success, and some did so two or 

three times. The small proportion (4 percent) who received job placement 

assistance only, meanwhile, attained suitable employment more quickly 

than those who received other types of services.

23

 22 Based on our assessment of the reliability of VA administrative data, we determined 

there may be some inaccuracies in data on the number of times veterans interrupted their 

programs. Therefore, we provide estimates rather than exact numbers when presenting 

these data. For more information on our data reliability assessment see appendix I. 

23 Under the VOW Act provision, veterans may receive only up to 12 additional months of 

benefits. By contrast, there is no specific limit on the number of additional months of 

benefits available to veterans who qualify under the pre-existing criteria. 
Page 15 GAO-14-61 VA Vocational Rehabilitation and Employment

While veterans may discontinue for a variety of reasons, analysis of 

administrative data, results from a VA customer satisfaction survey, and 

our interviews with VR&E staff and participants point to several key 

challenges that may affect veterans at any stage of the rehabilitation 

process.24

Veterans’ disabilities—especially those related to mental health—present 

challenges throughout the rehabilitation process. Among respondents to 

VA’s Voice of the Veteran benchmark survey who had withdrawn from or 

interrupted their rehabilitation programs, “medical difficulties” was the 

most frequently cited reason for doing so (about one-third cited this as a 

reason).25 Further, our analysis of VA administrative data found that 

veterans with mental health conditions were 12 percentage points less 

likely to be successful within 8 years of program entry than those without 

such conditions, the largest difference for any specific type of disability 

(see fig. 5).26

 24 Administrative data indicate that VR&E participants who received some employment 

services most commonly discontinued after failing to complete a training program, but also 

did so after receiving additional evaluation and planning services or during the job 

placement phase. In conducting this analysis we looked at either the veteran’s case status 

immediately preceding their final discontinuation or, in cases where veterans interrupted 

services before finally discontinuing, the case status immediately preceding this 

interruption.

VR&E managers, staff, and program participants we 

interviewed provided examples of how medical conditions can be an 

obstacle. For example, a VRC in one office explained that veterans who 

25 VA’s Voice of the Veteran survey measures veterans’ satisfaction with benefits and 

services received, including through the VR&E program. The results reported here are 

from the pilot survey fielded in October 2012 to January 2013; VA plans to field the survey 

on a regular, ongoing basis in the future. More specifically, the results on reasons for 

withdrawing or interrupting from the program are based on the responses of almost 200 

veterans who had withdrawn or interrupted after spending at least 60 days in the program. 

We assessed the methodology of the pilot survey and concluded that the results we are 

reporting are sufficiently reliable for our purposes. However, these results reflect the 

opinions of survey respondents and cannot be generalized to the broader population of 

VR&E participants. For more information see appendix I.

26 We chose a period of 8 years for this analysis because, as shown in figure 4, many 

veterans take that long to achieve a successful outcome. This period of analysis provided 

enough time to observe successful outcomes and included results for veterans who 

applied in both fiscal years 2003 and 2004. We also analyzed the associations between 

specific disabilities and the likelihood and rate of achieving program success at various 

points in time after application, while controlling for other factors that may affect outcomes. 

Even after controlling for these factors, we still found that having a mental health condition 

was associated with a smaller chance of success. For more details on our analysis and 

results, see appendix II.

A Variety of Challenges 

May Result in Many 

Veterans Not Completing 

the Program

Mental Health and Other 

Disabilities
Page 16 GAO-14-61 VA Vocational Rehabilitation and Employment

stop participating in a training program when they undergo surgery for 

their service-connected disability may lack motivation to resume training 

afterwards. A VRC in another office said veterans with post traumatic 

stress disorder (PTSD) face special challenges with maintaining 

employment because of difficulties interacting with co-workers. One 

veteran we interviewed said he completed a certificate program to 

become a heavy equipment operator, but was unable to obtain the 

required class D driver’s license because his PTSD made it difficult for 

him to deal with the noise and crowds at the testing site. Another veteran 

told us he trained for and was placed in a job in information technology. 

However, he said his job involves tasks for which he had not been 

trained, and his mental health conditions—including traumatic brain injury 

(TBI)—have made it hard for him to learn these tasks and may force him 

to leave his job.

Figure 5: Likelihood of Achieving Suitable Employment for Veterans with Selected 

Disabilities Compared to Those with Other Disabilities

Note: This figure presents estimates of the difference in the likelihood of success within 8 years for 

veterans with each disability compared to those without the disability. The figure does not include 

results for 6 other disabilities for which we did not find a statistically significant association with the 

likelihood of success, after controlling for other factors.

VR&E staff and veterans often find it challenging to develop employment 

plans that can realistically lead to success. Administrative data suggest 

there are difficulties with developing a sound plan. About half the veterans 

Developing a Realistic 

Employment Plan
Page 17 GAO-14-61 VA Vocational Rehabilitation and Employment

who started since fiscal year 2003 and received some employmentrelated services have stopped their programs and returned for additional 

evaluation and planning at least once, and some have done so multiple 

times.27

Analysis of administrative data indicates that the more VR&E counselors 

a veteran works with during his or her participation in the program, the 

less likely he or she is to achieve success. As noted above, veterans 

typically spend several years participating in the program, and most work 

with multiple counselors. The vast majority of those who entered the 

program since fiscal year 2003 and received some employment services 

have worked with at least two counselors, and a substantial proportion 

(about one-third) worked with four or more. We found that the chance of 

Managers or staff in all eight offices we contacted said some 

veterans enter the program with unrealistic employment goals, which 

makes vocational planning more challenging. For example, VR&E staff in 

one office said veterans may want to train for jobs in law enforcement, 

even though they have PTSD or another psychological condition that 

makes this goal unrealistic. Some veterans we interviewed cited 

challenges coming to agreement with their counselors on an employment 

plan. For example, one said he wanted to study theology, but his 

counselor felt that studying to become a teacher would be more 

appropriate, and he ultimately left the program because of this 

disagreement. In addition, managers or staff in all eight offices we 

contacted said veterans may come to the VR&E program with the goal of 

furthering their education rather than obtaining a job, or may have a 

misperception about the purpose of the VR&E program. For example, a 

VRC in one office said she must explain to veterans that VR&E only 

provides training when it is necessary to achieve employment. She feels 

veterans’ chance of success would be enhanced if they entered the 

program with specific employment goals in mind, rather than training 

goals. Input from veterans is consistent with this viewpoint. While most 

recently-enrolled respondents to the Voice of the Veteran survey said 

they were motivated to apply to the VR&E program by employmentrelated goals, a substantial number (about a quarter) said they applied 

primarily to further their education.

 27 Based on our assessment of the reliability of VA administrative data, we determined 

that there may be some inaccurate data regarding the number of times VR&E participants 

received evaluation and planning services. Therefore, we present estimates rather than 

exact numbers for these data. For more information on our assessment of VA data, see 

appendix I. 

Multiple VR&E Counselors
Page 18 GAO-14-61 VA Vocational Rehabilitation and Employment

success declines steadily as veterans work with more counselors (see fig. 

6).28

Figure 6: Likelihood of Achieving Suitable Employment for Veterans Working with 

Multiple Counselors

For example, veterans who had worked with two counselors were 8 

percentage points less likely to have attained suitable employment within 

8 years of entering the program, as compared to those who had worked 

with one counselor. Veterans who had worked with four counselors were 

27 percentage points less likely to have attained suitable employment.

Note: This figure presents estimates of the percentage difference in the likelihood of success within 8 

years for veterans who worked with more than 1 counselor compared to those who worked with only 

1 counselor.

According to the Voice of the Veteran survey, many veterans who 

withdrew from or interrupted their employment services cited “family 

obligations” (27 percent) and “financial difficulties” (18 percent) as a 

reason. Managers or staff in seven of the eight regional offices we 

contacted also said one or both of these are common issues for veterans. 

 28 We analyzed the association between multiple counselors and the likelihood and rate of 

achieving program success at various points in time following application, while controlling 

for other factors that may affect outcomes. Among other things we controlled for the 

number of times veterans discontinued from and re-entered the VR&E program, which

could be associated with the number of counselors they worked with. Even after 

controlling for these factors, we still found that working with multiple counselors was 

associated with a smaller chance of success. For more details on our analysis and results, 

see appendix II.

Family and Financial 

Obligations
Page 19 GAO-14-61 VA Vocational Rehabilitation and Employment

For example, according to managers in one office, veterans may have an 

immediate need to earn more money to support their families, so they quit 

training and take a job that may not be suitable given their disability. 

VRCs in another office told us a change in life circumstances—such as 

marriage, the birth of a child, or a death in the family—may make it more 

difficult to complete training. Veterans we interviewed also mentioned 

family obligations as a challenge. For example, one said she failed to 

complete her certification program in school administration because of the 

birth of her child.

Managers, staff, and veterans we interviewed cited challenges associated 

with the transition from military service to civilian employment. For 

example, managers or staff in five of the eight regional offices we 

contacted told us veterans may have difficulty translating military 

experience into terms that civilian employers can understand, or that they 

may need civilian credentials to get a job even with the expertise they 

gained in the military. A VRC in one office explained that, in the state 

served by his office, veterans who received technical training in the 

medical field while in the military may still need additional training to pass 

a state certification exam to work in a comparable civilian job. Along 

similar lines, one veteran said he had responsibilities for benefits 

counseling, training, and recruitment while in the military that are 

comparable to the skills needed in a civilian human resources job. 

However, when he sought employment with the federal government, his 

VR&E counselor could not help him translate his military experience into 

federal civilian employment. In addition, staff in three of the offices we 

contacted said civilian employers may not understand veterans’ 

disabilities and how to accommodate them. One veteran we interviewed 

told us he was struggling to perform his job duties because of his physical 

disabilities, but had not received needed accommodations from his 

employer. He also felt his VR&E counselor did not sufficiently explain the 

physical demands of the job in advance. Finally, staff in two offices said 

veterans may lack experience with civilian work culture, which can 

present difficulties with succeeding in a new job. For example, an EC in 

one office said veterans are accustomed to having standard operating 

procedures in the military, and may struggle in a civilian job that lacks 

such clear guidelines.

Transition from Military to 

Civilian Employment
Page 20 GAO-14-61 VA Vocational Rehabilitation and Employment

We found that veterans served by some regional offices were significantly 

less likely to succeed than those served by others, after controlling for a 

variety of other factors.29 However, we were unable to control for local 

economic conditions in our analyses,30 and managers or staff in 4 of 8 

offices we contacted said the local labor market can be a key challenge to 

placing veterans in suitable jobs. VA central office officials also said 

variations in local economic conditions, including variation in the 

unemployment rate and the availability of good-paying jobs, contribute to 

regional variation. For example, they cited the loss of manufacturing jobs 

in the area served by one office as part of the explanation for its lower 

than average success rate. VA officials also cited other potential factors in 

regional variation, such as offices’ ability to retain experienced staff and 

fluctuations in staffing levels due to retirements. For example, they said 

the VR&E workforce in one office with below average performance may 

be more transient than in other offices.

 29 We estimated the percentage suitably employed at several time points across the 56 

offices. For this estimate we used the last office a veteran was active in since it may be 

most closely connected to the last veteran status. This estimate takes into account the 

several different possible outcomes (placed in suitable employment, discontinued, still 

participating, and achieved independent living goal) in estimating the probability of 

achieving suitable employment.

30 Because we analyzed outcomes for veterans who entered and exited the program at 

different points in time over a 10 -year period, and local economic conditions varied over 

this time, it was not practical to include factors such as state or local unemployment rates 

in our analysis. In addition, measuring demand for labor across veterans’ diverse skills 

and local economies would require tailored measures that we were unable to obtain.

Regional Office Variation Due 

to Economic or Other Factors
Page 21 GAO-14-61 VA Vocational Rehabilitation and Employment

VA has identified limitations with its primary measure of VR&E success—

the rehabilitation rate—and is considering replacing it with a numeric 

target for positive outcomes. VA central and regional office officials noted 

that the rehabilitation rate goal can incentivize a VRC to hold onto an 

inactive case, even if the veteran has clearly left the program, because 

holding onto inactive cases instead of closing them can help regional 

offices and individual VRCs increase their rehabilitation rates and more 

easily reach their goals for a particular fiscal year.31

 31 The rehabilitation rate is the percentage of successful rehabilitations, out of the total of 

successful rehabilitations plus cases discontinued as unsuccessful. It excludes cases 

where veterans were not successfully rehabilitated but had maximum rehabilitation gain, 

and where veterans transferred to Post-9/11 GI Bill benefits. VA calculates an overall 

rehabilitation rate for all VR&E participants and separate rates for different sub-groups. 

For the purpose of this section of the report, we focused on the rate for participants who 

received employment-related services.

When staff delay 

closing cases like this, an office’s or individual’s rehabilitation rate may 

appear higher than it really is in one year and lower in another year, 

leading to an inaccurate picture of VA’s success in rehabilitating veterans. 

Furthermore, while cases remain in interrupted status, VRCs are still 

required to attempt contact with veterans every 60 days, which can limit 

the time spent working with veterans who are actively participating. In 

addition to the incentive to hold onto inactive cases, VA officials noted 

VA Has Made Limited 

Progress in Improving 

VR&E Performance 

Management, 

Workload 

Management, and 

Training

VA Faces Challenges with 

Measuring Employment 

Outcomes and Veteran 

Satisfaction

VR&E Outcome Measurement 

Presents a Challenge for VA
Page 22 GAO-14-61 VA Vocational Rehabilitation and Employment

that the rehabilitation rate does not capture all of the benefits VR&E 

provides to veterans. In particular, the current calculation excludes MRG 

cases, where the veteran obtained some value from VR&E services but 

did not obtain and maintain suitable employment.32 Because it does not 

explicitly include MRG cases in the assessment of VR&E performance, 

VA’s current measure does not fully reflect the value it provides to 

veterans. Another concern raised by some regional officials was that 

having a uniform rehabilitation rate goal for every office does not account 

for regional differences in factors affecting employment, such as local job 

market conditions.33

To address these and other issues, VA established four workgroups to 

review and make recommendations for improving VR&E’s individual 

employee performance standards.34 The resulting draft recommendations 

included eliminating VR&E’s rehabilitation rate measure and replacing it 

with a measure of the number of positive outcomes—specifically, the 

number of successful employment and independent living rehabilitations 

and MRG cases.35

While the current rehabilitation rate has limitations, the measure VA is 

considering as an alternative may not necessarily be a better way to 

monitor and demonstrate VR&E’s success. Specifically, VA is considering 

replacing its current outcome measure—which tracks those who obtained 

suitable jobs relative to those who did not—with a measure that tracks 

only the number of any positive outcomes. According to VA officials, 

Four regional offices were recommended to pilot the 

revised measure. According to a VA official, VA is expected to approve 

these draft recommendations by the end of fiscal year 2014. Also, in

comments provided after reviewing a draft of this report, VA stated that it 

is developing VA-wide and regional office measures of program success, 

and expects to implement the VA-wide measures in fiscal year 2015.

 32 VA can determine that the veteran (1) is employed with VR&E assistance, but not in a 

suitable job; (2) is employable, with VR&E assistance, but not employed; or (3) is not 

employable, but received assistance in making gains in self-management, self-advocacy, 

and independence in daily living.

33 However, VA officials said they may take local economic conditions into account in 

setting each office’s target number of veterans rehabilitated.

34 VA created VR&E manager, VRCs, ECs and support staff workgroups. 

35 VA already uses a similar measure in individual VRC performance standards: the 

number of veterans successfully rehabilitated, not including MRG cases.
Page 23 GAO-14-61 VA Vocational Rehabilitation and Employment

positive outcomes would include not just veterans who obtained suitable 

employment, but also veterans who obtained other benefits (such as 

MRG) from the VR&E program. Good practices for performance 

measurement suggest that measures should allow agencies to 

transparently report their success to the Congress and public, and identify 

and address performance variances across their programs.36

Aside from considering changes to its outcome measures, VA has taken 

other steps to help regional offices better manage their interrupted cases, 

but it is unclear how successful these efforts have been. For example, VA 

uses performance goals to hold regional offices and VRCs accountable 

for monitoring interrupted cases and keeping their proportions below 

certain thresholds. The current goal is 11 percent for regional offices and 

12 percent for individual VRCs. Also, VR&E’s central office has provided 

tools and training to help VR&E managers monitor the number and 

average time of cases in interrupted status, and percent in interrupted 

status over 180 days. VR&E managers at two regional offices we 

contacted noted that they had VRCs close old, inactive cases, regardless 

of the short-term impact on the office’s rehabilitation rate. VA reported 

that the overall percentage of interrupted cases declined from 10.4 

percent in fiscal year 2010 to 9.6 percent in fiscal year 2012. However, 

managing interrupted cases remains a challenge. According to VA data, 

in July 2013, 39 percent of more than 15,000 interrupted cases had been 

in that status for more than 180 days. In addition, 8 of 57 regional offices 

had at least 50 percent of their interrupted cases in that status for more 

than 180 days, ranging up to 70 percent at one office.

The 

proposed measure of specific numbers of positive outcomes lacks this 

transparency because it does not account for veterans who left the 

program without obtaining suitable employment or other discernable 

benefits. It also obscures VA’s success in achieving its core mission 

because it does not separate veterans who obtained suitable employment 

from those who received other, lesser benefits. Finally, measuring only 

numbers of positive outcomes is less effective for monitoring and 

managing performance across regional offices—that is, comparing each 

office’s relative rate of success to identify those that may serve as models 

or require further oversight.

 36 For example, see GAO, Standards for Internal Control in the Federal Government, 

GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999) and GAO, Managing for 

Results: Analytic Challenges in Measuring Performance, GAO/HEHS/GGD-97-138 

(Washington, D.C.: May 30, 1997).
Page 24 GAO-14-61 VA Vocational Rehabilitation and Employment

The mission of the VR&E program includes helping veterans maintain 

suitable employment, and VA holds itself accountable for achieving this 

part of the mission by counting job placements as rehabilitations if they 

are maintained for at least 60 days. According to several regional office 

managers and staff we interviewed, cases are typically closed when 

veterans have maintained employment for this 60-day minimum, as 

problems with a veteran maintaining employment were most likely to 

occur in the first 60 days of employment if they occurred at all. However, 

if a veteran appears to be at high risk of losing a job, for example, the 

VRC can keep the case open for a longer period to provide additional 

supports. Some VRCs noted that this can occur when a veteran has a 

serious mental health condition, for example.37 According to VA data on 

about 38,000 veterans who applied for VR&E between fiscal years 2003 

and 2012 and have been rehabilitated to employment, over one-third 

were tracked in employment for 6 months or longer before having their 

cases closed as rehabilitated, and some were tracked for 1 year or 

more.38

Research and practices at other federal employment programs suggest 

that tracking employment for 60 days may not be sufficient to gauge 

whether a veteran will be able to maintain employment. For example, a 

2013 National Organization on Disability study suggested that wounded 

warriors transitioning to civilian employment need support for an extended 

time after obtaining jobs. Meanwhile, the Department of Labor (Labor) 

measures individuals’ employment retention over a longer term, 180 

days.

A VA official noted that some of this time is spent completing 

documentation and approval of case closures. In addition, one regional 

VR&E manager noted that closure may be delayed when the veteran 

does not inform the regional office that he or she obtained a job.

39

 37 Some cases are required to be tracked longer. Veterans in the self-employment track 

must be followed up on for at least a year.

This measure applies to many of Labor’s employment programs, 

including its veterans employment programs.

38 Timeframes of fewer than 60 days were reported in 10 percent of the cases. Also, in 

about 8 percent of the approximately 38,000 cases, veterans may have obtained 

employment before completing training, which may lead to longer periods between job 

placement and case closure.

39 For an employment program participant, Labor defines employment retention as having 

a job in both the second and third calendar quarters after exiting the program. The 

measure is the proportion of such participants, out of all participants who were employed 

in the first quarter after exiting the program. 

VA Lacks Performance 

Measures of Veterans’ Longterm Employment Success
 

Page 25 GAO-14-61 VA Vocational Rehabilitation and Employment

VA officials stated that they plan to collect information on veterans’ longterm employment, but did not indicate whether they plan to develop 

performance measures to show the program’s impact on veterans’ longterm employment. Specifically, VA is considering contacting rehabilitated 

and discontinued VR&E participants at 6 and 12 months after they finish 

the program. This effort could yield information on employment over the 

longer term. VA plans to use this information to identify veterans who are 

having difficulties in maintaining employment, and possibly provide them 

additional supports. However, without developing performance measures, 

VA may be missing an opportunity to track the long-term effectiveness of 

the VR&E program.

VA has made progress toward holding itself accountable for VR&E 

participants’ satisfaction by implementing its new Voice of the Veteran 

survey, but the agency has not incorporated practices necessary to 

ensure the quality of benchmark results. VA intends to use survey data to 

inform program improvements and training developments, and potentially 

to report on veterans’ satisfaction with VR&E.40 In developing and piloting 

the survey,41 VA followed generally recognized survey practices, including 

detailed survey planning, comprehensive questionnaire development and 

testing, and extensive data collection activities, such as follow-up with 

veterans who did not respond to VR&E’s initial survey request.42

 

40 VA included a placeholder measure of veterans’ satisfaction with the VR&E program in 

its fiscal year 2012 Performance and Accountability Report, and noted that it would 

establish baseline performance or goals based on a veterans’ survey. At the time of our 

review, VR&E had not decided whether to use Voice of the Veteran survey data for this 

purpose.

However, VA did not perform a nonresponse analysis to determine the 

41 VA contracted the services of J.D. Power and Associates to develop and pilot the Voice 

of the Veteran survey. The total cost of VA’s pilot survey effort was about $977,192. VA 

plans to administer the Voice of the Veteran survey on an on-going basis, and officials 

anticipate the total contract cost to be about $1,462,500.

42 For generally recognized survey research practices, see Office of Management and 

Budget, Standards and Guidelines for Statistical Surveys, September 2006; The American 

Association for Public Opinion Research, 2011 Standard Definitions: Final Dispositions of 

Case Codes and Outcome Rates for Surveys 7th edition, AAPOR; and the American 

Statistical Association, What is a Survey, 2nd edition, 2004, 

https://www.whatisasurvey.info/. For more information on our assessment of the 

methodology and pilot results of the Voice of the Veteran surveys administered, see 

appendix I.

VA Generally Followed 

Recognized Practices in 

Designing Its Customer 

Satisfaction Survey, but the 

Quality of Benchmark Results 

is Uncertain
Page 26 GAO-14-61 VA Vocational Rehabilitation and Employment

effects of any nonresponse bias on the quality of its pilot survey data.43

Guidance from the Office of Management and Budget (OMB) states that 

agencies should perform nonresponse analysis when a survey’s unit 

response rate is below 80 percent, to identify the possibility of bias in a 

survey’s results, and measure, adjust for, report, and analyze 

nonresponse to assess its effects on data quality.44 VA’s plan for the pilot 

survey, as approved by the OMB, anticipated the need for this analysis, 

but its contract with the survey administrator made it optional.45 Although 

the response rates for the three pilot surveys of VR&E participants each 

fell below the 30 percent level specified in VA’s sample design, VA 

officials said they elected not to conduct a nonresponse analysis due to 

budget constraints.46 Moving forward, VA officials stated there is no 

expectation that a nonresponse analysis be performed for VA’s ongoing 

survey effort unless deemed necessary. Currently, VA’s approved 

contract for its ongoing survey effort does not include an option for a 

nonresponse analysis if response rates are low.

 43 Nonresponse occurs when members of the sample do not participate in the survey at all 

(unit nonresponse), or do not answer particular questions (item nonresponse). 

Nonresponse may result in nonresponse bias—systematic errors that result in under- or 

overestimation of a true value in survey results—because nonrespondents may have 

provided substantially different answers than those who did respond. A nonresponse 

analysis is an established survey research practice used to determine whether 

nonresponse bias has occurred. It can be performed using a variety of methods—for 

example, by randomly selecting a sample of survey nonrespondents, interviewing them to 

obtain missing answers to key survey questions, or imputing those answers from 

administrative records or other known. 

44 Research in survey methods suggests that—while response rates may not be highly 

correlated with the presence of nonresponse bias and are not considered, by themselves, 

to be sufficient indicators of the quality of survey results—response rates are a useful 

indicator of the risk of nonresponse bias. According to OMB guidelines, a survey’s 

response rate is a valuable data quality and performance indicator, and is associated with 

the generalizability of a survey’s results. See Office of Management and Budget, 

Guidance on Agency Survey and Statistical Information Collections, (Washington, D.C.: 

January 20, 2006).

45 Specifically, VA’s contract stated that the survey administrator would monitor the 

response rates of the surveys and provide reports that would “include recommendations 

(where necessary) as to whether or not a nonresponse bias analysis should be conducted 

to ascertain reasons or cause for low response rates.”

46 Three pilot surveys were administered to different sub-groups of VR&E participants. For 

more information on the surveys and their response rates, see appendix I. 
Page 27 GAO-14-61 VA Vocational Rehabilitation and Employment

VA has acted to strengthen VR&E workload management by 

commissioning a study of workload management issues, among other 

things. The study team collected and analyzed data on how VR&E staff at 

four regional offices spent their time on a variety of tasks, such as 

paperwork and client contacts. In its April 2011 report,47

Despite these efforts, some regional offices struggle with high caseloads 

that may affect quality of services to veterans, while others do not, 

suggesting potential shortcomings in the way VA allocates resources. 

VR&E managers and staff in four of the offices we contacted said high 

workloads impede their ability to spend sufficient time working with 

individual veterans. Meanwhile, VRCs in other offices we contacted said 

their workloads were manageable. VA administrative data indicate that 

average ratios of VR&E cases to VRCs vary considerably, as shown in 

figure 7. For example, while VA reported a national average of 139 cases 

per VRC as of March 31, 2013, regional office average caseloads ranged 

from 64 in Fargo, North Dakota to 579 in White River Junction, Vermont.

the study team 

recommended, among other things, that VA improve its approach for 

estimating its overall staffing needs. VA now has an ongoing initiative to 

assess the total number of VR&E staff needed across the regional offices. 

This initiative, intended for use in formulating future VR&E administrative 

budget requests, is expected to be completed by March 2014. Also, in 

response to an internal business process review, VA has made changes 

to VR&E procedures intended to ease VRCs’ workloads by reducing 

paperwork.

48

As of March 31, 2013, seven offices averaged fewer than 100 cases per 

VRC,49 while eight averaged more than 175 cases per VRC.50

 47 U.S. Department of Veterans Affairs, Vocational Rehabilitation and Employment 

Service, Work Measurement Study, Final Report (Washington, D.C.: April 21, 2011). The 

study was conducted for VA by the Millennium Corp. and New Editions Consulting, Inc.

48 According to VA, White River Junction had only one VRC, although another was added 

after March 31, 2013. In commenting on a draft of this report, VA stated that it expects 

another vacancy to be filled in January 2014, and White River Junction’s average to drop 

to about 160 cases per VRC.

49 In addition to Fargo, the offices in Wichita, Kansas; Togus, Maine; Lincoln, Nebraska; 

Pittsburgh, Pennsylvania; Huntington, West Virginia; and Manila, the Philippines averaged 

fewer than 100 cases per VRC.

VA Workload Management 

Efforts May Not Fully 

Address Challenges 

Facing Some Regional 

Offices
Page 28 GAO-14-61 VA Vocational Rehabilitation and Employment

Figure 7: Regional Office Caseloads for Vocational Rehabilitation Counselors—

Lowest, Highest and Overall Average—Fiscal Years 2008 through 2012 and March 

2013

VA officials cited factors that can contribute to the wide variations in 

caseload ratios across regional offices, such as different case 

management approaches or VRC attrition than can result in temporarily 

high caseloads. Further, some offices place VRCs at military treatment 

facilities and college campuses. VA officials noted that these VRCs—

because of the different service needs of servicemembers and veterans 

at these locations51

 50 In addition to White River Junction, the offices in Los Angeles, California; Hartford, 

Connecticut; Honolulu, Hawaii; Indianapolis, Indiana; Newark, New Jersey; Albuquerque, 

New Mexico; and Cleveland, Ohio averaged more than 175 cases per VRC. The 

Cleveland office had the next-highest ratio, with 206 cases per VRC.

—may have different caseload levels.

51 VRCs at military treatment facilities provide VR&E services to wounded, ill and injured 

servicemembers who are in the Integrated Disability Evaluation System. Through the 

VetSuccess on Campus program, VRCs at college campuses provide support to student 

veterans using VA education benefits, such as career and academic counseling and 

medical and mental health referrals.
Page 29 GAO-14-61 VA Vocational Rehabilitation and Employment

According to VA, the resource allocation model VA uses to set the fulltime equivalent (FTE) employee level in each office and assign any new 

FTE positions each year accounts for three factors that may affect staffing 

needs.52 For fiscal year 2013, the model was based on workload (50 

percent), outcome performance (25 percent), and accuracy performance 

(25 percent).53

• The model does not consider workloads associated with educational 

and vocational counseling cases,

However, VR&E managers in five of the eight regional 

offices we contacted identified gaps and issues with VA’s criteria for 

allocating VR&E staff across offices that could limit the model’s 

effectiveness in assigning resources where they are most needed. Issues 

cited by these officials included:

54

• The model’s use of performance measures may have the effect of 

taking staff away from poorer-performing offices, while increasing 

staffing at better-performing offices. This could make it harder for 

poorer-performing offices to improve their performance.

which may be disproportionately 

high in certain offices, particularly where there is a large military 

presence. For example, according to VA data, the San Diego office 

had over one-third of all educational and vocational counseling cases 

in fiscal year 2012. Regional offices expend staff resources on these 

cases, but do not receive credit for them in the resource allocation 

model.

55

 52 Before applying the model, VA allocates FTE positions to support the Integrated 

Disability Evaluation System (at military installations) and the VetSuccess on Campus 

program (at college campuses).

53 Workload factors include numbers of applications and cases in job ready status. 

Outcome performance factors include numbers of entitlement determinations and 

rehabilitations. Accuracy performance factors include the accuracy of entitlement 

determinations and program outcomes (rehabilitation or unsuccessful discontinuance).

54 In addition to VR&E services, VR&E can provide free educational and vocational 

counseling to certain veterans and their dependents, and servicemembers. These include 

veterans eligible for VA education and VR&E benefits, and servicemembers within 180 

days of expected discharge. Services include interest and aptitude testing; assistance in 

setting occupational goals; and assistance in identifying education and training institutions 

that can be used to achieve occupational goals. 

55 In previous work, we presented research that identified drawbacks of basing program 

funding on performance measures. GAO, Vocational Rehabilitation Funding Formula: 

Options for Improving Equity in State Grants and Considerations for Performance 

Incentives, GAO-09-798 (Washington, D.C.: September 30, 2009).
Page 30 GAO-14-61 VA Vocational Rehabilitation and Employment

According to leading practices for developing resource allocation models, 

agencies should ensure the credibility of the data used in such models. 

For example, the data should reflect correct assumptions about how staff 

time is actually spent. Also, agencies should consider ideas and 

information from people who work at the affected offices and other 

stakeholders when designing the model.56

In addition to issues with its resource allocation model, VA lacks 

information about regional offices’ varied approaches to managing their 

caseloads. VA’s administration of the VR&E program is decentralized; 

according to VA central office officials, each regional office’s management 

is free to decide how best to manage its office’s workloads. Thus, regional 

offices have adopted a variety of approaches to managing their VR&E 

workloads. Some offices generally assign a veteran to work with one VRC 

throughout the rehabilitation process, until the point at which the veteran 

is determined to be job ready. Meanwhile, other regional offices have 

VRCs specialize in one phase of the rehabilitation process, so a veteran 

works with multiple VRCs as he or she progresses through the program. 

For example, some VRCs may deal with just the front end of the 

process—determining veterans’ entitlement to services and preparing 

rehabilitation plans—while others focus on managing cases of veterans 

who are pursuing their rehabilitation plans.

For VR&E, this could include 

VR&E staff at regional offices. VA officials stated that they have not made 

significant revisions to the resource allocation model since fiscal year 

2003, but have made adjustments, such as adding the accuracy factor. 

However, VA officials noted that they plan to re-assess the VR&E 

resource allocation model after they have developed their new model for 

estimating overall VR&E staffing needs.

57

 56 GAO, DOJ Workforce Planning: Grant-Making Components Should Enhance the Utility 

of Their Staffing Models, 

Also, some regional offices 

have VRCs who specialize in certain types of cases, such as veterans in 

the independent living track. Another variation at some offices is to assign 

a VRC to veterans attending a specific college or other education and 

training institution.

GAO-13-92 (Washington, D.C.: December 14, 2012).

57 In some cases veterans may work with multiple VRCs even if an office does not take a 

specialization approach. For example, if a VRC retires, that VRC’s cases would be 

redistributed among other VRCs.
Page 31 GAO-14-61 VA Vocational Rehabilitation and Employment

Officials at the regional offices we contacted had different opinions about 

the efficacy of VRCs specializing by phase of the VR&E process. Some 

VR&E managers and staff cited advantages to allowing a single VRC to 

handle a case: for example, a VRC can (1) develop a stronger rapport 

with the veteran; and (2) avoid possible confusion, including among the

veterans, related to handoffs between VRCs. Other regional officials cited 

advantages of specialization. For example, the VRC can: (1) manage 

caseloads without being distracted by a larger range of tasks (for 

example, assessing entitlement, preparing rehabilitation plans and 

monitoring veterans’ progress on their plans); and (2) develop expertise in 

one phase of the VR&E process. VR&E managers at one of the regional 

offices we contacted stated that they plan to move away from a more 

specialized caseload management approach, citing improved efficiency 

from having one VRC handle a case. Meanwhile, managers in another 

office said they had recently moved towards a specialization approach 

because they feel it better enables them to handle their large caseload.

Federal internal control standards require that managers obtain 

operational data to ensure their programs are run effectively and 

efficiently.58

 58 See 

For VA, this could include data on how staff resources are 

being utilized across regional offices, and how such usage affects offices’ 

efficiency and effectiveness. It is especially important that VA understand 

regional offices’ approaches and their effects, given our finding that 

veterans who work with a number of different VRCs over time are less 

likely to be successful. Moreover, the April 2011 consultants’ report to VA 

on VR&E workload management—which also found wide variations in 

how VRCs at different offices allocated their time among different tasks—

recommended that VA assess regional office approaches to assigning 

tasks among VRCs in order to identify best practices for helping to 

rehabilitate veterans, while balancing standardization across offices with 

flexibility to serve each office’s service area. However, VA central office 

does not collect information on how regional offices manage their 

caseloads and assign tasks among VRCs, and has not assessed the 

effect of varied regional office approaches on case processing efficiency 

and veterans’ rehabilitation success. VR&E central office officials stated 

that they prefer to leave workload management to regional office 

managers because they are in the best position to determine which 

GAO/AIMD-00-21.3.1
Page 32 GAO-14-61 VA Vocational Rehabilitation and Employment

workload management approach will enable the office to meet its 

program goals.

In the course of our work, we found evidence that staff had experienced 

redundancy and critical gaps in their training; however, VA had taken 

recent steps to address the redundancy and most, but not all of the 

training gaps. Principles of effective strategic training and workforce 

development call for avoiding duplication, identifying workforce 

competency gaps, and addressing these gaps.59 Nonetheless, some of 

the VR&E regional managers or staff at six of the eight offices we 

contacted told us that the annual curriculum offered to VRCs and ECs 

required them to repeat courses. Their experience notwithstanding, we 

found that VA has recently been adding new courses and, by fiscal year 

2013, none of the courses on the curricula for experienced VRCs and 

ECs were redundant with those offered in the prior year. Further, with the 

addition of these new courses, VA is actively closing the nine critical 

training gaps identified in a June 2010 skill gap analysis report.60 The 

report noted that five gaps affect both new and experienced VRCs and 

four additional gaps affect only new staff. VBA contracted for this analysis 

in response to a report we issued in 2009, in which we found that VR&E 

had not collected data on the critical skills and competencies needed by 

VRCs to ensure they could meet the current and future needs of veterans 

in the program.61

Although VR&E closed many training gaps, our review of VR&E’s fiscal 

year 2013 curriculum identified other gaps related to job placement and 

In September 2013, VA informed us that it had begun 

providing training to staff to address these gaps and will have provided 

training to address all the gaps by the end of 2013, and in 2014 will 

provide two new courses for staff on mental health to improve their ability 

to assist veterans with PTSD and other mental illnesses. See appendix III 

for the gaps identified and training developed to address them.

 59 GAO, Human Capital: A Guide for Assessing Strategic Training and Development 

Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: March 2004).

60 For the VBA-sponsored study, see General Dynamics, Skills Survey Results Report for 

Vocational Rehabilitation and Employment (VR&E) Services (Orlando, FL: June 7, 2010).

61 GAO, VA Vocational Rehabilitation and Employment: Better Incentives, Workforce 

Planning, and Performance Reporting Could Improve Program, GAO-09-34 (Washington, 

D.C.: January 26, 2009).

VA Is Addressing 

Redundancy and Most 

Gaps in VR&E Staff 

Training
Page 33 GAO-14-61 VA Vocational Rehabilitation and Employment

accommodation for experienced VRCs. Staff or managers in five of the 

eight regional VR&E offices contacted also identified either job placement 

or workplace accommodation as areas in which additional training was 

needed. For example:

• Some regional managers and staff told us VRCs need more training in 

job placement strategies, employment laws, and tax incentives for 

employers.62

• Some regional managers and staff told us VRCs needed training in 

current workplace accommodations to better assist veterans. 

Workplace accommodations help ensure veterans with disabilities can 

successfully perform on the job and maintain their employment. A 

June 2012 VR&E-sponsored study confirmed the importance of 

workplace accommodations to help veterans attain and maintain longterm employment and recommended that VR&E either train or retrain 

its staff in workplace accommodation.

Job placement skills are critical to achieving VR&E’s 

main mission to ensure that veterans with disabilities who can be 

employed obtain and maintain suitable employment. While ECs are 

generally responsible for assisting veterans with job placement, VRCs 

may take on this task in offices that lack ECs, or when ECs simply are 

not able to provide these services to veterans needing them.

63

Although VA does not currently have specific plans to address either of 

these gaps in fiscal year 2014, VA officials informed us that the agency 

intends in the future to individually identify and address skill gaps for each 

VRC and EC. In the meantime, regarding job placement training, VA 

officials told us that VRCs may opt to take online training developed for its 

ECs, but that VA does not plan to require VRCs to take job placement 

training because VRCs receive it as part of their college or master’s 

degree programs. However, VRCs may not opt to take optional VA 

training as several VRCs and managers told us that VRCs lack time to 

take all the required training. As of July 2013, VA officials had not yet 

developed the curriculum for fiscal year 2014 and did not know if training 

on workplace accommodation for VRCs would be included.

 62Tax incentives, such as the Work Opportunity Tax Credit and the disabled access credit, 

are available to employers to encourage the hiring, retention, and accommodation of 

workers with disabilities. 

63 For the VR&E-sponsored study, see ICF International, VR&E Service Employment of 

Individuals with Severe Injuries Study: Final Report (Fairfax, VA: June 13, 2012).
Page 34 GAO-14-61 VA Vocational Rehabilitation and Employment

As for VA’s plans to identify and address each VRC’s and EC’s skill gaps, 

this effort is part of its strategic initiative to provide competency-based 

training for 16 critical positions throughout the agency. VA officials 

informed us that under this initiative, which began in 2010, the agency

plans to design and provide tailored training and other performance 

support that will be based on an assessment of each critical employee’s 

current proficiency level. VR&E officials expect the outcomes of this effort 

to include: (1) individual performance standards, (2) tailored performance 

support, (3) an assessment baseline and developmental roadmap (i.e., 

training plan), and (4) a tailored National Training Curriculum and 

remediation plan for each VRC and EC. Furthermore, through the 

collection and management of competency data, VR&E officials hope to 

track the knowledge, skills, abilities, education, and experience required 

for specific positions and recruit, train, distribute, and professionally 

develop personnel based on this competency framework. VA officials 

reported that as of fiscal year 2012, 47 percent of employees in critical 

and key occupations had participated in a competency-based training 

program within the last 12 months, and that in June 2013 the agency took 

an initial step to provide individualized training for VRCs. The program 

and plans VA described are consistent with best practices in human 

capital management.

VA has taken steps to improve its management of the VR&E program—

ultimately with the goal of helping more veterans with disabilities attain 

suitable employment—but its ongoing efforts may not sufficiently address 

certain key management issues. With respect to performance measures, 

VA’s efforts to address problems with its existing rehabilitation rate 

measure may actually result in the adoption of a less transparent and 

meaningful outcome measure, as it is not clear how measuring the 

number of positive outcomes will provide the context necessary to 

understand program success. Furthermore, while VA plans to collect 

more information about longer-term outcomes, absent a performance 

measure VA may miss an opportunity to hold staff accountable and make 

adjustments to the program to promote success in maintaining 

employment—consistent with VR&E’s central mission. In addition, while 

VA’s ongoing customer satisfaction survey has the potential to provide 

valuable information, without performing additional analyses to determine 

whether responses actually reflect the experiences of VR&E participants 

overall, the agency cannot be sure that its survey will ultimately produce 

sound results for informing program management and training 

investments.

Conclusions
Page 35 GAO-14-61 VA Vocational Rehabilitation and Employment

VA has also made strides related to workload management and training, 

but remaining shortcomings can negatively affect services provided to 

veterans. Because VA’s formula for allocating staff among regional offices 

rewards higher-performing offices, which can make it harder for lowerperforming offices to improve, but does not consider educational and 

vocational counseling cases, which are concentrated in certain regions, 

some offices’ ability to serve veterans may be negatively affected. 

Lacking information on the varied approaches that regional offices are 

using to manage workloads also undermines VA’s ability to help offices 

optimize their use of staff to serve veterans in the most effective manner. 

Finally, training of VR&E staff is an area in which VA has made 

substantial improvements, but absent additional training on job placement 

and accommodations, counselors may be hindered in their ability to help 

veterans find and maintain employment.

The Secretary of Veterans Affairs should direct the Under Secretary for 

Benefits to take the following actions:

1. In any revised set of national and regional performance measures for 

the VR&E program, include measures of (a) the proportion of program 

participants successfully rehabilitated into employment, and (b) the 

proportion of participants who obtain other benefits from VR&E 

services.

2. Develop new measures of long-term employment that go beyond the 

minimum 60 days of post-placement monitoring that is currently 

required. In developing measures, consider the feasibility of using 

results from planned post-closure surveys of veterans as a data 

source.

3. Consistent with generally accepted survey practices and as warranted 

by survey response rates, conduct nonresponse analysis of the 

results of VA’s ongoing Voice of the Veteran customer satisfaction 

surveys.

4. In revisiting VA’s formula for allocating VR&E staff among the regional 

offices, (a) assess the inclusion of factors related to regional office 

performance and if warranted remove them from the formula, and (b) 

assess the exclusion of any factor related to the number of 

educational counseling cases in each regional office and If warranted 

add such a factor.

5. Collect information on the regional offices’ approaches for managing 

their VR&E workloads, assess the advantages and disadvantages of 

Recommendations for 

Executive Action
Page 36 GAO-14-61 VA Vocational Rehabilitation and Employment

these approaches, and use the results of this assessment to provide 

guidance to the offices on potential best practices or options to 

consider.

6. Provide additional training to all individual VRCs on job placement 

strategies and workplace accommodations, potentially as part of the 

effort to develop a competency-based training approach.

We provided a draft of this report to VA for review and comment. In its 

written comments, produced as appendix IV in this report, VA agreed with 

our conclusions and generally concurred with our recommendations. 

Specifically, VA concurred with 5 of our 6 recommendations, and noted 

steps it plans to take to address them, as follows. 

• Regarding our recommendation to measure the proportions of 

VR&E participants achieving employment and other benefits, VA 

said that while ratio-based metrics will not be included in its 

revised employee-level performance measures, they will be 

included in the revised national-level measures. 

• Regarding our recommendation to conduct nonresponse bias 

analysis of its Voice of the Veteran survey results, VA said that—if 

funding permits—it will modify its survey administration contract to 

include such an analysis.

• With regard to our recommendation to assess certain aspects of 

its formula for allocating staffing levels across regional offices, VA 

said it plans to revisit the formula and make any changes 

necessary.

• With regard to our recommendation to assess regional offices’ 

workload management approaches, VA said it plans to collect and 

analyze data on the various approaches and inform the regions 

about best practices.

• With regard to our recommendation to close certain gaps in staff 

training, VA said it plans to develop and make available to staff 

new training on employment and accommodations. 

VA agreed in principle with our recommendation to develop performance 

measures related to VR&E participants’ longer-term employment 

success. Specifically, VA agreed there is value in looking at long-term 

outcomes, and noted its ongoing longitudinal study of a sample of 

participants and its plans to collect data on veterans’ post-closure service 

needs. However, VA also pointed out that VR&E counselors already work 

with veterans for many years, including up to 18 months following training

completion. As such, VA indicated it does not see the value in directing 

limited resources to implementing and executing additional postplacement measures or services that will extend program participation 

Agency Comments 

and Our Evaluation
Page 37 GAO-14-61 VA Vocational Rehabilitation and Employment

and may create program dependencies. We acknowledge that VA in 

some cases monitors veterans for 6 months or more after they are placed 

in a job (as we note on page 24 of the report, this occurred in over onethird of the cases we examined). We also recognize the potential 

challenges of additional post-employment monitoring. At the same time, 

we believe that unless VA holds itself accountable for the longer-term 

outcomes of all participants who get hired into a suitable job, it may miss 

opportunities to make adjustments that better promote job maintenance. 

We encourage VA to explore cost-effective means of checking on 

veterans’ employment status at some point more than 60 days after job 

placement. We did not intend and do not believe this measure would 

necessarily entail extending program services or participation beyond 

current program practices. VA also provided technical comments, which 

we incorporated in our report as appropriate. 

We are sending copies of this report to appropriate congressional 

committees, the Secretary of Veterans Affairs, and other interested 

parties. In addition, this report will also be available at no charge on 

GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-7215 or bertonid@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff members who made key 

contributions to this report are listed in appendix V.

Daniel Bertoni

Director, Education, Workforce and Income Security Issues
Appendix I: Objectives, Scope, and 

Methodology

Page 38 GAO-14-61 VA Vocational Rehabilitation and Employment

The objectives of this report were to examine (1) the outcomes for 

veterans seeking employment through the Vocational Rehabilitation and 

Employment (VR&E) program, and (2) the progress the Department of 

Veterans Affairs (VA) has made in addressing critical management 

issues. To address these objectives, we reviewed relevant federal laws, 

regulations, and guidance, as well as VA documentation related to the 

VR&E program and recent management initiatives. We also reviewed 

several external studies and evaluations of the VR&E program, and 

determined that their methodologies were sufficient to allow us to report 

certain findings from these studies. We reviewed various criteria for 

effective program management, including in the areas of performance 

management and staff training. We analyzed VA administrative data on 

VR&E participants who applied to the program between fiscal years 2003 

and 2012 (see below for more information on our efforts to analyze and 

ensure the reliability of these data), and interviewed VA central office staff 

responsible for the VR&E program, VR&E managers and staff in VA 

regional benefits offices (see below for details on office selection), and 

representatives of the Disabled American Veterans and the Paralyzed 

Veterans of America. Finally, to obtain input from veterans, we conducted 

phone interviews with veterans who had participated in the VR&E 

program (see below for details on how we developed our survey and 

selected veterans) and analyzed data from a VA customer satisfaction 

survey, after reviewing the methods VA used to develop and pilot the 

survey (see below for more detail on our assessment of the survey).

We analyzed data from VA’s Corporate WINRS (CWINRS) 1 system, 

which VA uses to track veteran cases through the VR&E process and 

manage program costs.2

 1 The “WINRS” part of the acronym represents the first five regional offices that tested the 

original system. These offices include Waco, Texas; Indianapolis, Indiana; Newark, New 

Jersey; Roanoke, Virginia; and Seattle, Washington.

In our analysis, we included veterans who 

initially applied to the VR&E program between fiscal years 2003—the 

earliest year for which reliable data are available—and 2012. Unless 

otherwise noted, we included only veterans who received VR&E services 

designed to help veterans prepare for and obtain suitable jobs, and not 

those who received other services—such as educational and vocational 

2 In addition to CWINRS data, we also analyzed data on the specific type of training 

veterans received from VA’s Benefits Delivery Network and data on regional office staffing 

from VA’s ProClarity system.

Appendix I: Objectives, Scope, and 

Methodology

Analysis of VA 

Administrative Data
Appendix I: Objectives, Scope, and 

Methodology

Page 39 GAO-14-61 VA Vocational Rehabilitation and Employment

counseling—which are also provided by VR&E staff. We analyzed data 

on the characteristics of VR&E participants (e.g., age, type of disability); 

the services they received from the VR&E program (e.g., evaluation, 

training); and their outcomes.

To assess outcomes, we placed VR&E participants who received 

employment-related services into one of four outcome categories: (1) 

rehabilitated to employment, including those who at least once had a 

case status of rehabilitated associated with achieving employment; (2) 

rehabilitated to independent living, including those who never had an 

employment-related rehabilitation but had least one rehabilitated status 

associated with achieving an independent living goal; (3) discontinued, 

including those who never had any rehabilitated status, and whose last 

status as of the end of fiscal year 2012 was discontinued;3

We found the CWINRS data to be sufficiently reliable for the purposes of 

our report. To assess the reliability of the data, we reviewed 

documentation on the system, interviewed VA officials, and performed 

electronic testing. VA officials told us that CWINRS became a single, 

national database during 2002, and that prior data are from separate, not 

necessarily uniform, systems operated by individual regional offices and 

are therefore less reliable. Based on this information, we chose to 

analyze data only for fiscal years 2003 through 2012. Also, through 

electronic testing we identified potential errors with outcome data for 1 

percent of cases that received some employment-related services—for 

example, cases closed as rehabilitations for reasons which could not 

logically be associated with a rehabilitation. Because this made it 

impossible for us to assign these cases to one of our four outcome 

categories, we excluded them from our analysis of veterans’ outcomes. 

Finally, VA officials told us there may be some inaccuracies in data 

related to the status of cases—in particular for data prior to fiscal year 

2008—because of a programming design flaw that allowed correct case 

and (4) still 

participating, including those who never had a rehabilitated status and 

whose last status as of the end of fiscal year 2012 indicated some type of 

ongoing participation in the program. (See app. II for a description and 

detailed results of our analysis of the association between VR&E 

participants’ outcomes and their characteristics and services received.)

 3 The discontinued category also includes veterans who started employment-related 

services but were later found not eligible.
Appendix I: Objectives, Scope, and 

Methodology

Page 40 GAO-14-61 VA Vocational Rehabilitation and Employment

status data in CWINRS to be inadvertently overridden with obsolete data 

from another VA system. Therefore, we use approximations rather than 

exact numbers when reporting the number of separate occasions on 

which veterans received certain services through the VR&E program, 

such as evaluation and planning assistance, job ready services, and 

rehabilitation to employment.

To obtain perspectives on the challenges VR&E staff face in helping to 

rehabilitate veterans, we interviewed VR&E managers and staffs at 8 of 

VA’s 57 regional offices.4

• Geography: We selected at least one office in each of the four 

Veterans Benefits Administration areas.

We selected offices to achieve diversity among 

offices in the following characteristics:

• Workload: Based on VR&E administrative data, we ranked offices by 

the number of initial fiscal year 2011 VR&E applications filed from 

largest to smallest numbers of applications.

• Outcomes: Based on VA’s ASPIRE VR&E performance dashboard,

we ranked offices by their Employment Rehabilitation Rate5

• Timeliness: Based on VA’s ASPIRE VR&E performance dashboard, 

we ranked offices according to the average days required to complete 

the evaluation and planning phase during September 2012 from 

lowest to highest.

in 

September 2012 from highest to lowest percentage rate.

• Unemployment: We considered the November 2012 unemployment 

rate for each office’s state, as reported by the Bureau of Labor 

Statistics.

See figure 8 for details about the characteristics of each regional office 

we contacted.

 4 VA renamed the Washington, D.C., regional office as the National Capital Region 

Benefits Office in November 2012. For purposes of our review, we refer to the National 

Capital Region Benefits Office as a regional office. 

5 The Employment Rehabilitation Rate differs from VR&E’s general Rehabilitation Rate in 

that it does not include veterans placed in the Independent Living Program.

Regional Office Interviews
Appendix I: Objectives, Scope, and 

Methodology

Page 41 GAO-14-61 VA Vocational Rehabilitation and Employment

Figure 8: Regional Offices Contacted and Their Characteristics

Note: Rankings are out of 58 offices for which VA reported data. VA reported data separately for the 

Cheyenne, Wyoming, office, which is considered part of the Denver regional office.

a

We used state unemployment rates as a proxy for unemployment in a regional office’s jurisdiction. 

Some regional offices have jurisdictions covering part of a state; for example, 4 states have more 

than one office each. The national unemployment rate for November 2012 was 7.7 percent, according 

to the Bureau of Labor Statistics.
Appendix I: Objectives, Scope, and 

Methodology

Page 42 GAO-14-61 VA Vocational Rehabilitation and Employment

We visited the regional offices in Washington, D.C., and Oakland, 

California, and conducted telephone interviews with VR&E managers and 

staff at the other six offices. At each of the eight offices, we interviewed 

the VR&E Officer (and one or more Assistant VR&E Officers at four 

offices); at least three VRCs; and at least one EC.6 At our in-person visits 

to Washington and Oakland, we also interviewed representatives of the 

Disabled American Veterans and California Department of Veterans 

Affairs. Our interview results cannot be generalized to all regional offices’ 

VR&E operations.

We conducted phone interviews with 17 veterans who had participated in 

the VR&E program, including 9 who had discontinued from the program 

and 8 who had achieved rehabilitation to employment. We pretested our 

interview questions twice with each group, and used the results of the 

pretests in our final analysis. We asked questions about the veterans’ 

employment, their interactions with their VR&E counselors, and their 

satisfaction with various VR&E services received. We randomly selected 

veterans for interviews, but cannot generalize the results of the interviews 

to the overall population of VR&E participants. To select our sample of 

veterans, we used CWINRS data to identify VR&E participants who had 

received some employment-related services and had either discontinued 

or achieved an employment-related rehabilitation between April 1, 2012 

and September 30, 2012. We attempted to contact 49 discontinued 

veterans and 32 rehabilitated veterans after sending an advance 

informational email. These contact attempts resulted in complete 

interviews with 17 veterans.

We obtained aggregate results from the pilot of VA’s Voice of the Veteran 

customer satisfaction survey directly though VA’s dashboard—a central 

database that contains the agency’s performance metrics and customer 

satisfaction reports. This survey was administered to random samples of 

veterans from different customer populations. Each sample received a 

questionnaire reflecting the VA benefits and services they received. We 

obtained data from the three questionnaires targeted to VR&E 

participants: the Enrollment survey, the Servicing survey, and the 

 6 We interviewed the VR&E Officer at the Boston, Massachusetts regional office, who 

manages VR&E operations at the Manchester, New Hampshire regional office. Also, 

Manchester did not have an EC at the time of our review.

Interviews with Veterans

Analysis of Voice of the 

Veteran Survey
Appendix I: Objectives, Scope, and 

Methodology

Page 43 GAO-14-61 VA Vocational Rehabilitation and Employment

Escaped Beneficiary survey. All three VR&E pilot surveys were fielded 

from October 2012 to January 2013. Table 2 provides background 

information on each of these surveys, including a description of the 

survey’s purpose and target sample population.

Table 2: Description of Vocational Rehabilitation and Employment’s Voice of the Veteran Pilot Surveys

VR&E pilot survey Survey purpose Target population

Enrollment survey Obtained veterans’ views on benefit 

eligibility and application process, benefit 

entitlement, benefit information, and VA 

personnel.

Veterans who had an initial meeting with their VR&E counselor 

and were granted a decision regarding their entitlement in the 

60 days before the survey field period. This includes those 

who (1) apply and attend the initial appointment to the program 

and pursue (entitled), (2) apply and attend the initial 

appointment to the program and do not pursue (entitled), and 

(3) apply and attend the initial appointment to the program (not 

entitled).

Servicing survey Obtained veterans’ views on benefit 

entitlement, benefit information, and VA 

personnel. 

Veterans who have entered and been enrolled in one of the 

five tracks for at least 60 days before the survey field period. 

This may include veterans who (1) are currently participating, 

(2) been rehabilitated, (3) did not fully complete the program, 

and (4) have reached maximum rehabilitation gain and could 

not proceed in the program. 

Escaped Beneficiary 

survey

Obtained veterans’ views on benefit 

entitlement, benefit information, VA 

personnel, and the reasons for deciding not 

to complete the benefit application process 

and continue to pursue the benefit. 

Veterans who dropped out of the program prior to completing 

a rehabilitation plan. This may include applicants who (1) 

never attended the initial meeting with a counselor, (2) were 

determined to be entitled and did not complete a rehabilitation 

plan, and (3) started but did not complete rehabilitation.

Source: VA survey documentation.

Based on interviews with agency officials and contract representatives for 

the Voice of the Veteran surveys, and our review of documentation of the 

surveys’ methodology, we concluded that these three VR&E surveys 

were generally developed and piloted in accordance with generally 

recognized survey research practices, and that the results we cite in this 

report are sufficiently reliable for our analysis. We used standards and 

guidelines issued by the Office of Management and Budget (OMB) to 

assess the methods VA used to design and administer the surveys.7

 7 See Office of Management and Budget, Standards and Guidelines for Statistical Surveys

(September 2006).

In 

addition, we reviewed survey guidelines of generally recognized 
Appendix I: Objectives, Scope, and 

Methodology

Page 44 GAO-14-61 VA Vocational Rehabilitation and Employment

professional organizations such as the American Association for Public 

Opinion Research and the American Statistical Association.8

However, we concluded that the pilot survey results cited should be 

interpreted as representative of only those veterans responding to the 

survey. Response rates for the VR&E surveys were low—between 16

percent and 25 percent—and fell below the expected 30 percent rates 

specified in VA’s sample design. Table 3 provides more information on 

sample outcomes. While response rates may not be highly correlated with 

the presence of nonresponse bias and are not considered, by 

themselves, to be sufficient indicators of the quality of survey results, 

response rates are a useful indicator of the risk of nonresponse bias. 

According to OMB’s guidelines, a survey’s response rate is a valuable 

data quality and performance indicator, and is associated with the 

generalizability of a survey’s results. In addition, OMB’s guidelines state 

that a nonresponse analysis should be performed when a survey’s unit 

response rate is below 80 percent, to identify the possibility of bias in a 

survey’s results, and that agencies should appropriately measure, adjust 

for, report, and analyze nonresponse to assess its effect on data quality. 

Because VA did not perform a nonresponse analysis to determine if 

systematic differences exist between respondents and nonrespondents, it 

is not possible to generalize the results to the broader population of 

VR&E participants. Also, because the effects of any nonresponse bias on 

data quality were not assessed, it is not known if any systematic 

differences exist between nonrespondents and respondents.

Table 3: Sample Outcomes for Vocational Rehabilitation and Employment’s Voice of the Veteran Pilot Surveys

Veteran 

population 

(annual)

Veterans 

sampled

Target number 

of survey 

completions

Usable surveys 

completed a

Proportion 

of target 

completed

Response 

rate b

Enrollment survey

c

77,998 4,599 3,000 939 31% 20%

Servicing survey 121,722 8,781 3,000 2,161 72% 25%

Escaped beneficiary survey 11,700 5,000 1,500 817 54% 16% 

Source: VA Voice of the Veteran Survey documentation.

 

8 See the American Association for Public Opinion Research, 2011 Standard Definitions: 

Final Dispositions of Case Codes and Outcome Rates for Surveys 7th edition; and the 

American Statistical Association, What is a Survey, 2nd edition (2004), 

https://www.whatisasurvey.info/. 
Appendix I: Objectives, Scope, and 

Methodology

Page 45 GAO-14-61 VA Vocational Rehabilitation and Employment

a

The “target number of survey completions” was identified in VA’s initial sample design, and was 

based on VA’s expected response rates.

b

The “proportion of target completed” was calculated by dividing the number of “usable surveys 

completed” by the “target number of survey completions.” 

c

The “response rate” was calculated by dividing the number of “usable surveys completed” by the 

number of “veterans sampled.” 
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 46 GAO-14-61 VA Vocational Rehabilitation and Employment

We used a statistical method known as survival analysis to estimate 

Vocational Rehabilitation and Employment (VR&E) participants’ likelihood 

of achieving success, their program completion times, and the factors that 

were associated with rehabilitation. Survival modeling methods provide a 

simple and efficient way to estimate the rate of event occurrence when 

the duration times of the events are “censored,” meaning that they are 

unobserved over some portion of the time period of interest. For example, 

a rehabilitation time would be censored if a veteran had been in a 

program for 2 years but had neither dropped out nor finished the program 

at the time of analysis. Although the veteran’s rehabilitation time is at 

least 2 years, the ultimate time to rehabilitation or discontinuation is not 

yet known (censored). We used a specific type of survival method, Cox 

regression models, to identify predictors of program completion—or what 

the Department of Veterans Affairs (VA) defines as “rehabilitation”—and 

estimate the rate at which rehabilitation occurred at various times after 

application among various groups of veterans.

Cox models allowed us to analyze the data without needing to make 

strong assumptions about the rehabilitation process. We did not need to 

assume that the likelihood or rate of rehabilitation changed over time in 

specific ways, as we would with other survival methods. The flexibility of 

Cox models comes at some cost of interpretation, however. The 

mathematics of Cox models make it more difficult to estimate the 

probability of rehabilitation by certain follow-up times (known as “failure 

probabilities”). To avoid the complex computation required for these 

estimates, we generally report simpler “Kaplan-Meier” estimates of 

rehabilitation probabilities in the body of the report—a form of survival 

analyses that allows us to report on likelihood of program completion, but 

without controlling for other covariates. To corroborate these results, we 

use Cox models to estimate the “hazard rate” of rehabilitation in this 

appendix, which is loosely equal to the rate of rehabilitation at a certain 

time after starting a VR&E program, while controlling for other factors. 

The results of this analysis are presented in this appendix and essentially 

support the same substantive findings about factors associated with the 

chance of rehabilitation discussed in the body of this report.

For the survival analysis using Cox models and Kaplan-Meier methods, 

we used data from the VR&E program’s Corporate WINRS system, which 

includes archived data on program participants’ demographic 

characteristics, services received, and outcomes achieved. We selected 

data on veterans who began their first VR&E programs in fiscal year 2003 

through fiscal year 2012. We limited our analysis to this time period after 

Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Source Data and 

Measurement of Key 

Variables
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 47 GAO-14-61 VA Vocational Rehabilitation and Employment

VA identified reliability problems with older data. Appendix I describes the 

source data and their reliability in more detail.

In initial exploratory analysis, we considered various factors that could be 

associated with successful program completion and duration. These 

factors (which we also may refer to as covariates, variables, or predictors) 

included age, sex, education, branch of service, number of counselors, 

number of discontinuations, severity of disability, type of disability, year of 

first application to the program, VR&E services provided (“service track”), 

and the regional offices that managed veterans’ cases.

A key strength of the VR&E administrative data is its volume. The total 

number of veterans available for analysis, using the population defined 

above, was 133,679. Table 4, below, shows how the data were 

distributed across years of program entry and program outcomes. The 

large sample size available for analysis allowed us to estimate valid and 

precise relationships between completion times and a large number of 

covariates. We confirmed the adequacy of the sample by examining the 

cell sample sizes in tabulations of potential covariates.

Table 4: Sample Size by Fiscal Year of Program Entry and Outcomes

N

Percent of 

Sample

Fiscal Year of First Application

2003 16,726 12.5

2004 14,587 10.9

2005 14,653 11.0

2006 13,636 10.2

2007 14,667 11.0

2008 16,078 12.0

2009 14,490 10.8

2010 11,264 8.4

2011 10,220 7.7

2012 7,358 5.5

Program Outcome

Rehabilitated to employment 38,664 28.9

Censored

Program in progress 68,462 51.2

Discontinued 23,271 17.4
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 48 GAO-14-61 VA Vocational Rehabilitation and Employment

N

Percent of 

Sample

Transferred to Independent Living track and 

achieved rehabilitation

1,351 1.0

Outcome not recorded 1,931 1.4

Total 133,679 100.0

Source: GAO analysis of VR&E administrative data. 

Note: All percentages do not add to 100 due to rounding.

We assessed the degree of correlation among the initial list of covariates, 

in order to identify those that would be redundant or introduce the 

possibility of spurious estimates. Most of the potential covariates were not 

strongly intercorrelated, except for the number of counselors and 

discontinuations, the number of counselors and regional offices, and age 

and era of service. The pair-wise correlations for these covariates ranged 

from 0.37 to 0.52. Including all of these covariates in our model would 

have inflated the estimated relationships between them and program 

completion times. Therefore, we included age and number of counselors 

in the final list of covariates, and excluded era of service, number of 

discontinuations, and number of offices from our model.

Measuring most of the covariates of interest was straightforward, and can 

be determined for a subset of meaningful predictors, as shown in Table 6 

below. However, measuring certain covariates—such as regional offices, 

certain VR&E services, and disability type—required more complex 

measurement decisions.

Because veterans can be served by a number of regional offices over 

time, measuring the offices serving each veteran is challenging. In our 

data, veterans’ cases were managed by more than 4,100 permutations of 

regional offices. The large number of permutations made it difficult to 

meaningfully analyze variation across offices, and, in any case, produced 

sample sizes that were too small for modeling purposes. To simplify the 

measurement, we focused on the last office where each veteran was 

enrolled, as a measure of regional office at program completion. Because 

veterans could have discontinued their programs multiple times and could 

have been served by as many as six offices, the last office was a logical 

proxy for the office that could have influenced the final outcome most 

strongly.

Because we selected only veterans from the VA administrative data who 

received employment-related services, this group received job training or 
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 49 GAO-14-61 VA Vocational Rehabilitation and Employment

job placement assistance. We created a single measure of services 

received by creating categories for specific services related to 

employment, such as job placement services, and specific types of job 

training received, such as high school or college education. Table 5 below 

identifies the services we measured.

Finally, veterans could have been diagnosed with one or more disabilities, 

such as mental health conditions, ear disorders, and/or cardiovascular 

disorders. This allows for a potentially large number of unique 

combinations of disorders, which makes it difficult to create mutually 

exclusive and exhaustive groups. To avoid this problem, we compared 

veterans who had each type of disorder to veterans who did not have the 

disorder, using binary indicators for the presence or absence of each 

diagnosis. While the possibility of multiple disabilities makes these 

indicators mutually inclusive, our measurement scheme provided a 

concise way to collapse the many unique combinations of disorders.

The purpose of survival modeling is to estimate the probability of an event 

by various follow-up times or the average rate of event occurrence at any 

one follow-up time. Because these two quantities are mathematically 

equivalent, we used two different methods that produce similar results but 

vary in their ease of interpretation.

In the body of this report, we provide Kaplan-Meier estimates of the 

probability of rehabilitation by 8 years after the time of first application, 

known as a “failure” probability in survival analysis. Let tj denote one of j = 

1, 2, ..., k rehabilitation times observed in the data. Then the KaplanMeier failure estimate at tj is given by

F(t) = 1 − �∏ nj− dj

nj j:tj<t � ,

where nj is the number of veterans who are “at risk” of rehabilitation and 

dj is the number of rehabilitations observed at tj. We estimated failure 

probabilities for various subgroups of veterans, in order to describe the 

association between these factors and the chance of rehabilitation by 5 

and 8 years.

To corroborate these associations and control for alternative variables 

that could account for them, we used multivariate survival models—and 

the Cox Model for Proportional Hazards, in particular—to estimate the 

average rate of event occurrence at any one time in an observation 

Survival Model and 

Statistical Assumptions
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 50 GAO-14-61 VA Vocational Rehabilitation and Employment

period, also known as the “hazard rate.”1

The formula for our Cox regression model was

In our analysis, the event of 

interest is rehabilitation, and the time period of observation is fiscal year 

2003 through fiscal year 2012. In our analysis, we defined censored 

outcomes as a veteran’s remaining in the program at the end of fiscal 

year 2012 or having discontinued from a program without yet re-applying 

(see table 5 above).

λi(t)= λ0(t)exp(xi b) ,

where λi(t, xi) is the hazard rate for veteran i at time t, xi = xi1 ,..., xip is a 

vector of p covariates, and xib is a linear combination of covariates and 

coefficients, with no intercept term. As described above and in table 6, the 

covariates included: age, sex, job training services, the interaction of sex 

and job training services, education, branch of service, number of 

counselors, severity of disability, and disability type. The term λ0(t) is the 

“baseline hazard rate,” or the rate implied when all covariates equal zero. 

(Note that the equation simplifies to λ0(t) when this is true.)

The Cox Model assumes that the hazard rates are proportional across 

values of the covariates throughout the observation period. If the hazard 

in one group exceeds that of a reference group, the model assumes that 

the proportional difference in rates will remain constant, and that the 

difference in hazard rates across groups does not decline, increase, or 

reverse direction over time. For example, the model assumes that the 

rehabilitation rate for veterans with mental health conditions differs from 

the rate for veterans without this type of condition by a constant amount 

throughout the observation period.

However, because a particular data generation process may not satisfy 

the proportional hazards assumption, we plotted the estimated hazards 

for various values of each covariate over time and formally tested the 

assumption for each covariate using statistical tests based on the model’s 

Shoenfeld residuals. We found a non-zero slope in a generalized linear 

regression of the Schoenfeld residuals on functions of time for several 

variables. A non-zero slope indicates that the hazards are not 

 1 For additional information on survival modeling methods, please see Paul D. Allison, 

Survival Analysis Using SAS®: A Practical Guide, 2d ed. (Cary, NC: SAS Institute, Inc., 

2010).
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 51 GAO-14-61 VA Vocational Rehabilitation and Employment

proportional across values of the covariates over time.2

To identify the most serious departures from the proportional hazard 

assumption, we plotted the cumulative hazard function over time for the 

covariates that failed the Schoenfeld test. These plots suggested that the 

last office and first year of application covariates had strongly 

disproportional hazards. The remaining variables only failed to meet the 

proportional hazard assumption in a category coded for “missing data,” 

and the other variables that appeared to be disproportional were not 

meaningfully associated with rehabilitation times.

A global test 

across all variables further suggested that the hazard function was not 

proportional.

To adjust for most seriously non-proportional hazards, we stratified the 

model by last office and first year of application. Stratification allowed us 

to control for these two factors while also allowing their hazard functions 

to vary over time across offices and years of application. After stratifying 

by last office and first year of application, the proportional hazard 

assumption for the model was more consistent with our data.

Our analysis identified a number of factors that had strong and 

statistically significant associations with rehabilitation times, as shown in 

Table 5.

Veterans took more time to complete their programs if multiple counselors 

worked with them. Average completion times increased at a higher rate 

as the number of counselors increased from 1 to 5 than from 6 to 9 or 

more. For example, veterans with 2 counselors had average rehabilitation 

times that were 29 percent slower, on average, than veterans with 1 

counselor. In contrast, veterans with 9 or more counselors had average 

rehabilitation times that were about 1 percent slower than veterans with 8 

counselors. In exploratory analysis, we found that veterans who more 

often discontinued and re-entered the program and who were served by 

multiple regional offices were more likely to work with multiple counselors, 

and that discontinuing more frequently was associated with slower rates 

of rehabilitation. However, when we estimated Cox models that controlled 

for the number of times discontinued and the number of regional offices 

 2 Allison, 175-176.

Results
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 52 GAO-14-61 VA Vocational Rehabilitation and Employment

used, we found that the number of counselors still had a meaningful 

association with success. Nevertheless, a reciprocal relationship, in which 

longer program times are associated with more counselors, may also 

explain this pattern.

The presence of mental health conditions substantially decreased 

completion rates. Veterans with mental health conditions completed their 

programs at a rate that was about 27 percent slower on average than 

veterans who did not have mental health conditions.3

As expected, the type of services veterans received was among the 

strongest predictors of completion times. Veterans who received job 

placement services finished their programs about 4.5 times more quickly 

than veterans who were pursuing an undergraduate education. This is 

consistent with the nature of VR&E’s job placement services, which focus 

on veterans who already have the skills to pursue employment. Veterans 

who received apprenticeships, non-vocational training, unpaid work at a 

federal agency, vocational or technical education, or non-degree college 

credit hours finished their programs about 1.6 to 2.4 times more quickly 

than veterans pursuing an undergraduate college degree. In contrast, the 

average completion rate for veterans pursuing a graduate degree was 

only 28 percent quicker than for veterans pursuing an undergraduate 

degree. As a whole, these results suggest that veterans pursuing 

undergraduate or graduate degrees require more time to complete their 

programs than veterans receiving other training.

The amount of prior education was associated with faster program 

completion rates. As compared to veterans with only a high school 

degree, veterans with some college, an undergraduate degree, or a 

graduate degree had completion rates that were 1.2 to 1.5 times faster, 

on average. Because these estimates control for the type of services 

received and the type and severity of disabilities, veterans with more prior 

education appear to have finished more quickly regardless of the training 

they received or the nature of their disabilities.

Finally, veterans with disability ratings of less than 100 percent finished 

their programs more quickly than veterans with disability ratings of 100 

percent, with completion rates that were about 1.5 times more quickly on 

 3 The 95 percent confidence interval of this hazard ratio ranged from 0.72 to 0.75.
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 53 GAO-14-61 VA Vocational Rehabilitation and Employment

average.4

Table 5: Estimated Hazard Ratios of Time to Vocational Rehabilitation and 

Employment Program Completion

This pattern may reflect the fact that veterans with more severe 

disabilities face greater challenges to timely program completion, even 

holding constant factors such as type of training, prior education, and 

age.

Variable Hazard Ratio

95% 

Lower 

Bound

95% 

Upper 

Bound

Prior Education

High School 1.000 

(Referent)

Less than High School 0.923 0.812 c 1.05

Some College 1.230 1.200 c 1.261

College 1.449 1.394 d 1.506

Graduate 1.461 1.330 c 1.606

Missing Data 2.585 2.503 c 2.669

Number of Case Managers

1 1.000 

(Referent)

2 0.712 0.677 c 0.748

3 0.471 0.447 c 0.495

4 0.336 0.318 c 0.354

5 0.230 0.216 c 0.244

6 0.159 0.147 c 0.172

7 0.115 0.102 c 0.130

8 0.088 0.071 c 0.109

9+ 0.073 0.054 c 0.010

Type of Services Receiveda

Undergraduate 1.000 

(Referent)

Job Placement Only 4.463 4.263 c 4.673

Missing 1.079 1.012 e 1.150

Graduate School 1.282 1.218 c 1.349

College, Non-Degree, Credit Hour 1.679 1.576 c 1.789

 4 The 95 percent confidence interval of this hazard ratio ranges from 1.4 to 1.6.
Appendix II: Statistical Analysis of 

Administrative Data on Vocational 

Rehabilitation and Employment Outcomes

Page 54 GAO-14-61 VA Vocational Rehabilitation and Employment

Variable Hazard Ratio

95% 

Lower 

Bound

95% 

Upper 

Bound

Non-standard degree and/or voc/tech 2.376 2.291 c 2.463

Nonpaid work fed agency 1.738 1.564 c 1.930

Other: HS/Farm, Special Rehab 1.523 1.187 d 1.955

OJT 3.028 2.671 c 3.433

Apprenticeship 1.635 1.309 c 2.043

Non-Vocational 1.800 1.642 c 1.973

Unpaid OJT, fe, home, facility, shelter 

etc.

1.708 1.442 c 2.024

Type of Disabilityb

Auditory 1.104 1.081 c 1.129

Cardiovascular 1.048 1.023 c 1.073

Gynecological 1.089 1.043 c 1.137

Infectious/Immune 

Disease

0.941 0.893 e 0.990

Mental health / PTSD 0.731 0.716 c 0.747

Neurological 0.948 0.928 c 0.968

Respiratory 1.075 1.052 d 1.099

Skin 1.073 1.050 d 1.096

Severity of Disability

Full Disability (100 percent) 1.000 

(Referent)

Partial Disability (0-90 percent) 1.492 1.419 c 1.570

PT/Missing 0.760 0.682 c 0.848

Source: GAO analysis of VR&E administrative data

Note: Entries are exponentiated coefficients and 95 percent confidence intervals from a Cox 

proportional hazard model of VR&E program completion, treating rehabilitation to employment as the 

outcome of interest and all other events as censored. The model included additional covariates not 

listed in this table, as described in the body of this appendix. The hazard function was stratified by 

regional office and year of first application. N = 132,042

a

Estimates are summarized for male veterans, due to an interaction term in the model for sex and 

type of services received. The results reported here for men are substantially similar to the results for 

both sexes when the interaction term is excluded.

b

Estimates apply to the presence or absence of each disorder. Because veterans can have one or 

more disorders, and many combinations of disorders are possible, we did not use mutually exclusive 

and exhaustive categories.

c p<0.001

d p<0.01

e p<0.05
Appendix III: Vocational Rehabilitation 

Counselor Training Gaps Identified by VA in 

2010 and the Agency’s Efforts to Address

Page 55 GAO-14-61 VA Vocational Rehabilitation and Employment

Training gaps Courses 

Fiscal year course 

provided

New VRCs

How to provide services to veterans recently 

discharged from the military

How to provide services to veterans with severe 

disabilities

How to determine when extended evaluation is 

necessary and conduct subsequent individualized 

extended evaluation plan development

How to closeout cases for different veteran outcomes 

(e.g., for veterans who achieved suitable employment 

or maximum rehabilitation gain)

40-hour training course 2013

For new and experienced VRCs 

How to provide services to veterans with traumatic 

brain injuries (TBI)

1-hour course on mild TBI 2012

2-hour course on moderate to severe TBI 2013

2-hour course on vocational implications of TBI 2013 

How to provide services to veterans with posttraumatic stress disorder (PTSD)1.5 hour course on PTSD: The VR&E Counselor's Role 2012

2-hour course on mental health awareness 2014 a

3-hour course on mental health techniques 2014 a

How to develop an independent living (IL) plan 1-hour course on IL (plan) construction 2012

1-hour course on IL program guidance 2013 b

How to conduct a preliminary and/or comprehensive IL 

assessment

15-hour course on IL 2014

How to input benefit awards into the computer system 

(e.g., inputting the decision to provide a veteran a 

subsistence allowance)

c

2-hour course on VR&E fiscal accuracy and integrity

Two 1-hour courses on VR&E processing of 

subsistence allowance awards

d

1-hour course on employment adjustment allowances

d

2012

d

Source: GAO review of VA-sponsored June 2010 skill gap analysis and additional information provided by VA.

a

The new courses on mental health will address PTSD and other mental illnesses and are intended to 

help increase successful VR&E outcomes for veterans with these conditions.

b

VA added this course to the fiscal year 2013 curriculum to address changes to its guidance for 

independent living and delivered the training in September, 2013.

c

VA has developed the course and will assign it as part of the fiscal year 2014 curriculum to be 

completed by the end of the first fiscal quarter (i.e., by December 31, 2013).

d

Appendix III: Vocational Rehabilitation 

Counselor Training Gaps Identified by VA in 

2010 and the Agency’s Efforts to Address 

VA used existing courses to train VR&E staff on inputting benefit awards into its computer system.
Appendix IV: Comments from the Department 

of Veterans Affairs

Page 56 GAO-14-61 VA Vocational Rehabilitation and Employment

Appendix IV: Comments from the 

Department of Veterans Affairs
Appendix IV: Comments from the Department

of Veterans Affairs

Page 57 GAO-14-61 VA Vocational Rehabilitation and Employment
Appendix IV: Comments from the Department 

of Veterans Affairs

Page 58 GAO-14-61 VA Vocational Rehabilitation and Employment
Appendix IV: Comments from the Department 

of Veterans Affairs

Page 59 GAO-14-61 VA Vocational Rehabilitation and Employment
Appendix V: GAO Contact and Staff 

Acknowledgments

Page 60 GAO-14-61 VA Vocational Rehabilitation and Employment

Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov

In addition to the contact listed above, individuals making key 

contributions to this report were Michele Grgich (Assistant Director), 

Jessica Botsford, Frederick Caison, Elizabeth Curda, Kevin Daly, Holly 

Dye, Julie DeVault, Brenda Farrell, Alex Galuten, Shannon Grabich, Mitch 

Karpman, Kirsten B. Lauber, Lorin Obler, Carl Ramirez, Almeta Spencer, 

Jeff Tessin, Greg Whitney, and Ashanta Williams.

Appendix V: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments
Related GAO Products

Page 61 GAO-14-61 VA Vocational Rehabilitation and Employment

VA Vocational Rehabilitation and Employment Program: Improved 

Oversight of Independent Living Services and Supports Is Needed.

GAO-13-474. Washington, D.C.: June 7, 2013.

Veterans’ Employment and Training: Better Targeting, Coordinating, and 

Reporting Needed to Enhance Program Effectiveness. GAO-13-29. 

Washington, D.C.: December 13, 2012.

Federal Information System Controls Audit Manual (FISCAM). 

GAO-09-232G. Washington, D.C.: February 2009.

VA Vocational Rehabilitation and Employment: Better Incentives, 

Workforce Planning, and Performance Reporting Could Improve 

Program. GAO-09-34. Washington, D.C.: January 26, 2009.

Multiple Agencies Provide Assistance to Service-disabled Veterans or 

Entrepreneurs, but Specific Needs Are Difficult to Identify and 

Coordination Is Weak. GAO-09-11R. Washington, D.C.: October 15, 

2008.

Disabled Veterans’ Employment: Additional Planning, Monitoring, and 

Data Collection Efforts Would Improve Assistance. GAO-07-1020. 

Washington, D.C.: September 12, 2007.

Highlights of a GAO Forum: Modernizing Federal Disability Policy. 

GAO-07-934SP. Washington, D.C.: August 3, 2007.

VA Vocational Rehabilitation and Employment: Service Contract 

Management Is Improving, but Challenges Remain. GAO-07-568R. 

Washington, D.C.: April 23, 2007.

Federal Disability Assistance: Wide Array of Programs Needs to Be 

Examined in Light of 21st Century Challenges. GAO-05-626. Washington, 

D.C.: June 2, 2005.

Vocational Rehabilitation: VA Has Opportunities to Improve Services, but 

Faces Significant Challenges. GAO-05-572T. Washington, D.C.: April 20, 

2005.

VA Vocational Rehabilitation and Employment Program: GAO Comments 

on Key Task Force Findings and Recommendations. GAO-04-853. 

Washington, D.C.: June 15, 2004.

Related GAO Products
Related GAO Products

Page 62 GAO-14-61 VA Vocational Rehabilitation and Employment

Internal Controls Management and Evaluation Tool. GAO-01-1008G. 

Washington, D.C.: August 2001.

Standards for Internal Control in the Federal Government. 

GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.

(131210)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.