VA Vocational Rehabilitation and Employment: Further Performance and Workload Management Improvements Are Needed
Released: 2014-01-14T00:00:00

Letter

Background

VR&E Entitlement and Eligibility

To be entitled to VR&E services and related benefits, veterans generally must (1) have at least a 20 percent disability rating from the VA and (2) be in need of rehabilitation because of an employment handicap caused in substantial part by a service-connected disability. If needed to achieve employment, entitled veterans may receive a maximum of 48 months of vocational rehabilitation services and an additional 18 months of employment services, which include counseling, placement, and postplacement services. Veterans are typically eligible to receive these services within a 12-year period beginning on either: (1) the date of separation from military service, or (2) the date the veteran received a disability rating from VA, whichever is later. In certain circumstances, the 48-month limit and the 12-year period of eligibility may be extended.

Availability of Additional Benefits pre- and post- VOW Act

Under certain circumstances, veterans who have successfully completed a VR&E program and achieved rehabilitation may qualify for additional VR&E benefits and services. Prior to the VOW Act, veterans who successfully completed a VR&E program could receive additional services if (1) their service-connected condition worsened to the extent that they could no longer perform the duties of the occupation for which they were previously rehabilitated, or (2) their occupation was no longer suitable because of their current capabilities and employment handicap. The VOW Act created another pathway for previously rehabilitated veterans to receive additional services. Under section 233 of the VOW Act, veterans who have exhausted their right to regular unemployment compensation under state law may receive up to 12 additional months of VR&E services and benefits following a successful rehabilitation, as long as they are otherwise eligible and they begin the additional program within 6 months after they exhausted unemployment compensation.This provision took effect on June 1, 2012, and veterans must apply no later than March 31, 2014 to qualify.

Administrative Structure

Within VA’s Veterans Benefits Administration’s (VBA) Office of Economic Opportunity in Washington D.C., the central VR&E Office is responsible for overseeing the VR&E program. VR&E services are provided by managers and staff at 57 regional offices and 179 satellite offices, under the management of VBA’s central Office of Field Operations, which manages workload and performance to achieve targets set in coordination with the central VR&E Office. VR&E positions typically include (1) a VR&E officer who oversees the VR&E field office and its staff; (2) vocational rehabilitation counselors (VRC) who work directly with veterans to assess their entitlement, develop their rehabilitation plans, and manage their progress; (3) employment coordinators (EC) who focus on assisting veterans with job search and placement, which also includes employer outreach to facilitate contact between veterans and employers; and (4) other staff to support the administration of the program. VR&E field staff are also located at select college campuses to help veterans successfully complete their training and become employed, and at military sites to help servicemembers with disabilities as they begin their transition to veteran status and the civilian workforce.

As of March 2013, 1,281 field staff were administering the VR&E program, of which about 70 percent (890) were VRCs. VR&E’s total caseload has generally increased every year from 98,015 at the end of fiscal year 2008 to 124,100 on March 31, 2013. Over the same period, the number of VRCs has also generally been growing. As shown in figure 1, the increase in staffing has helped to bring down the average caseload per VRC, which grew from 137 cases to 152 cases between fiscal years 2008 and 2009, then steadily decreased to 139 cases as of March 31, 2013.

VR&E Process

The provision of services under the VR&E program begins with the veteran’s application for services, a determination of entitlement and eligibility, and a visit to a regional or satellite office for further assessment.

If a veteran is entitled and eligible to receive VR&E services and found to be employable, the case manager, VRC and veteran will work together to select one of the four VR&E employment tracks best suited for the veteran, to identify a suitable employment goal, and to incorporate that goal and the needed services and benefits to achieve it into a rehabilitation plan.

VR&E Performance Management

VA measures the performance of its VR&E program across several dimensions, and uses performance goals and data to help manage the program. VA provides transparency on its performance through (1) its annual performance and accountability report; (2) its annual budget VA’s submission to the Congress; and (3) a database on VA’s website.VR&E measures cover the dimensions of program outcomes, timeliness, and accuracy (see table 1).

VA also holds the regional offices that administer the VR&E program accountable through performance goals, both for regional VR&E management and for individual VRC and EC staff members. To accomplish this, VA uses the above goals, as well as additional measures and goals. For example, individual VRCs are evaluated on case production and participants’ outcomes based on the general and serious employment handicap rehabilitation rates; the number of successful rehabilitations they complete; and the percentage of their caseloads in interrupted status.

About Half of Veterans Seeking Employment Are Successful, Although Program Participants Face Many Challenges

About Half of Veterans Find Suitable Jobs, but Time Frames Can Be Lengthy

About half of the veterans who entered the VR&E program in fiscal year 2003 and received employment-related services have obtained suitable jobs. Of the 45,520 veterans who applied in fiscal year 2003, 16,726 received some type of employment-related services. Among veterans in this group, 7,686 had achieved the program goal of suitable employment by the end of fiscal year 2012. Many others—4,841—had discontinued (stopped participating) and not yet re-applied. Although these veterans have not achieved suitable employment, a substantial number are considered by VA to have attained MRG and therefore some benefits from participating. In fact, according to VA data, some of these veterans were ultimately placed in a job, but the job was not seen as suitable given their disability. Most of the remaining veterans who started in fiscal year 2003 and received employment-related services were still participating in VR&E services, while a few had transferred to the independent living program within VR&E and achieved rehabilitation. (See fig. 3.)

While the amount of time veterans spend in the VR&E program varies substantially among participants, it can be lengthy for many veterans.

Among those who started in fiscal year 2003, the average time from application to successful placement in a suitable job was 4 years 7 months. While about half of these veterans completed the program within 3 to 6 years, many took considerably longer (see fig. 4). Furthermore, it appears that time frames for completing the program have not changed substantially for more recent applicants. For example, looking at VR&E participants who successfully attained suitable employment during fiscal year 2012, we found their average time from application to completion was also 4 years 7 months, with a range similar to that for fiscal year 2003 applicants. Meanwhile, among veterans who entered in fiscal year 2003 and have discontinued without yet returning, the average time from application to the final discontinuation was very similar—5 years.

Factors contributing to varied but generally lengthy time frames include the services veterans receive, as well as the fact that many extend their overall time in the program by interrupting or discontinuing along the way. Of those veterans who started since fiscal year 2003 and received some employment services, most (about 70 percent) pursued a 4-year college degree, and these veterans typically took longer to complete the program than those who pursued other types of training such as community college degrees or vocational training. In addition, we estimate that among veterans who started since fiscal year 2003 and achieved suitable employment, about one-quarter interrupted their programs at least once.

Almost the same proportion discontinued and returned to the program at least once before ultimately achieving success, and some did so two or three times. The small proportion (4 percent) who received job placement assistance only, meanwhile, attained suitable employment more quickly than those who received other types of services.

Once veterans attain suitable employment, they rarely come back for additional services, even though they may be entitled to such services if, for example, their disability worsens or they qualify under new expanded criteria established by the VOW Act. Of veterans who entered the program in fiscal year 2003 and were placed in suitable jobs, 4 percent have applied for and 2 percent have actually received additional employment-related VR&E services, with 3 years on average between completing the program and re-applying. The data on applying again following rehabilitation are similar for veterans who entered in fiscal years 2004 and 2005. Moreover, the expanded eligibility criteria authorized by the VOW Act have not so far had a substantial impact on the number of veterans returning for additional services. According to data provided by VA, between May 2012 and April 2013 only 10 veterans nationwide received additional VR&E services under these new VOW Act criteria. Managers or staff in a number of regional offices told us the new criteria would not significantly expand the population of veterans eligible for additional VR&E services: several noted that almost all the veterans who would qualify under the new criteria are already eligible under the old criteria. Some also said it is preferable to obtain additional services under the old criteria, which provide more months of additional services.

A Variety of Challenges May Result in Many Veterans Not Completing the Program

While veterans may discontinue for a variety of reasons, analysis of administrative data, results from a VA customer satisfaction survey, and our interviews with VR&E staff and participants point to several key challenges that may affect veterans at any stage of the rehabilitation process.

Mental Health and Other Disabilities

VA’s Voice of the Veteran survey measures veterans’ satisfaction with benefits and services received, including through the VR&E program. The results reported here are from the pilot survey fielded in October 2012 to January 2013; VA plans to field the survey on a regular, ongoing basis in the future. More specifically, the results on reasons for withdrawing or interrupting from the program are based on the responses of almost 200 veterans who had withdrawn or interrupted after spending at least 60 days in the program. We assessed the methodology of the pilot survey and concluded that the results we are reporting are sufficiently reliable for our purposes. However, these results reflect the opinions of survey respondents and cannot be generalized to the broader population of VR&E participants. For more information see appendix I. stop participating in a training program when they undergo surgery for their service-connected disability may lack motivation to resume training afterwards. A VRC in another office said veterans with post traumatic stress disorder (PTSD) face special challenges with maintaining employment because of difficulties interacting with co-workers. One veteran we interviewed said he completed a certificate program to become a heavy equipment operator, but was unable to obtain the required class D driver’s license because his PTSD made it difficult for him to deal with the noise and crowds at the testing site. Another veteran told us he trained for and was placed in a job in information technology. However, he said his job involves tasks for which he had not been trained, and his mental health conditions—including traumatic brain injury (TBI)—have made it hard for him to learn these tasks and may force him to leave his job.

VR&E staff and veterans often find it challenging to develop employment plans that can realistically lead to success. Administrative data suggest there are difficulties with developing a sound plan. About half the veterans who started since fiscal year 2003 and received some employment- related services have stopped their programs and returned for additional evaluation and planning at least once, and some have done so multiple times. Managers or staff in all eight offices we contacted said some veterans enter the program with unrealistic employment goals, which makes vocational planning more challenging. For example, VR&E staff in one office said veterans may want to train for jobs in law enforcement, even though they have PTSD or another psychological condition that makes this goal unrealistic. Some veterans we interviewed cited challenges coming to agreement with their counselors on an employment plan. For example, one said he wanted to study theology, but his counselor felt that studying to become a teacher would be more appropriate, and he ultimately left the program because of this disagreement. In addition, managers or staff in all eight offices we contacted said veterans may come to the VR&E program with the goal of furthering their education rather than obtaining a job, or may have a misperception about the purpose of the VR&E program. For example, a VRC in one office said she must explain to veterans that VR&E only provides training when it is necessary to achieve employment. She feels veterans’ chance of success would be enhanced if they entered the program with specific employment goals in mind, rather than training goals. Input from veterans is consistent with this viewpoint. While most recently-enrolled respondents to the Voice of the Veteran survey said they were motivated to apply to the VR&E program by employment- related goals, a substantial number (about a quarter) said they applied primarily to further their education.

Multiple VR&E Counselors

Analysis of administrative data indicates that the more VR&E counselors a veteran works with during his or her participation in the program, the less likely he or she is to achieve success. As noted above, veterans typically spend several years participating in the program, and most work with multiple counselors. The vast majority of those who entered the program since fiscal year 2003 and received some employment services have worked with at least two counselors, and a substantial proportion (about one-third) worked with four or more. We found that the chance of success declines steadily as veterans work with more counselors (see fig. 6). For example, veterans who had worked with two counselors were 8 percentage points less likely to have attained suitable employment within 8 years of entering the program, as compared to those who had worked with one counselor. Veterans who had worked with four counselors were 27 percentage points less likely to have attained suitable employment.

According to the Voice of the Veteran survey, many veterans who withdrew from or interrupted their employment services cited “family obligations” (27 percent) and “financial difficulties” (18 percent) as a reason. Managers or staff in seven of the eight regional offices we contacted also said one or both of these are common issues for veterans.

For example, according to managers in one office, veterans may have an immediate need to earn more money to support their families, so they quit training and take a job that may not be suitable given their disability. VRCs in another office told us a change in life circumstances—such as marriage, the birth of a child, or a death in the family—may make it more difficult to complete training. Veterans we interviewed also mentioned family obligations as a challenge. For example, one said she failed to complete her certification program in school administration because of the birth of her child.

Transition from Military to Civilian Employment

Managers, staff, and veterans we interviewed cited challenges associated with the transition from military service to civilian employment. For example, managers or staff in five of the eight regional offices we contacted told us veterans may have difficulty translating military experience into terms that civilian employers can understand, or that they may need civilian credentials to get a job even with the expertise they gained in the military. A VRC in one office explained that, in the state served by his office, veterans who received technical training in the medical field while in the military may still need additional training to pass a state certification exam to work in a comparable civilian job. Along similar lines, one veteran said he had responsibilities for benefits counseling, training, and recruitment while in the military that are comparable to the skills needed in a civilian human resources job. However, when he sought employment with the federal government, his VR&E counselor could not help him translate his military experience into federal civilian employment. In addition, staff in three of the offices we contacted said civilian employers may not understand veterans’ disabilities and how to accommodate them. One veteran we interviewed told us he was struggling to perform his job duties because of his physical disabilities, but had not received needed accommodations from his employer. He also felt his VR&E counselor did not sufficiently explain the physical demands of the job in advance. Finally, staff in two offices said veterans may lack experience with civilian work culture, which can present difficulties with succeeding in a new job. For example, an EC in one office said veterans are accustomed to having standard operating procedures in the military, and may struggle in a civilian job that lacks such clear guidelines.

Regional Office Variation Due to Economic or Other Factors

We found that veterans served by some regional offices were significantly less likely to succeed than those served by others, after controlling for a variety of other factors. However, we were unable to control for local economic conditions in our analyses, and managers or staff in 4 of 8 offices we contacted said the local labor market can be a key challenge to placing veterans in suitable jobs. VA central office officials also said variations in local economic conditions, including variation in the unemployment rate and the availability of good-paying jobs, contribute to regional variation. For example, they cited the loss of manufacturing jobs in the area served by one office as part of the explanation for its lower than average success rate. VA officials also cited other potential factors in regional variation, such as offices’ ability to retain experienced staff and fluctuations in staffing levels due to retirements. For example, they said the VR&E workforce in one office with below average performance may be more transient than in other offices.

VA Has Made Limited Progress in Improving VR&E Performance Management, Workload Management, and Training

VA Faces Challenges with Measuring Employment Outcomes and Veteran Satisfaction

VR&E Outcome Measurement Presents a Challenge for VA

VA has identified limitations with its primary measure of VR&E success— the rehabilitation rate—and is considering replacing it with a numeric target for positive outcomes. VA central and regional office officials noted that the rehabilitation rate goal can incentivize a VRC to hold onto an inactive case, even if the veteran has clearly left the program, because holding onto inactive cases instead of closing them can help regional offices and individual VRCs increase their rehabilitation rates and more easily reach their goals for a particular fiscal year. When staff delay closing cases like this, an office’s or individual’s rehabilitation rate may appear higher than it really is in one year and lower in another year, leading to an inaccurate picture of VA’s success in rehabilitating veterans. Furthermore, while cases remain in interrupted status, VRCs are still required to attempt contact with veterans every 60 days, which can limit the time spent working with veterans who are actively participating. In addition to the incentive to hold onto inactive cases, VA officials noted that the rehabilitation rate does not capture all of the benefits VR&E provides to veterans. In particular, the current calculation excludes MRG cases, where the veteran obtained some value from VR&E services but did not obtain and maintain suitable employment. Because it does not explicitly include MRG cases in the assessment of VR&E performance, VA’s current measure does not fully reflect the value it provides to veterans. Another concern raised by some regional officials was that having a uniform rehabilitation rate goal for every office does not account for regional differences in factors affecting employment, such as local job market conditions.

To address these and other issues, VA established four workgroups to review and make recommendations for improving VR&E’s individual employee performance standards. The resulting draft recommendations included eliminating VR&E’s rehabilitation rate measure and replacing it with a measure of the number of positive outcomes—specifically, the number of successful employment and independent living rehabilitations and MRG cases. Four regional offices were recommended to pilot the revised measure. According to a VA official, VA is expected to approve these draft recommendations by the end of fiscal year 2014. Also, in comments provided after reviewing a draft of this report, VA stated that it is developing VA-wide and regional office measures of program success, and expects to implement the VA-wide measures in fiscal year 2015.

While the current rehabilitation rate has limitations, the measure VA is considering as an alternative may not necessarily be a better way to monitor and demonstrate VR&E’s success. Specifically, VA is considering replacing its current outcome measure—which tracks those who obtained suitable jobs relative to those who did not—with a measure that tracks only the number of any positive outcomes. According to VA officials, positive outcomes would include not just veterans who obtained suitable employment, but also veterans who obtained other benefits (such as MRG) from the VR&E program. Good practices for performance measurement suggest that measures should allow agencies to transparently report their success to the Congress and public, and identify and address performance variances across their programs. The proposed measure of specific numbers of positive outcomes lacks this transparency because it does not account for veterans who left the program without obtaining suitable employment or other discernable benefits. It also obscures VA’s success in achieving its core mission because it does not separate veterans who obtained suitable employment from those who received other, lesser benefits. Finally, measuring only numbers of positive outcomes is less effective for monitoring and managing performance across regional offices—that is, comparing each office’s relative rate of success to identify those that may serve as models or require further oversight.

Aside from considering changes to its outcome measures, VA has taken other steps to help regional offices better manage their interrupted cases, but it is unclear how successful these efforts have been. For example, VA uses performance goals to hold regional offices and VRCs accountable for monitoring interrupted cases and keeping their proportions below certain thresholds. The current goal is 11 percent for regional offices and 12 percent for individual VRCs. Also, VR&E’s central office has provided tools and training to help VR&E managers monitor the number and average time of cases in interrupted status, and percent in interrupted status over 180 days. VR&E managers at two regional offices we contacted noted that they had VRCs close old, inactive cases, regardless of the short-term impact on the office’s rehabilitation rate. VA reported that the overall percentage of interrupted cases declined from 10.4 percent in fiscal year 2010 to 9.6 percent in fiscal year 2012. However, managing interrupted cases remains a challenge. According to VA data, in July 2013, 39 percent of more than 15,000 interrupted cases had been in that status for more than 180 days. In addition, 8 of 57 regional offices had at least 50 percent of their interrupted cases in that status for more than 180 days, ranging up to 70 percent at one office.

VA Lacks Performance Measures of Veterans’ Long- term Employment Success

The mission of the VR&E program includes helping veterans maintain suitable employment, and VA holds itself accountable for achieving this part of the mission by counting job placements as rehabilitations if they are maintained for at least 60 days. According to several regional office managers and staff we interviewed, cases are typically closed when veterans have maintained employment for this 60-day minimum, as problems with a veteran maintaining employment were most likely to occur in the first 60 days of employment if they occurred at all. However, if a veteran appears to be at high risk of losing a job, for example, the VRC can keep the case open for a longer period to provide additional supports. Some VRCs noted that this can occur when a veteran has a serious mental health condition, for example. According to VA data on about 38,000 veterans who applied for VR&E between fiscal years 2003 and 2012 and have been rehabilitated to employment, over one-third were tracked in employment for 6 months or longer before having their cases closed as rehabilitated, and some were tracked for 1 year or more. A VA official noted that some of this time is spent completing documentation and approval of case closures. In addition, one regional VR&E manager noted that closure may be delayed when the veteran does not inform the regional office that he or she obtained a job.

Research and practices at other federal employment programs suggest that tracking employment for 60 days may not be sufficient to gauge whether a veteran will be able to maintain employment. For example, a 2013 National Organization on Disability study suggested that wounded warriors transitioning to civilian employment need support for an extended time after obtaining jobs. Meanwhile, the Department of Labor (Labor) measures individuals’ employment retention over a longer term, 180 days. This measure applies to many of Labor’s employment programs, including its veterans employment programs.

VA officials stated that they plan to collect information on veterans’ long- term employment, but did not indicate whether they plan to develop performance measures to show the program’s impact on veterans’ long- term employment. Specifically, VA is considering contacting rehabilitated and discontinued VR&E participants at 6 and 12 months after they finish the program. This effort could yield information on employment over the longer term. VA plans to use this information to identify veterans who are having difficulties in maintaining employment, and possibly provide them additional supports. However, without developing performance measures, VA may be missing an opportunity to track the long-term effectiveness of the VR&E program.

VA Generally Followed Recognized Practices in Designing Its Customer Satisfaction Survey, but the Quality of Benchmark Results is Uncertain

VA has made progress toward holding itself accountable for VR&E participants’ satisfaction by implementing its new Voice of the Veteran survey, but the agency has not incorporated practices necessary to ensure the quality of benchmark results. VA intends to use survey data to inform program improvements and training developments, and potentially to report on veterans’ satisfaction with VR&E. In developing and piloting the survey, VA followed generally recognized survey practices, including detailed survey planning, comprehensive questionnaire development and testing, and extensive data collection activities, such as follow-up with veterans who did not respond to VR&E’s initial survey request.However, VA did not perform a nonresponse analysis to determine the effects of any nonresponse bias on the quality of its pilot survey data. Guidance from the Office of Management and Budget (OMB) states that agencies should perform nonresponse analysis when a survey’s unit response rate is below 80 percent, to identify the possibility of bias in a survey’s results, and measure, adjust for, report, and analyze nonresponse to assess its effects on data quality. VA’s plan for the pilot survey, as approved by the OMB, anticipated the need for this analysis, but its contract with the survey administrator made it optional. Although the response rates for the three pilot surveys of VR&E participants each fell below the 30 percent level specified in VA’s sample design, VA officials said they elected not to conduct a nonresponse analysis due to budget constraints. Moving forward, VA officials stated there is no expectation that a nonresponse analysis be performed for VA’s ongoing survey effort unless deemed necessary. Currently, VA’s approved contract for its ongoing survey effort does not include an option for a nonresponse analysis if response rates are low.

VA Workload Management Efforts May Not Fully Address Challenges Facing Some Regional Offices

VA has acted to strengthen VR&E workload management by commissioning a study of workload management issues, among other things. The study team collected and analyzed data on how VR&E staff at four regional offices spent their time on a variety of tasks, such as paperwork and client contacts. In its April 2011 report, the study team recommended, among other things, that VA improve its approach for estimating its overall staffing needs. VA now has an ongoing initiative to assess the total number of VR&E staff needed across the regional offices. This initiative, intended for use in formulating future VR&E administrative budget requests, is expected to be completed by March 2014. Also, in response to an internal business process review, VA has made changes to VR&E procedures intended to ease VRCs’ workloads by reducing paperwork.

Despite these efforts, some regional offices struggle with high caseloads that may affect quality of services to veterans, while others do not, suggesting potential shortcomings in the way VA allocates resources. VR&E managers and staff in four of the offices we contacted said high workloads impede their ability to spend sufficient time working with individual veterans. Meanwhile, VRCs in other offices we contacted said their workloads were manageable. VA administrative data indicate that average ratios of VR&E cases to VRCs vary considerably, as shown in figure 7. For example, while VA reported a national average of 139 cases per VRC as of March 31, 2013, regional office average caseloads ranged from 64 in Fargo, North Dakota to 579 in White River Junction, Vermont. As of March 31, 2013, seven offices averaged fewer than 100 cases per VRC, while eight averaged more than 175 cases per VRC.

According to VA, White River Junction had only one VRC, although another was added after March 31, 2013. In commenting on a draft of this report, VA stated that it expects another vacancy to be filled in January 2014, and White River Junction’s average to drop to about 160 cases per VRC.

In some cases veterans may work with multiple VRCs even if an office does not take a specialization approach. For example, if a VRC retires, that VRC’s cases would be redistributed among other VRCs.

Officials at the regional offices we contacted had different opinions about the efficacy of VRCs specializing by phase of the VR&E process. Some VR&E managers and staff cited advantages to allowing a single VRC to handle a case: for example, a VRC can (1) develop a stronger rapport with the veteran; and (2) avoid possible confusion, including among the veterans, related to handoffs between VRCs. Other regional officials cited advantages of specialization. For example, the VRC can: (1) manage caseloads without being distracted by a larger range of tasks (for example, assessing entitlement, preparing rehabilitation plans and monitoring veterans’ progress on their plans); and (2) develop expertise in one phase of the VR&E process. VR&E managers at one of the regional offices we contacted stated that they plan to move away from a more specialized caseload management approach, citing improved efficiency from having one VRC handle a case. Meanwhile, managers in another office said they had recently moved towards a specialization approach because they feel it better enables them to handle their large caseload.

Federal internal control standards require that managers obtain operational data to ensure their programs are run effectively and efficiently. For VA, this could include data on how staff resources are being utilized across regional offices, and how such usage affects offices’ efficiency and effectiveness. It is especially important that VA understand regional offices’ approaches and their effects, given our finding that veterans who work with a number of different VRCs over time are less likely to be successful. Moreover, the April 2011 consultants’ report to VA on VR&E workload management—which also found wide variations in how VRCs at different offices allocated their time among different tasks— recommended that VA assess regional office approaches to assigning tasks among VRCs in order to identify best practices for helping to rehabilitate veterans, while balancing standardization across offices with flexibility to serve each office’s service area. However, VA central office does not collect information on how regional offices manage their caseloads and assign tasks among VRCs, and has not assessed the effect of varied regional office approaches on case processing efficiency and veterans’ rehabilitation success. VR&E central office officials stated that they prefer to leave workload management to regional office managers because they are in the best position to determine which workload management approach will enable the office to meet its program goals.

VA Is Addressing Redundancy and Most Gaps in VR&E Staff Training

In the course of our work, we found evidence that staff had experienced redundancy and critical gaps in their training; however, VA had taken recent steps to address the redundancy and most, but not all of the training gaps. Principles of effective strategic training and workforce development call for avoiding duplication, identifying workforce competency gaps, and addressing these gaps. Nonetheless, some of the VR&E regional managers or staff at six of the eight offices we contacted told us that the annual curriculum offered to VRCs and ECs required them to repeat courses. Their experience notwithstanding, we found that VA has recently been adding new courses and, by fiscal year 2013, none of the courses on the curricula for experienced VRCs and ECs were redundant with those offered in the prior year. Further, with the addition of these new courses, VA is actively closing the nine critical training gaps identified in a June 2010 skill gap analysis report. The report noted that five gaps affect both new and experienced VRCs and four additional gaps affect only new staff. VBA contracted for this analysis in response to a report we issued in 2009, in which we found that VR&E had not collected data on the critical skills and competencies needed by VRCs to ensure they could meet the current and future needs of veterans In September 2013, VA informed us that it had begun in the program.providing training to staff to address these gaps and will have provided training to address all the gaps by the end of 2013, and in 2014 will provide two new courses for staff on mental health to improve their ability to assist veterans with PTSD and other mental illnesses. See appendix III for the gaps identified and training developed to address them.

Although VR&E closed many training gaps, our review of VR&E’s fiscal year 2013 curriculum identified other gaps related to job placement and accommodation for experienced VRCs. Staff or managers in five of the eight regional VR&E offices contacted also identified either job placement or workplace accommodation as areas in which additional training was needed. For example:

Some regional managers and staff told us VRCs need more training in job placement strategies, employment laws, and tax incentives for Job placement skills are critical to achieving VR&E’s employers.main mission to ensure that veterans with disabilities who can be employed obtain and maintain suitable employment. While ECs are generally responsible for assisting veterans with job placement, VRCs may take on this task in offices that lack ECs, or when ECs simply are not able to provide these services to veterans needing them.

Some regional managers and staff told us VRCs needed training in current workplace accommodations to better assist veterans. Workplace accommodations help ensure veterans with disabilities can successfully perform on the job and maintain their employment. A June 2012 VR&E-sponsored study confirmed the importance of workplace accommodations to help veterans attain and maintain long- term employment and recommended that VR&E either train or retrain its staff in workplace accommodation.

Although VA does not currently have specific plans to address either of these gaps in fiscal year 2014, VA officials informed us that the agency intends in the future to individually identify and address skill gaps for each VRC and EC. In the meantime, regarding job placement training, VA officials told us that VRCs may opt to take online training developed for its ECs, but that VA does not plan to require VRCs to take job placement training because VRCs receive it as part of their college or master’s degree programs. However, VRCs may not opt to take optional VA training as several VRCs and managers told us that VRCs lack time to take all the required training. As of July 2013, VA officials had not yet developed the curriculum for fiscal year 2014 and did not know if training on workplace accommodation for VRCs would be included.

As for VA’s plans to identify and address each VRC’s and EC’s skill gaps, this effort is part of its strategic initiative to provide competency-based training for 16 critical positions throughout the agency. VA officials informed us that under this initiative, which began in 2010, the agency plans to design and provide tailored training and other performance support that will be based on an assessment of each critical employee’s current proficiency level. VR&E officials expect the outcomes of this effort to include: (1) individual performance standards, (2) tailored performance support, (3) an assessment baseline and developmental roadmap (i.e., training plan), and (4) a tailored National Training Curriculum and remediation plan for each VRC and EC. Furthermore, through the collection and management of competency data, VR&E officials hope to track the knowledge, skills, abilities, education, and experience required for specific positions and recruit, train, distribute, and professionally develop personnel based on this competency framework. VA officials reported that as of fiscal year 2012, 47 percent of employees in critical and key occupations had participated in a competency-based training program within the last 12 months, and that in June 2013 the agency took an initial step to provide individualized training for VRCs. The program and plans VA described are consistent with best practices in human capital management.

Conclusions

VA has taken steps to improve its management of the VR&E program— ultimately with the goal of helping more veterans with disabilities attain suitable employment—but its ongoing efforts may not sufficiently address certain key management issues. With respect to performance measures, VA’s efforts to address problems with its existing rehabilitation rate measure may actually result in the adoption of a less transparent and meaningful outcome measure, as it is not clear how measuring the number of positive outcomes will provide the context necessary to understand program success. Furthermore, while VA plans to collect more information about longer-term outcomes, absent a performance measure VA may miss an opportunity to hold staff accountable and make adjustments to the program to promote success in maintaining employment—consistent with VR&E’s central mission. In addition, while VA’s ongoing customer satisfaction survey has the potential to provide valuable information, without performing additional analyses to determine whether responses actually reflect the experiences of VR&E participants overall, the agency cannot be sure that its survey will ultimately produce sound results for informing program management and training investments.

VA has also made strides related to workload management and training, but remaining shortcomings can negatively affect services provided to veterans. Because VA’s formula for allocating staff among regional offices rewards higher-performing offices, which can make it harder for lower- performing offices to improve, but does not consider educational and vocational counseling cases, which are concentrated in certain regions, some offices’ ability to serve veterans may be negatively affected. Lacking information on the varied approaches that regional offices are using to manage workloads also undermines VA’s ability to help offices optimize their use of staff to serve veterans in the most effective manner. Finally, training of VR&E staff is an area in which VA has made substantial improvements, but absent additional training on job placement and accommodations, counselors may be hindered in their ability to help veterans find and maintain employment.

Recommendations for Executive Action

The Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take the following actions: 1. In any revised set of national and regional performance measures for the VR&E program, include measures of (a) the proportion of program participants successfully rehabilitated into employment, and (b) the proportion of participants who obtain other benefits from VR&E services. 2. Develop new measures of long-term employment that go beyond the minimum 60 days of post-placement monitoring that is currently required. In developing measures, consider the feasibility of using results from planned post-closure surveys of veterans as a data source. 3. Consistent with generally accepted survey practices and as warranted by survey response rates, conduct nonresponse analysis of the results of VA’s ongoing Voice of the Veteran customer satisfaction surveys. 4. In revisiting VA’s formula for allocating VR&E staff among the regional offices, (a) assess the inclusion of factors related to regional office performance and if warranted remove them from the formula, and (b) assess the exclusion of any factor related to the number of educational counseling cases in each regional office and If warranted add such a factor. 5. Collect information on the regional offices’ approaches for managing their VR&E workloads, assess the advantages and disadvantages of these approaches, and use the results of this assessment to provide guidance to the offices on potential best practices or options to consider. 6. Provide additional training to all individual VRCs on job placement strategies and workplace accommodations, potentially as part of the effort to develop a competency-based training approach.

Agency Comments and Our Evaluation

We provided a draft of this report to VA for review and comment. In its written comments, produced as appendix IV in this report, VA agreed with our conclusions and generally concurred with our recommendations. Specifically, VA concurred with 5 of our 6 recommendations, and noted steps it plans to take to address them, as follows.

Regarding our recommendation to measure the proportions of VR&E participants achieving employment and other benefits, VA said that while ratio-based metrics will not be included in its revised employee-level performance measures, they will be included in the revised national-level measures.

Regarding our recommendation to conduct nonresponse bias analysis of its Voice of the Veteran survey results, VA said that—if funding permits—it will modify its survey administration contract to include such an analysis.

With regard to our recommendation to assess certain aspects of its formula for allocating staffing levels across regional offices, VA said it plans to revisit the formula and make any changes necessary.

With regard to our recommendation to assess regional offices’ workload management approaches, VA said it plans to collect and analyze data on the various approaches and inform the regions about best practices.

With regard to our recommendation to close certain gaps in staff training, VA said it plans to develop and make available to staff new training on employment and accommodations.

VA agreed in principle with our recommendation to develop performance measures related to VR&E participants’ longer-term employment success. Specifically, VA agreed there is value in looking at long-term outcomes, and noted its ongoing longitudinal study of a sample of participants and its plans to collect data on veterans’ post-closure service needs. However, VA also pointed out that VR&E counselors already work with veterans for many years, including up to 18 months following training completion. As such, VA indicated it does not see the value in directing limited resources to implementing and executing additional post- placement measures or services that will extend program participation and may create program dependencies. We acknowledge that VA in some cases monitors veterans for 6 months or more after they are placed in a job (as we note on page 24 of the report, this occurred in over one- third of the cases we examined). We also recognize the potential challenges of additional post-employment monitoring. At the same time, we believe that unless VA holds itself accountable for the longer-term outcomes of all participants who get hired into a suitable job, it may miss opportunities to make adjustments that better promote job maintenance. We encourage VA to explore cost-effective means of checking on veterans’ employment status at some point more than 60 days after job placement. We did not intend and do not believe this measure would necessarily entail extending program services or participation beyond current program practices. VA also provided technical comments, which we incorporated in our report as appropriate.

We are sending copies of this report to appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, this report will also be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to examine (1) the outcomes for veterans seeking employment through the Vocational Rehabilitation and Employment (VR&E) program, and (2) the progress the Department of Veterans Affairs (VA) has made in addressing critical management issues. To address these objectives, we reviewed relevant federal laws, regulations, and guidance, as well as VA documentation related to the VR&E program and recent management initiatives. We also reviewed several external studies and evaluations of the VR&E program, and determined that their methodologies were sufficient to allow us to report certain findings from these studies. We reviewed various criteria for effective program management, including in the areas of performance management and staff training. We analyzed VA administrative data on VR&E participants who applied to the program between fiscal years 2003 and 2012 (see below for more information on our efforts to analyze and ensure the reliability of these data), and interviewed VA central office staff responsible for the VR&E program, VR&E managers and staff in VA regional benefits offices (see below for details on office selection), and representatives of the Disabled American Veterans and the Paralyzed Veterans of America. Finally, to obtain input from veterans, we conducted phone interviews with veterans who had participated in the VR&E program (see below for details on how we developed our survey and selected veterans) and analyzed data from a VA customer satisfaction survey, after reviewing the methods VA used to develop and pilot the survey (see below for more detail on our assessment of the survey).

Analysis of VA Administrative Data

We analyzed data from VA’s Corporate WINRS (CWINRS) system, which VA uses to track veteran cases through the VR&E process and manage program costs. In our analysis, we included veterans who initially applied to the VR&E program between fiscal years 2003—the earliest year for which reliable data are available—and 2012. Unless otherwise noted, we included only veterans who received VR&E services designed to help veterans prepare for and obtain suitable jobs, and not those who received other services—such as educational and vocational counseling—which are also provided by VR&E staff. We analyzed data on the characteristics of VR&E participants (e.g., age, type of disability); the services they received from the VR&E program (e.g., evaluation, training); and their outcomes.

To assess outcomes, we placed VR&E participants who received employment-related services into one of four outcome categories: (1) rehabilitated to employment, including those who at least once had a case status of rehabilitated associated with achieving employment; (2) rehabilitated to independent living, including those who never had an employment-related rehabilitation but had least one rehabilitated status associated with achieving an independent living goal; (3) discontinued, including those who never had any rehabilitated status, and whose last status as of the end of fiscal year 2012 was discontinued; and (4) still participating, including those who never had a rehabilitated status and whose last status as of the end of fiscal year 2012 indicated some type of ongoing participation in the program. (See app. II for a description and detailed results of our analysis of the association between VR&E participants’ outcomes and their characteristics and services received.)

We found the CWINRS data to be sufficiently reliable for the purposes of our report. To assess the reliability of the data, we reviewed documentation on the system, interviewed VA officials, and performed electronic testing. VA officials told us that CWINRS became a single, national database during 2002, and that prior data are from separate, not necessarily uniform, systems operated by individual regional offices and are therefore less reliable. Based on this information, we chose to analyze data only for fiscal years 2003 through 2012. Also, through electronic testing we identified potential errors with outcome data for 1 percent of cases that received some employment-related services—for example, cases closed as rehabilitations for reasons which could not logically be associated with a rehabilitation. Because this made it impossible for us to assign these cases to one of our four outcome categories, we excluded them from our analysis of veterans’ outcomes. Finally, VA officials told us there may be some inaccuracies in data related to the status of cases—in particular for data prior to fiscal year 2008—because of a programming design flaw that allowed correct case status data in CWINRS to be inadvertently overridden with obsolete data from another VA system. Therefore, we use approximations rather than exact numbers when reporting the number of separate occasions on which veterans received certain services through the VR&E program, such as evaluation and planning assistance, job ready services, and rehabilitation to employment.

Regional Office Interviews

To obtain perspectives on the challenges VR&E staff face in helping to rehabilitate veterans, we interviewed VR&E managers and staffs at 8 of VA’s 57 regional offices. We selected offices to achieve diversity among offices in the following characteristics:

Geography: We selected at least one office in each of the four Veterans Benefits Administration areas.

Workload: Based on VR&E administrative data, we ranked offices by the number of initial fiscal year 2011 VR&E applications filed from largest to smallest numbers of applications.

Outcomes: Based on VA’s ASPIRE VR&E performance dashboard, we ranked offices by their Employment Rehabilitation RateSeptember 2012 from highest to lowest percentage rate.

Timeliness: Based on VA’s ASPIRE VR&E performance dashboard, we ranked offices according to the average days required to complete the evaluation and planning phase during September 2012 from lowest to highest.

Unemployment: We considered the November 2012 unemployment rate for each office’s state, as reported by the Bureau of Labor Statistics.

See figure 8 for details about the characteristics of each regional office we contacted.

We visited the regional offices in Washington, D.C., and Oakland, California, and conducted telephone interviews with VR&E managers and staff at the other six offices. At each of the eight offices, we interviewed the VR&E Officer (and one or more Assistant VR&E Officers at four offices); at least three VRCs; and at least one EC. At our in-person visits to Washington and Oakland, we also interviewed representatives of the Disabled American Veterans and California Department of Veterans Affairs. Our interview results cannot be generalized to all regional offices’ VR&E operations.

Interviews with Veterans

We conducted phone interviews with 17 veterans who had participated in the VR&E program, including 9 who had discontinued from the program and 8 who had achieved rehabilitation to employment. We pretested our interview questions twice with each group, and used the results of the pretests in our final analysis. We asked questions about the veterans’ employment, their interactions with their VR&E counselors, and their satisfaction with various VR&E services received. We randomly selected veterans for interviews, but cannot generalize the results of the interviews to the overall population of VR&E participants. To select our sample of veterans, we used CWINRS data to identify VR&E participants who had received some employment-related services and had either discontinued or achieved an employment-related rehabilitation between April 1, 2012 and September 30, 2012. We attempted to contact 49 discontinued veterans and 32 rehabilitated veterans after sending an advance informational email. These contact attempts resulted in complete interviews with 17 veterans.

Analysis of Voice of the Veteran Survey

We obtained aggregate results from the pilot of VA’s Voice of the Veteran customer satisfaction survey directly though VA’s dashboard—a central database that contains the agency’s performance metrics and customer satisfaction reports. This survey was administered to random samples of veterans from different customer populations. Each sample received a questionnaire reflecting the VA benefits and services they received. We obtained data from the three questionnaires targeted to VR&E participants: the Enrollment survey, the Servicing survey, and the Escaped Beneficiary survey. All three VR&E pilot surveys were fielded from October 2012 to January 2013. Table 2 provides background information on each of these surveys, including a description of the survey’s purpose and target sample population.

Based on interviews with agency officials and contract representatives for the Voice of the Veteran surveys, and our review of documentation of the surveys’ methodology, we concluded that these three VR&E surveys were generally developed and piloted in accordance with generally recognized survey research practices, and that the results we cite in this report are sufficiently reliable for our analysis. We used standards and guidelines issued by the Office of Management and Budget (OMB) to assess the methods VA used to design and administer the surveys. In addition, we reviewed survey guidelines of generally recognized professional organizations such as the American Association for Public Opinion Research and the American Statistical Association.

However, we concluded that the pilot survey results cited should be interpreted as representative of only those veterans responding to the survey. Response rates for the VR&E surveys were low—between 16 percent and 25 percent—and fell below the expected 30 percent rates specified in VA’s sample design. Table 3 provides more information on sample outcomes. While response rates may not be highly correlated with the presence of nonresponse bias and are not considered, by themselves, to be sufficient indicators of the quality of survey results, response rates are a useful indicator of the risk of nonresponse bias. According to OMB’s guidelines, a survey’s response rate is a valuable data quality and performance indicator, and is associated with the generalizability of a survey’s results. In addition, OMB’s guidelines state that a nonresponse analysis should be performed when a survey’s unit response rate is below 80 percent, to identify the possibility of bias in a survey’s results, and that agencies should appropriately measure, adjust for, report, and analyze nonresponse to assess its effect on data quality. Because VA did not perform a nonresponse analysis to determine if systematic differences exist between respondents and nonrespondents, it is not possible to generalize the results to the broader population of VR&E participants. Also, because the effects of any nonresponse bias on data quality were not assessed, it is not known if any systematic differences exist between nonrespondents and respondents.

Appendix II: Statistical Analysis of Administrative Data on Vocational Rehabilitation and Employment Outcomes

We used a statistical method known as survival analysis to estimate Vocational Rehabilitation and Employment (VR&E) participants’ likelihood of achieving success, their program completion times, and the factors that were associated with rehabilitation. Survival modeling methods provide a simple and efficient way to estimate the rate of event occurrence when the duration times of the events are “censored,” meaning that they are unobserved over some portion of the time period of interest. For example, a rehabilitation time would be censored if a veteran had been in a program for 2 years but had neither dropped out nor finished the program at the time of analysis. Although the veteran’s rehabilitation time is at least 2 years, the ultimate time to rehabilitation or discontinuation is not yet known (censored). We used a specific type of survival method, Cox regression models, to identify predictors of program completion—or what the Department of Veterans Affairs (VA) defines as “rehabilitation”—and estimate the rate at which rehabilitation occurred at various times after application among various groups of veterans.

Cox models allowed us to analyze the data without needing to make strong assumptions about the rehabilitation process. We did not need to assume that the likelihood or rate of rehabilitation changed over time in specific ways, as we would with other survival methods. The flexibility of Cox models comes at some cost of interpretation, however. The mathematics of Cox models make it more difficult to estimate the probability of rehabilitation by certain follow-up times (known as “failure probabilities”). To avoid the complex computation required for these estimates, we generally report simpler “Kaplan-Meier” estimates of rehabilitation probabilities in the body of the report—a form of survival analyses that allows us to report on likelihood of program completion, but without controlling for other covariates. To corroborate these results, we use Cox models to estimate the “hazard rate” of rehabilitation in this appendix, which is loosely equal to the rate of rehabilitation at a certain time after starting a VR&E program, while controlling for other factors. The results of this analysis are presented in this appendix and essentially support the same substantive findings about factors associated with the chance of rehabilitation discussed in the body of this report.

Source Data and Measurement of Key Variables

For the survival analysis using Cox models and Kaplan-Meier methods, we used data from the VR&E program’s Corporate WINRS system, which includes archived data on program participants’ demographic characteristics, services received, and outcomes achieved. We selected data on veterans who began their first VR&E programs in fiscal year 2003 through fiscal year 2012. We limited our analysis to this time period after VA identified reliability problems with older data. Appendix I describes the source data and their reliability in more detail.

In initial exploratory analysis, we considered various factors that could be associated with successful program completion and duration. These factors (which we also may refer to as covariates, variables, or predictors) included age, sex, education, branch of service, number of counselors, number of discontinuations, severity of disability, type of disability, year of first application to the program, VR&E services provided (“service track”), and the regional offices that managed veterans’ cases.

A key strength of the VR&E administrative data is its volume. The total number of veterans available for analysis, using the population defined above, was 133,679. Table 4, below, shows how the data were distributed across years of program entry and program outcomes. The large sample size available for analysis allowed us to estimate valid and precise relationships between completion times and a large number of covariates. We confirmed the adequacy of the sample by examining the cell sample sizes in tabulations of potential covariates.

We assessed the degree of correlation among the initial list of covariates, in order to identify those that would be redundant or introduce the possibility of spurious estimates. Most of the potential covariates were not strongly intercorrelated, except for the number of counselors and discontinuations, the number of counselors and regional offices, and age and era of service. The pair-wise correlations for these covariates ranged from 0.37 to 0.52. Including all of these covariates in our model would have inflated the estimated relationships between them and program completion times. Therefore, we included age and number of counselors in the final list of covariates, and excluded era of service, number of discontinuations, and number of offices from our model.

Measuring most of the covariates of interest was straightforward, and can be determined for a subset of meaningful predictors, as shown in Table 6 below. However, measuring certain covariates—such as regional offices, certain VR&E services, and disability type—required more complex measurement decisions.

Because veterans can be served by a number of regional offices over time, measuring the offices serving each veteran is challenging. In our data, veterans’ cases were managed by more than 4,100 permutations of regional offices. The large number of permutations made it difficult to meaningfully analyze variation across offices, and, in any case, produced sample sizes that were too small for modeling purposes. To simplify the measurement, we focused on the last office where each veteran was enrolled, as a measure of regional office at program completion. Because veterans could have discontinued their programs multiple times and could have been served by as many as six offices, the last office was a logical proxy for the office that could have influenced the final outcome most strongly.

Because we selected only veterans from the VA administrative data who received employment-related services, this group received job training or job placement assistance. We created a single measure of services received by creating categories for specific services related to employment, such as job placement services, and specific types of job training received, such as high school or college education. Table 5 below identifies the services we measured.

Finally, veterans could have been diagnosed with one or more disabilities, such as mental health conditions, ear disorders, and/or cardiovascular disorders. This allows for a potentially large number of unique combinations of disorders, which makes it difficult to create mutually exclusive and exhaustive groups. To avoid this problem, we compared veterans who had each type of disorder to veterans who did not have the disorder, using binary indicators for the presence or absence of each diagnosis. While the possibility of multiple disabilities makes these indicators mutually inclusive, our measurement scheme provided a concise way to collapse the many unique combinations of disorders.

Survival Model and Statistical Assumptions

The purpose of survival modeling is to estimate the probability of an event by various follow-up times or the average rate of event occurrence at any one follow-up time. Because these two quantities are mathematically equivalent, we used two different methods that produce similar results but vary in their ease of interpretation. where nj is the number of veterans who are “at risk” of rehabilitation and dj is the number of rehabilitations observed at tj. We estimated failure probabilities for various subgroups of veterans, in order to describe the association between these factors and the chance of rehabilitation by 5 and 8 years.

To corroborate these associations and control for alternative variables that could account for them, we used multivariate survival models—and the Cox Model for Proportional Hazards, in particular—to estimate the average rate of event occurrence at any one time in an observation period, also known as the “hazard rate.”interest is rehabilitation, and the time period of observation is fiscal year 2003 through fiscal year 2012. In our analysis, we defined censored outcomes as a veteran’s remaining in the program at the end of fiscal year 2012 or having discontinued from a program without yet re-applying (see table 5 above).

The formula for our Cox regression model was λi(t)= λ0(t)exp(xi b) , where λi(t, xi) is the hazard rate for veteran i at time t, xi = xi1 ,..., xip is a vector of p covariates, and xib is a linear combination of covariates and coefficients, with no intercept term. As described above and in table 6, the covariates included: age, sex, job training services, the interaction of sex and job training services, education, branch of service, number of counselors, severity of disability, and disability type. The term λ0(t) is the “baseline hazard rate,” or the rate implied when all covariates equal zero. (Note that the equation simplifies to λ0(t) when this is true.)

The Cox Model assumes that the hazard rates are proportional across values of the covariates throughout the observation period. If the hazard in one group exceeds that of a reference group, the model assumes that the proportional difference in rates will remain constant, and that the difference in hazard rates across groups does not decline, increase, or reverse direction over time. For example, the model assumes that the rehabilitation rate for veterans with mental health conditions differs from the rate for veterans without this type of condition by a constant amount throughout the observation period.

However, because a particular data generation process may not satisfy the proportional hazards assumption, we plotted the estimated hazards for various values of each covariate over time and formally tested the assumption for each covariate using statistical tests based on the model’s Shoenfeld residuals. We found a non-zero slope in a generalized linear regression of the Schoenfeld residuals on functions of time for several variables. A non-zero slope indicates that the hazards are not proportional across values of the covariates over time.across all variables further suggested that the hazard function was not proportional.

To identify the most serious departures from the proportional hazard assumption, we plotted the cumulative hazard function over time for the covariates that failed the Schoenfeld test. These plots suggested that the last office and first year of application covariates had strongly disproportional hazards. The remaining variables only failed to meet the proportional hazard assumption in a category coded for “missing data,” and the other variables that appeared to be disproportional were not meaningfully associated with rehabilitation times.

To adjust for most seriously non-proportional hazards, we stratified the model by last office and first year of application. Stratification allowed us to control for these two factors while also allowing their hazard functions to vary over time across offices and years of application. After stratifying by last office and first year of application, the proportional hazard assumption for the model was more consistent with our data.

Results

Our analysis identified a number of factors that had strong and statistically significant associations with rehabilitation times, as shown in Table 5.

Veterans took more time to complete their programs if multiple counselors worked with them. Average completion times increased at a higher rate as the number of counselors increased from 1 to 5 than from 6 to 9 or more. For example, veterans with 2 counselors had average rehabilitation times that were 29 percent slower, on average, than veterans with 1 counselor. In contrast, veterans with 9 or more counselors had average rehabilitation times that were about 1 percent slower than veterans with 8 counselors. In exploratory analysis, we found that veterans who more often discontinued and re-entered the program and who were served by multiple regional offices were more likely to work with multiple counselors, and that discontinuing more frequently was associated with slower rates of rehabilitation. However, when we estimated Cox models that controlled for the number of times discontinued and the number of regional offices used, we found that the number of counselors still had a meaningful association with success. Nevertheless, a reciprocal relationship, in which longer program times are associated with more counselors, may also explain this pattern.

The presence of mental health conditions substantially decreased completion rates. Veterans with mental health conditions completed their programs at a rate that was about 27 percent slower on average than veterans who did not have mental health conditions.

As expected, the type of services veterans received was among the strongest predictors of completion times. Veterans who received job placement services finished their programs about 4.5 times more quickly than veterans who were pursuing an undergraduate education. This is consistent with the nature of VR&E’s job placement services, which focus on veterans who already have the skills to pursue employment. Veterans who received apprenticeships, non-vocational training, unpaid work at a federal agency, vocational or technical education, or non-degree college credit hours finished their programs about 1.6 to 2.4 times more quickly than veterans pursuing an undergraduate college degree. In contrast, the average completion rate for veterans pursuing a graduate degree was only 28 percent quicker than for veterans pursuing an undergraduate degree. As a whole, these results suggest that veterans pursuing undergraduate or graduate degrees require more time to complete their programs than veterans receiving other training.

The amount of prior education was associated with faster program completion rates. As compared to veterans with only a high school degree, veterans with some college, an undergraduate degree, or a graduate degree had completion rates that were 1.2 to 1.5 times faster, on average. Because these estimates control for the type of services received and the type and severity of disabilities, veterans with more prior education appear to have finished more quickly regardless of the training they received or the nature of their disabilities.

Finally, veterans with disability ratings of less than 100 percent finished their programs more quickly than veterans with disability ratings of 100 percent, with completion rates that were about 1.5 times more quickly on average.disabilities face greater challenges to timely program completion, even holding constant factors such as type of training, prior education, and age.

Appendix III: Vocational Rehabilitation Counselor Training Gaps Identified by VA in 2010 and the Agency’s Efforts to Address

Training gaps New VRCs How to provide services to veterans recently discharged from the military How to provide services to veterans with severe disabilities How to determine when extended evaluation is necessary and conduct subsequent individualized extended evaluation plan development How to closeout cases for different veteran outcomes (e.g., for veterans who achieved suitable employment or maximum rehabilitation gain)

For new and experienced VRCs How to provide services to veterans with traumatic brain injuries (TBI)

How to provide services to veterans with post- traumatic stress disorder (PTSD)

How to conduct a preliminary and/or comprehensive IL assessment How to input benefit awards into the computer system (e.g., inputting the decision to provide a veteran a subsistence allowance)

2-hour course on VR&E fiscal accuracy and integrity Two 1-hour courses on VR&E processing of subsistence allowance awards 1-hour course on employment adjustment allowances VA added this course to the fiscal year 2013 curriculum to address changes to its guidance for independent living and delivered the training in September, 2013. VA has developed the course and will assign it as part of the fiscal year 2014 curriculum to be completed by the end of the first fiscal quarter (i.e., by December 31, 2013).

VA used existing courses to train VR&E staff on inputting benefit awards into its computer system.

Appendix IV: Comments from the Department of Veterans Affairs

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact listed above, individuals making key contributions to this report were Michele Grgich (Assistant Director), Jessica Botsford, Frederick Caison, Elizabeth Curda, Kevin Daly, Holly Dye, Julie DeVault, Brenda Farrell, Alex Galuten, Shannon Grabich, Mitch Karpman, Kirsten B. Lauber, Lorin Obler, Carl Ramirez, Almeta Spencer, Jeff Tessin, Greg Whitney, and Ashanta Williams.

Related GAO Products

VA Vocational Rehabilitation and Employment Program: Improved Oversight of Independent Living Services and Supports Is Needed. GAO-13-474. Washington, D.C.: June 7, 2013.

Veterans’ Employment and Training: Better Targeting, Coordinating, and Reporting Needed to Enhance Program Effectiveness. GAO-13-29. Washington, D.C.: December 13, 2012.

Federal Information System Controls Audit Manual (FISCAM). GAO-09-232G. Washington, D.C.: February 2009.

VA Vocational Rehabilitation and Employment: Better Incentives, Workforce Planning, and Performance Reporting Could Improve Program. GAO-09-34. Washington, D.C.: January 26, 2009.

Multiple Agencies Provide Assistance to Service-disabled Veterans or Entrepreneurs, but Specific Needs Are Difficult to Identify and Coordination Is Weak. GAO-09-11R. Washington, D.C.: October 15, 2008.

Disabled Veterans’ Employment: Additional Planning, Monitoring, and Data Collection Efforts Would Improve Assistance. GAO-07-1020. Washington, D.C.: September 12, 2007.

Highlights of a GAO Forum: Modernizing Federal Disability Policy. GAO-07-934SP. Washington, D.C.: August 3, 2007.

VA Vocational Rehabilitation and Employment: Service Contract Management Is Improving, but Challenges Remain. GAO-07-568R. Washington, D.C.: April 23, 2007.

Federal Disability Assistance: Wide Array of Programs Needs to Be Examined in Light of 21st Century Challenges. GAO-05-626. Washington, D.C.: June 2, 2005.

Vocational Rehabilitation: VA Has Opportunities to Improve Services, but Faces Significant Challenges. GAO-05-572T. Washington, D.C.: April 20, 2005.

VA Vocational Rehabilitation and Employment Program: GAO Comments on Key Task Force Findings and Recommendations. GAO-04-853. Washington, D.C.: June 15, 2004.

Internal Controls Management and Evaluation Tool. GAO-01-1008G. Washington, D.C.: August 2001.

Standards for Internal Control in the Federal Government. GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.