VA Vocational Rehabilitation and Employment: Further Performance and Workload Management Improvements Are Needed
Published: 2014-01-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Veterans with disabilities face special challenges to finding employment-- resulting, for example, from the veterans' physical or mental health, or negative attitudes or stereotypes on the part of some employers. VA's VR&E program aims to help veterans with disabilities obtain and maintain suitable employment--compatible with their disabilities--through services such as training and job search assistance. The VOW to Hire Heroes Act of 2011 directed GAO to review the program. GAO examined (1) the outcomes for veterans seeking employment through the program, and (2) the progress VA has made in addressing critical management issues. GAO reviewed relevant laws, regulations, and guidance as well as recent studies; reviewed various program management criteria; analyzed VA administrative data on veterans who entered the program between fiscal years 2003 and 2012; interviewed staff at the VA central office and 8 regional offices; interviewed a random but nongeneralizable sample of 17 program participants; and analyzed data from a VA survey of participants.
section_title: What GAO Found
paragraphs: About half of the almost 17,000 veterans who entered the Department of Veterans Affairs' (VA) Vocational Rehabilitation and Employment (VR&E) program in fiscal year 2003 and received employment-related services were placed in suitable jobs, one-third left the program, and most of the others are still participating. It often took veterans 6 years or more to achieve success, due in part to veterans often leaving the program temporarily. Interviews with VR&E staff and participants and administrative data GAO reviewed suggest veterans face numerous challenges that affect their ability to obtain employment, especially related to mental health conditions, working with multiple VR&E counselors over time, and civilian employers' limited understanding of military work experience.
VA has taken steps to improve VR&E performance management, workload management, and staff training, but weaknesses remain. With regard to performance management, VA has an ongoing initiative to revise its approach for measuring rehabilitation success at the individual employee, regional, and national levels. However, the new approach VA is considering for employees reflects only the number, not the rate of successful outcomes, and therefore would not provide sufficient context for understanding program success. VA has not yet developed its new approaches for assessing rehabilitation success regionally and nationally. Also, VA began surveying participants' satisfaction with the program and plans to use the results to manage performance; while VA has generally followed good survey design practices, the agency has not fully assessed the reliability of early customer satisfaction results. In terms of workload management, VA has taken steps to reduce paperwork burdens on regional offices. However, several offices still reported heavy workloads and noted that VA's formula for allocating staff among offices does not consider other staff duties affecting workloads, such as education counseling. In addition, VA has not studied the relative effectiveness and efficiency of regional offices' approaches for assigning staff to manage workloads. Finally, with respect to training, VA has addressed redundancy and most gaps in training for VR&E staff, but gaps remain in the areas of job placement assistance and workplace accommodations.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA reflect success rates in revised performance measures, ensure the reliability of its customer satisfaction survey results, re-visit its staff allocation formula, study staff assignments, and close certain gaps in its training for staff. In its comments, VA generally concurred with these recommendations and noted steps it plans to take to address them.

section_title: Letter
paragraphs: About 3 million veterans—including more than 700,000 from the Iraq and  Afghanistan era—have a disability connected to their military service,  and many face special challenges obtaining employment. Veterans with  disabilities may face employment challenges related to physical  impairments, less visible mental health conditions, or negative attitudes  and stereotypes among some employers. The Department of Veterans  Affairs (VA) operates the Vocational Rehabilitation and Employment  (VR&E) program to help veterans with service-connected disabilities  obtain and maintain suitable employment, which it considers to be  employment consistent with their abilities, aptitudes, and interests. The  program provides services including vocational assessment, education  and training, and job placement. In fiscal year 2012, about 121,000  veterans participated in the program at a cost of almost $1 billion.
To improve the transition of veterans to civilian employment, Congress  passed the VOW to Hire Heroes Act of 2011 (VOW Act). In addition to  provisions intended to promote veterans’ employment, including  expanded access to VR&E benefits for certain unemployed veterans, the  VOW Act mandated that GAO review the VR&E program.(1) the outcomes for veterans seeking employment through the VR&E  program, and (2) the progress VA has made in addressing critical  management issues. To address these objectives, we reviewed relevant  federal laws, regulations, and guidance as well as documentation related  to recent management initiatives. We analyzed VA administrative data on  VR&E participants who applied to the program between fiscal years 2003  and 2012, including their characteristics, services received, and outcomes  obtained. We assessed the reliability of these data through electronic  testing and interviews with VA staff, and found them sufficiently reliable  for our reporting purposes. We interviewed VA central office staff  responsible for the VR&E program, as well as VR&E managers and staff  in 8 of VA’s 57 regional offices. We selected offices to achieve diversity in  VR&E caseload, performance in job placement and case processing  timeliness, state unemployment rate, and geographic location. We  conducted phone interviews with a random but non-generalizable sample  of 17 veterans who had participated in the VR&E program and exited with  either a successful or unsuccessful outcome. Finally, we obtained  selected data resulting from a pilot VA customer satisfaction survey that  includes responses from VR&E participants, after reviewing the survey’s  methodology and finding it to be sufficiently reliable for our purposes  We conducted this performance audit from October 2012 to January 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Additional information on our  scope and methodology is provided in appendix I.
subsections: section_title: Background
subsections: section_title: VR&E Entitlement and Eligibility
paragraphs: To be entitled to VR&E services and related benefits, veterans generally  must (1) have at least a 20 percent disability rating from the VA and (2)  be in need of rehabilitation because of an employment handicap caused  in substantial part by a service-connected disability. If needed to  achieve employment, entitled veterans may receive a maximum of 48  months of vocational rehabilitation services and an additional 18 months  of employment services, which include counseling, placement, and  postplacement services. Veterans are typically eligible to receive these  services within a 12-year period beginning on either: (1) the date of  separation from military service, or (2) the date the veteran received a  disability rating from VA, whichever is later. In certain circumstances, the  48-month limit and the 12-year period of eligibility may be extended.
section_title: Availability of Additional Benefits pre- and post- VOW Act
paragraphs: Under certain circumstances, veterans who have successfully completed  a VR&E program and achieved rehabilitation may qualify for additional  VR&E benefits and services. Prior to the VOW Act, veterans who  successfully completed a VR&E program could receive additional  services if (1) their service-connected condition worsened to the extent  that they could no longer perform the duties of the occupation for which  they were previously rehabilitated, or (2) their occupation was no longer  suitable because of their current capabilities and employment handicap.  The VOW Act created another pathway for previously rehabilitated  veterans to receive additional services. Under section 233 of the VOW  Act, veterans who have exhausted their right to regular unemployment  compensation under state law may receive up to 12 additional months of  VR&E services and benefits following a successful rehabilitation, as long  as they are otherwise eligible and they begin the additional program  within 6 months after they exhausted unemployment compensation.This provision took effect on June 1, 2012, and veterans must apply no  later than March 31, 2014 to qualify.
section_title: Administrative Structure
paragraphs: Within VA’s Veterans Benefits Administration’s (VBA) Office of Economic  Opportunity in Washington D.C., the central VR&E Office is responsible  for overseeing the VR&E program. VR&E services are provided by  managers and staff at 57 regional offices and 179 satellite offices, under  the management of VBA’s central Office of Field Operations, which  manages workload and performance to achieve targets set in  coordination with the central VR&E Office. VR&E positions typically  include (1) a VR&E officer who oversees the VR&E field office and its  staff; (2) vocational rehabilitation counselors (VRC) who work directly with  veterans to assess their entitlement, develop their rehabilitation plans,  and manage their progress; (3) employment coordinators (EC) who focus  on assisting veterans with job search and placement, which also includes  employer outreach to facilitate contact between veterans and employers;  and (4) other staff to support the administration of the program. VR&E  field staff are also located at select college campuses to help veterans  successfully complete their training and become employed, and at military  sites to help servicemembers with disabilities as they begin their transition  to veteran status and the civilian workforce.
As of March 2013, 1,281 field staff were administering the VR&E  program, of which about 70 percent (890) were VRCs. VR&E’s total  caseload has generally increased every year from 98,015 at the end of  fiscal year 2008 to 124,100 on March 31, 2013. Over the same period,  the number of VRCs has also generally been growing. As shown in figure  1, the increase in staffing has helped to bring down the average caseload  per VRC, which grew from 137 cases to 152 cases between fiscal years  2008 and 2009, then steadily decreased to 139 cases as of March 31,  2013.
section_title: VR&E Process
paragraphs: The provision of services under the VR&E program begins with the  veteran’s application for services, a determination of entitlement and  eligibility, and a visit to a regional or satellite office for further  assessment.
If a veteran is entitled and eligible to receive VR&E  services and found to be employable, the case manager, VRC and  veteran will work together to select one of the four VR&E employment  tracks best suited for the veteran, to identify a suitable employment goal,  and to incorporate that goal and the needed services and benefits to  achieve it into a rehabilitation plan.
section_title: VR&E Performance Management
paragraphs: VA measures the performance of its VR&E program across several  dimensions, and uses performance goals and data to help manage the  program. VA provides transparency on its performance through (1) its  annual performance and accountability report; (2) its annual budget   VA’s  submission to the Congress; and (3) a database on VA’s website.VR&E measures cover the dimensions of program outcomes, timeliness,  and accuracy (see table 1).
VA also holds the regional offices that administer the VR&E program  accountable through performance goals, both for regional VR&E  management and for individual VRC and EC staff members. To  accomplish this, VA uses the above goals, as well as additional measures  and goals. For example, individual VRCs are evaluated on case  production and participants’ outcomes based on the general and serious  employment handicap rehabilitation rates; the number of successful  rehabilitations they complete; and the percentage of their caseloads in  interrupted status.
section_title: About Half of Veterans Seeking Employment Are Successful, Although Program Participants Face Many Challenges
subsections: section_title: About Half of Veterans Find Suitable Jobs, but Time Frames Can Be Lengthy
paragraphs: About half of the veterans who entered the VR&E program in fiscal year  2003 and received employment-related services have obtained suitable  jobs. Of the 45,520 veterans who applied in fiscal year 2003, 16,726  received some type of employment-related services. Among veterans in  this group, 7,686 had achieved the program goal of suitable employment  by the end of fiscal year 2012. Many others—4,841—had discontinued  (stopped participating) and not yet re-applied. Although these veterans  have not achieved suitable employment, a substantial number are  considered by VA to have attained MRG and therefore some benefits  from participating. In fact, according to VA data, some of these veterans  were ultimately placed in a job, but the job was not seen as suitable given  their disability. Most of the remaining veterans who started in fiscal year  2003 and received employment-related services were still participating in  VR&E services, while a few had transferred to the independent living  program within VR&E and achieved rehabilitation. (See fig. 3.)
While the amount of time veterans spend in the VR&E program varies  substantially among participants, it can be lengthy for many veterans.
Among those who started in fiscal year 2003, the average time from  application to successful placement in a suitable job was 4 years 7  months. While about half of these veterans completed the program within  3 to 6 years, many took considerably longer (see fig. 4). Furthermore, it  appears that time frames for completing the program have not changed  substantially for more recent applicants. For example, looking at VR&E  participants who successfully attained suitable employment during fiscal  year 2012, we found their average time from application to completion  was also 4 years 7 months, with a range similar to that for fiscal year  2003 applicants. Meanwhile, among veterans who entered in fiscal year  2003 and have discontinued without yet returning, the average time from  application to the final discontinuation was very similar—5 years.
Factors contributing to varied but generally lengthy time frames include  the services veterans receive, as well as the fact that many extend their  overall time in the program by interrupting or discontinuing along the way.  Of those veterans who started since fiscal year 2003 and received some  employment services, most (about 70 percent) pursued a 4-year college  degree, and these veterans typically took longer to complete the program  than those who pursued other types of training such as community  college degrees or vocational training. In addition, we estimate that  among veterans who started since fiscal year 2003 and achieved suitable  employment, about one-quarter interrupted their programs at least once.
Almost the same proportion discontinued and returned to the program at  least once before ultimately achieving success, and some did so two or  three times. The small proportion (4 percent) who received job placement  assistance only, meanwhile, attained suitable employment more quickly  than those who received other types of services.
Once veterans attain suitable employment, they rarely come back for  additional services, even though they may be entitled to such services if,  for example, their disability worsens or they qualify under new expanded  criteria established by the VOW Act. Of veterans who entered the  program in fiscal year 2003 and were placed in suitable jobs, 4 percent  have applied for and 2 percent have actually received additional  employment-related VR&E services, with 3 years on average between  completing the program and re-applying. The data on applying again  following rehabilitation are similar for veterans who entered in fiscal years  2004 and 2005. Moreover, the expanded eligibility criteria authorized by  the VOW Act have not so far had a substantial impact on the number of  veterans returning for additional services. According to data provided by  VA, between May 2012 and April 2013 only 10 veterans nationwide  received additional VR&E services under these new VOW Act criteria.  Managers or staff in a number of regional offices told us the new criteria  would not significantly expand the population of veterans eligible for  additional VR&E services: several noted that almost all the veterans who  would qualify under the new criteria are already eligible under the old  criteria. Some also said it is preferable to obtain additional services under  the old criteria, which provide more months of additional services.
section_title: A Variety of Challenges May Result in Many Veterans Not Completing the Program
paragraphs: While veterans may discontinue for a variety of reasons, analysis of  administrative data, results from a VA customer satisfaction survey, and  our interviews with VR&E staff and participants point to several key  challenges that may affect veterans at any stage of the rehabilitation  process.
subsections: section_title: Mental Health and Other Disabilities
paragraphs: VA’s Voice of the Veteran survey measures veterans’ satisfaction with benefits and  services received, including through the VR&E program. The results reported here are  from the pilot survey fielded in October 2012 to January 2013; VA plans to field the survey  on a regular, ongoing basis in the future. More specifically, the results on reasons for  withdrawing or interrupting from the program are based on the responses of almost 200  veterans who had withdrawn or interrupted after spending at least 60 days in the program.  We assessed the methodology of the pilot survey and concluded that the results we are  reporting are sufficiently reliable for our purposes. However, these results reflect the  opinions of survey respondents and cannot be generalized to the broader population of  VR&E participants. For more information see appendix I. stop participating in a training program when they undergo surgery for  their service-connected disability may lack motivation to resume training  afterwards. A VRC in another office said veterans with post traumatic  stress disorder (PTSD) face special challenges with maintaining  employment because of difficulties interacting with co-workers. One  veteran we interviewed said he completed a certificate program to  become a heavy equipment operator, but was unable to obtain the  required class D driver’s license because his PTSD made it difficult for  him to deal with the noise and crowds at the testing site. Another veteran  told us he trained for and was placed in a job in information technology.  However, he said his job involves tasks for which he had not been  trained, and his mental health conditions—including traumatic brain injury  (TBI)—have made it hard for him to learn these tasks and may force him  to leave his job.
VR&E staff and veterans often find it challenging to develop employment  plans that can realistically lead to success. Administrative data suggest  there are difficulties with developing a sound plan. About half the veterans  who started since fiscal year 2003 and received some employment- related services have stopped their programs and returned for additional  evaluation and planning at least once, and some have done so multiple  times. Managers or staff in all eight offices we contacted said some  veterans enter the program with unrealistic employment goals, which  makes vocational planning more challenging. For example, VR&E staff in  one office said veterans may want to train for jobs in law enforcement,  even though they have PTSD or another psychological condition that  makes this goal unrealistic. Some veterans we interviewed cited  challenges coming to agreement with their counselors on an employment  plan. For example, one said he wanted to study theology, but his  counselor felt that studying to become a teacher would be more  appropriate, and he ultimately left the program because of this  disagreement. In addition, managers or staff in all eight offices we  contacted said veterans may come to the VR&E program with the goal of  furthering their education rather than obtaining a job, or may have a  misperception about the purpose of the VR&E program. For example, a  VRC in one office said she must explain to veterans that VR&E only  provides training when it is necessary to achieve employment. She feels  veterans’ chance of success would be enhanced if they entered the  program with specific employment goals in mind, rather than training  goals. Input from veterans is consistent with this viewpoint. While most  recently-enrolled respondents to the Voice of the Veteran survey said  they were motivated to apply to the VR&E program by employment- related goals, a substantial number (about a quarter) said they applied  primarily to further their education.
section_title: Multiple VR&E Counselors
paragraphs: Analysis of administrative data indicates that the more VR&E counselors  a veteran works with during his or her participation in the program, the  less likely he or she is to achieve success. As noted above, veterans  typically spend several years participating in the program, and most work  with multiple counselors. The vast majority of those who entered the  program since fiscal year 2003 and received some employment services  have worked with at least two counselors, and a substantial proportion  (about one-third) worked with four or more. We found that the chance of  success declines steadily as veterans work with more counselors (see fig.  6). For example, veterans who had worked with two counselors were 8  percentage points less likely to have attained suitable employment within  8 years of entering the program, as compared to those who had worked  with one counselor. Veterans who had worked with four counselors were  27 percentage points less likely to have attained suitable employment.
According to the Voice of the Veteran survey, many veterans who  withdrew from or interrupted their employment services cited “family  obligations” (27 percent) and “financial difficulties” (18 percent) as a  reason. Managers or staff in seven of the eight regional offices we  contacted also said one or both of these are common issues for veterans.
For example, according to managers in one office, veterans may have an  immediate need to earn more money to support their families, so they quit  training and take a job that may not be suitable given their disability.  VRCs in another office told us a change in life circumstances—such as  marriage, the birth of a child, or a death in the family—may make it more  difficult to complete training. Veterans we interviewed also mentioned  family obligations as a challenge. For example, one said she failed to  complete her certification program in school administration because of the  birth of her child.
section_title: Transition from Military to Civilian Employment
paragraphs: Managers, staff, and veterans we interviewed cited challenges associated  with the transition from military service to civilian employment. For  example, managers or staff in five of the eight regional offices we  contacted told us veterans may have difficulty translating military  experience into terms that civilian employers can understand, or that they  may need civilian credentials to get a job even with the expertise they  gained in the military. A VRC in one office explained that, in the state  served by his office, veterans who received technical training in the  medical field while in the military may still need additional training to pass  a state certification exam to work in a comparable civilian job. Along  similar lines, one veteran said he had responsibilities for benefits  counseling, training, and recruitment while in the military that are  comparable to the skills needed in a civilian human resources job.  However, when he sought employment with the federal government, his  VR&E counselor could not help him translate his military experience into  federal civilian employment. In addition, staff in three of the offices we  contacted said civilian employers may not understand veterans’  disabilities and how to accommodate them. One veteran we interviewed  told us he was struggling to perform his job duties because of his physical  disabilities, but had not received needed accommodations from his  employer. He also felt his VR&E counselor did not sufficiently explain the  physical demands of the job in advance. Finally, staff in two offices said  veterans may lack experience with civilian work culture, which can  present difficulties with succeeding in a new job. For example, an EC in  one office said veterans are accustomed to having standard operating  procedures in the military, and may struggle in a civilian job that lacks  such clear guidelines.
section_title: Regional Office Variation Due to Economic or Other Factors
paragraphs: We found that veterans served by some regional offices were significantly  less likely to succeed than those served by others, after controlling for a  variety of other factors. However, we were unable to control for local  economic conditions in our analyses, and managers or staff in 4 of 8  offices we contacted said the local labor market can be a key challenge to  placing veterans in suitable jobs. VA central office officials also said  variations in local economic conditions, including variation in the  unemployment rate and the availability of good-paying jobs, contribute to  regional variation. For example, they cited the loss of manufacturing jobs  in the area served by one office as part of the explanation for its lower  than average success rate. VA officials also cited other potential factors in  regional variation, such as offices’ ability to retain experienced staff and  fluctuations in staffing levels due to retirements. For example, they said  the VR&E workforce in one office with below average performance may  be more transient than in other offices.
section_title: VA Has Made Limited Progress in Improving VR&E Performance Management, Workload Management, and Training
subsections: section_title: VA Faces Challenges with Measuring Employment Outcomes and Veteran Satisfaction
subsections: section_title: VR&E Outcome Measurement Presents a Challenge for VA
paragraphs: VA has identified limitations with its primary measure of VR&E success— the rehabilitation rate—and is considering replacing it with a numeric  target for positive outcomes. VA central and regional office officials noted  that the rehabilitation rate goal can incentivize a VRC to hold onto an  inactive case, even if the veteran has clearly left the program, because  holding onto inactive cases instead of closing them can help regional  offices and individual VRCs increase their rehabilitation rates and more  easily reach their goals for a particular fiscal year. When staff delay  closing cases like this, an office’s or individual’s rehabilitation rate may  appear higher than it really is in one year and lower in another year,  leading to an inaccurate picture of VA’s success in rehabilitating veterans.  Furthermore, while cases remain in interrupted status, VRCs are still  required to attempt contact with veterans every 60 days, which can limit  the time spent working with veterans who are actively participating. In  addition to the incentive to hold onto inactive cases, VA officials noted  that the rehabilitation rate does not capture all of the benefits VR&E  provides to veterans. In particular, the current calculation excludes MRG  cases, where the veteran obtained some value from VR&E services but  did not obtain and maintain suitable employment. Because it does not  explicitly include MRG cases in the assessment of VR&E performance,  VA’s current measure does not fully reflect the value it provides to  veterans. Another concern raised by some regional officials was that  having a uniform rehabilitation rate goal for every office does not account  for regional differences in factors affecting employment, such as local job  market conditions.
To address these and other issues, VA established four workgroups to  review and make recommendations for improving VR&E’s individual  employee performance standards. The resulting draft recommendations  included eliminating VR&E’s rehabilitation rate measure and replacing it  with a measure of the number of positive outcomes—specifically, the  number of successful employment and independent living rehabilitations  and MRG cases. Four regional offices were recommended to pilot the  revised measure. According to a VA official, VA is expected to approve  these draft recommendations by the end of fiscal year 2014. Also, in  comments provided after reviewing a draft of this report, VA stated that it  is developing VA-wide and regional office measures of program success,  and expects to implement the VA-wide measures in fiscal year 2015.
While the current rehabilitation rate has limitations, the measure VA is  considering as an alternative may not necessarily be a better way to  monitor and demonstrate VR&E’s success. Specifically, VA is considering  replacing its current outcome measure—which tracks those who obtained  suitable jobs relative to those who did not—with a measure that tracks  only the number of any positive outcomes. According to VA officials,  positive outcomes would include not just veterans who obtained suitable  employment, but also veterans who obtained other benefits (such as  MRG) from the VR&E program. Good practices for performance  measurement suggest that measures should allow agencies to  transparently report their success to the Congress and public, and identify  and address performance variances across their programs. The  proposed measure of specific numbers of positive outcomes lacks this  transparency because it does not account for veterans who left the  program without obtaining suitable employment or other discernable  benefits. It also obscures VA’s success in achieving its core mission  because it does not separate veterans who obtained suitable employment  from those who received other, lesser benefits. Finally, measuring only  numbers of positive outcomes is less effective for monitoring and  managing performance across regional offices—that is, comparing each  office’s relative rate of success to identify those that may serve as models  or require further oversight.
Aside from considering changes to its outcome measures, VA has taken  other steps to help regional offices better manage their interrupted cases,  but it is unclear how successful these efforts have been. For example, VA  uses performance goals to hold regional offices and VRCs accountable  for monitoring interrupted cases and keeping their proportions below  certain thresholds. The current goal is 11 percent for regional offices and  12 percent for individual VRCs. Also, VR&E’s central office has provided  tools and training to help VR&E managers monitor the number and  average time of cases in interrupted status, and percent in interrupted  status over 180 days. VR&E managers at two regional offices we  contacted noted that they had VRCs close old, inactive cases, regardless  of the short-term impact on the office’s rehabilitation rate. VA reported  that the overall percentage of interrupted cases declined from 10.4  percent in fiscal year 2010 to 9.6 percent in fiscal year 2012. However,  managing interrupted cases remains a challenge. According to VA data,  in July 2013, 39 percent of more than 15,000 interrupted cases had been  in that status for more than 180 days. In addition, 8 of 57 regional offices  had at least 50 percent of their interrupted cases in that status for more  than 180 days, ranging up to 70 percent at one office.
section_title: VA Lacks Performance Measures of Veterans’ Long- term Employment Success
paragraphs: The mission of the VR&E program includes helping veterans maintain  suitable employment, and VA holds itself accountable for achieving this  part of the mission by counting job placements as rehabilitations if they  are maintained for at least 60 days. According to several regional office  managers and staff we interviewed, cases are typically closed when  veterans have maintained employment for this 60-day minimum, as  problems with a veteran maintaining employment were most likely to  occur in the first 60 days of employment if they occurred at all. However,  if a veteran appears to be at high risk of losing a job, for example, the  VRC can keep the case open for a longer period to provide additional  supports. Some VRCs noted that this can occur when a veteran has a  serious mental health condition, for example. According to VA data on  about 38,000 veterans who applied for VR&E between fiscal years 2003  and 2012 and have been rehabilitated to employment, over one-third  were tracked in employment for 6 months or longer before having their  cases closed as rehabilitated, and some were tracked for 1 year or  more. A VA official noted that some of this time is spent completing  documentation and approval of case closures. In addition, one regional  VR&E manager noted that closure may be delayed when the veteran  does not inform the regional office that he or she obtained a job.
Research and practices at other federal employment programs suggest  that tracking employment for 60 days may not be sufficient to gauge  whether a veteran will be able to maintain employment. For example, a  2013 National Organization on Disability study suggested that wounded  warriors transitioning to civilian employment need support for an extended  time after obtaining jobs. Meanwhile, the Department of Labor (Labor)  measures individuals’ employment retention over a longer term, 180  days. This measure applies to many of Labor’s employment programs, including its veterans employment programs.
VA officials stated that they plan to collect information on veterans’ long- term employment, but did not indicate whether they plan to develop  performance measures to show the program’s impact on veterans’ long- term employment. Specifically, VA is considering contacting rehabilitated  and discontinued VR&E participants at 6 and 12 months after they finish  the program. This effort could yield information on employment over the  longer term. VA plans to use this information to identify veterans who are  having difficulties in maintaining employment, and possibly provide them  additional supports. However, without developing performance measures,  VA may be missing an opportunity to track the long-term effectiveness of  the VR&E program.
section_title: VA Generally Followed Recognized Practices in Designing Its Customer Satisfaction Survey, but the Quality of Benchmark Results is Uncertain
paragraphs: VA has made progress toward holding itself accountable for VR&E  participants’ satisfaction by implementing its new Voice of the Veteran  survey, but the agency has not incorporated practices necessary to  ensure the quality of benchmark results. VA intends to use survey data to  inform program improvements and training developments, and potentially  to report on veterans’ satisfaction with VR&E. In developing and piloting  the survey, VA followed generally recognized survey practices, including  detailed survey planning, comprehensive questionnaire development and  testing, and extensive data collection activities, such as follow-up with  veterans who did not respond to VR&E’s initial survey request.However, VA did not perform a nonresponse analysis to determine the  effects of any nonresponse bias on the quality of its pilot survey data.  Guidance from the Office of Management and Budget (OMB) states that  agencies should perform nonresponse analysis when a survey’s unit  response rate is below 80 percent, to identify the possibility of bias in a  survey’s results, and measure, adjust for, report, and analyze  nonresponse to assess its effects on data quality. VA’s plan for the pilot  survey, as approved by the OMB, anticipated the need for this analysis,  but its contract with the survey administrator made it optional. Although  the response rates for the three pilot surveys of VR&E participants each  fell below the 30 percent level specified in VA’s sample design, VA  officials said they elected not to conduct a nonresponse analysis due to  budget constraints. Moving forward, VA officials stated there is no  expectation that a nonresponse analysis be performed for VA’s ongoing  survey effort unless deemed necessary. Currently, VA’s approved  contract for its ongoing survey effort does not include an option for a  nonresponse analysis if response rates are low.
section_title: VA Workload Management Efforts May Not Fully Address Challenges Facing Some Regional Offices
paragraphs: VA has acted to strengthen VR&E workload management by  commissioning a study of workload management issues, among other  things. The study team collected and analyzed data on how VR&E staff at  four regional offices spent their time on a variety of tasks, such as  paperwork and client contacts. In its April 2011 report, the study team  recommended, among other things, that VA improve its approach for  estimating its overall staffing needs. VA now has an ongoing initiative to  assess the total number of VR&E staff needed across the regional offices.  This initiative, intended for use in formulating future VR&E administrative  budget requests, is expected to be completed by March 2014. Also, in  response to an internal business process review, VA has made changes  to VR&E procedures intended to ease VRCs’ workloads by reducing  paperwork.
Despite these efforts, some regional offices struggle with high caseloads  that may affect quality of services to veterans, while others do not,  suggesting potential shortcomings in the way VA allocates resources.  VR&E managers and staff in four of the offices we contacted said high  workloads impede their ability to spend sufficient time working with  individual veterans. Meanwhile, VRCs in other offices we contacted said  their workloads were manageable. VA administrative data indicate that  average ratios of VR&E cases to VRCs vary considerably, as shown in  figure 7. For example, while VA reported a national average of 139 cases  per VRC as of March 31, 2013, regional office average caseloads ranged  from 64 in Fargo, North Dakota to 579 in White River Junction, Vermont. As of March 31, 2013, seven offices averaged fewer than 100 cases per  VRC, while eight averaged more than 175 cases per VRC.
According to VA, White River Junction had only one VRC, although another was added  after March 31, 2013. In commenting on a draft of this report, VA stated that it expects  another vacancy to be filled in January 2014, and White River Junction’s average to drop  to about 160 cases per VRC.
In some cases veterans may work with multiple VRCs even if an office does not take a  specialization approach. For example, if a VRC retires, that VRC’s cases would be  redistributed among other VRCs.
Officials at the regional offices we contacted had different opinions about  the efficacy of VRCs specializing by phase of the VR&E process. Some  VR&E managers and staff cited advantages to allowing a single VRC to  handle a case: for example, a VRC can (1) develop a stronger rapport  with the veteran; and (2) avoid possible confusion, including among the  veterans, related to handoffs between VRCs. Other regional officials cited  advantages of specialization. For example, the VRC can: (1) manage  caseloads without being distracted by a larger range of tasks (for  example, assessing entitlement, preparing rehabilitation plans and  monitoring veterans’ progress on their plans); and (2) develop expertise in  one phase of the VR&E process. VR&E managers at one of the regional  offices we contacted stated that they plan to move away from a more  specialized caseload management approach, citing improved efficiency  from having one VRC handle a case. Meanwhile, managers in another  office said they had recently moved towards a specialization approach  because they feel it better enables them to handle their large caseload.
Federal internal control standards require that managers obtain  operational data to ensure their programs are run effectively and  efficiently. For VA, this could include data on how staff resources are  being utilized across regional offices, and how such usage affects offices’  efficiency and effectiveness. It is especially important that VA understand  regional offices’ approaches and their effects, given our finding that  veterans who work with a number of different VRCs over time are less  likely to be successful. Moreover, the April 2011 consultants’ report to VA  on VR&E workload management—which also found wide variations in  how VRCs at different offices allocated their time among different tasks— recommended that VA assess regional office approaches to assigning  tasks among VRCs in order to identify best practices for helping to  rehabilitate veterans, while balancing standardization across offices with  flexibility to serve each office’s service area. However, VA central office  does not collect information on how regional offices manage their  caseloads and assign tasks among VRCs, and has not assessed the  effect of varied regional office approaches on case processing efficiency  and veterans’ rehabilitation success. VR&E central office officials stated  that they prefer to leave workload management to regional office  managers because they are in the best position to determine which  workload management approach will enable the office to meet its  program goals.
section_title: VA Is Addressing Redundancy and Most Gaps in VR&E Staff Training
paragraphs: In the course of our work, we found evidence that staff had experienced  redundancy and critical gaps in their training; however, VA had taken  recent steps to address the redundancy and most, but not all of the  training gaps. Principles of effective strategic training and workforce  development call for avoiding duplication, identifying workforce  competency gaps, and addressing these gaps. Nonetheless, some of  the VR&E regional managers or staff at six of the eight offices we  contacted told us that the annual curriculum offered to VRCs and ECs  required them to repeat courses. Their experience notwithstanding, we  found that VA has recently been adding new courses and, by fiscal year  2013, none of the courses on the curricula for experienced VRCs and  ECs were redundant with those offered in the prior year. Further, with the  addition of these new courses, VA is actively closing the nine critical  training gaps identified in a June 2010 skill gap analysis report. The  report noted that five gaps affect both new and experienced VRCs and  four additional gaps affect only new staff. VBA contracted for this analysis  in response to a report we issued in 2009, in which we found that VR&E  had not collected data on the critical skills and competencies needed by  VRCs to ensure they could meet the current and future needs of veterans   In September 2013, VA informed us that it had begun  in the program.providing training to staff to address these gaps and will have provided  training to address all the gaps by the end of 2013, and in 2014 will  provide two new courses for staff on mental health to improve their ability  to assist veterans with PTSD and other mental illnesses. See appendix III  for the gaps identified and training developed to address them.
Although VR&E closed many training gaps, our review of VR&E’s fiscal  year 2013 curriculum identified other gaps related to job placement and  accommodation for experienced VRCs. Staff or managers in five of the  eight regional VR&E offices contacted also identified either job placement  or workplace accommodation as areas in which additional training was  needed. For example:
Some regional managers and staff told us VRCs need more training in  job placement strategies, employment laws, and tax incentives for   Job placement skills are critical to achieving VR&E’s  employers.main mission to ensure that veterans with disabilities who can be  employed obtain and maintain suitable employment. While ECs are  generally responsible for assisting veterans with job placement, VRCs  may take on this task in offices that lack ECs, or when ECs simply are  not able to provide these services to veterans needing them.
Some regional managers and staff told us VRCs needed training in  current workplace accommodations to better assist veterans.  Workplace accommodations help ensure veterans with disabilities can  successfully perform on the job and maintain their employment. A  June 2012 VR&E-sponsored study confirmed the importance of  workplace accommodations to help veterans attain and maintain long- term employment and recommended that VR&E either train or retrain  its staff in workplace accommodation.
Although VA does not currently have specific plans to address either of  these gaps in fiscal year 2014, VA officials informed us that the agency  intends in the future to individually identify and address skill gaps for each  VRC and EC. In the meantime, regarding job placement training, VA  officials told us that VRCs may opt to take online training developed for its  ECs, but that VA does not plan to require VRCs to take job placement  training because VRCs receive it as part of their college or master’s  degree programs. However, VRCs may not opt to take optional VA  training as several VRCs and managers told us that VRCs lack time to  take all the required training. As of July 2013, VA officials had not yet  developed the curriculum for fiscal year 2014 and did not know if training  on workplace accommodation for VRCs would be included.
As for VA’s plans to identify and address each VRC’s and EC’s skill gaps,  this effort is part of its strategic initiative to provide competency-based  training for 16 critical positions throughout the agency. VA officials  informed us that under this initiative, which began in 2010, the agency  plans to design and provide tailored training and other performance  support that will be based on an assessment of each critical employee’s  current proficiency level. VR&E officials expect the outcomes of this effort  to include: (1) individual performance standards, (2) tailored performance  support, (3) an assessment baseline and developmental roadmap (i.e.,  training plan), and (4) a tailored National Training Curriculum and  remediation plan for each VRC and EC. Furthermore, through the  collection and management of competency data, VR&E officials hope to  track the knowledge, skills, abilities, education, and experience required  for specific positions and recruit, train, distribute, and professionally  develop personnel based on this competency framework. VA officials  reported that as of fiscal year 2012, 47 percent of employees in critical  and key occupations had participated in a competency-based training  program within the last 12 months, and that in June 2013 the agency took  an initial step to provide individualized training for VRCs. The program  and plans VA described are consistent with best practices in human  capital management.
section_title: Conclusions
paragraphs: VA has taken steps to improve its management of the VR&E program— ultimately with the goal of helping more veterans with disabilities attain  suitable employment—but its ongoing efforts may not sufficiently address  certain key management issues. With respect to performance measures,  VA’s efforts to address problems with its existing rehabilitation rate  measure may actually result in the adoption of a less transparent and  meaningful outcome measure, as it is not clear how measuring the  number of positive outcomes will provide the context necessary to  understand program success. Furthermore, while VA plans to collect  more information about longer-term outcomes, absent a performance  measure VA may miss an opportunity to hold staff accountable and make  adjustments to the program to promote success in maintaining  employment—consistent with VR&E’s central mission. In addition, while  VA’s ongoing customer satisfaction survey has the potential to provide  valuable information, without performing additional analyses to determine  whether responses actually reflect the experiences of VR&E participants  overall, the agency cannot be sure that its survey will ultimately produce  sound results for informing program management and training  investments.
VA has also made strides related to workload management and training,  but remaining shortcomings can negatively affect services provided to  veterans. Because VA’s formula for allocating staff among regional offices  rewards higher-performing offices, which can make it harder for lower- performing offices to improve, but does not consider educational and  vocational counseling cases, which are concentrated in certain regions,  some offices’ ability to serve veterans may be negatively affected.  Lacking information on the varied approaches that regional offices are  using to manage workloads also undermines VA’s ability to help offices  optimize their use of staff to serve veterans in the most effective manner.  Finally, training of VR&E staff is an area in which VA has made  substantial improvements, but absent additional training on job placement  and accommodations, counselors may be hindered in their ability to help  veterans find and maintain employment.
section_title: Recommendations for Executive Action
paragraphs: The Secretary of Veterans Affairs should direct the Under Secretary for  Benefits to take the following actions:  1.  In any revised set of national and regional performance measures for  the VR&E program, include measures of (a) the proportion of program  participants successfully rehabilitated into employment, and (b) the  proportion of participants who obtain other benefits from VR&E  services.  2.  Develop new measures of long-term employment that go beyond the  minimum 60 days of post-placement monitoring that is currently  required. In developing measures, consider the feasibility of using  results from planned post-closure surveys of veterans as a data  source.  3.  Consistent with generally accepted survey practices and as warranted  by survey response rates, conduct nonresponse analysis of the  results of VA’s ongoing Voice of the Veteran customer satisfaction  surveys.  4.  In revisiting VA’s formula for allocating VR&E staff among the regional  offices, (a) assess the inclusion of factors related to regional office  performance and if warranted remove them from the formula, and (b)  assess the exclusion of any factor related to the number of  educational counseling cases in each regional office and If warranted  add such a factor.  5.  Collect information on the regional offices’ approaches for managing  their VR&E workloads, assess the advantages and disadvantages of  these approaches, and use the results of this assessment to provide  guidance to the offices on potential best practices or options to  consider.  6.  Provide additional training to all individual VRCs on job placement  strategies and workplace accommodations, potentially as part of the  effort to develop a competency-based training approach.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to VA for review and comment. In its  written comments, produced as appendix IV in this report, VA agreed with  our conclusions and generally concurred with our recommendations.  Specifically, VA concurred with 5 of our 6 recommendations, and noted  steps it plans to take to address them, as follows.
Regarding our recommendation to measure the proportions of  VR&E participants achieving employment and other benefits, VA  said that while ratio-based metrics will not be included in its  revised employee-level performance measures, they will be  included in the revised national-level measures.
Regarding our recommendation to conduct nonresponse bias  analysis of its Voice of the Veteran survey results, VA said that—if  funding permits—it will modify its survey administration contract to  include such an analysis.
With regard to our recommendation to assess certain aspects of  its formula for allocating staffing levels across regional offices, VA  said it plans to revisit the formula and make any changes  necessary.
With regard to our recommendation to assess regional offices’  workload management approaches, VA said it plans to collect and  analyze data on the various approaches and inform the regions  about best practices.
With regard to our recommendation to close certain gaps in staff  training, VA said it plans to develop and make available to staff  new training on employment and accommodations.
VA agreed in principle with our recommendation to develop performance  measures related to VR&E participants’ longer-term employment  success. Specifically, VA agreed there is value in looking at long-term  outcomes, and noted its ongoing longitudinal study of a sample of  participants and its plans to collect data on veterans’ post-closure service  needs. However, VA also pointed out that VR&E counselors already work  with veterans for many years, including up to 18 months following training  completion. As such, VA indicated it does not see the value in directing  limited resources to implementing and executing additional post- placement measures or services that will extend program participation  and may create program dependencies. We acknowledge that VA in  some cases monitors veterans for 6 months or more after they are placed  in a job (as we note on page 24 of the report, this occurred in over one- third of the cases we examined). We also recognize the potential  challenges of additional post-employment monitoring. At the same time,  we believe that unless VA holds itself accountable for the longer-term  outcomes of all participants who get hired into a suitable job, it may miss  opportunities to make adjustments that better promote job maintenance.  We encourage VA to explore cost-effective means of checking on  veterans’ employment status at some point more than 60 days after job  placement.  We did not intend and do not believe this measure would  necessarily entail extending program services or participation beyond  current program practices. VA also provided technical comments, which  we incorporated in our report as appropriate.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Veterans Affairs, and other interested  parties. In addition, this report will also be available at no charge on  GAO’s website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or bertonid@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff members who made key  contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to examine (1) the outcomes for  veterans seeking employment through the Vocational Rehabilitation and  Employment (VR&E) program, and (2) the progress the Department of  Veterans Affairs (VA) has made in addressing critical management  issues. To address these objectives, we reviewed relevant federal laws,  regulations, and guidance, as well as VA documentation related to the  VR&E program and recent management initiatives. We also reviewed  several external studies and evaluations of the VR&E program, and  determined that their methodologies were sufficient to allow us to report  certain findings from these studies. We reviewed various criteria for  effective program management, including in the areas of performance  management and staff training. We analyzed VA administrative data on  VR&E participants who applied to the program between fiscal years 2003  and 2012 (see below for more information on our efforts to analyze and  ensure the reliability of these data), and interviewed VA central office staff  responsible for the VR&E program, VR&E managers and staff in VA  regional benefits offices (see below for details on office selection), and  representatives of the Disabled American Veterans and the Paralyzed  Veterans of America. Finally, to obtain input from veterans, we conducted  phone interviews with veterans who had participated in the VR&E  program (see below for details on how we developed our survey and  selected veterans) and analyzed data from a VA customer satisfaction  survey, after reviewing the methods VA used to develop and pilot the  survey (see below for more detail on our assessment of the survey).
subsections: section_title: Analysis of VA Administrative Data
paragraphs: We analyzed data from VA’s Corporate WINRS (CWINRS)  system,  which VA uses to track veteran cases through the VR&E process and  manage program costs. In our analysis, we included veterans who  initially applied to the VR&E program between fiscal years 2003—the  earliest year for which reliable data are available—and 2012. Unless  otherwise noted, we included only veterans who received VR&E services  designed to help veterans prepare for and obtain suitable jobs, and not  those who received other services—such as educational and vocational  counseling—which are also provided by VR&E staff. We analyzed data  on the characteristics of VR&E participants (e.g., age, type of disability);  the services they received from the VR&E program (e.g., evaluation,  training); and their outcomes.
To assess outcomes, we placed VR&E participants who received  employment-related services into one of four outcome categories: (1)  rehabilitated to employment, including those who at least once had a  case status of rehabilitated associated with achieving employment; (2)  rehabilitated to independent living, including those who never had an  employment-related rehabilitation but had least one rehabilitated status  associated with achieving an independent living goal; (3) discontinued,  including those who never had any rehabilitated status, and whose last  status as of the end of fiscal year 2012 was discontinued; and (4) still  participating, including those who never had a rehabilitated status and  whose last status as of the end of fiscal year 2012 indicated some type of  ongoing participation in the program. (See app. II for a description and  detailed results of our analysis of the association between VR&E  participants’ outcomes and their characteristics and services received.)
We found the CWINRS data to be sufficiently reliable for the purposes of  our report. To assess the reliability of the data, we reviewed  documentation on the system, interviewed VA officials, and performed  electronic testing. VA officials told us that CWINRS became a single,  national database during 2002, and that prior data are from separate, not  necessarily uniform, systems operated by individual regional offices and  are therefore less reliable. Based on this information, we chose to  analyze data only for fiscal years 2003 through 2012. Also, through  electronic testing we identified potential errors with outcome data for 1  percent of cases that received some employment-related services—for  example, cases closed as rehabilitations for reasons which could not  logically be associated with a rehabilitation. Because this made it  impossible for us to assign these cases to one of our four outcome  categories, we excluded them from our analysis of veterans’ outcomes.  Finally, VA officials told us there may be some inaccuracies in data  related to the status of cases—in particular for data prior to fiscal year  2008—because of a programming design flaw that allowed correct case  status data in CWINRS to be inadvertently overridden with obsolete data  from another VA system. Therefore, we use approximations rather than  exact numbers when reporting the number of separate occasions on  which veterans received certain services through the VR&E program,  such as evaluation and planning assistance, job ready services, and  rehabilitation to employment.
section_title: Regional Office Interviews
paragraphs: To obtain perspectives on the challenges VR&E staff face in helping to  rehabilitate veterans, we interviewed VR&E managers and staffs at 8 of  VA’s 57 regional offices. We selected offices to achieve diversity among  offices in the following characteristics:
Geography: We selected at least one office in each of the four  Veterans Benefits Administration areas.
Workload: Based on VR&E administrative data, we ranked offices by  the number of initial fiscal year 2011 VR&E applications filed from  largest to smallest numbers of applications.
Outcomes: Based on VA’s ASPIRE VR&E performance dashboard,  we ranked offices by their Employment Rehabilitation RateSeptember 2012 from highest to lowest percentage rate.
Timeliness: Based on VA’s ASPIRE VR&E performance dashboard,  we ranked offices according to the average days required to complete  the evaluation and planning phase during September 2012 from  lowest to highest.
Unemployment: We considered the November 2012 unemployment  rate for each office’s state, as reported by the Bureau of Labor  Statistics.
See figure 8 for details about the characteristics of each regional office  we contacted.
We visited the regional offices in Washington, D.C., and Oakland,  California, and conducted telephone interviews with VR&E managers and  staff at the other six offices. At each of the eight offices, we interviewed  the VR&E Officer (and one or more Assistant VR&E Officers at four  offices); at least three VRCs; and at least one EC. At our in-person visits  to Washington and Oakland, we also interviewed representatives of the  Disabled American Veterans and California Department of Veterans  Affairs. Our interview results cannot be generalized to all regional offices’  VR&E operations.
section_title: Interviews with Veterans
paragraphs: We conducted phone interviews with 17 veterans who had participated in  the VR&E program, including 9 who had discontinued from the program  and 8 who had achieved rehabilitation to employment. We pretested our  interview questions twice with each group, and used the results of the  pretests in our final analysis. We asked questions about the veterans’  employment, their interactions with their VR&E counselors, and their  satisfaction with various VR&E services received. We randomly selected  veterans for interviews, but cannot generalize the results of the interviews  to the overall population of VR&E participants. To select our sample of  veterans, we used CWINRS data to identify VR&E participants who had  received some employment-related services and had either discontinued  or achieved an employment-related rehabilitation between April 1, 2012  and September 30, 2012. We attempted to contact 49 discontinued  veterans and 32 rehabilitated veterans after sending an advance  informational email. These contact attempts resulted in complete  interviews with 17 veterans.
section_title: Analysis of Voice of the Veteran Survey
paragraphs: We obtained aggregate results from the pilot of VA’s Voice of the Veteran  customer satisfaction survey directly though VA’s dashboard—a central  database that contains the agency’s performance metrics and customer  satisfaction reports. This survey was administered to random samples of  veterans from different customer populations. Each sample received a  questionnaire reflecting the VA benefits and services they received. We  obtained data from the three questionnaires targeted to VR&E  participants: the Enrollment survey, the Servicing survey, and the  Escaped Beneficiary survey. All three VR&E pilot surveys were fielded  from October 2012 to January 2013. Table 2 provides background  information on each of these surveys, including a description of the  survey’s purpose and target sample population.
Based on interviews with agency officials and contract representatives for  the Voice of the Veteran surveys, and our review of documentation of the  surveys’ methodology, we concluded that these three VR&E surveys  were generally developed and piloted in accordance with generally  recognized survey research practices, and that the results we cite in this  report are sufficiently reliable for our analysis. We used standards and  guidelines issued by the Office of Management and Budget (OMB) to  assess the methods VA used to design and administer the surveys. In  addition, we reviewed survey guidelines of generally recognized  professional organizations such as the American Association for Public  Opinion Research and the American Statistical Association.
However, we concluded that the pilot survey results cited should be  interpreted as representative of only those veterans responding to the  survey. Response rates for the VR&E surveys were low—between 16  percent and 25 percent—and fell below the expected 30 percent rates  specified in VA’s sample design. Table 3 provides more information on  sample outcomes. While response rates may not be highly correlated with  the presence of nonresponse bias and are not considered, by  themselves, to be sufficient indicators of the quality of survey results,  response rates are a useful indicator of the risk of nonresponse bias.  According to OMB’s guidelines, a survey’s response rate is a valuable  data quality and performance indicator, and is associated with the  generalizability of a survey’s results. In addition, OMB’s guidelines state  that a nonresponse analysis should be performed when a survey’s unit  response rate is below 80 percent, to identify the possibility of bias in a  survey’s results, and that agencies should appropriately measure, adjust  for, report, and analyze nonresponse to assess its effect on data quality.  Because VA did not perform a nonresponse analysis to determine if  systematic differences exist between respondents and nonrespondents, it  is not possible to generalize the results to the broader population of  VR&E participants. Also, because the effects of any nonresponse bias on  data quality were not assessed, it is not known if any systematic  differences exist between nonrespondents and respondents.
section_title: Appendix II: Statistical Analysis of Administrative Data on Vocational Rehabilitation and Employment Outcomes
paragraphs: We used a statistical method known as survival analysis to estimate  Vocational Rehabilitation and Employment (VR&E) participants’ likelihood  of achieving success, their program completion times, and the factors that  were associated with rehabilitation. Survival modeling methods provide a  simple and efficient way to estimate the rate of event occurrence when  the duration times of the events are “censored,” meaning that they are  unobserved over some portion of the time period of interest. For example,  a rehabilitation time would be censored if a veteran had been in a  program for 2 years but had neither dropped out nor finished the program  at the time of analysis. Although the veteran’s rehabilitation time is at  least 2 years, the ultimate time to rehabilitation or discontinuation is not  yet known (censored). We used a specific type of survival method, Cox  regression models, to identify predictors of program completion—or what  the Department of Veterans Affairs (VA) defines as “rehabilitation”—and  estimate the rate at which rehabilitation occurred at various times after  application among various groups of veterans.
Cox models allowed us to analyze the data without needing to make  strong assumptions about the rehabilitation process. We did not need to  assume that the likelihood or rate of rehabilitation changed over time in  specific ways, as we would with other survival methods. The flexibility of  Cox models comes at some cost of interpretation, however. The  mathematics of Cox models make it more difficult to estimate the  probability of rehabilitation by certain follow-up times (known as “failure  probabilities”). To avoid the complex computation required for these  estimates, we generally report simpler “Kaplan-Meier” estimates of  rehabilitation probabilities in the body of the report—a form of survival  analyses that allows us to report on likelihood of program completion, but  without controlling for other covariates. To corroborate these results, we  use Cox models to estimate the “hazard rate” of rehabilitation in this  appendix, which is loosely equal to the rate of rehabilitation at a certain  time after starting a VR&E program, while controlling for other factors.  The results of this analysis are presented in this appendix and essentially  support the same substantive findings about factors associated with the  chance of rehabilitation discussed in the body of this report.
subsections: section_title: Source Data and Measurement of Key Variables
paragraphs: For the survival analysis using Cox models and Kaplan-Meier methods,  we used data from the VR&E program’s Corporate WINRS system, which  includes archived data on program participants’ demographic  characteristics, services received, and outcomes achieved. We selected  data on veterans who began their first VR&E programs in fiscal year 2003  through fiscal year 2012. We limited our analysis to this time period after  VA identified reliability problems with older data. Appendix I describes the  source data and their reliability in more detail.
In initial exploratory analysis, we considered various factors that could be  associated with successful program completion and duration. These  factors (which we also may refer to as covariates, variables, or predictors)  included age, sex, education, branch of service, number of counselors,  number of discontinuations, severity of disability, type of disability, year of  first application to the program, VR&E services provided (“service track”),  and the regional offices that managed veterans’ cases.
A key strength of the VR&E administrative data is its volume. The total  number of veterans available for analysis, using the population defined  above, was 133,679. Table 4, below, shows how the data were  distributed across years of program entry and program outcomes. The  large sample size available for analysis allowed us to estimate valid and  precise relationships between completion times and a large number of  covariates. We confirmed the adequacy of the sample by examining the  cell sample sizes in tabulations of potential covariates.
We assessed the degree of correlation among the initial list of covariates,  in order to identify those that would be redundant or introduce the  possibility of spurious estimates. Most of the potential covariates were not  strongly intercorrelated, except for the number of counselors and  discontinuations, the number of counselors and regional offices, and age  and era of service. The pair-wise correlations for these covariates ranged  from 0.37 to 0.52. Including all of these covariates in our model would  have inflated the estimated relationships between them and program  completion times. Therefore, we included age and number of counselors  in the final list of covariates, and excluded era of service, number of  discontinuations, and number of offices from our model.
Measuring most of the covariates of interest was straightforward, and can  be determined for a subset of meaningful predictors, as shown in Table 6  below. However, measuring certain covariates—such as regional offices,  certain VR&E services, and disability type—required more complex  measurement decisions.
Because veterans can be served by a number of regional offices over  time, measuring the offices serving each veteran is challenging. In our  data, veterans’ cases were managed by more than 4,100 permutations of  regional offices. The large number of permutations made it difficult to  meaningfully analyze variation across offices, and, in any case, produced  sample sizes that were too small for modeling purposes. To simplify the  measurement, we focused on the last office where each veteran was  enrolled, as a measure of regional office at program completion. Because  veterans could have discontinued their programs multiple times and could  have been served by as many as six offices, the last office was a logical  proxy for the office that could have influenced the final outcome most  strongly.
Because we selected only veterans from the VA administrative data who  received employment-related services, this group received job training or  job placement assistance. We created a single measure of services  received by creating categories for specific services related to  employment, such as job placement services, and specific types of job  training received, such as high school or college education. Table 5 below  identifies the services we measured.
Finally, veterans could have been diagnosed with one or more disabilities,  such as mental health conditions, ear disorders, and/or cardiovascular  disorders. This allows for a potentially large number of unique  combinations of disorders, which makes it difficult to create mutually  exclusive and exhaustive groups. To avoid this problem, we compared  veterans who had each type of disorder to veterans who did not have the  disorder, using binary indicators for the presence or absence of each  diagnosis. While the possibility of multiple disabilities makes these  indicators mutually inclusive, our measurement scheme provided a  concise way to collapse the many unique combinations of disorders.
section_title: Survival Model and Statistical Assumptions
paragraphs: The purpose of survival modeling is to estimate the probability of an event  by various follow-up times or the average rate of event occurrence at any  one follow-up time. Because these two quantities are mathematically  equivalent, we used two different methods that produce similar results but  vary in their ease of interpretation. where nj is the number of veterans who are “at risk” of rehabilitation and  dj is the number of rehabilitations observed at tj. We estimated failure  probabilities for various subgroups of veterans, in order to describe the  association between these factors and the chance of rehabilitation by 5  and 8 years.
To corroborate these associations and control for alternative variables  that could account for them, we used multivariate survival models—and  the Cox Model for Proportional Hazards, in particular—to estimate the  average rate of event occurrence at any one time in an observation  period, also known as the “hazard rate.”interest is rehabilitation, and the time period of observation is fiscal year  2003 through fiscal year 2012. In our analysis, we defined censored  outcomes as a veteran’s remaining in the program at the end of fiscal  year 2012 or having discontinued from a program without yet re-applying  (see table 5 above).
The formula for our Cox regression model was  λi(t)= λ0(t)exp(xi b) ,  where λi(t, xi) is the hazard rate for veteran i at time t, xi = xi1 ,..., xip is a  vector of p covariates, and xib is a linear combination of covariates and  coefficients, with no intercept term. As described above and in table 6, the  covariates included: age, sex, job training services, the interaction of sex  and job training services, education, branch of service, number of  counselors, severity of disability, and disability type. The term λ0(t) is the  “baseline hazard rate,” or the rate implied when all covariates equal zero.  (Note that the equation simplifies to λ0(t) when this is true.)
The Cox Model assumes that the hazard rates are proportional across  values of the covariates throughout the observation period. If the hazard  in one group exceeds that of a reference group, the model assumes that  the proportional difference in rates will remain constant, and that the  difference in hazard rates across groups does not decline, increase, or  reverse direction over time. For example, the model assumes that the  rehabilitation rate for veterans with mental health conditions differs from  the rate for veterans without this type of condition by a constant amount  throughout the observation period.
However, because a particular data generation process may not satisfy  the proportional hazards assumption, we plotted the estimated hazards  for various values of each covariate over time and formally tested the  assumption for each covariate using statistical tests based on the model’s  Shoenfeld residuals. We found a non-zero slope in a generalized linear  regression of the Schoenfeld residuals on functions of time for several  variables. A non-zero slope indicates that the hazards are not  proportional across values of the covariates over time.across all variables further suggested that the hazard function was not  proportional.
To identify the most serious departures from the proportional hazard  assumption, we plotted the cumulative hazard function over time for the  covariates that failed the Schoenfeld test. These plots suggested that the  last office and first year of application covariates had strongly  disproportional hazards. The remaining variables only failed to meet the  proportional hazard assumption in a category coded for “missing data,”  and the other variables that appeared to be disproportional were not  meaningfully associated with rehabilitation times.
To adjust for most seriously non-proportional hazards, we stratified the  model by last office and first year of application. Stratification allowed us  to control for these two factors while also allowing their hazard functions  to vary over time across offices and years of application. After stratifying  by last office and first year of application, the proportional hazard  assumption for the model was more consistent with our data.
section_title: Results
paragraphs: Our analysis identified a number of factors that had strong and  statistically significant associations with rehabilitation times, as shown in  Table 5.
Veterans took more time to complete their programs if multiple counselors  worked with them. Average completion times increased at a higher rate  as the number of counselors increased from 1 to 5 than from 6 to 9 or  more. For example, veterans with 2 counselors had average rehabilitation  times that were 29 percent slower, on average, than veterans with 1  counselor. In contrast, veterans with 9 or more counselors had average  rehabilitation times that were about 1 percent slower than veterans with 8  counselors. In exploratory analysis, we found that veterans who more  often discontinued and re-entered the program and who were served by  multiple regional offices were more likely to work with multiple counselors,  and that discontinuing more frequently was associated with slower rates  of rehabilitation. However, when we estimated Cox models that controlled  for the number of times discontinued and the number of regional offices  used, we found that the number of counselors still had a meaningful  association with success. Nevertheless, a reciprocal relationship, in which  longer program times are associated with more counselors, may also  explain this pattern.
The presence of mental health conditions substantially decreased  completion rates. Veterans with mental health conditions completed their  programs at a rate that was about 27 percent slower on average than  veterans who did not have mental health conditions.
As expected, the type of services veterans received was among the  strongest predictors of completion times. Veterans who received job  placement services finished their programs about 4.5 times more quickly  than veterans who were pursuing an undergraduate education. This is  consistent with the nature of VR&E’s job placement services, which focus  on veterans who already have the skills to pursue employment. Veterans  who received apprenticeships, non-vocational training, unpaid work at a  federal agency, vocational or technical education, or non-degree college  credit hours finished their programs about 1.6 to 2.4 times more quickly  than veterans pursuing an undergraduate college degree. In contrast, the  average completion rate for veterans pursuing a graduate degree was  only 28 percent quicker than for veterans pursuing an undergraduate  degree. As a whole, these results suggest that veterans pursuing  undergraduate or graduate degrees require more time to complete their  programs than veterans receiving other training.
The amount of prior education was associated with faster program  completion rates. As compared to veterans with only a high school  degree, veterans with some college, an undergraduate degree, or a  graduate degree had completion rates that were 1.2 to 1.5 times faster,  on average. Because these estimates control for the type of services  received and the type and severity of disabilities, veterans with more prior  education appear to have finished more quickly regardless of the training  they received or the nature of their disabilities.
Finally, veterans with disability ratings of less than 100 percent finished  their programs more quickly than veterans with disability ratings of 100  percent, with completion rates that were about 1.5 times more quickly on  average.disabilities face greater challenges to timely program completion, even  holding constant factors such as type of training, prior education, and  age.
section_title: Appendix III: Vocational Rehabilitation Counselor Training Gaps Identified by VA in 2010 and the Agency’s Efforts to Address
paragraphs: Training gaps     New VRCs  How to provide services to veterans recently  discharged from the military  How to provide services to veterans with severe  disabilities  How to determine when extended evaluation is  necessary and conduct subsequent individualized  extended evaluation plan development  How to closeout cases for different veteran outcomes  (e.g., for veterans who achieved suitable employment  or maximum rehabilitation gain)
For new and experienced VRCs   How to provide services to veterans with traumatic  brain injuries (TBI)
How to provide services to veterans with post- traumatic stress disorder (PTSD)
How to conduct a preliminary and/or comprehensive IL  assessment  How to input benefit awards into the computer system   (e.g., inputting the decision to provide a veteran a  subsistence allowance)
2-hour course on VR&E fiscal accuracy and integrity Two 1-hour courses on VR&E processing of  subsistence allowance awards 1-hour course on employment adjustment allowances VA added this course to the fiscal year 2013 curriculum to address changes to its guidance for  independent living and delivered the training in September, 2013.  VA has developed the course and will assign it as part of the fiscal year 2014 curriculum to be  completed by the end of the first fiscal quarter (i.e., by December 31, 2013).
VA used existing courses to train VR&E staff on inputting benefit awards into its computer system.
section_title: Appendix IV: Comments from the Department of Veterans Affairs
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact listed above, individuals making key  contributions to this report were Michele Grgich (Assistant Director),  Jessica Botsford, Frederick Caison, Elizabeth Curda, Kevin Daly, Holly  Dye, Julie DeVault, Brenda Farrell, Alex Galuten, Shannon Grabich, Mitch  Karpman, Kirsten B. Lauber, Lorin Obler, Carl Ramirez, Almeta Spencer,  Jeff Tessin, Greg Whitney, and Ashanta Williams.
section_title: Related GAO Products
paragraphs: VA Vocational Rehabilitation and Employment Program: Improved  Oversight of Independent Living Services and Supports Is Needed.  GAO-13-474. Washington, D.C.: June 7, 2013.
Veterans’ Employment and Training: Better Targeting, Coordinating, and  Reporting Needed to Enhance Program Effectiveness. GAO-13-29.  Washington, D.C.: December 13, 2012.
Federal Information System Controls Audit Manual (FISCAM).  GAO-09-232G. Washington, D.C.: February 2009.
VA Vocational Rehabilitation and Employment: Better Incentives,  Workforce Planning, and Performance Reporting Could Improve  Program. GAO-09-34. Washington, D.C.: January 26, 2009.
Multiple Agencies Provide Assistance to Service-disabled Veterans or  Entrepreneurs, but Specific Needs Are Difficult to Identify and  Coordination Is Weak. GAO-09-11R. Washington, D.C.: October 15,  2008.
Disabled Veterans’ Employment: Additional Planning, Monitoring, and  Data Collection Efforts Would Improve Assistance. GAO-07-1020.  Washington, D.C.: September 12, 2007.
Highlights of a GAO Forum: Modernizing Federal Disability Policy.  GAO-07-934SP. Washington, D.C.: August 3, 2007.
VA Vocational Rehabilitation and Employment: Service Contract  Management Is Improving, but Challenges Remain. GAO-07-568R.  Washington, D.C.: April 23, 2007.
Federal Disability Assistance: Wide Array of Programs Needs to Be  Examined in Light of 21st Century Challenges. GAO-05-626. Washington,  D.C.: June 2, 2005.
Vocational Rehabilitation: VA Has Opportunities to Improve Services, but  Faces Significant Challenges. GAO-05-572T. Washington, D.C.: April 20,  2005.
VA Vocational Rehabilitation and Employment Program: GAO Comments  on Key Task Force Findings and Recommendations. GAO-04-853.  Washington, D.C.: June 15, 2004.
Internal Controls Management and Evaluation Tool. GAO-01-1008G.  Washington, D.C.: August 2001.
Standards for Internal Control in the Federal Government.  GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.