Veterans' Benefits: Increased Focus on Evaluation and Accountability Would Enhance Training and Performance Management for Claims Processors
Released: 2008-05-27T00:00:00

Letter

Background

In fiscal year 2007, the Department of Veterans Affairs (VA) paid about $37.5 billion in disability compensation and pension benefits to more than 3.6 million veterans and their families. Through its disability compensation program, the VBA pays monthly benefits to veterans with service- connected disabilities (injuries or diseases incurred or aggravated while on active military duty). Monthly benefit amounts vary according to the severity of the disability. Through its pension benefit program, VBA pays monthly benefits to wartime veterans with low incomes who are either elderly or permanently and totally disabled for reasons not service- connected. In addition, VBA pays dependency and indemnity compensation to some deceased veterans’ spouses, children, and parents and to survivors of servicemembers who died while on active duty.

When a veteran submits a benefits claim to any of VBA’s 57 regional offices, a Veterans Service Representative (VSR) is responsible for obtaining the relevant evidence to evaluate the claim. For disability compensation benefits, such evidence includes veterans’ military service records, medical examinations, and treatment records from VA medical facilities and private providers. Once a claim is developed (i.e., has all the necessary evidence), a Rating Veterans Service Representative (RVSR) evaluates the claim, determines whether the claimant is eligible for benefits, and assigns a disability rating based on degree of impairment. The rating determines the amount of benefits the veteran will receive. For the pension program, claims processing staff review the veteran’s military, financial, and other records to determine eligibility. Eligible veterans receive monthly pension benefit payments based on the difference between their countable income, as determined by VA, and the maximum pension amounts as updated annually by statute. In fiscal year 2007, VBA employed over 4,100 VSRs and about 1,800 RVSRs to administer the disability compensation and pension programs’ caseload of almost 3.8 million claims.

In 2001 the VA Claims Processing Task Force noted that VSRs were responsible for understanding almost 11,000 separate benefit delivery tasks, such as tasks in claims establishment, claims development, public contacts, and appeals. To improve VBA’s workload controls, accuracy rates, and timeliness, the Task Force recommended that VA divide these tasks among a number of claims processing teams with defined functions. To that end, in fiscal year 2002, VBA developed the Claims Processing Improvement model that created six claims processing teams, based on phases of the claims process. (See table 1.)

According to one VA official, new claims processing staff generally begin as VSRs and typically have a probationary period of about one year. After their probationary period ends, staff can either continue to qualify to become senior VSRs or apply for RVSR positions. VSRs are also given the option to rotate to other VSR claim teams to gain a broader understanding of the claims process.

VBA Has a Uniform Training Curriculum for New Claims Processors and an Annual Training Requirement for All Claims Processors, but Staff Are Not Held Accountable for Meeting This Requirement

VBA has established a standardized curriculum for training new VSRs and RVSRs on how to process claims, and it has an 80-hour annual training requirement for both new and experienced staff; however, it does not hold individual staff accountable for meeting this requirement. VBA has designed a uniform curriculum for training new VSRs and RVSRs that is implemented in three phases—initial orientation training, a 3-week training session referred to as centralized training, and comprehensive on- the-job and classroom training after centralizing training. It also requires all staff to meet an annual 80-hour training requirement. To ensure that staff meet this requirement, each regional office must develop an annual training plan, which can contain a mix of training topics identified by VBA central office and by the regional office. However, individual staff members are not held accountable for meeting their training requirement.

Training for New Staff Is Conducted in Three Stages Using a Uniform Curriculum

VBA has a highly structured, three-phased program for all new claims processors designed to deliver standardized training, regardless of training location or individual instructors. (See fig. 1.) For example, each topic included in this training program contains a detailed lesson plan with review exercises, student handouts, and copies of slides used during the instructor’s presentation. Each phase in this program is designed to both introduce new material and reinforce material from the previous phase, according to a VBA official.

According to VBA policy, the first phase of training for new VSRs and RVSRs is prerequisite training. New VSRs and RVSRs begin prerequisite training at their home regional office as soon as they begin working. Prerequisite training lays the foundation for future training by introducing new VSRs to topics such as the software applications used to process and track claims, medical terminology, the system for maintaining and filing a case folder, and the process for requesting medical records. Although VBA specifies the topics that must be covered during prerequisite training, regional offices can choose the format for the training and the time frame. New VSRs and RVSRs typically spend 2 to 3 weeks completing prerequisite training in their home office before they begin the second program phase, centralized training.

During what is referred to as centralized training, new VSRs and RVSRs spend 3 weeks in intensive classroom training. Participants from multiple regional offices are typically brought together in centralized training sessions, which may occur at their home regional office, another regional office, or the Veterans Benefits Academy in Baltimore, Maryland. According to VBA officials in three of the four offices we visited, bringing together VSRs and RVSRS from different regional offices helps to promote networking opportunities, while VBA officials from two of these offices also stated that it provides a nationwide perspective on VBA. Centralized training provides an overview of the technical aspects of the VSR and RVSR positions. Training instructors should follow the prescribed schedule and curriculum dictating when and how material is taught. For example, for a particular topic, the instructor’s guide explains the length of the lesson, the instructional method, and the materials required; lays out the information that must be covered; and provides exercises to review the material. (See fig. 2 for a sample of an instructor’s guide from the centralized training curriculum.) Centralized training classes have at least three instructors, but the actual number can vary depending on the size of the group. VBA’s goal is to maintain a minimum ratio of instructors to students.

The first week of centralized training for VSRs focuses on key concepts, such as security, privacy and records management; terminology; and job tools, such as the policy manual and software applications. The final 2 weeks of training focus on the different roles and responsibilities of VSRs on the Pre-determination and Post-determination teams in processing claims. To practice processing different types of claims and processing claims from start to finish, VSRs work on either real claims or hypothetical claims specifically designed for training. Centralized training for new RVSRs—many of whom have been promoted from the VSR position— focuses on topics such as systems of the human body, how to review medical records, and how to interpret a medical exam. According to staff in one site we visited, RVSRs new to VBA also take VSR centralized training or its equivalent to learn the overall procedures for processing claims.

To accommodate the influx of new staff it must train, in fiscal year 2007 VBA substantially increased the frequency of centralized training and is increasing student capacity at the Veterans Benefits Academy. During fiscal year 2007, VBA held 67 centralized training sessions for 1,458 new VSRs and RVSRs. Centralized training sessions were conducted at 26 different regional offices during fiscal year 2007, in addition to the Veterans Benefits Academy. By comparison, during fiscal year 2006, VBA held 27 centralized training sessions for 678 new claims processors.

To implement centralized training, VBA relies on qualified regional office staff who have received training on how to be an instructor. According to VBA officials, centralized training instructors may be Senior VSRs, RVSRs, supervisors, or other staff identified by regional office managers as having the capability and the right personality to be effective instructors. Potential instructors have certain training requirements. First, they must complete the week-long Instructor Development Course, which covers the ways different adults learn, the process for developing lesson plans, and the use of different training methods and media. During this course, participants are videotaped and given feedback on their presentation style. In addition, each time instructors teach a centralized training session, they are supposed to take the 2.5 day Challenge Curriculum Course, designed to update instructors on changes to the curriculum and general training issues. Between October 2006 and February 2008, about 250 VSRs and RVSRs from regional offices completed the Instructor Development Course, and VBA officials reported that, given the influx of new VSRs and RVSRs, they are increasing the number of times this course is offered in order to train more instructors. Instructors can teach centralized training sessions in their home office, another regional office, or the Veterans Benefits Academy.

When new VSRs and RVSRs return to their home office after centralized training, they are required to begin their third phase of training, which is supposed to include on-the-job, classroom, and computer-based training, all conducted by and at their regional office. In the regional offices we visited, managers indicated that new VSRs and RVSRs typically take about 6 to 12 months after they return from centralized training to complete all the training requirements for new staff. During this final phase, new claims processing staff cover more advanced topics, building on what they learned in centralized training. Under the supervision of experienced claims processors, they work on increasingly complex types of real claims. On-the-job training is supplemented in the offices we visited by regular classroom training that follows a required curriculum of courses developed by VBA’s Compensation and Pension Service, specifically for new VSRs and RVSRs. For example, new VSRs might complete a class in processing burial claims and then spend time actually processing such claims. The amount of time spent working on each type of claim varies from a couple of days to a few weeks, depending on the complexity of the claim. On-the-job training is also supposed to be supplemented with modules from the Training and Performance Support System (TPSS), an interactive on-line system that can be used by staff individually or in a group. TPSS modules provide detailed lessons, practice cases, and tests for VSRs and RVSRs. Modules for new VSRs cover topics such as burial benefits and medical terminology; RVSR modules cover topics such as the musculoskeletal system, general medical terminology, and introduction to post-traumatic stress disorder.

New and Experienced Staff Have an Annual Training Requirement, and Regional Offices Develop Training Plans That Cover a Mix of Topics Identified Centrally and Locally

A policy established by VBA’s Compensation and Pension Service requires both new and experienced VSRs and RVSRs to complete a minimum of 80 hours of technical training annually, double the number VBA requires of its employees in other technical positions. VBA officials said this higher training requirement for VSRs and RVSRs is justified because their jobs are particularly complex and they must work with constantly changing policies and procedures.

The 80-hour training requirement has two parts. At least 60 hours must come from a list of core technical training topics identified by the central office of the Compensation and Pension Service. For example, core topics for VSRs in fiscal year 2007 included establishing veteran status and asbestos claims development; topics for RVSRs included due process provisions and eye-vision issues. VBA specifies more core topics than are necessary to meet the 60-hour requirement, so regional offices can choose those topics most relevant to their needs. They can also choose the training method used to address each topic, such as classroom or TPSS training. (See app. II for the list of core technical training topics for fiscal year 2007.) Regional offices determine the training topics that are used to meet the remaining 20 hours, based on local needs and input. Regional offices may select topics from the list of core technical training topics or identify other topics on their own.

The four regional offices we visited varied in the extent to which they utilized their discretion to choose topics outside the core technical training topics in fiscal year 2007. Two sites selected the required 60 hours of training from the core requirements and identified their own topics for the remaining 20 hours. In the other two sites, almost all the training provided to staff in fiscal year 2007 was based on topics from the list of core requirements. An official in one regional office, for example, said that his office used its full 20 hours to provide training on new and emerging issues that are not covered by the core technical training topics, as well as training to address error prone areas. An official in another regional office said the core requirements satisfied staff training needs in fiscal year 2007, possibly because this regional office had a large proportion of new staff and the core topics are focused on the needs of new staff.

Regional offices must develop training plans each year that indicate which courses will actually be provided to staff to enable them to meet the 80- hour training requirement. The training plan is a list of courses that the regional office plans to offer throughout the year, as well as the expected length and number and types of participants in each course. In the regional offices we visited, when managers develop their training plans, they solicit input from supervisors of VSRs and RVSRs and typically also consider national or local error trend data. Regional offices must submit their plans to the VBA central office at the beginning of each fiscal year for review and feedback. Central office officials review the plans to determine whether (1) the regional office will deliver at least 60 hours of training on the required core topics, (2) the additional topics identified by the regional office are appropriate, and (3) staff in similar positions within an office receive the same level and type of training. According to central office officials, they provide feedback to the regional offices on their current plans as well as guidance on what topics to include in the next year’s training plans. Regional offices can adjust their training plans throughout the year to address shifting priorities and unexpected training needs. For example, a regional office may add or remove courses from the plan in response to changing trends in errors or policy changes resulting from legal decisions. (See app. III for excerpts from the fiscal year 2007 training plans from the regional offices we visited.)

While regional offices have discretion over the methods they use to provide training, the four offices we visited relied primarily on classroom training in fiscal year 2007. In each of these offices, at least 80 percent of the total fiscal year 2007 training hours completed by all claims processors was in the form of classroom instruction (see fig. 3). Officials in two of the regional offices we visited said they used lesson plans provided by the Compensation and Pension Service and adapted these plans to the needs of their staff; one regional office developed its own courses. An official in one office said they sometimes invite guest speakers, and an official in another regional office said that classroom training is sometimes delivered as part of team meetings. The offices we visited generally made little use of other training methods. Only one office used TPSS for its training more than 1 percent of the time. Two offices used self-instruction—such as reading memos from VBA central office—for about 10 percent of their training, and no office used videos for more than 1 percent of their training. The central office usually communicates immediate policy and regulatory changes through memos called Fast Letters, which may be discussed in team meetings or may just be read by staff individually.

Staff Are Not Held Accountable for Meeting Their Training Requirement

Because the agency has no policy outlining consequences for individual staff who do not complete their 80 hours of training per year, individual staff are not held accountable for meeting their annual training requirement, and at present, VBA central office lacks the ability to track training completed by individual staff members. According to VBA officials, however, the agency is in the process of implementing an automated system that should allow it to track the training each staff member completes. Officials reported that this system is expected to be implemented during fiscal year 2008. VBA officials reported that this system will be able to record the number of training hours and the courses completed for each individual, staff position, and regional office. One official said the central office and regional office supervisors will have the ability to monitor training completed by individual staff members, but that central office will likely not monitor the training completed by each individual staff member, even though it may monitor the training records for a sample of staff members. Furthermore, despite the absence of a VBA- wide tracking system, managers in two of the regional offices we visited reported using locally developed tracking methods to determine the number of training hours their staff had completed.

While individuals are not held accountable, VBA reported taking some steps to ensure that staff complete the required number of training hours. VBA central office periodically reviews the aggregated number of training hours completed at each regional office to determine whether the office is on track to meet the training requirement. According to a VBA official, managers in offices where the staff is not on track to complete 80 hours of training during the year can be reprimanded by a higher-level manager, and if their staff do not meet the aggregate training hours at the end of the fiscal year, managers could face negative consequences in their performance assessments.

VBA Is Taking Steps to Strategically Plan Its Training for Staff, but Does Not Adequately Evaluate Training and May Be Falling Short in Design and Implementation

VBA is taking steps to strategically plan its training for VSRs and RVSRs including the establishment of a training board to assess VBA’s training needs. VBA has also made some effort to evaluate its training for new staff, but does not require regional offices to collect feedback from staff on any of the training they provide. Although some regional offices collect some training feedback, it is not shared with VBA central office. Both new and experienced staff we interviewed did, in fact, report some problems with their training. A number of new staff raised issues with how consistently their training curriculum was implemented. Experienced staff differed in their assessments of the VBA’s annual training requirement, with some indicating they struggle to meet this requirement because of workload pressures or that training topics are sometimes redundant or not relevant to their position.

VBA Is Taking Steps to Strategically Plan Its Training

VBA is taking steps to strategically plan its training for claims processors, in accordance with generally accepted practices identified by GAO. (See app. I for a detailed description of these generally accepted practices.)

Aligning Training with the Agency’s Mission and Goals

VBA has made an effort to align training with the agency’s mission and goals. According to VBA documents, in fiscal year 2004 an Employee Training and Learning Board (board) was established to ensure that training decisions within the VBA are coordinated; support the agency’s strategic and business plans, goals and objectives; and are in accordance with the policy and vision of VBA. Some of the board’s responsibilities include establishing training priorities and reviewing regional office and annual training plans.

Identifying the Skills and Competencies Needed by the Workforce

VBA has identified the skills and competencies needed by VBA’s claims processing workforce. VBA developed a decision tree and task analysis of the claims process, which GAO experts in the field of training told us made it possible to understand and map both the claims process and the decisions associated with it that supported the development of VBA’s training curriculum.

Determining the Appropriate Level of Investment in Training and Prioritizing Funding

VBA is taking steps to determine the appropriate level of investment in training and prioritize funding. According to VBA documents, some of the board’s responsibilities include developing annual training budget recommendations and identifying and recommending training initiatives to the Under Secretary of Benefits. VBA officials also reported developing several documents that made a business case for different aspects of VBA’s training, such as VA’s annual budget and the task analysis of the VSR and RVSR job positions.

Considering Government Reforms and Initiatives

According to one VBA official, the agency identifies regulatory, statutory, and administrative changes as well as any legal or judicial decisions that affect how VBA does business and issues guidance letters, or Fast Letters, which can be sent out several times a year, to notify regional offices of these changes. Also, as a result of Congress authorizing an increase in its number of full-time employees and VBA’s succession planning efforts, VBA has increased the number of centralized training sessions for new staff and has also increased the number of Instructor Development Courses offered to potential centralized training instructors. As a result, VBA is taking steps to consider government reforms and initiatives to improve its management and performance when planning its training.

VBA Collects Feedback on Centralized Training, but Regional Offices Do not Always Collect Feedback on the Training They Provide

According to accepted practices, federal agencies should also evaluate their training programs and demonstrate how these efforts help employees, rather than just focusing on activities or processes (such as number of training participants or hours of training). VBA has made some efforts to evaluate its training for claims processors. During the 3-week centralized training session for new staff, VBA solicits daily feedback from participants using forms that experts in the training field consider well- constructed and well-balanced. According to one GAO expert, the forms generally employ the correct principles to determine the effectiveness of the training and ascertain whether the instructor effectively presented the material (see fig. 4). VBA officials told us that they have used this feedback to improve centralized training for new staff. Management at one regional office cited the decision to separate training curricula for VSRs on Pre- determination teams and VSRs on Post-determination teams as an example of a change based on this feedback.

Although VBA evaluates centralized training, it does not require regional offices to obtain feedback from participants on any of the training they provide to new and experienced staff. In a previous GAO report, VA staff told us that new training materials they develop are evaluated before being implemented. However, none of the regional offices we visited consistently collect feedback on the training they conduct. Supervisors from three of the regional offices we visited told us that they collect feedback on some of the training their office conducts, but this feedback largely concerns the performance of the instructor. Participants are generally not asked for feedback on course content. Moreover, regional offices we visited that do, to some degree, collect feedback do not share this information with VBA.

VBA’s Training Curriculum for New Staff Appears Generally Well Designed, but Some Staff Raised Issues Concerning Its Implementation

According to GAO experts in the training field, VBA’s training curriculum for new staff appears well designed. VBA’s curriculum for new staff conforms to adult learning principles, carefully defining all pertinent terms and concepts, and providing abundant and realistic examples of claims work. GAO experts also determined that VBA’s training for those who teach the curriculum for new staff was well designed and would enable experienced claims processors to become competent trainers because they are coached on teaching theory and have multiple opportunities to practice their teaching skills and receive feedback.

Many of the new staff at all four sites we visited reported that centralized training provided them with a good foundation of knowledge and prepared them for additional training conducted by their regional office. Also, regional office managers from three offices we visited told us that centralized training affords new staff the opportunity to network with other new staff at different regional offices, which imbues a sense of how their positions fit in the organization. However, some staff reported that VBA’s implementation of their centralized training was not always consistent. A number of staff at three regional offices reported that during their centralized training the instructors sometimes taught different ways of performing the same procedures or disagreed on claim procedures. Regional office officials told us that while centralized training instructors attempt to teach consistently through the use of standardized training materials, certain procedures can be done differently in different regional offices while adhering to VBA policy. For example, regional offices may differ on what to include in veteran notification letters. VBA officials also told us that centralized training conducted at the regional offices may not be as consistent as centralized training conducted at the Veterans Benefits Academy. According to these officials, unlike the regional offices, the Veterans Benefits Academy has on-site training experts to guide and ensure that instructors are teaching the curriculum consistently.

New staff also gave mixed assessments about how training was conducted at their home office after they returned from centralized training. While some staff at all of the regional offices we visited told us that the additional training better prepared them to perform their jobs, with on-the- job training identified as a useful learning tool, others told us that the training could not always be completed in a timely manner due to regional office priorities. Some management and staff at two of the regional offices we visited reported that, because of workload pressures, some of their RVSRs had to interrupt their training to perform VSR duties. Also, a few new staff indicated that VBA’s TPSS was somewhat difficult to use. Although TPSS was developed to provide consistent technical training designed to improve the accuracy of claims ratings, a number of staff at all of the regional offices we visited reported that TPSS was too theoretical. For example, some staff said it provided too much information and no practical exercises in applying the knowledge. Some staff also noted that certain material in TPSS was out-of-date with policy changes such as how to order medical examinations. Some staff at three of the regional offices also reported that TPSS was not always useful in training staff, in part, because TPSS does not use real cases. Three of the regional offices reported using TPSS for less than 1 percent of their training and VSRs at one regional office were unaware of what TPSS was.

Experienced Staff Expressed Mixed Views of the Design and Implementation of Their Training

At all of the regional offices we visited, staff we spoke with generally noted that training enables them to keep up-to-date on changes in laws and regulations as well as provides opportunities for obtaining refresher training on claims procedures they perform infrequently. However, regional office staff we spoke with differed in their assessment of the 80- hour requirement. Some regional office staff said the number of training hours required was appropriate, while others suggested that VBA adopt a graduated approach, with the most experienced staff being required to complete fewer hours than new staff. VBA officials told us that, in 2007, the Compensation and Pension Service reviewed their annual training requirements and determined the 80-hour annual training requirement was appropriate. However, the officials we spoke with could not identify the criteria that were used to make these determinations. Furthermore, VBA management does not systematically collect feedback from staff evaluating the usefulness of the training they must receive to meet this requirement. Consequently, when determining the appropriateness of the 80-hour requirement, VBA has not taken into account the views of staff to gauge the effect the requirement has on them.

Experienced staff had mixed views on training provided by the regional office. Staff at three regional offices said the core technical training topics set by the Compensation and Pension Service are really designed for newer staff and do not change much from year to year, and therefore experienced staff end up repeating courses. Also, a number of staff at all of the regional offices we visited told us some regional office training was not relevant for those with more experience. Conversely, other regional office staff note that although training topics may be the same from year to year, a person can learn something new each time the course is covered. Some VBA officials and regional office managers also noted that some repetition of courses is good for several reasons. Staff may not see a particular issue very often in their day-to-day work and can benefit from refreshers. Also, regional office managers at one office told us that the core technical training topics could be modified to reflect changes in policy so that courses are less repetitive for experienced staff.

Many experienced staff also reported having difficulty meeting the 80-hour annual training requirement due to workload pressures. Many of the experienced staff we spoke with, at each of the regional offices we visited, told us that there is a constant struggle between office production goals and training goals. For example, office production goals can affect the availability of the regional office’s instructors. A number of staff from one regional office noted that instructors were unable to spend time teaching because of their heavy workloads and because instructors’ training preparation hours do not count toward the 80-hour training requirement. Staff at another regional office told us that, due to workload pressures, staff may rush through training and may not get as much out of it as they should.

Performance Management System for Claims Processors Generally Conforms to Accepted Practices, but May Not Clearly Differentiate between Performance Levels

Performance Management System for Claims Processors Is Generally Consistent with Accepted Practices

The elements used to evaluate individual VSRs’ and RVSRs’ performance appear to be generally aligned with VBA’s organizational performance measures, something prior GAO work has identified as a well-recognized practice for effective performance management systems (see app. I). Aligning individual and organizational performance measures helps staff see the connection between their daily work activities and their organization’s goals and the importance of their roles and responsibilities in helping to achieve these goals. VSRs must be evaluated on four critical elements: quality, productivity, workload management, and customer service. RVSRs are evaluated on quality, productivity, and customer service. In addition, VBA central office requires regional offices to evaluate their staff on at least one non-critical element. The central office has provided a non-critical element called cooperation and organizational support, and although regional offices are not required to use this particular element, all four offices we visited did so (see table 2). For each element, there are three defined levels of performance: exceptional, fully successful, or less than fully successful. Table 2 refers only to the fully successful level of performance for each element.

Three critical elements in particular—quality, workload management, and productivity—are aligned with VBA’s organizational performance measures (see table 3). According to VA’s strategic plan, one key organizational performance measure for VBA is overall accuracy in rating disability claims. This organizational measure is aligned with the quality element for VSRs and RVSRs, which is assessed by measuring the accuracy of their claims-processing work. An individual performance element designed to motivate staff to process claims accurately should, in turn, help VBA meet its overall accuracy goal. Two other key performance measures for VBA are the average number of days that open disability claims have been pending and the average number of days it takes to process disability claims. VSRs are evaluated on their workload management, a measure of whether they complete designated claims- related tasks within specific deadlines. Individual staff performance in this element is linked to the agency’s ability to manage its claims workload and process claims within goal time frames. Finally, a performance measure that VBA uses to evaluate the claims-processing divisions within its regional offices—and that, according to VBA, relates to the organization’s overall mission—is production, or the number of compensation and pension claims processed by each office in a given time period. Individual VSRs and RVSRs are evaluated on their productivity, i.e., the number of claims-related tasks they complete per day. Higher productivity by individual staff should result in more claims being processed by each regional office and by VBA overall.

Providing and Routinely Using Performance Information to Track Organizational Priorities

Providing objective performance information to individuals helps show progress in achieving organizational goals and allows individuals to manage their performance during the year by identifying performance gaps and improvement opportunities. Regional offices are supposed to use the critical and non-critical performance elements to evaluate and provide feedback to their staff. Supervisors are required to provide at least one progress review to their VSRs and RVSRs each year, indicating how their performance on each element compares to the defined standards for fully successful performance. In the offices we visited, supervisors typically provide some feedback to staff on a monthly basis. For example, VSRs in the Atlanta regional office receive a memo on their performance each month showing their production in terms of average weighted actions per day, their accuracy percentage based on a review of a sample of cases, and how their performance compared to the minimum requirements for production and accuracy. If staff members fall below the fully successful level in a critical element at any time during the year, a performance improvement plan must be implemented to help the staff member improve.

Connecting Performance Expectations to Crosscutting Goals

Performance elements related to collaboration or teamwork can help reinforce behaviors and actions that support crosscutting goals and provide a consistent message to all employees about how they are expected to achieve results. VSR and RVSR performance related to customer service is evaluated partly based on whether any valid complaints have been received about a staff member’s interaction with their colleagues. And performance related to the cooperation and organizational support element is based on whether staff members’ interaction with their colleagues is professional and constructive.

Using Competencies to Provide a Fuller Assessment of Performance

Competencies, which define the skills and supporting behaviors that individuals are expected to exhibit to carry out their work effectively, can provide a fuller assessment of an individual’s performance. In addition to elements that are evaluated in purely quantitative terms, VBA uses a cooperation and organizational support element for VSRs and RVSRs that requires supervisors to assess whether their staff are exhibiting a number of behaviors related to performing well as a claims processor.

Involving Employees and Stakeholders to Gain Ownership of the Performance Management System

Actively involving employees and stakeholders in developing the performance management system and providing ongoing training on the system helps increase their understanding and ownership of the organizational goals and objectives. For example, VA worked with the union representing claims processors to develop an agreement about its basic policies regarding performance management. Also, VBA indicated that it planned to pilot revisions to how productivity is measured for VSRs in a few regional offices, partly so VSRs would have a chance to provide feedback on the changes.

VA’s System May Not Clearly Differentiate between Performance Levels

Clear differentiation between staff performance levels is also an accepted practice for effective performance management systems. Systems that do not result in meaningful distinctions between different levels of performance fail to give (1) employees the constructive feedback they need to improve, and (2) managers the information they need to reward top performers and address performance issues. GAO has previously reported that, in order to provide meaningful distinctions in performance for experienced staff, agencies should use performance rating scales with at least three levels, and scales with four or five levels are preferable because they allow for even greater differentiation between performance levels. If staff members are concentrated in just one or two of multiple performance levels, however, the system may not be making meaningful distinctions in performance.

VA’s performance appraisal system has the potential to clearly differentiate between staff performance levels. Each fiscal year, regional offices give their staff a rating on each critical and non-critical performance element using a three-point scale—exceptional, fully successful, or less than fully successful. Based on a VA-wide formula, the combination of ratings across these elements is converted into one of VA’s five overall performance levels: outstanding, excellent, fully successful, minimally satisfactory, and unsatisfactory (see fig. 5). Regional offices may award financial bonuses to staff on the basis of their end-of-year performance category. Prior to fiscal year 2006, VA used two performance levels—successful and unacceptable—to characterize each staff member’s overall performance. To better differentiate between the overall performance levels of staff, VA abandoned this pass-fail system in that year, choosing instead to use a five-level scale.

However, there is evidence to suggest that the performance management system for VSRs and RVSRs may not clearly or accurately differentiate among staff’s performance. VBA central office officials and managers in two of the four regional offices we visited raised concerns with VA’s formula for translating ratings on individual performance elements into an overall performance rating. These officials said that under this formula it is more difficult for staff to be placed in certain overall performance categories than others, even if staff’s performance truly does fall within one of those categories. Indeed, at least 90 percent of all claims processors in the regional offices we visited were placed in either the outstanding or the fully successful category in fiscal year 2007. (Fig. 6 shows the distribution of overall performance ratings for claims processors in each office.)

Central and regional office managers noted that, in particular, it is difficult for staff to receive an overall rating of excellent. Managers in one office said there are staff whose performance is better than fully successful but not quite outstanding, but under the formula it is difficult for these staff to be placed in the excellent category as the managers feel they should be. An excellent rating requires exceptional ratings in all the critical elements and a fully successful rating in at least one non-critical element. However, according to staff we interviewed, virtually all staff who are exceptional in the critical elements are also exceptional in all non-critical element(s), so they appropriately end up in the outstanding category. On the other hand, the overall rating for staff who receive a fully successful rating on just one of the critical elements—even if they are rated exceptional in all the other elements—drops down to fully successful. Managers in one regional office commented that the system would produce more accurate overall performance ratings if staff were given an overall rating of excellent when they had, for example, exceptional ratings on three of five overall elements and fully successful ratings on the other two.

An official in VA’s Office of Human Resources Management acknowledged that there may be an issue with the agency’s formula. Although neither VBA nor VA central office officials have examined the distribution of VSRs and RVSRs across the five overall performance ratings, VA indicated it is considering changes to the system designed to allow for greater differentiation in performance ratings. For example, one possible change would be to use a five-point scale for rating individual elements—probably mirroring the five overall performance rating categories of outstanding, excellent, fully successful, minimally satisfactory, and unsatisfactory— rather than the current three-point scale. Under the proposed change, a staff member who was generally performing at the excellent but not outstanding level could get excellent ratings in all the elements and receive an overall rating of excellent. This change must still be negotiated with several stakeholder groups, according to the VA official we interviewed.

Conclusions

In many ways, VBA has developed a training program for its new staff that is consistent with accepted training practices in the federal government. However, because VBA does not centrally evaluate or collect feedback on training provided by its regional offices, it lacks the information needed to determine if training provided at regional offices is useful and what improvements, if any, may be needed. Ultimately, this information would help VBA determine if 80 hours of training annually is the right amount, particularly for its experienced staff, and whether experienced staff members are receiving training that is relevant for their positions. Identifying the right amount of training is crucial for the agency as it tries to address its claims backlog. An overly burdensome training requirement needlessly may take staff away from claims processing, while too little training could contribute to processing inaccuracies. Also, without collecting feedback on regional office training, VBA may not be aware of issues with the implementation of its TPSS, the on-line training tool designed to ensure consistency across offices in technical training. Setting aside the issue of how many hours of training should be required, VBA does not hold its staff accountable for fulfilling their training requirement. As a result, VBA is missing an opportunity to clearly convey to staff the importance of managing their time to meet training requirements as well as production and accuracy goals. With the implementation of its new learning management system, VBA should soon have the ability to track training completed by individual staff members, making it possible to hold them accountable for meeting the training requirement.

As with its training program for VSRs and RVSRs, the VA is not examining the performance management system for claims processors as closely as it should. VBA is generally using the right elements to evaluate its claims processors’ performance, and the performance appraisals have the potential to give managers information they can use to recognize and reward higher levels of performance. However, evidence suggests the formula used to place VSRs and RVSRs into overall performance categories may not clearly and accurately differentiate among staff’s performance levels. Absent additional examination of the distribution of claims processors among overall performance categories, VA lacks a clear picture of whether its system is working as intended and whether any adjustments are needed.

Recommendations for Executive Action

The Secretary of Veterans Affairs should direct VBA to: Collect and review feedback from staff on the training conducted at the if the 80-hour annual training requirement is appropriate for all VSRs and RVSRs; the extent to which regional offices provide training that is relevant to VSRs’ and RVSRs’ work, given varying levels of staff experience; and whether regional offices find the TPSS a useful learning tool and, if not, what adjustments are needed to make it more useful; and Use information from its new learning management system to hold individual VSRs and RVSRs accountable for completing whatever annual training requirement it determines is appropriate.

The Secretary of Veterans Affairs should also examine the distribution of claims processing staff across overall performance categories to determine if its performance appraisal system clearly differentiates between overall performance levels, and if necessary adjust its system to ensure that it makes clear distinctions.

Agency Comments

We provided a draft of this report to the Secretary of Veterans Affairs for review and comment. In VA’s written comments (see app. IV), the agency agreed with our conclusions and concurred with our recommendations. For example, VBA plans to consult with regional office staff to evaluate its annual 80-hour training requirement and will examine if staff performance ratings clearly differentiate between overall performance levels. VA also provided technical comments that were incorporated as appropriate.

We are sending copies of this report to the Secretary of Veterans Affairs, relevant congressional committees, and others who are interested. We will also provide copies to others on request. The report is also available at no charge on GAO’s Web site at http://www.gao.gov.

Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Contact points for the Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

We were asked to determine: (1) What training is provided to new and experienced claims processors and how uniform is this training? (2) To what extent has the Veterans Benefits Administration (VBA) developed a strategic approach to planning training for claims processors and how well is their training designed, implemented, and evaluated? And (3) To what extent is the performance management system for claims processors consistent with generally accepted performance management practices in the public sector? To answer these questions, we reviewed documents and data from the central office of the Department of Veterans Affairs’ Veterans Benefits Administration (VBA) and interviewed VBA central office officials. We conducted site visits to and collected data from four VBA regional offices, and visited the Veterans Benefits Academy. We also interviewed officials from the American Federation of Government Employees, the labor union that represents Veterans Service Representatives (VSR) and Rating Veterans Service Representatives (RVSR). We compared VBA’s training and performance management systems to accepted human capital principles and criteria compiled by GAO. We conducted this performance audit from September 2007 through May 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Regional Office Site Visits

We conducted site visits to 4 of VBA’s 57 regional offices—Atlanta; Baltimore; Milwaukee; and Portland, Oregon. We judgmentally selected these offices to achieve some diversity in geographic location, number of staff, and claims processing accuracy rates, and what we report about these sites may not necessarily be representative of any other regional offices or all regional offices (see fig. 7).

During our site visits, we interviewed regional office managers, supervisors of VSRs and RVSRs, VSRs, and RVSRs about the training and performance management practices in their offices. The VSRs and RVSRs we interviewed at the four regional offices had varying levels of experience at VBA. Regional office managers selected the staff we interviewed. We also observed a demonstration of VBA’s on-line learning tool, the Training and Performance Support System (TPSS), and collected data from the regional offices on, for example, the training they provided during fiscal year 2007. In conjunction with our visit to the Baltimore regional office, we also visited VBA’s Veterans Benefits Academy, where we observed classes for VSRs and RVSRs and interviewed the director of the Academy.

Assessment of VBA’s Training for Claims Processors

To determine whether VBA’s training program is consistent with accepted training practices in the public sector, we relied partly on a guide developed by GAO that lays out principles that federal agencies should follow to ensure their training is effective. This guide was developed in collaboration with government officials and experts in the private sector, academia, and nonprofit organizations; and in conjunction with a review of laws, regulations and literature on training and development issues, including previous GAO reports. The guide lays out the four broad components of the training and development process (see fig. 8).

The guide also provides key questions for federal agencies to consider in assessing their performance in each component. (See table 4 for a sample of these questions.)

In addition, GAO training experts reviewed VBA materials, including training curricula, lesson plans, and course evaluation forms, to determine if these materials are consistent with accepted training practices.

Assessment of VBA’s Performance Management System for Claims Processors

In assessing the performance management system for VSRs and RVSRs, we relied primarily on a set of accepted practices of effective public sector performance management systems that has been compiled by GAO. To identify these accepted practices, GAO reviewed its prior reports on performance management that drew on the experiences of public sector organizations both in the United States and abroad. For the purpose of this review, we focused on the six accepted practices most relevant for VBA’s claims-processing workforce (see table 5).

Appendix II: Fiscal Year 2007 Core Technical Training Requirements for VSRs and RVSRs

Course title or topic

Additional Issue Specific Lesson Plans are under development. (Lesson plans can be taken from the Centralized Training Curriculum found on the C&P Intranet Training Site. If used as provided they do not require C&P review and approval.

These plans can and often should be modified to focus in on a particular narrow issue of training need. Modified lesson plans are to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training.

Any Challenge-oriented original lesson plan developed by Station personnel is to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training.)

C&P Service Broadcasts that may be provided during the course of the FY may be substituted in place of any training scheduled on an hour by hour basis.

60 Hours of the required 80 Hours will be selected from the suggested topics above. The remaining 20 hours will be selected at the Stations discretion based upon their own individual quality review. (Training provided from the above topics can be focused on a particular aspect of the topic; i.e. Cold Injuries and Rating Hypertension from Cardiovascular issues could be separate classes)

Participation in Agency Advancement Programs (i.e., LEAD, LVA) does not substitute for Required training requirements.

Additional Issue Specific Lesson Plans are under development. (Lesson plans can be taken from the Centralized Training Curriculum found on the C&P Intranet Training Site. If used as provided they do not require C&P review and approval.

These plans can and often should be modified to focus in on a particular narrow issue of training need. Modified lesson plans are to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training.

Any Challenge-oriented original lesson plan developed by Station personnel is to be submitted to C&P Service for review and approval at least 30 days prior to delivery of training.)

C&P Service Broadcasts that may be provided during the course of the FY may be substituted in place of any training scheduled on an hour by hour basis.

Drill Pay Waivers Pension Awards Processing & BDN Hospital Reductions Burial Benefits Death Pension Accrued Benefits Accrued Awards & the BDN Apportionments Special Monthly Pension Helpless Child Incompetency/Fiduciary Arrangements Claims Processing Auto Allowance and Adaptive Equipment Special Adapted Housing Special Home Adaptation Grants Incarcerated Veterans Processing Write Outs FOIA/Privacy Act Telephone & Interview Techniques Telephone Development IRIS Introduction to VACOLS Education Benefits Insurance Benefits National Cemetery VR&E Benefits Loan Guaranty Benefits General Benefits – FAQs Suicidal Caller Guidance Non-Receipt of BDN Payments Mail Handling Income & Net Worth Determinations Bootcamp test and review of VSR Readiness Guide (2 HRS Required) Reference Material Training and Navigation (1 HR Required) Appeals and Ancillary Benefits Ready to Rate Development Customer Service FNOD Info and PMC Process Intro to Appeals Process DRO Selection Letter Income Adjustment Materials Income Adjustments 60 Hours of the required 80 Hours will be selected from the suggested topics above. The remaining 20 hours will be selected at the Stations discretion based upon their own individual quality review.

Overview of VA Mission Reference Materials: Manual Training & WARMS C&P Website Claims Folder Maintenance Records Management POA/Service Organizations Compensation Original Compensation Claims Non-Original Compensation Claims VA Form 21-526, App. For Compensation or Pension Establishing Veteran Status Claims Recognition Duty to Assist Selecting the Correct Worksheet for VA Exams Issue Specific Claim Development Asbestos Claim Development Herbicide Claim Development POW Claim Development Radiation Claim Development PTSD Claim Development Undiagnosed Illness Claim Development Dependency Contested Claims Deemed Valid and Common-law Marriage Continuous Cohabitation Pension Intro. To Disability Pension Overview of SHARE (SSA) Administrative Decision Process Character of Discharge Line of Duty – Willful Misconduct Claims Development Workload Management Utilizing WIPP DEA Training (req. added 4/06)

Intro to Ratings Paragraph 29 & 30 Ratings Ratings & the BDN BDN 301 Interface Video PCGL Award Letters PCGL Dependents & the BDN Compensation Offsets Drill Pay Waivers Star Reporter Pension Awards Processing & the BDN Hospital Reductions Burial Benefits Disallowance Processing DIC Benefits Death Pension Accrued Benefits Accrued Awards & the BDN Apportionment Special Monthly Pension Helpless Child Incompetency/Fiduciary Arrangements Claims Processing Automobile Allowance and Adaptive Equipment Specially Adapted Housing and Special Home Adaptation Grants Incarceration Processing Computer Write Outs DEA Training (req. added 4/06)

Public Contact Team Training: FOIA/Privacy Act Communication Skills Telephone Development Inquiry Routing and Information System (IRIS) Intro to VACOLS Other VBA Business Lines Customer Service Insurance Education (2 hrs)

Intro to Appeals Process VACOLS http://cptraining.vba.va.gov/ C&P_Training/vsr/VSR_ Curriculum.htm#att http://cptraining.vba.va.gov/ C&P_Training/vsr/VSR_ Curriculum.htm#iam.

Appendix III: Excerpts from Fiscal Year 2007 Training Plans for Four Regional Offices

Each training plan we reviewed contained the same informational categories, some of which were what courses were offered by the regional office, whether or not the course was conducted, and how many employees completed the training. Although the fiscal year 2007 training plans we reviewed include data on whether and when the course was actually completed, the initial training plans submitted at the beginning of the fiscal year of course do not have this information. The lists provided below include the first 25 courses listed on each plan alphabetically, a small sample of the courses that the regional offices reported they completed for the fiscal year.

Appendix IV: Comments from the Department of Veterans Affairs

Appendix V: GAO Contact and Staff Acknowledgments

Daniel Bertoni (202) 512-7215 bertonid@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Clarita Mrena, Assistant Director; Lorin Obler, Analyst-in-Charge; Carolyn S. Blocker; and David Forgosh made major contributions to this report; Margaret Braley, Peter Del Toro, Chris Dionis, Janice Latimer, and Carol Willett provided guidance; Walter Vance assisted with study design; Charles Willson helped draft the report; and Roger Thomas provided legal advice.

Related GAO Products

Veterans’ Benefits: Improved Management Would Enhance VA’s Pension Program. GAO-08-112. Washington, D.C.: February 14, 2008.

Veterans’ Disability Benefits: Claims Processing Challenges Persist, while VA Continues to Take Steps to Address Them. GAO-08-473T. Washington, D.C.: February 14, 2008.

Disabled Veterans’ Employment: Additional Planning, Monitoring, and Data Collection Efforts Would Improve Assistance. GAO-07-1020. Washington, D.C.: September 12, 2007.

Veterans’ Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions. GAO-03-1045. Washington, D.C.: September 30, 2003.

Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government. GAO-03-893G. Washington, D.C.: July 2003.

Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success. GAO-03-488. Washington D.C.: March 14, 2003.

Major Management Challenges and Program Risks: Department of Veterans Affairs. GAO-03-110. Washington, D.C.: January 1, 2003.

Veterans’ Benefits: Claims Processing Timeliness Performance Measures Could Be Improved. GAO-03-282. Washington, D.C.: December 19, 2002.

Veterans’ Benefits: Quality Assurance for Disability Claims and Appeals Processing Can Be Further Improved. GAO-02-806. Washington, D.C.: August 16, 2002.

Veterans’ Benefits: Training for Claims Processors Needs Evaluation. GAO-01-601. Washington, D.C.: May 31, 2001.

Veterans Benefits Claims: Further Improvements Needed in Claims- Processing Accuracy. GAO/HEHS-99-35. Washington, D.C.: March 1, 1999.