Veterans Benefits: Training for Claims Processors Needs Evaluation
Published: 2001-05-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Veterans Benefits Administration (VBA) developed a computer-assisted training program, known as the Training and Performance Support System (TPSS), to help its employees become more accurate in processing disability compensation and pension claims. The program seeks to provide uniform and consistent training to employees in 57 regional offices. Although VBA's long-term goal is to attain a 96 percent accuracy rate for claims processing, VBA reported an accuracy rate of only 59 percent for fiscal year 2000. This report reviews (1) the status of the TPSS program's development and implementation and (2) the extent to which TPSS will meet its objectives.
section_title: What GAO Found
paragraphs: GAO found that despite VBA's objective to centrally develop a standardized training program, significant delays in the development of TPSS are hindering the program's ability to provide standardized training to claims processing employees. According to VBA's current schedule, the full development of the program will not be completed until at least 2004, or about two years later than VBA had planned. Although VBA provided nine training modules to regional offices to begin the program, the extent to which the offices implemented them varied considerably. Many offices reported that workload pressures prevented them from fully using the modules. GAO found that TPSS might not fully achieve its objectives. For instance, TPSS training modules may not be available in time to train new employees hired to replace employees who are expected to retire in the future. Furthermore, TPSS may not reduce training time, as envisioned by VBA.

section_title: Letter
paragraphs: A top priority of the Veterans Benefits Administration (VBA) is to improve its employees’ accuracy in processing disability compensation and pension claims. While VBA’s long-term goal is to attain a 96-percent accuracy rate for claims processing, VBA reported an accuracy rate of only 59 percent for fiscal year 2000. To improve the accuracy rate, VBA’s primary training initiative is a computer-assisted training program known as the Training and Performance Support System (TPSS). The program’s purpose is to provide uniform and consistent training to employees in 57 regional offices in order to improve the accuracy and consistency of disability claims processing nationwide. The need to invest in human capital through better training was emphasized in 1993 when a VBA Blue Ribbon Panel reported that VBA needed to improve the training for its regional office employees. In 1998, VBA committed itself to develop the TPSS program, and, as of March 2001, had spent or obligated about $18.6 million of the estimated overall program cost of $32 million.
You asked us to provide you with information on (1) the status of the TPSS program’s development and implementation and (2) the extent to which TPSS will meet its objectives. To address these issues, we met with VBA officials and reviewed VBA documents. We surveyed all regional offices and visited six of them to determine their experiences with and their views on the TPSS program. At the six offices we visited, we talked with regional office officials and employees who had taken TPSS training. We did not evaluate the effectiveness of any type of training, but we obtained the views of regional offices on the benefits or adequacy of TPSS. (See app. I for more information on scope and methodology.) We conducted our review from June 2000 through March 2001 in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: VBA’s compensation program pays monthly benefits, based on degree of disability, to veterans who have service-connected disabilities (injuries or diseases incurred or aggravated while on active military duty), regardless of whether these veterans are employed or have earnings. The pension program pays monthly benefits, based on financial need, to wartime veterans who have low incomes and are permanently and totally disabled for reasons not service-connected. When veterans submit disability claims, VBA’s 57 regional offices must follow newly defined requirements for assisting veterans in obtaining evidence to support their claims. For a disability compensation claim, the regional office must (1) obtain the veteran’s existing service medical records and, if the veteran has furnished information sufficient to locate such records, other relevant service records; (2) obtain records of relevant medical treatment or examination of the veteran provided at VA health-care facilities or at VA’s expense; (3) obtain any other relevant records held by a federal department or agency; and (4) provide a medical examination or obtain a medical opinion, if necessary to make a decision on the claim.
Regional offices have three primary claims-processing positions for which TPSS is intended to provide training. Veterans Service Representatives obtain evidence required for veterans’ claims. Rating Veterans Service Representatives (hereafter referred to as Rating Specialists) analyze the evidence, evaluate service-connected impairments, and assign a rating for the degree to which the veteran is disabled. Decision Review Officers attempt to resolve veterans’ disagreements with regional office decisions and can grant benefits based on the same evidence used by the Rating Specialist. Two of these regional office positions are relatively new—the Veterans Service Representative position and the Decision Review Officer position were implemented during 1998-99 and 2000, respectively.
If a veteran disagrees with a regional office decision, the veteran may file an appeal asking VA’s Board of Veterans’ Appeals to review the decision.The Board makes VA’s final decision on such appeals and can grant or deny benefits, but in some cases the Board may return (remand) the case to the regional office to obtain further evidence and reconsider the decision. In these cases, the regional office either grants the claim or returns it to the Board for a final VA decision.
Our High-Risk Series states that human capital challenges pose a high risk for VBA, as well as other agencies across the federal government.Specifically, our High-Risk Series states that VBA’s training and recruitment programs may not be adequate to ensure a sufficient workforce of competent claims processors, and the lack of such a workforce could undermine efforts to reduce current problems of claims- processing backlogs and errors. We have published a human capital self- assessment checklist providing a systematic approach to identifying and addressing human capital issues. Our checklist states that an agency should make appropriate investments in education, training, and other developmental opportunities to help its employees build the competencies needed to achieve the agency’s “shared vision”—the mission, vision for the future, core values, goals and objectives, and strategies that define the agency’s direction and expectations for itself and its people. Such investments require an explicit workforce planning strategy, linked to the agency’s strategic and program planning efforts. These planning efforts should identify, among other things, the knowledge, skills, and abilities needed to position an agency for its future. In addition, they should provide an explicit link between the agency’s training offerings and curriculums and the competencies identified by the agency for mission accomplishment.
In VA’s Departmental Performance Plan for fiscal year 2001, VBA links the TPSS program to its strategic goal of providing world-class service to veterans and their families. In working toward this goal, VBA’s top priority is to improve the technical accuracy of claims-processing, and VBA states that the TPSS program is one of its most important initiatives for achieving this objective. To develop TPSS training modules for a specific claims-processing position, the job and tasks for that position must be analyzed in detail. After this analysis, VBA estimates that it takes 14 to 16 months to develop each module, which consists of computer- assisted instruction packages and written instructional materials designed to support cooperative learning by employees who train in groups of two or three. Under VBA’s direction, contractors develop the modules with the help of subject matter experts from VBA’s regional and central offices. Before a module is released, selected regional office employees take the training on a trial basis to validate its effectiveness. Each training module includes a pretest for determining whether an employee needs to take the training. Each module also includes a post-test requiring students to process test cases to determine their mastery of the tasks and work processes. Using regional office employees, these pretests and post-tests are also validated before the modules are released.
section_title: TPSS Program Schedule Has Been Delayed and Regional Office Implementation Has Varied
paragraphs: The design and development of the TPSS program will take longer than planned to complete. According to VBA’s current schedule, the development of the full TPSS program will not be completed until at least calendar year 2004, or about 2 years later than VBA had planned. Also, based on our survey of regional offices, 49 of the 57 regional offices had used TPSS, but great variation existed in the extent to which they had implemented the available TPSS modules to train employees. In fact, until December 2000, VBA had not established a requirement that regional offices use the TPSS modules. In general, the regional offices reported that the most significant hindrance to using the modules was lack of time for training due to workload pressures.
subsections: section_title: TPSS Program Is Behind Schedule
paragraphs: In early 2000, VBA projected that the development of the full TPSS program would be completed during calendar year 2002. However, as of March 2001, the earliest projected completion date for the full program would be calendar year 2004 (see table 1). As of March 2001, VBA had provided the regional offices with 9 training modules out of an estimated 50 modules expected to constitute the full TPSS program.
According to VBA, delays in the TPSS program can be attributed to two primary causes—the creation of new claims-processing positions and legislative changes in VBA’s duty to assist veterans in obtaining evidence to support their claims. VBA is waiting for the duties of the new Decision Review Officer position to stabilize before conducting the task analysis necessary to develop the training modules for this position. In addition, after completing the task analysis for the new Veterans Service Representative position, VBA learned it will need to develop many more training modules than originally expected because the position encompasses more tasks and greater complexity than anticipated.Moreover, VBA has decided to give priority to developing modules for the Veterans Service Representative position and thus delay the development of training modules for other categories of employees. Finally, because the Congress enacted legislative changes in the duty-to-assist requirements in November 2000, VBA decided to include these new requirements in the remaining basic rating modules. Doing so delayed releasing those modules to the regional offices.
section_title: Regional Office Implementation of TPSS Varied Widely
paragraphs: At the time of our survey in October 2000, the extent to which regional offices had used the six TPSS modules that were available varied widely. Of the 57 regional offices, 8 had not used any of the available TPSS modules to train employees. Of the 49 that had used TPSS, their use varied greatly. For example, 41 regional offices had used the original compensation module, but only 11 had used the routine future examinations module (see table 2). Only 5 regional offices had used all of the available modules to train employees, and 10 regional offices had used only one module.
VBA had strongly encouraged, but not required, regional offices to use TPSS in their training programs. However, VBA conducted a survey of its own in October 2000 and found that some regional offices still were not using any TPSS modules. As a result, in December 2000, VBA issued a letter mandating the use of TPSS in all regional offices. According to VBA officials, the December mandate required that regional offices use the available basic rating modules to train all new Rating Specialists and use future modules for Veterans Service Representatives to train all new employees filling this position. VBA officials told us that these new training requirements for specific types of employees have also been communicated to the regional offices via nationwide conference calls.
Based on responses to our survey from the 49 regional offices that had used at least one TPSS module, the most significant hindrance to implementing the modules was lack of time for training due to workload pressures. About 47 percent of these regional offices said that workload pressures affected their use of TPSS to at least a moderate extent. As we reported in May 2000, persistent backlogs of pending claims have resulted in veterans having to endure long waits to receive decisions on their claims. Consequently, regional offices face pressure not only to improve accuracy in processing claims, but also to reduce the backlog of pending claims. A contractor evaluating VBA training reported in April 2000 that “training time is too easily given up to the press of productivity.” The six modules that had been released at the time of our survey were designed to require a total of about 24 workdays for an employee to complete.
section_title: TPSS May Not Fully Achieve Some of Its Objectives
paragraphs: In 1998, VBA prepared a rationale supporting the adoption of the TPSS program. In this rationale, VBA outlined several objectives of the TPSS program related to human capital management, including providing training for new employees replacing the expected wave of retirees, shortening the training period for employees, and improving employees’ claims-processing accuracy and consistency. We found that many TPSS modules will not be available when needed to train new employees and that TPSS may not substantially reduce the training period for employees. Also, VBA has not developed a formal plan to evaluate the program’s impact on claims-processing accuracy or consistency.
subsections: section_title: Many TPSS Modules Will Not Be Available When Needed to Train New Claims Processing Employees
paragraphs: In its rationale for the TPSS program, VBA stated that the program would allow VBA to effectively train employees hired and promoted as replacements for the expected wave of retiring claims processors. However, given the slippage in the TPSS schedule, new employees have been and will be working in these positions before many of the TPSS modules are available for their training. Of the 4,000 Veterans Service Representatives and Rating Specialists that were employed as of fiscal year 2000, VBA expects about 1,000 (25 percent) to retire between 2000- 2005. In response, VBA already has hired a total of about 600 new Veterans Service Representatives and Rating Specialists between fiscal years 1998 and 2000. Additionally, in fiscal year 2001, VBA anticipates hiring 224 Veterans Service Representatives and 325 Rating Specialists. These new workers need to be trained in order to carry out the duties of their predecessors. However, as of late calendar year 2000, VBA did not expect to release any TPSS modules to train new Veterans Service Representatives until late in calendar year 2002 or in 2003.
Recognizing that the regional offices had an immediate need to train new Veterans Service Representatives, the Under Secretary for Benefits indicated that VBA needed to provide some modules earlier than projected. As a result, VBA currently plans to release five modules that are more narrowly scoped than originally planned for Veterans Service Representatives in November 2001 (see table 1). The remaining modules for Veterans Service Representatives are to be released intermittently during 2002 through 2004.
At the same time, VBA also is exploring alternative means of providing standardized training to compensate for the delays in developing TPSS modules. For Veterans Service Representatives, VBA is developing a reference manual for new hires and also is revising a training guide it has on its internal computer network to make it more interactive, comprehensive, and user friendly. VBA expects to complete these efforts in March and April 2001, respectively. Additionally, it is considering using the VBA Satellite Network and video teleconferencing facilities to provide training for both Rating Specialists and Veterans Service Representatives. The extent to which these alternatives may eliminate the need to develop some future TPSS modules has not been determined.
Furthermore, VBA has established a formal 12-week basic training course for all new Rating Specialists beginning in March 2001 and for all new Veterans Service Representatives in April 2001. To provide this training, VBA established training centers for each of the nine Service Delivery Networks under which the regional offices are organized. After new trainees have been given training in specific topics, such as VA terminology, by the regional offices, they will alternate between 3 weeks of intensive training at the training centers and 3-week periods of training in their regional offices. Training in the regional offices will incorporate available TPSS modules.
section_title: TPSS May Not Substantially Shorten the Training Period for Employees to Become Fully Proficient
paragraphs: In its rationale for the TPSS program, VBA may have overstated the extent to which the TPSS modules can be expected to reduce the employees’ training period. VBA assumed that the TPSS basic rating modules would enable new Rating Specialists to become fully productive in 6 to 8 months rather than 2 years. This assumption presumed that TPSS would significantly reduce the need for on-the-job training provided by regional offices, which historically has constituted the majority of training. Although the regional offices that used the modules generally said in our survey that the TPSS modules do an adequate job of providing basic knowledge, most of these regional offices also said that traditional forms of local training will remain essential or very important.
Regional office officials and employees told us that TPSS cannot by itself adequately train employees in the many complexities, subjective issues, and variations in circumstances encountered in individual claims. In a similar vein, a VBA contractor evaluating TPSS training said that TPSS could not fully cover the complexity of the work of Rating Specialists. Such in-depth training can only be gained through on-the-job training and coaching and mentoring while employees are gaining actual experience in processing claims. In our site visits, regional office officials said they believe that achieving full proficiency as a Rating Specialist will require about 2 years, even with TPSS. VBA officials, however, said that such conclusions may be premature because at the time of our review, the regional offices had had relatively limited exposure to TPSS. They also said that they believe the new 12-week training course incorporating TPSS will help new trainees become proficient faster than in the past. In addition, the officials said that they plan to evaluate the results of the new 12-week program.
section_title: Impact of TPSS Can Be Further Evaluated
paragraphs: In its rationale for adopting the TPSS program, VBA stated that TPSS would help improve the accuracy and consistency of claims processing; however, VBA has no formal plan to assess the impact of TPSS on the accuracy rate or on consistency. Assessing the specific impact of TPSS on accuracy and consistency is difficult. While it may not be feasible for VBA to isolate the effect of TPSS on accuracy or consistency, opportunities exist for VBA to collect data that could help provide some indications of the possible extent of such effects. As stated in our self-assessment checklist for helping agencies address human capital issues, an agency should make determinations of the impact of its training programs.
subsections: section_title: Impact on Accuracy
paragraphs: Assessing the specific impact of TPSS on the accuracy of claims- processing performance is difficult. In its fiscal year 2001 budget submission, VBA stated that it is reasonable to anticipate that the TPSS program will improve the national accuracy rate for disability claims processing by 8 percentage points. However, according to VBA officials, this projected increase in the accuracy rate is not based on an analysis linking employee training to claims-processing accuracy, but rather on professional judgment. The ability to isolate the specific impact of TPSS is difficult because other initiatives implemented by VBA may also affect claims-processing performance. For instance, VBA’s new accuracy measurement system can also affect claims-processing performance because it provides information on specific types of errors. This information can be used to correct the causes of errors, thereby helping to improve the overall accuracy rate. In addition, researchers have found it difficult to link TPSS training to changes in employees’ claims-processing performance. For example, researchers cautioned that while their findings indicated that TPSS training did result in knowledge transfer (over 95 percent of the students who took TPSS module post-tests passed the tests), their evaluation results could not substantiate whether TPSS training had improved standardization or increased claims-processing productivity across regional offices.
Using its accuracy measurement system, VBA is able to track overall trends in the claims-processing accuracy rate, which may provide a general indicator of whether TPSS is helping to improve accuracy at the national level. However, it may be possible for VBA to collect data that could provide further indications of the impact of TPSS. In fiscal year 2001, VBA is planning to pilot test a new initiative—Systematic Individual Performance Assessment (SIPA)—to assess performance quality and identify training needs at the employee level. SIPA will complement the existing accuracy measurement system by reviewing the quality of a representative sample of the individual employee’s work products. Such data on individual employees’ performance could potentially help provide indications of whether TPSS is having an impact on accuracy. For example, VBA could use such data to monitor the accuracy levels of individual employees as they progress through the TPSS curriculum, as well as compare the accuracy levels of employees who completed training on certain TPSS modules with those employees who did not. VBA is considering using data from SIPA to assess the impact of TPSS, but VBA has not developed a formal evaluation plan.
section_title: Impact on Consistency
paragraphs: VBA in its rationale for the TPSS program stated that the program is intended to improve claims-processing consistency among the regional offices. VBA is considering performing assessments to measure claims- processing consistency, but it has not developed a formal evaluation plan. Such assessments are needed to obtain data that could provide indications of whether TPSS is helping to improve consistency. In 1997, the National Academy of Public Administration issued a report that, among other things, addressed inconsistency in VBA’s claims processing. The report stated that VBA needed to identify the degree of subjectivity expected for various medical issues, set consistency standards, and measure the level of consistency as part of the quality review process or through testing of control cases in several regional offices. Performing such an assessment could provide a baseline against which future assessments of consistency could be compared to measure progress in achieving consistency. While such assessments would not provide conclusive evidence on the impact of TPSS, they potentially could provide an indication of whether TPSS is helping to improve consistency.
section_title: Conclusions
paragraphs: While we support VBA’s objective to centrally develop a standardized training program, significant delays in the development of TPSS are hindering the program’s ability to achieve its objective of providing standardized training to claims processing employees. Given the large number of recent and expected new employees, we support VBA’s effort to expedite the development of modules aimed at these employees. At the same time, we support its exploration of alternative ways to provide standardized training to employees for whom TPSS modules are not yet available. Moreover, because TPSS is still evolving, opportunities exist to reassess the need for TPSS modules whose topics are adequately covered through these alternative means. Furthermore, we recognize that it may not be feasible to measure the exact extent to which TPSS improves claims-processing accuracy or consistency. However, opportunities exist for VBA to collect data that could help provide some indications of the possible extent of such effects. Such information could serve as an indicator of the impact of TPSS on accuracy and consistency and help VBA demonstrate the value of the program to the Congress and to veterans.
section_title: Recommendations for Executive Action
paragraphs: To further strengthen VBA’s ability to provide timely, standardized training to its regional office employees who process disability claims, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits to take the following actions.
As VBA explores alternative means, such as the VBA Satellite Network, to deliver timely standardized training, it should determine whether the topics that are covered through such alternative means could be eliminated from plans for future TPSS modules.
VBA should develop formal plans for evaluating the impact of TPSS on claims-processing accuracy and consistency. For example, VBA could use individual performance data obtained from SIPA to provide indications of the impact of TPSS on accuracy. VBA could also conduct a baseline assessment of consistency, such as testing control cases in regional offices, and then conduct periodic assessments to provide indications of whether TPSS is helping to improve consistency. The information regarding the impact of TPSS should be included in the annual reports submitted under the Government Performance and Results Act of 1993.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from VA (see app. II). In its comments, VA concurred with both of our recommendations. With regard to our first recommendation, VA stated that VBA will continue exploring alternative means for providing standardized training and will evaluate whether training on certain topics is better suited to TPSS or alternative means such as the VBA Satellite Network. With regard to our second recommendation, VA stated that when the SIPA program is implemented in late 2001 and 2002, VBA intends to develop formal plans to use it as a tool in evaluating the impact of TPSS on accuracy and consistency. We support these efforts.
In its comments, VA also stated that while some variance exists to the TPSS schedule produced in fiscal year 1998, VBA has given assurance that it will meet the original final delivery date in fiscal year 2004. However, in its fiscal year 2001 budget submission, VBA showed that the TPSS program would be completed in calendar year 2002. Accordingly, we are reporting that the TPSS program is behind schedule.
As agreed with your office, unless you publicly announce it contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Honorable Christopher H. Smith, Chairman of the House Committee on Veterans’ Affairs; the Honorable Arlen Specter, Chairman of the Senate Committee on Veterans’ Affairs; the Honorable John D. Rockefeller IV, Ranking Minority Member of the Senate Committee on Veterans’ Affairs; other appropriate congressional committees; the Honorable Anthony J. Principi, Secretary of the Department of Veterans Affairs; and other interested parties. We will also make copies available to others upon request.
If you have questions about this report, please call me on (202) 512-7101 or Irene Chu on (202) 512-7102. Other key contributors were Ira Spears, Steve Morris, Paul Wright, and Patrick di Battista.
section_title: Appendix I: Scope and Methodology
subsections: section_title: To determine regional offices’ experiences with and views on the Training and Performance Support System (TPSS) program, as well as other types of training, we sent a questionnaire in October 2000 to each of the Veterans Benefits Administration’s (VBA) 57 regional offices. All 57 offices responded to our survey. To examine the extent and manner of TPSS usage in more detail, we visited six regional offices: Des Moines, Iowa; Roanoke, Virginia; Pittsburgh, Pennsylvania; St. Petersburg, Florida; San Diego, California; and Seattle, Washington. The six offices we visited were judgmentally selected to provide a mix of offices based on a combination of factors, including regional office workloads, Service Delivery Network affiliation, geographic location, and variations in the regional offices’ implementation and usage of TPSS. While we did not attempt to evaluate the effectiveness of the TPSS program, or other forms of training, we discussed the benefits and limitations of these training approaches with officials of the regional offices we visited and VBA headquarters.