Report to the Ranking Democratic

Member, Committee on Veterans'

Affairs, House of Representatives

United States General Accounting Office

GAO

May 2001 VETERANS'

BENEFITS

Training for Claims

Processors Needs

Evaluation

GAO-01-601
Page i GAO-01-601 Veterans' Benefits

Letter 1

Appendix I Scope and Methodology 16

Appendix II Comments From the Department of Veterans Affairs 17

Tables

Table 1: VBA’s Schedule for Releasing TPSS Modules to Regional

Offices 6

Table 2: Number and Percentage of Regional Offices Using TPSS as

of September 30, 2000 8

Abbreviations

TPSS Training and Performance Support System

VA Department of Veterans Affairs

VBA Veterans Benefits Administration

Contents
Page 1 GAO-01-601 Veterans' Benefits

May 31, 2001

The Honorable Lane Evans

Ranking Democratic Member

Committee on Veterans’ Affairs

House of Representatives

Dear Mr. Evans:

A top priority of the Veterans Benefits Administration (VBA) is to improve

its employees’ accuracy in processing disability compensation and pension

claims. While VBA’s long-term goal is to attain a 96-percent accuracy rate

for claims processing, VBA reported an accuracy rate of only 59 percent

for fiscal year 2000. To improve the accuracy rate, VBA’s primary training

initiative is a computer-assisted training program known as the Training

and Performance Support System (TPSS). The program’s purpose is to

provide uniform and consistent training to employees in 57 regional

offices in order to improve the accuracy and consistency of disability

claims processing nationwide. The need to invest in human capital through

better training was emphasized in 1993 when a VBA Blue Ribbon Panel

reported that VBA needed to improve the training for its regional office

employees. In 1998, VBA committed itself to develop the TPSS program,

and, as of March 2001, had spent or obligated about $18.6 million of the

estimated overall program cost of $32 million.

You asked us to provide you with information on (1) the status of the

TPSS program’s development and implementation and (2) the extent to

which TPSS will meet its objectives. To address these issues, we met with

VBA officials and reviewed VBA documents. We surveyed all regional

offices and visited six of them to determine their experiences with and

their views on the TPSS program. At the six offices we visited, we talked

with regional office officials and employees who had taken TPSS training.

We did not evaluate the effectiveness of any type of training, but we

obtained the views of regional offices on the benefits or adequacy of TPSS.

(See app. I for more information on scope and methodology.) We

conducted our review from June 2000 through March 2001 in accordance

with generally accepted government auditing standards.

The development and implementation schedule for the TPSS program is at

least 2 years behind schedule. In February 2000, VBA projected that TPSS

would be fully developed and implemented by the middle of calendar year

United States General Accounting Office

Washington, DC 20548

Results in Brief
Page 2 GAO-01-601 Veterans' Benefits

2002. However, as of March 2001, the earliest projected completion date

for TPSS would be calendar year 2004. As of March 2001, VBA had

provided the regional offices with 9 training modules out of an estimated

50 modules that would be needed to fully implement TPSS. Data from our

survey showed that, as of October 2000, 8 of the 57 regional offices had not

used any of the available modules to train employees, and the other 49

regional offices had used the modules to varying extents. Until December

2000, VBA had not established a requirement that regional offices use the

TPSS modules. In general, the regional offices reported that the most

significant hindrance to using the modules was lack of time for training

due to workload pressures.

Early indications are that TPSS may not fully achieve its objectives. For

example, VBA planned to use TPSS as the means for providing

standardized training to new employees whom VBA currently is hiring or

plans to hire in the near future to replace the wave of employees expected

to retire during the next several years. However, many TPSS training

modules will not be available in time to train new employees. Also, VBA

viewed TPSS as the means for significantly reducing the training period

required for employees to become fully proficient in their jobs. However,

in our site visits, regional office officials said they believe that TPSS will

not significantly reduce the training period and that traditional forms of

training, such as local on-the-job training and coaching and mentoring, will

continue to be a significant and necessary learning source. Furthermore,

although VBA believes that TPSS will improve claims-processing accuracy

and consistency, VBA does not have a formal plan to evaluate the impact

of TPSS on the accuracy rate or on consistency. While it may be difficult to

determine the exact impact of TPSS, avenues are available for VBA to

collect data that could provide indications of whether TPSS is helping to

improve accuracy and consistency.

This report contains recommendations to the Secretary of the Department

of Veterans Affairs (VA) concerning actions needed to provide timely

standardized training and provide indicators of the impact of TPSS on

accuracy and consistency.

VA provided comments on a draft of this report. In its comments, VA

agreed with our recommendations.

VBA’s compensation program pays monthly benefits, based on degree of

disability, to veterans who have service-connected disabilities (injuries or

diseases incurred or aggravated while on active military duty), regardless

Background
Page 3 GAO-01-601 Veterans' Benefits

of whether these veterans are employed or have earnings. The pension

program pays monthly benefits, based on financial need, to wartime

veterans who have low incomes and are permanently and totally disabled

for reasons not service-connected. When veterans submit disability claims,

VBA’s 57 regional offices must follow newly defined requirements for

assisting veterans in obtaining evidence to support their claims.1

 For a

disability compensation claim, the regional office must (1) obtain the

veteran’s existing service medical records and, if the veteran has furnished

information sufficient to locate such records, other relevant service

records; (2) obtain records of relevant medical treatment or examination

of the veteran provided at VA health-care facilities or at VA’s expense; (3)

obtain any other relevant records held by a federal department or agency;

and (4) provide a medical examination or obtain a medical opinion, if

necessary to make a decision on the claim.

Regional offices have three primary claims-processing positions for which

TPSS is intended to provide training. Veterans Service Representatives

obtain evidence required for veterans’ claims. Rating Veterans Service

Representatives (hereafter referred to as Rating Specialists) analyze the

evidence, evaluate service-connected impairments, and assign a rating for

the degree to which the veteran is disabled. Decision Review Officers

attempt to resolve veterans’ disagreements with regional office decisions

and can grant benefits based on the same evidence used by the Rating

Specialist. Two of these regional office positions are relatively new—the

Veterans Service Representative position and the Decision Review Officer

position were implemented during 1998-99 and 2000, respectively.

If a veteran disagrees with a regional office decision, the veteran may file

an appeal asking VA’s Board of Veterans’ Appeals to review the decision.2

The Board makes VA’s final decision on such appeals and can grant or

deny benefits, but in some cases the Board may return (remand) the case

to the regional office to obtain further evidence and reconsider the

 1

These requirements are contained in the Veterans Claims Assistance Act of 2000 (P.L. 106-

475, Nov. 9, 2000).

2

In fiscal year 2000, veterans filed such appeals in about 5 percent of all initial decisions

involving ratings.
Page 4 GAO-01-601 Veterans' Benefits

decision.3 In these cases, the regional office either grants the claim or

returns it to the Board for a final VA decision.4

Our High-Risk Series states that human capital challenges pose a high risk

for VBA, as well as other agencies across the federal government.5

Specifically, our High-Risk Series states that VBA’s training and

recruitment programs may not be adequate to ensure a sufficient

workforce of competent claims processors, and the lack of such a

workforce could undermine efforts to reduce current problems of claimsprocessing backlogs and errors. We have published a human capital selfassessment checklist providing a systematic approach to identifying and

addressing human capital issues.6

 Our checklist states that an agency

should make appropriate investments in education, training, and other

developmental opportunities to help its employees build the competencies

needed to achieve the agency’s “shared vision”—the mission, vision for the

future, core values, goals and objectives, and strategies that define the

agency’s direction and expectations for itself and its people. Such

investments require an explicit workforce planning strategy, linked to the

agency’s strategic and program planning efforts. These planning efforts

should identify, among other things, the knowledge, skills, and abilities

needed to position an agency for its future. In addition, they should

provide an explicit link between the agency’s training offerings and

curriculums and the competencies identified by the agency for mission

accomplishment.

In VA’s Departmental Performance Plan for fiscal year 2001, VBA links the

TPSS program to its strategic goal of providing world-class service to

veterans and their families.7

 In working toward this goal, VBA’s top

 3

Not every case returned by the Board indicates that the regional office made an error. For

example, cases may be returned because the veteran submits new evidence or because

changes occur in regulations after an appealed case is sent to the Board.

4

 Veterans have two levels of appeal beyond the Board—first to the U.S. Court of Appeals

for Veterans Claims and then to the Court of Appeals for the Federal Circuit.

5

High-Risk Series: An Update (GAO-01-263, Jan. 2001).

6

Human Capital: A Self-Assessment Checklist for Agency Leaders (GAO/OCG-00-14G, Sept.

2000).

7

The Government Performance and Results Act of 1993 requires VA, as well as other

federal agencies, to clearly define its mission, set goals, and measure performance. VA

must submit to the Congress annual performance plans and annual reports on its success

in achieving program performance goals.
Page 5 GAO-01-601 Veterans' Benefits

priority is to improve the technical accuracy of claims-processing, and

VBA states that the TPSS program is one of its most important initiatives

for achieving this objective.8

 To develop TPSS training modules for a

specific claims-processing position, the job and tasks for that position

must be analyzed in detail. After this analysis, VBA estimates that it takes

14 to 16 months to develop each module, which consists of computerassisted instruction packages and written instructional materials designed

to support cooperative learning by employees who train in groups of two

or three. Under VBA’s direction, contractors develop the modules with the

help of subject matter experts from VBA’s regional and central offices.

Before a module is released, selected regional office employees take the

training on a trial basis to validate its effectiveness. Each training module

includes a pretest for determining whether an employee needs to take the

training. Each module also includes a post-test requiring students to

process test cases to determine their mastery of the tasks and work

processes. Using regional office employees, these pretests and post-tests

are also validated before the modules are released.

The design and development of the TPSS program will take longer than

planned to complete. According to VBA’s current schedule, the

development of the full TPSS program will not be completed until at least

calendar year 2004, or about 2 years later than VBA had planned. Also,

based on our survey of regional offices, 49 of the 57 regional offices had

used TPSS, but great variation existed in the extent to which they had

implemented the available TPSS modules to train employees. In fact, until

December 2000, VBA had not established a requirement that regional

offices use the TPSS modules. In general, the regional offices reported that

the most significant hindrance to using the modules was lack of time for

training due to workload pressures.

In early 2000, VBA projected that the development of the full TPSS

program would be completed during calendar year 2002.9

 However, as of

 8

TPSS training is in addition to local regional office training, such as classroom instruction,

on-the-job training, and coaching and mentoring and VBA training provided via VBA’s

Satellite Broadcast Network and the Video Teleconferencing System shared with the Board

of Veterans’ Appeals.

9

VBA published the projected schedule in its budget submission for fiscal year 2001, dated

February 2000.

TPSS Program

Schedule Has Been

Delayed and Regional

Office

Implementation Has

Varied

TPSS Program Is Behind

Schedule
Page 6 GAO-01-601 Veterans' Benefits

March 2001, the earliest projected completion date for the full program

would be calendar year 2004 (see table 1). As of March 2001, VBA had

provided the regional offices with 9 training modules out of an estimated

50 modules expected to constitute the full TPSS program.

Table 1: VBA’s Schedule for Releasing TPSS Modules to Regional Offices

TPSS modules

Estimated completion date shown

in VBA’s fiscal year 2001 budget

Actual

release date

Planned

release date

(as of March

2001)

Basic rating modules for new Rating Specialists Sept. 2000

Original compensation Jan. 1999

Original pension Jan. 1999

Original dependency and indemnity compensation Jan. 1999

Routine future examinations Dec. 1999

Hospitalization and convalescence reviews Dec. 1999

Reopened compensation Feb. 2001

New claim for pension Feb. 2001

Reopened dependency and indemnity compensation Feb. 2001

Due process June 2001

Ancillary benefits

All basic rating modules were to be

completed by Sept. 2000.

June 2002

Advanced rating modules for experienced Rating

Specialists and Decision Review Officers

March 2002

Appeals May 1998

Other modulesa

All advanced rating modules were to

be completed by Mar. 2002. Unknown

Medical modules for Rating SpecialistsbSept. 2000

Medical modules (first group) Oct. 2001

Medical modules (second group) May 2002

Medical modules (third group)

All medical modules were to be

completed by Sept. 2000.

May 2003

Modules for Veterans Service Representatives Dec. 2001

Original compensation Nov. 2001

Original pension Nov. 2001

Dependency benefits Nov. 2001

Death pension Nov. 2001

Income adjustments Nov. 2001

Fiduciary issuescUnknown

Other modules (approximately 15 modules)d

All modules for Veterans Service

Representatives were to be

completed by Dec. 2001.

2002-2004

a

VBA has suspended the development of the other advanced rating modules until fiscal year 2002.

The number of other modules to be developed has not yet been determined.

b

VBA has identified a preliminary list of 17 medical modules, such as medical terminology,

musculoskeletal, and organs of sense.

c

Regional offices have “Field” Veterans Service Representatives who specialize in the fiduciary

responsibilities of persons who hold or administer VA benefits in trust for beneficiaries. The modules

for fiduciary issues were to be completed by June 2002, according to the budget for fiscal year 2001.
Page 7 GAO-01-601 Veterans' Benefits

d

VBA anticipates developing 15 other modules for Veterans Service Representatives that include

such topics as helpless child benefits, aid and attendance for nursing home patients, and burial

benefits.

Source: VBA.

According to VBA, delays in the TPSS program can be attributed to two

primary causes—the creation of new claims-processing positions and

legislative changes in VBA’s duty to assist veterans in obtaining evidence

to support their claims. VBA is waiting for the duties of the new Decision

Review Officer position to stabilize before conducting the task analysis

necessary to develop the training modules for this position. In addition,

after completing the task analysis for the new Veterans Service

Representative position, VBA learned it will need to develop many more

training modules than originally expected because the position

encompasses more tasks and greater complexity than anticipated.10

Moreover, VBA has decided to give priority to developing modules for the

Veterans Service Representative position and thus delay the development

of training modules for other categories of employees.11 Finally, because

the Congress enacted legislative changes in the duty-to-assist requirements

in November 2000, VBA decided to include these new requirements in the

remaining basic rating modules. Doing so delayed releasing those modules

to the regional offices.

At the time of our survey in October 2000, the extent to which regional

offices had used the six TPSS modules that were available varied widely.

Of the 57 regional offices, 8 had not used any of the available TPSS

modules to train employees. Of the 49 that had used TPSS, their use varied

greatly. For example, 41 regional offices had used the original

compensation module, but only 11 had used the routine future

 10VBA began a demonstration project in 1996 that led to the adoption of the TPSS program

in 1998. In this demonstration project, VBA developed the Appeals training module for

Rating Specialists and conducted the job and task analysis necessary to develop the basic

rating modules for Rating Specialists.

11VBA shifted funding away from the advanced rating modules for Rating Specialists in

order to accelerate the development of modules for Veterans Service Representatives. VBA

also slowed the development of modules for fiduciary issues. According to VBA officials,

resource issues prevent VBA from developing TPSS modules for all claims-processing

positions at the same time. Beyond the issue of annual funding availability, developing

TPSS modules requires the time of regional office employees for developing and validating

module content and testing. According to VBA, concurrent development of modules for all

claims-processing positions would drain too many resources from the field.

Regional Office

Implementation of TPSS

Varied Widely
Page 8 GAO-01-601 Veterans' Benefits

examinations module (see table 2). Only 5 regional offices had used all of

the available modules to train employees, and 10 regional offices had used

only one module.

Table 2: Number and Percentage of Regional Offices Using TPSS as of September

30, 2000

TPSS modules

released as of

September 30, 2000

Number of

regional offices

that had used

module

Percentage of

57 regional

offices that had

used module

Original compensation 41 72

Appeals 33 58

Original pension 33 58

Original dependency and indemnity compensation 26 46

Hospitalization and convalescence 12 21

Routine future exams 11 19

Source: GAO survey.

VBA had strongly encouraged, but not required, regional offices to use

TPSS in their training programs. However, VBA conducted a survey of its

own in October 2000 and found that some regional offices still were not

using any TPSS modules. As a result, in December 2000, VBA issued a

letter mandating the use of TPSS in all regional offices. According to VBA

officials, the December mandate required that regional offices use the

available basic rating modules to train all new Rating Specialists and use

future modules for Veterans Service Representatives to train all new

employees filling this position. VBA officials told us that these new

training requirements for specific types of employees have also been

communicated to the regional offices via nationwide conference calls.

Based on responses to our survey from the 49 regional offices that had

used at least one TPSS module, the most significant hindrance to

implementing the modules was lack of time for training due to workload

pressures. About 47 percent of these regional offices said that workload

pressures affected their use of TPSS to at least a moderate extent. As we

reported in May 2000, persistent backlogs of pending claims have resulted

in veterans having to endure long waits to receive decisions on their

claims.12 Consequently, regional offices face pressure not only to improve

 12Veterans Benefits Administration: Problems and Challenges Facing Disability Claims

Processing (GAO/T-HEHS/AIMD-00-146, May 2000).
Page 9 GAO-01-601 Veterans' Benefits

accuracy in processing claims, but also to reduce the backlog of pending

claims. A contractor evaluating VBA training reported in April 2000 that

“training time is too easily given up to the press of productivity.”

13 The six

modules that had been released at the time of our survey were designed to

require a total of about 24 workdays for an employee to complete.

In 1998, VBA prepared a rationale supporting the adoption of the TPSS

program.14 In this rationale, VBA outlined several objectives of the TPSS

program related to human capital management, including providing

training for new employees replacing the expected wave of retirees,

shortening the training period for employees, and improving employees’

claims-processing accuracy and consistency.15 We found that many TPSS

modules will not be available when needed to train new employees and

that TPSS may not substantially reduce the training period for employees.

Also, VBA has not developed a formal plan to evaluate the program’s

impact on claims-processing accuracy or consistency.

In its rationale for the TPSS program, VBA stated that the program would

allow VBA to effectively train employees hired and promoted as

replacements for the expected wave of retiring claims processors.

However, given the slippage in the TPSS schedule, new employees have

been and will be working in these positions before many of the TPSS

modules are available for their training. Of the 4,000 Veterans Service

Representatives and Rating Specialists that were employed as of fiscal

year 2000, VBA expects about 1,000 (25 percent) to retire between 2000-

 13VBA contracted with the University of Illinois to conduct a three-phase evaluation of the

TPSS program. The first phase, completed in April 1999, focused on the quality and

implementation of the appeals training module. The second phase, completed in April 2000,

attempted, among other things, to examine whether appeals module training reduced the

rate at which the Board of Veterans’ Appeals returned decisions to regional offices for

further development. Researchers concluded they could not isolate the effect of the

appeals module on the return rate. The final report for the third phase of the TPSS

evaluation is not yet available. The third phase continued the examination of the

implementation and effect of the appeals module and also examined how TPSS training fits

into regional office training programs.

14The rationale for the program was set forth in VBA’s September 1998 Capital Investment

Proposal for TPSS.

15Because of the judgment and subjectivity involved in many claims, two different Rating

Specialists could review the same case and reach differing decisions that might be

considered “accurate,” but would not be “consistent.”

TPSS May Not Fully

Achieve Some of Its

Objectives

Many TPSS Modules Will

Not Be Available When

Needed to Train New

Claims Processing

Employees
Page 10 GAO-01-601 Veterans' Benefits

2005. In response, VBA already has hired a total of about 600 new Veterans

Service Representatives and Rating Specialists between fiscal years 1998

and 2000. Additionally, in fiscal year 2001, VBA anticipates hiring 224

Veterans Service Representatives and 325 Rating Specialists. These new

workers need to be trained in order to carry out the duties of their

predecessors. However, as of late calendar year 2000, VBA did not expect

to release any TPSS modules to train new Veterans Service

Representatives until late in calendar year 2002 or in 2003.

Recognizing that the regional offices had an immediate need to train new

Veterans Service Representatives, the Under Secretary for Benefits

indicated that VBA needed to provide some modules earlier than

projected. As a result, VBA currently plans to release five modules that are

more narrowly scoped than originally planned for Veterans Service

Representatives in November 2001 (see table 1). The remaining modules

for Veterans Service Representatives are to be released intermittently

during 2002 through 2004.

At the same time, VBA also is exploring alternative means of providing

standardized training to compensate for the delays in developing TPSS

modules. For Veterans Service Representatives, VBA is developing a

reference manual for new hires and also is revising a training guide it has

on its internal computer network to make it more interactive,

comprehensive, and user friendly. VBA expects to complete these efforts

in March and April 2001, respectively. Additionally, it is considering using

the VBA Satellite Network and video teleconferencing facilities to provide

training for both Rating Specialists and Veterans Service Representatives.

The extent to which these alternatives may eliminate the need to develop

some future TPSS modules has not been determined.

Furthermore, VBA has established a formal 12-week basic training course

for all new Rating Specialists beginning in March 2001 and for all new

Veterans Service Representatives in April 2001. To provide this training,

VBA established training centers for each of the nine Service Delivery

Networks under which the regional offices are organized.16 After new

trainees have been given training in specific topics, such as VA

 16VBA has structured its 57 regional offices into nine Service Delivery Networks. The

regional offices that compose each Service Delivery Network are expected to collaborate,

provide mutual support, share resources, use team-based principles, and share collective

responsibility and accountability for the Service Delivery Network’s overall performance of

all assigned work.
Page 11 GAO-01-601 Veterans' Benefits

terminology, by the regional offices, they will alternate between 3 weeks

of intensive training at the training centers and 3-week periods of training

in their regional offices. Training in the regional offices will incorporate

available TPSS modules.

In its rationale for the TPSS program, VBA may have overstated the extent

to which the TPSS modules can be expected to reduce the employees’

training period. VBA assumed that the TPSS basic rating modules would

enable new Rating Specialists to become fully productive in 6 to 8 months

rather than 2 years. This assumption presumed that TPSS would

significantly reduce the need for on-the-job training provided by regional

offices, which historically has constituted the majority of training.

Although the regional offices that used the modules generally said in our

survey that the TPSS modules do an adequate job of providing basic

knowledge, most of these regional offices also said that traditional forms

of local training will remain essential or very important.

Regional office officials and employees told us that TPSS cannot by itself

adequately train employees in the many complexities, subjective issues,

and variations in circumstances encountered in individual claims. In a

similar vein, a VBA contractor evaluating TPSS training said that TPSS

could not fully cover the complexity of the work of Rating Specialists.

Such in-depth training can only be gained through on-the-job training and

coaching and mentoring while employees are gaining actual experience in

processing claims. In our site visits, regional office officials said they

believe that achieving full proficiency as a Rating Specialist will require

about 2 years, even with TPSS. VBA officials, however, said that such

conclusions may be premature because at the time of our review, the

regional offices had had relatively limited exposure to TPSS. They also

said that they believe the new 12-week training course incorporating TPSS

will help new trainees become proficient faster than in the past. In

addition, the officials said that they plan to evaluate the results of the new

12-week program.

In its rationale for adopting the TPSS program, VBA stated that TPSS

would help improve the accuracy and consistency of claims processing;

however, VBA has no formal plan to assess the impact of TPSS on the

accuracy rate or on consistency. Assessing the specific impact of TPSS on

accuracy and consistency is difficult. While it may not be feasible for VBA

to isolate the effect of TPSS on accuracy or consistency, opportunities

exist for VBA to collect data that could help provide some indications of

TPSS May Not

Substantially Shorten the

Training Period for

Employees to Become

Fully Proficient

Impact of TPSS Can Be

Further Evaluated
Page 12 GAO-01-601 Veterans' Benefits

the possible extent of such effects. As stated in our self-assessment

checklist for helping agencies address human capital issues, an agency

should make determinations of the impact of its training programs.

Assessing the specific impact of TPSS on the accuracy of claimsprocessing performance is difficult. In its fiscal year 2001 budget

submission, VBA stated that it is reasonable to anticipate that the TPSS

program will improve the national accuracy rate for disability claims

processing by 8 percentage points. However, according to VBA officials,

this projected increase in the accuracy rate is not based on an analysis

linking employee training to claims-processing accuracy, but rather on

professional judgment. The ability to isolate the specific impact of TPSS is

difficult because other initiatives implemented by VBA may also affect

claims-processing performance. For instance, VBA’s new accuracy

measurement system can also affect claims-processing performance

because it provides information on specific types of errors.17 This

information can be used to correct the causes of errors, thereby helping to

improve the overall accuracy rate. In addition, researchers have found it

difficult to link TPSS training to changes in employees’ claims-processing

performance. For example, researchers cautioned that while their findings

indicated that TPSS training did result in knowledge transfer (over 95

percent of the students who took TPSS module post-tests passed the

tests), their evaluation results could not substantiate whether TPSS

training had improved standardization or increased claims-processing

productivity across regional offices.

Using its accuracy measurement system, VBA is able to track overall

trends in the claims-processing accuracy rate, which may provide a

general indicator of whether TPSS is helping to improve accuracy at the

national level. However, it may be possible for VBA to collect data that

could provide further indications of the impact of TPSS. In fiscal year

2001, VBA is planning to pilot test a new initiative—Systematic Individual

Performance Assessment (SIPA)—to assess performance quality and

identify training needs at the employee level. SIPA will complement the

existing accuracy measurement system by reviewing the quality of a

representative sample of the individual employee’s work products. Such

 17In fiscal year 1999, VBA implemented a new system for measuring the accuracy of claims

processing. Under the new system, the error rate includes not only incorrect decisions on

whether to grant or deny benefits but also procedural and technical errors such as the

failure to include all required documentation in the case file or to properly notify veterans

of decisions.

Impact on Accuracy
Page 13 GAO-01-601 Veterans' Benefits

data on individual employees’ performance could potentially help provide

indications of whether TPSS is having an impact on accuracy. For

example, VBA could use such data to monitor the accuracy levels of

individual employees as they progress through the TPSS curriculum, as

well as compare the accuracy levels of employees who completed training

on certain TPSS modules with those employees who did not. VBA is

considering using data from SIPA to assess the impact of TPSS, but VBA

has not developed a formal evaluation plan.

VBA in its rationale for the TPSS program stated that the program is

intended to improve claims-processing consistency among the regional

offices. VBA is considering performing assessments to measure claimsprocessing consistency, but it has not developed a formal evaluation plan.

Such assessments are needed to obtain data that could provide indications

of whether TPSS is helping to improve consistency. In 1997, the National

Academy of Public Administration issued a report that, among other

things, addressed inconsistency in VBA’s claims processing. The report

stated that VBA needed to identify the degree of subjectivity expected for

various medical issues, set consistency standards, and measure the level of

consistency as part of the quality review process or through testing of

control cases in several regional offices. Performing such an assessment

could provide a baseline against which future assessments of consistency

could be compared to measure progress in achieving consistency. While

such assessments would not provide conclusive evidence on the impact of

TPSS, they potentially could provide an indication of whether TPSS is

helping to improve consistency.

While we support VBA’s objective to centrally develop a standardized

training program, significant delays in the development of TPSS are

hindering the program’s ability to achieve its objective of providing

standardized training to claims processing employees. Given the large

number of recent and expected new employees, we support VBA’s effort

to expedite the development of modules aimed at these employees. At the

same time, we support its exploration of alternative ways to provide

standardized training to employees for whom TPSS modules are not yet

available. Moreover, because TPSS is still evolving, opportunities exist to

reassess the need for TPSS modules whose topics are adequately covered

through these alternative means. Furthermore, we recognize that it may

not be feasible to measure the exact extent to which TPSS improves

claims-processing accuracy or consistency. However, opportunities exist

for VBA to collect data that could help provide some indications of the

possible extent of such effects. Such information could serve as an

Impact on Consistency

Conclusions
Page 14 GAO-01-601 Veterans' Benefits

indicator of the impact of TPSS on accuracy and consistency and help

VBA demonstrate the value of the program to the Congress and to

veterans.

To further strengthen VBA’s ability to provide timely, standardized training

to its regional office employees who process disability claims, we

recommend that the Secretary of Veterans Affairs direct the Under

Secretary for Benefits to take the following actions.

• As VBA explores alternative means, such as the VBA Satellite Network, to

deliver timely standardized training, it should determine whether the

topics that are covered through such alternative means could be

eliminated from plans for future TPSS modules.

• VBA should develop formal plans for evaluating the impact of TPSS on

claims-processing accuracy and consistency. For example, VBA could use

individual performance data obtained from SIPA to provide indications of

the impact of TPSS on accuracy. VBA could also conduct a baseline

assessment of consistency, such as testing control cases in regional

offices, and then conduct periodic assessments to provide indications of

whether TPSS is helping to improve consistency. The information

regarding the impact of TPSS should be included in the annual reports

submitted under the Government Performance and Results Act of 1993.

We received written comments on a draft of this report from VA (see app.

II). In its comments, VA concurred with both of our recommendations.

With regard to our first recommendation, VA stated that VBA will continue

exploring alternative means for providing standardized training and will

evaluate whether training on certain topics is better suited to TPSS or

alternative means such as the VBA Satellite Network. With regard to our

second recommendation, VA stated that when the SIPA program is

implemented in late 2001 and 2002, VBA intends to develop formal plans to

use it as a tool in evaluating the impact of TPSS on accuracy and

consistency. We support these efforts.

In its comments, VA also stated that while some variance exists to the

TPSS schedule produced in fiscal year 1998, VBA has given assurance that

it will meet the original final delivery date in fiscal year 2004. However, in

its fiscal year 2001 budget submission, VBA showed that the TPSS program

would be completed in calendar year 2002. Accordingly, we are reporting

that the TPSS program is behind schedule.

Recommendations for

Executive Action

Agency Comments

and Our Evaluation
Page 15 GAO-01-601 Veterans' Benefits

As agreed with your office, unless you publicly announce it contents

earlier, we plan no further distribution of this report until 30 days after its

issue date. At that time, we will send copies of this report to the Honorable

Christopher H. Smith, Chairman of the House Committee on Veterans’

Affairs; the Honorable Arlen Specter, Chairman of the Senate Committee

on Veterans’ Affairs; the Honorable John D. Rockefeller IV, Ranking

Minority Member of the Senate Committee on Veterans’ Affairs; other

appropriate congressional committees; the Honorable Anthony J. Principi,

Secretary of the Department of Veterans Affairs; and other interested

parties. We will also make copies available to others upon request.

If you have questions about this report, please call me on (202) 512-7101 or

Irene Chu on (202) 512-7102. Other key contributors were Ira Spears, Steve

Morris, Paul Wright, and Patrick di Battista.

Sincerely yours,

Cynthia A. Bascetta

Director, Education, Workforce,

 and Income Security Issues
Appendix I: Scope and Methodology

Page 16 GAO-01-601 Veterans' Benefits

To determine regional offices’ experiences with and views on the Training and

Performance Support System (TPSS) program, as well as other types of training, we

sent a questionnaire in October 2000 to each of the Veterans Benefits Administration’s

(VBA) 57 regional offices.1

 All 57 offices responded to our survey. To examine the

extent and manner of TPSS usage in more detail, we visited six regional offices: Des

Moines, Iowa; Roanoke, Virginia; Pittsburgh, Pennsylvania; St. Petersburg, Florida;

San Diego, California; and Seattle, Washington. The six offices we visited were

judgmentally selected to provide a mix of offices based on a combination of factors,

including regional office workloads, Service Delivery Network affiliation, geographic

location, and variations in the regional offices’ implementation and usage of TPSS.

While we did not attempt to evaluate the effectiveness of the TPSS program, or other

forms of training, we discussed the benefits and limitations of these training

approaches with officials of the regional offices we visited and VBA headquarters.

 1

Although VBA has an office in Cheyenne, Wyoming, this office reports administratively through the

Denver Regional Office, and for accuracy measurement, the Cheyenne office is treated as part of the

Denver Regional Office. Therefore, we sent a questionnaire to the Denver Regional Office only.

Appendix I: Scope and Methodology
Appendix II: Comments From the Department

of Veterans Affairs

Page 17 GAO-01-601 Veterans' Benefits

Appendix II: Comments From the

Department of Veterans Affairs
Appendix II: Comments From the Department

of Veterans Affairs

Page 18 GAO-01-601 Veterans' Benefits
Appendix II: Comments From the Department

of Veterans Affairs

(207103) Page 19 GAO-01-601 Veterans' Benefits

The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs