Veterans' Benefits Claims: Further Improvements Needed in Claims-Processing Accuracy
Published: 1999-03-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Department of Veterans Affairs' (VA) regional offices' (RO) accuracy in processing disability claims, focusing on: (1) the extent of improvements made by the Systematic Technical Accuracy Review (STAR) system in measuring claims-processing accuracy; (2) additional efforts needed to strengthen the system; and (3) challenges the Veterans Benefits Administration (VBA) faces in meeting goals for improving claims-processing accuracy.
section_title: What GAO Found
paragraphs: GAO noted that: (1) the new STAR system represents an important step forward by VBA in measuring the accuracy of compensation and pension claims processing; (2) compared with the previous system, STAR focuses more on RO decisions that are likely to contain processing errors, uses a stricter method for computing accuracy rates, provides more data on the performance of organizational levels within VBA, collects more data on processing errors, and stores more accuracy review results in a centralized database; (3) even so, VBA can further strengthen STAR's ability to identify error-prone cases and claims-processing weaknesses so that it can take corrective actions; (4) VBA needs to better pinpoint error-prone cases and weaknesses in the development of evidence by collecting more specific data on the types of medical characteristics and deficiencies in medical evidence that are most prevalent in incorrect decisions; (5) VBA can also better address vulnerabilities in the integrity of accuracy data; (6) STAR reviewers in ROs do not have sufficient separation of duties or adequate independence to meet government standards for internal controls or program performance audits; (7) these shortcomings raise concern about the integrity of STAR accuracy data, which are a key factor in the performance measurement system designed by VBA to meet the requirements of the Government Performance and Results Act (GPRA) of 1993; (8) while such system improvements are necessary, they alone are not sufficient for VBA to meet its goal for improving accuracy; (9) using the STAR pilot test's 64-percent accuracy rate as a baseline, VBA's goal is to achieve a 93-percent accuracy rate by fiscal year 2004; (10) VBA faces management challenges that it must address successfully in order to meet its accuracy improvement goal; (11) to do this, VBA recognizes that: (a) its newly implemented performance measurement system must hold program managers accountable for performance; and (b) the training program under development must effectively train the current RO workforce as well as the many new employees who will have to be hired in the coming decade to replace those who retire; and (12) it is too early to determine whether VBA's efforts to meet these challenges will be successful.

section_title: Letter
paragraphs: In fiscal year 1997, the Department of Veterans Affairs (VA) paid about $19 billion in compensation and pension benefits to more than 3 million veterans and their dependents and survivors. The compensation program pays monthly benefits to veterans with service-connected disabilities (injuries or diseases incurred or aggravated while on active military duty). Veterans with service-connected disabilities are entitled to compensation benefits even if they are working and regardless of the amount they earn. By contrast, the pension program pays monthly benefits to wartime veterans who have low incomes and are permanently and totally disabled for reasons not connected to their service.
VA has 58 regional offices (RO) that process veterans’ compensation and pension claims and decide whether to award benefits. The ROs develop evidence and adjudicate these claims under program guidance and policy provided by VA’s Veterans Benefits Administration (VBA). In recent years, the accuracy of RO claims processing has been the subject of concern and attention within VA and from the Congress and veterans’ service organizations. Although VBA had been reporting until recently that ROs process claims accurately more than 95 percent of the time, questions arose about RO accuracy because, when dissatisfied veterans appealed ROs’ initial decisions, the Board of Veterans’ Appeals during fiscal years 1993-97 reversed about 19 percent of the appealed decisions and remanded about 47 percent back to ROs for further development and reconsideration. In fiscal year 1998, VBA pilot tested its new accuracy measurement system, known as the Systematic Technical Accuracy Review (STAR) system. Using the STAR system, VBA found an accuracy rate of only 64 percent for RO initial decisions, indicating that VBA needs to give more attention to ensuring that ROs make the correct decision the first time so that veterans need not make unnecessary appeals or be unnecessarily delayed in receiving benefits owed them.
VBA implemented the STAR system nationwide in October 1998. VBA sees STAR as an improvement in its ability to measure accuracy and identify and correct the causes of claims-processing errors. As agreed with your office, this report addresses (1) the extent of improvements made by STAR in measuring claims-processing accuracy, (2) additional efforts needed to strengthen the system, and (3) challenges VBA faces in meeting goals for improving claims-processing accuracy.
In conducting our review, we spoke with officials of and reviewed reports and policy guidance by VBA, the Board of Veterans’ Appeals, and 15 ROs. We also received available data from VBA, the Board, and the ROs. Furthermore, we contacted and reviewed documents from several veterans’ service organizations, the National Academy of Public Administration, the Veterans’ Claims Adjudication Commission, and VA’s Office of Inspector General. We also obtained information on the quality assurance programs of several other organizations, including the Social Security Administration (SSA), which administers the largest federal disability benefits program. We conducted our review between October 1997 and December 1998 in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: VBA’s Compensation and Pension Service, located at VA headquarters, formulates the policy and guidance used by the RO staff who receive, develop, and evaluate veterans’ compensation and pension claims. The compensation program pays monthly benefits to veterans with service-connected disabilities (injuries or diseases incurred or aggravated while on active military duty). Veterans with service-connected disabilities are entitled to compensation benefits even if they are working and regardless of the amount they earn. By contrast, the pension program pays monthly benefits to wartime veterans who have low incomes and are permanently and totally disabled for reasons not connected to their service. In compensation cases, the payment amount varies according to degree of disability; in pension cases, the amount varies according to financial need. When veterans are unable to manage their affairs, benefit payments are made to guardians who serve as their fiduciary representatives.
Adjudicating an original disability claim involves two basic functions—“authorization” and “disability rating.” Authorization involves obtaining records from the military services and information from the veterans, such as medical records and information on income and dependents. Disability rating involves establishing whether a veteran’s impairment is service-connected and, if so, evaluating the veteran’s degree of disability. VBA considers claims requiring a disability rating to be the core workload of the compensation and pension program, and as a group, cases requiring a disability rating are considered to be the most error-prone in the program. In order to rate (or evaluate) a veteran’s disability, ROs often determine that they need medical evidence in addition to evidence obtained from the veteran’s physicians and other medical providers. In such cases, they send veterans to the Veterans Health Administration (VHA) for physical or mental examinations by VHA physicians.
From the medical evidence, ROs rate a veteran’s disability using VA’s Schedule for Rating Disabilities, which lists physical and mental conditions and assigns a disability rating to each condition. Under this schedule, the degree of disability is expressed in 10-percent increments up to 100-percent disability. A veteran can also receive a “zero-percent” disability rating, which means the veteran’s condition is service-connected but not severe enough to qualify for compensation payments on the basis of the medical criteria specified in the rating schedule. If the veteran’s condition later worsens, he or she may reapply, asking VA to increase the rating from zero to 10 percent or more.
Evaluating the degree of disability for some conditions, such as mental impairments, can require adjudicators to make subjective judgments that are not always clear-cut. For veterans with multiple impairments, the RO must rate each impairment separately and then combine them into a composite rating. After a veteran is placed on the rolls, his or her condition or circumstances may change in ways that can result in adjustments to the RO’s initial decision. For example, a veteran may file a claim for an increase in degree of disability if his or her medical condition deteriorates. Or nonmedical issues may arise that require an adjustment to the initial decision but do not require a disability rating in order to make the new decision. Such cases could arise from changes in the status of the veteran’s dependents or changes in the income of a veteran receiving pension benefits.
After the RO notifies the veteran of its decision, the veteran, if dissatisfied, may ask for a hearing before an RO hearing officer. The veteran also may file a notice of disagreement with the RO and then file an appeal asking for a review of the RO’s decision by the Board of Veterans’ Appeals, which makes VA’s final decisions on appeals on behalf of the Secretary. The Board may conduct a hearing if the veteran requests one. In deciding appeals, the Board can grant benefits (reverse the RO decision), deny benefits (affirm the RO decision), or remand (or return) the case to the RO to develop further evidence and reconsider the claim. After further development of a remanded claim, the RO either awards the claim or returns it to the Board for a decision.
Before 1989, the Board’s decisions on appeals were final. In that year, however, the Court of Veterans Appeals—established by the Veterans’ Judicial Review Act of 1988 (P.L. 100-687, Nov. 18, 1988)—began to hear cases. As a result, the Board is no longer the final step in the claims adjudication process. When a veteran disagrees with a decision of the Board, the veteran may now appeal to the Court, which is independent of VA. Additionally, either veterans or VA may appeal decisions of the Court of Veterans Appeals to the Court of Appeals for the Federal Circuit.
Since veterans began appealing Board decisions to the Court of Veterans Appeals, according to a court official, the Court has remanded more than 4,500 decisions back to the Board for further development and reconsideration. According to the same official, this represents about 59 percent of the Board’s decisions that were appealed to the Court, excluding dismissed cases. In turn, ROs have felt the repercussions of these Court decisions as evidenced by significant increases in the Board’s reversals and remands of appealed RO decisions. Before the advent of the Court, the Board historically had annually awarded benefits in 12 to 14 percent of appealed RO decisions and had annually remanded another 12 to 24 percent back to ROs for further development. However, in the years since the advent of the Court, the Board has annually awarded benefits in about 14 to 20 percent of the cases it reviewed and remanded another 31 to 51 percent back to ROs for further development. Despite these increases in awards and remands by the Board, VBA had continued to report—until STAR was implemented—that ROs were accurately processing compensation and pension claims more than 95 percent of the time. (See app. I for more details on the reversal and remand rates of the Court and the Board and on the accuracy rates reported by VBA.)
VBA considers a disability claim to have been accurately processed if basic eligibility has been determined correctly, the case file contains all required medical and nonmedical documentary evidence, the RO’s decision on service-connection and the rating given to each impairment are correct, the payment amount is correct, and the RO properly notified the veteran of the outcome of his or her claim.
Under the accuracy measurement system that was in operation from fiscal year 1992 through fiscal year 1997, VBA headquarters annually reviewed approximately 100 cases randomly selected from the cases completed by each of 57 ROs. These cases were selected from the entire universe of compensation and pension work products completed by the ROs. Using this procedure, VBA produced a national accuracy rate with a reasonable level of statistical precision. While each year’s sample was too small for VBA to produce accuracy rates for each RO with a reasonable level of statistical precision, VBA required each RO to self-review 300 to 900 cases annually, depending on the size of the RO. These RO self-reviews were to provide ROs with information needed to improve quality, not to compute accuracy rates for measuring performance.
Statistical precision refers to the amount of uncertainty in an estimate that results from sampling variability at a given level of confidence. For example, if a sample that has a 95-percent confidence level and a precision level of plus or minus 5 percentage points yields an estimated accuracy rate of 70 percent, this means that one can be 95-percent confident that the true accuracy rate is between 65 percent and 75 percent. accountability for the SDN’s overall performance of all work assigned to it. In meeting the requirements of the Results Act, VBA headquarters will measure each SDN’s performance, and each SDN will assess the performance of its ROs. This measurement will be made on the basis of five performance factors: claims-processing accuracy (as determined by STAR), timeliness of claims processing, unit cost, customer satisfaction, and employee satisfaction and development.
section_title: VBA Has Improved Its Measurement of Claims-Processing Accuracy
paragraphs: The new STAR system represents an important step forward by VBA in measuring the accuracy of compensation and pension claims processing and in providing data to identify error-prone cases and correct the causes of errors, including those that result in reversals and remands by the Board of Veterans’ Appeals. Compared with the previous accuracy measurement system that VBA had been using since 1992, the STAR system is a step forward because it focuses more on RO decisions that are likely to contain claims-processing errors, uses a stricter method for computing accuracy rates, provides more data on the performance of additional organizational levels within VBA, collects more data on errors, and stores the results of more accuracy reviews in a centralized database for further review and analysis.
Whereas VBA had been reporting more than 95-percent accuracy under the previous accuracy measurement system, VBA, in its pilot test of STAR, reported that only 64 percent of veterans’ claims were processed accurately. A primary reason for this difference is that the pilot test focused only on the most complex and more error-prone RO work products, those involving disability rating decisions. By contrast, the previous system drew its sample of cases from the entire universe of RO work products, including those not requiring an assessment of disability and, therefore, less error-prone. The newly implemented STAR system continues to focus on claims that involve disability ratings, but it also includes a sample of cases that address issues typically not requiring disability ratings and a separate sample of cases involving guardianship issues for veterans unable to represent themselves. Separate accuracy rates are computed for each of these two other samples.
Another reason that the STAR pilot test found an accuracy rate of 64 percent rather than 95 percent as reported under the previous system is STAR’s stricter accuracy rate computation method. Under the previous system, VBA categorized each error into one of three areas of the claims adjudication process: (1) case control and development, (2) decision elements, and (3) notification to the veteran. Thus, if a case had one error, VBA would record this error under the appropriate area and show the two other areas as error-free. After reviewing all cases, VBA computed separate accuracy rates for each of the three claims adjudication areas and then determined an overall accuracy rate by calculating the arithmetic mean (or average) of the three accuracy rates. Under STAR, however, VBA does not compute separate accuracy rates for the three areas of the claims adjudication process. If a case has any errors in any area of the claims adjudication process, the entire case is counted as incorrect for accuracy rate computation purposes. This approach tends to result in a lower accuracy rate than under the previous system. (See app. II for a hypothetical example demonstrating how STAR’s computation method can result in a lower accuracy rate.)
In addition to focusing more on error-prone RO decisions and using a stricter accuracy rate computation method, STAR provides accuracy rates with reasonable statistical precision not only for the nation as a whole but also for each SDN. Under the previous system, VBA headquarters had reviewed about 5,700 cases annually. Its sampling methodology allowed VBA to produce an accuracy rate with reasonable statistical precision for the nation as a whole. Under STAR, VBA headquarters will review about 7,400 cases annually. Its sampling methodology will enable VBA to provide accuracy rates with reasonable statistical precision for the nation and each SDN for the sample of cases requiring disability ratings and the sample of cases typically not requiring such ratings (see app. II for SDN sample sizes and statistical precision data). However, the sample of cases involving guardianship issues will be too small to provide the same level of statistical precision.
VBA originally considered designing STAR so that VBA headquarters also could produce accuracy rates for each RO but dropped this option because it would have required VBA headquarters to review an additional 50,000 cases annually. Instead, VBA opted to require each RO to review samples of its own work products using STAR review procedures. As in the headquarters review, these RO self-reviews will produce accuracy rates with reasonable statistical precision for the sample of cases requiring disability ratings and the sample of cases typically not requiring such ratings. However, the sample of cases involving guardianship issues will be too small to produce accuracy rates with the same level of statistical precision. Nationwide, the ROs will review about 44,000 randomly selected cases (see app. II for RO sample sizes and statistical precision data). VBA estimates that every 1,000 cases in these samples require about 1.0 full-time equivalent review staff per year.
STAR is also an improvement over the previous accuracy measurement system because it provides more precise information on the inaccuracies it identifies. Under the previous system, VBA’s database essentially captured only whether a decision did or did not contain errors. By contrast, STAR requires reviewers to answer a standardized series of questions about whether the RO’s actions and decisions were correct or incorrect in various steps of claims processing. The reviewers enter their answers to these questions, along with brief narrative comments, in the STAR database. In addition, because the need for further development of evidence is a primary reason that the Board of Veterans’ Appeals remands many cases to ROs, STAR asks reviewers to identify deficient evidence categories, such as private medical evidence, VA medical center records, and service records. Also, because the Board remands many cases to ROs to obtain further medical examinations by VHA physicians, STAR asks reviewers to indicate whether deficiencies in medical evidence supporting the decision relate to VHA medical examinations. These data on deficiencies in evidence are entered in the STAR database. The database also identifies cases involving five special conditions that have medical implications: prisoner of war, radiation exposure, Gulf War veteran, Agent Orange exposure, and posttraumatic stress syndrome.
Additionally, STAR’s database captures the results of accuracy reviews conducted by both VBA headquarters and the ROs, whereas under the previous system, VBA’s database captured only the results of accuracy reviews conducted by VBA headquarters. VBA planned to implement in February 1999 a new centralized database on its internal network (intranet) system that will permit both VBA headquarters and the ROs to input the results of all STAR reviews into the database. Capturing RO data will enrich the data available to analyze error trends, and both VBA headquarters and the ROs will have access to the full complement of data through the intranet.
section_title: VBA Can Strengthen Its Ability to Identify Error-Prone Cases and Address Vulnerabilities in the Integrity of Accuracy Reviews
paragraphs: Although STAR represents a significant step forward in VBA’s ability to measure claims-processing accuracy and identify error-prone cases, VBA can take further steps to improve this ability. These steps involve collecting additional data for identifying and correcting error-prone cases and addressing vulnerabilities in the integrity of accuracy reviews.
subsections: section_title: VBA Can Improve the Ability to Identify Error-Prone Cases
paragraphs: Even with the improvements provided by STAR, VBA’s ability to identify error-prone cases and target corrective actions is constrained by the limited data that it captures on the medical characteristics of claimants whose claims are processed incorrectly and on why medical evidence is deficient. Data captured on claimants’ medical characteristics is currently limited to identifying whether a veteran was a prisoner of war, served in the Gulf War, or had posttraumatic stress syndrome, radiation exposure, or Agent Orange exposure. More detailed medical characteristics data could help pinpoint the specific types of claims in which errors occur. Also, although STAR captures data on whether medical evidence and medical examinations are adequate, it does not record statistical data identifying why reviewers found the evidence or examinations supporting RO final decisions to be deficient. Such data also could help pinpoint the types of corrective actions that need to be taken to improve the accuracy of RO decisions.
Limited studies by VBA demonstrate how capturing additional data in the STAR database on medical issues could help VBA focus on corrective actions that can reduce claims-processing errors and in turn reduce remands from the Board of Veterans’ Appeals. In 1996, VBA and the Board of Veterans’ Appeals jointly conducted a limited study of remanded cases and reported that inadequate medical examinations were the most frequent reason for remands and that a majority of the remanded cases involved the need for specialty examinations, such as orthopedic, psychiatric, neurologic, audiologic, and ear-nose-throat examinations. Also, in 1996, the Milwaukee RO reviewed claims that were awarded by the RO’s hearing officers after the claims were initially denied. Of the cases in which the RO’s hearing officers reversed the initial decision, the Milwaukee RO captured data on the specific conditions, such as orthopedic impairments, that were involved in significant numbers of cases, and using such data, the RO identified specific corrective actions. According to Milwaukee RO officials, this helped reduce the RO’s remand rate from the Board of Veterans’ Appeals. From fiscal year 1995 to fiscal year 1998, the Milwaukee RO reduced its remand rate from about 40 percent to about 21 percent, one of the lowest remand rates in the nation.
SSA, which administers the largest federal disability program, has a quality assurance system that captures detailed data on claimants’ medical characteristics and on weaknesses in evidence. SSA has found that such data are helpful in identifying error-prone cases and targeting corrective actions. For each case reviewed, SSA’s system captures data on the specific body systems involved, such as musculoskeletal, respiratory, cardiovascular, and mental systems. Further, using codes from the International Classification of Diseases, SSA’s system identifies each claimant’s specific impairments. Additionally, when medical evidence is judged not adequate, SSA’s system records the specific medical specialty area in which evidence was lacking, such as orthopedic, psychiatric, and neurologic areas, and it identifies the specific type of test, study, or other medical evidence that was lacking.
Such data, according to an SSA quality assurance official, not only helps to identify error-prone cases but can pinpoint specific evidentiary weaknesses for cases involving specific body systems or impairments. Also, this official stated that spending resources up front to capture such data can reduce the need to conduct time-consuming special studies later to understand why certain types of cases are being processed incorrectly. According to the SSA quality assurance unit, the depth of the data collected from quality assurance reviews also enables it to assess the implementation of new or revised policies, perform analyses and make recommendations for operational and systems corrective actions, and provide broad levels of management information, such as information by categories of impairments.
VBA agrees that the STAR system deployed at the beginning of fiscal year 1999 provides a sound start for beginning to address claims-processing accuracy issues. VBA officials acknowledge, however, that they realized when STAR was deployed that continuous improvement should be sought to enhance its effectiveness. These VBA officials stated that VBA is open to considering the collection of additional data in order to enhance STAR.
section_title: VBA Can Further Address Vulnerabilities in the Integrity of Performance Data
paragraphs: To ensure integrity in the operation of government programs, standards for internal controls call for separation of key duties, and standards for performance audits call for those who review and evaluate a program’s performance to be organizationally independent of the program’s managers. Under STAR, however, the RO staff who review the accuracy of RO decisions are themselves responsible for making such decisions, and they report to RO managers responsible for claims processing. Such an arrangement does not meet the standard for separation of duties, nor does it meet the independence standard. Both the RO reviewers and their managers have an inherent self-interest in having as high an accuracy rate as possible. This self-interest derives from the fact that accuracy is one of five factors that determine RO performance scores, which VBA measures to meet Results Act requirements. Thus, without adequate separation of duties or adequate independence for RO reviewers, the integrity of both the STAR review process and the resulting accuracy rates and performance data reported under the Results Act are called into question.
The potential effect of impaired objectivity on performance data is exemplified by findings reported by VA’s Inspector General in September 1998. Because of concern about the accuracy of data used to meet Results Act requirements, the Inspector General examined the integrity of certain data used for Results Act reports. In this review, the Inspector General found instances in which RO staff had manipulated data on the timeliness of RO claims processing in order to make performance appear to be better than it actually was. The Inspector General found that weaknesses in internal controls had contributed to the lack of integrity in the timeliness data reported under the Results Act. During our review, some RO staff made comments on the integrity of accuracy reviews that parallel the findings of the Inspector General. These RO staff told us that ROs are biased against identifying their own errors. They also stated that ROs in the past, after selecting samples of cases to review, had sometimes “sanitized” or fixed problems in the case files before the cases underwent quality review.
No data are available to indicate the extent to which RO reviewers might attempt to overlook errors and sanitize case files to conceal errors in the approximately 44,000 cases that ROs review annually under STAR. However, to the extent that such efforts may occur, the accuracy rates reported by the ROs would be overstated. Furthermore, any attempts by ROs to conceal errors and overstate their accuracy rates could also result in an overstatement of the accuracy rates that VBA reports for SDNs and the nation. This vulnerability in VBA’s data exists because the sample of 7,400 cases that VBA reviews annually is selected directly from the approximately 44,000 cases reviewed by the ROs. VBA reviews its sample of 7,400 cases after the ROs complete their own reviews of these same cases. VBA believes that it can detect most attempts to sanitize case files because such attempts would likely require extensive backdating of corrected case file documents, which VBA believes would be difficult to conceal. VBA acknowledges, however, that it cannot ensure that it would detect every such attempt in the cases it reviews. To the extent that VBA may not detect all such attempts, the accuracy rates it reports for SDNs and the nation would be overstated.
Ensuring the integrity of accuracy data will require that staff who review claims-processing accuracy neither are responsible for claims processing nor report to program managers responsible for claims processing. VBA stated that resource restrictions prevent establishing independent accuracy review units either in the ROs or at VA headquarters; however, unless VBA provides adequate separation of duties and organizational independence for accuracy reviewers, potential questions about the integrity of accuracy-related performance data will likely persist. By contrast, we found that SSA has quality assurance units at its headquarters and in each of its 10 regional offices that are organizationally independent of program management. The independent quality assurance unit in SSA headquarters has overall responsibility for assessing disability claims-processing accuracy. To do this, it oversees the operation of the independent regional quality assurance units that review the accuracy of statistically random samples of the disability decisions rendered by 54 state agencies that process disability claims for SSA.
VBA contends that it would be impractical to establish independent accuracy review units in VBA’s 58 ROs, many of which are relatively small in size. Establishing independent STAR units in ROs would be more practical if only a relatively small number of large ROs processed all compensation and pension claims. Under the present structure, however, a more workable long-term solution could involve establishing an independent headquarters unit responsible for conducting all reviews used to determine the accuracy rates that go into the calculation of overall performance scores for VBA headquarters, SDNs, and ROs.
section_title: VBA Faces Challenges in Meeting Its Goal for Improving Claims-Processing Accuracy
paragraphs: VBA has set a goal of achieving a claims-processing accuracy rate of 93 percent by fiscal year 2004. This would be almost 30 percentage points higher than the baseline rate of 64 percent established in the 1998 pilot test of STAR. VBA acknowledges, however, that the STAR system on its own cannot ensure that VBA will meet its accuracy goal. Beyond any improvements that VBA might make in the STAR system, VBA acknowledges that there are challenges it must address successfully in order to meet its goal for improving accuracy. These challenges include effectively establishing accountability for accuracy improvement and developing an effective training program for the current and future workforce.
subsections: section_title: Establishing Stricter Accountability
paragraphs: In May 1998, VBA identified several root causes of quality problems in processing disability compensation and pension claims. One such cause was a lack of employee accountability. VBA plans to focus on quality and accountability with a quality assurance system that provides clear and fair accountability at all organizational levels. To accomplish this goal, VBA is implementing the “balanced scorecard” that scores the performance of VBA headquarters, SDNs, and ROs on the basis of five performance factors: claims-processing accuracy (as determined by STAR), timeliness of claims processing, unit cost, customer satisfaction, and employee satisfaction and development.
With the goal of achieving a 93-percent accuracy rate by fiscal year 2004, VBA believes its balanced scorecard approach will, among other things, drive organizational change, provide employees with feedback on measures they can influence, and link the performance appraisal and reward systems to organizational performance measures. VBA plans to use the balanced scorecard to give RO managers incentives to work as teams in their SDNs with a focus on meeting balanced scorecard performance measures, including accuracy. The extent to which this strategy will improve accountability and accuracy cannot yet be determined.
section_title: Developing More Effective Training
paragraphs: In our discussions with RO staff, many stated that VBA had not provided adequate training for claims adjudicators. They stated, for example, that there was confusion in the ROs on how to process cases because of apparent conflicts between decisions of the Court of Veterans Appeals and VA’s regulations and guidance. They also stated that too much of their training was determined locally, resulting in inconsistent training among the ROs. VBA acknowledged shortcomings in training and stated that it had not fared well in preparing its workforce, with a resultant decline in technical accuracy. VBA acknowledged the need for an effective, centralized, and comprehensive training program that provides the background necessary for its decisionmakers to render decisions according to the statutes and regulations mandated for claims adjudication.
Such training is important not only for current employees but also for the many new employees whom VBA will have to hire to replace retiring employees. According to VBA, it may lose up to 30 percent of its workforce to retirement by fiscal year 2003. To develop a training program for RO staff, VBA plans to identify the necessary employee skills and work processes for every decisionmaking position, implement skill certification or credentialing for these positions, and implement performance-based training connected to measurable outcomes. VBA has already developed a computer-based training module for processing appeals and is working on modules for original disability claims, service-connected death indemnity benefits, and pensions. VBA also plans to produce additional modules, including one for training RO staff when they first assume disability rating responsibilities. Whether these training efforts will enable VBA to meet its accuracy goal cannot yet be determined.
section_title: Conclusions and Recommendations
paragraphs: Although VBA had been reporting until recently that ROs were processing claims accurately more than 95 percent of the time, the STAR pilot test in fiscal year 1998 revealed that the accuracy rate for decisions involving disability ratings was much lower, about 64 percent. This confirmed that VBA needs to give more attention to ensuring that ROs make the correct decision the first time. Making the correct decision the first time would mean that veterans could avoid having to make unnecessary appeals and would not be unnecessarily delayed in receiving benefits owed them.
Although the new STAR system represents genuine improvement in VBA’s ability to measure accuracy and identify error-prone cases, VBA needs to make further progress in collecting data for identifying difficult cases, assessing adjudication difficulties, and developing corrective actions. Despite its newly implemented STAR system, without further refinements in the data collected on errors, significant inaccuracies are likely to persist because VBA is constrained in its ability to pinpoint error-prone cases and identify corrective actions. Moreover, the data produced from STAR reviews will be suspect because of weaknesses in internal controls and lack of adherence to performance audit standards. We believe this can potentially undermine progress made under STAR.
To further strengthen VBA’s ability to identify error-prone cases, ensure the integrity of accuracy rate-related performance data reported under the Results Act, and keep the Congress informed about VBA’s progress in addressing challenges that must be met in order to improve accuracy, we recommend that the Secretary of the Department of Veterans Affairs direct the Under Secretary for Benefits to take the following actions.
For RO disability decisions found to be in error, revise STAR to collect more detailed medical characteristics data, such as the human body systems, the specific impairments, and the specific deficiencies in medical evidence involved in these disability claims, so that VA can identify and focus corrective actions on specific problems that RO adjudicators have in correctly evaluating certain types of medical conditions or in correctly determining whether medical evidence is adequate to make a decision. Implement a claims-processing accuracy review function that meets the government’s internal control standard on separation of duties and the program performance audit standard on organizational independence. In the annual Results Act reports, inform the Congress on VBA’s progress in (1) establishing stricter employee accountability for the achievement of performance goals and (2) developing more effective training for claims adjudicators.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on our draft report, VA stated that it found the report to be a fair and balanced appraisal. VA concurred that its process for assessing claims accuracy is critical and stated that continued urgent action is required for VA to meet its own and its stakeholders’ expectations. VA stated that our recommendations were generally constructive but had concern about our first two recommendations.
The first recommendation in our draft report was that VA “revise STAR to include the collection of more detailed medical characteristics data on the human body systems, and specific impairments involved in disability claims as well as data on specific deficiencies in medical evidence and examinations.” VA interpreted our recommendation to mean that STAR should collect data on the quality of examinations conducted by VHA. However, this was not the intent of our recommendation. The intent was for STAR to collect additional data that would help VA better identify (1) specific types of medical conditions that RO adjudicators have difficulty evaluating correctly and (2) specific types of inadequacies in medical evidence that are most prevalent in incorrect decisions. This would provide a means for VA to develop corrective actions addressing the causes of errors in the evaluation of medical conditions and of failure to collect adequate medical evidence to make a supportable decision. We clarified the recommendation and our discussion of this issue in our report.
The second recommendation in our draft report was that VA “implement a claim processing accuracy review function that meets the government’s internal control standard on separation of duties and the program performance audit standard on organizational independence.” VA’s primary concern about this recommendation was that current budget constraints make it impractical to adopt approaches that would fully satisfy these standards—for example, establishing a single, large centralized review unit to assess all quality issues, including individual RO quality. However, while current budget constraints may present problems in finding ways to fully meet the standards immediately, we believe meeting these standards as expeditiously as possible should be a continuing priority in VA’s future planning process. Until the standards are met, the integrity of VA’s claims-processing accuracy data will remain questionable. As VA stated in its comments, “Effective reviews require an organizational commitment to dedicate the necessary resources to the review process.”
With regard to the second recommendation, VA also stated that while the STAR system is a compromise reflecting resource constraints, it has some distinct advantages compared with quality reviews performed by a consolidated, independent review unit. VA cited the value of having reviews performed by local staff in each RO. Our recommendation would not preclude local reviews, which we agree are important. Even if a single, central unit were established for the purpose of assessing the degree to which each RO processes claims accurately, it would still be critical for local RO management to gather detailed local data on claims processing to understand fully how to correct local processing problems. This function, however, is different from local reviewers conducting accuracy reviews of their own RO’s decisions, which our recommendation is intended to eliminate.
VA also stated that it is concerned that a “permanent” independent review staff would become stagnant. We disagree because the staff who perform reviews would not have to be permanently assigned to the unit but could instead be rotated to avert staff stagnation. VA furthermore expressed concern about the cost and increased potential for losing active case files that would result from mailing many more thousands of case files from the 58 ROs to a central review site. This concern, however, does not negate the need to meet the standards for separation of duties and organizational independence. Also, the concern could potentially be lessened by other measures. For example, the Congressional Commission on Servicemembers and Veterans Transition Assistance in its January 1999 report applauded VBA for consolidating the administration of its education and loan programs into fewer than 10 ROs but pointed out that VBA has made no effort to make a similar consolidation of the adjudication of compensation claims. If VBA were ever to consolidate the adjudication of claims into a few relatively large ROs, it would be more practical to locate an independent STAR unit in each of these ROs to review the accuracy of cases each one processed. Each RO STAR unit would then need to mail to a central review unit only a relatively small random sample of the cases it reviewed so that the central unit could ensure the reviews’ appropriateness and consistency.
VA’s comments are printed in appendix III.
As agreed with your office, we plan no further distribution of this report until 7 days from its date of issue, unless you publicly announce its contents earlier. We will then send copies to the Chairman of the House Committee on Veterans’ Affairs, the Secretary of the Department of Veterans Affairs, other congressional committees, and others who are interested. We will also make copies available to others upon request. If you have any questions about this report, please call me at (202) 512-7101 or Irene P. Chu, Assistant Director, at (202) 512-7102. Other major contributors to this report were Ira B. Spears, Mark Trapani, Connie D. Wilson, Paul C. Wright, and Deborah L. Edwards.
section_title: Historical Data on VBA’s Claims-Processing Accuracy Rates and on Award and Remand Rates of the Board of Veterans’ Appeals
paragraphs: Before the Veterans Benefits Administration (VBA) implemented the Systematic Technical Accuracy Review (STAR) measurement system, it reported that regional offices (RO) accurately processed and adjudicated disability compensation and disability pension claims more than 95 percent of the time during fiscal years 1993-97 (see table I.1).
The validity of such high accuracy rates, however, seemed inconsistent with the results of decisions made by the Board of Veterans’ Appeals when veterans appealed unfavorable RO decisions. The Board of Veterans’ Appeals awarded benefits or remanded cases for further development more than 60 percent of the time when veterans appealed RO decisions during fiscal years 1993-97 (see table I.2).
Only a small proportion of RO decisions are appealed to the Board. For example, in fiscal year 1997, veterans filed notices of disagreement in about 14 percent of the disability compensation claims processed by ROs (see table I.3). The number of cases appealed, however, is less than the number of cases in which veterans file a notice of disagreement with VA. In some cases, after notices of disagreement are filed, ROs award the benefits sought, or some veterans decide not to continue with their appeals if the RO again denies benefits at this point. In fiscal year 1997, the Board received initial substantive appeals equivalent to about 5 percent of all disability compensation claims processed by ROs.
Disability compensation claims processed by ROs (original and reopened)
section_title: Sample Sizes and Accuracy Rate Computation Methods for Compensation and Pension Cases
subsections: section_title: Sample Sizes
paragraphs: Under the pre-STAR accuracy measurement system, VBA annually reviewed approximately 5,700 compensation and pension cases, or approximately 100 cases randomly selected from the cases completed by each of 57 ROs.These cases were selected from the entire universe of compensation and pension work products completed by the ROs. Using this procedure, VBA annually produced, with a reasonable level of statistical precision, a national accuracy rate for the entire body of compensation and pension work done by the ROs during the prior year. The sample of approximately 100 cases selected for each RO was too small to produce accuracy rates for each RO with a reasonable level of statistical precision. However, VBA required each RO to self-review a sample of 300 to 900 cases annually, depending on the size of the RO. These RO self-reviews were intended to provide the RO with information needed to improve quality, not to compute accuracy rates for VBA to measure performance.
Under STAR, VBA annually reviews 7,371 compensation and pension cases for the nine service delivery networks (SDN), and the 57 ROs self-review about 44,000 cases. These cases are made up of three separate samples: (1) rating-related work products; (2) authorization work products that require significant development, review, and administrative decision or award action but may not involve any rating-related action; and (3) principal guardianship files, referred to as fiduciary cases. (See table II.1 for SDN and RO sample sizes.) For rating-related work products and authorization work products that typically do not require rating-related action, the sampling methodology will allow VBA to produce accuracy rates with a reasonable level of statistical precision for the nation and each SDN. However, the sample of fiduciary cases is too small to provide accuracy rates with the same level of statistical precision. Similarly, for cases that are self-reviewed by ROs, the sampling methodology will allow each RO to produce accuracy rates with a reasonable level of statistical precision for rating-related work products and authorization work products typically not requiring ratings. Again, however, the sample of fiduciary cases is too small to provide accuracy rates with the same level of statistical precision.
The totals in this row represent total sample size for each SDN and each RO.
The totals in this row represent national total sample size for 9 SDNs and 57 ROs.
section_title: Accuracy Rate Computation Methods
paragraphs: For each case reviewed under the previous accuracy measurement system, VBA categorized each error into one of three areas of the claims adjudication process: (1) control and development of the claim, (2) decision elements, and (3) notification to the veteran. Thus, for example, if a case had only one error, VBA would record this error under the appropriate area of the claims adjudication process and would show the two other areas as error-free for that case. After all cases were reviewed, VBA would compute an accuracy rate for each of the three areas in the claims adjudication process. To arrive at an overall accuracy rate for the three areas combined, VBA computed their arithmetic mean (or average). For example, table II.2 shows a hypothetical outcome for accuracy reviews of 10 cases. Under the control and development area, one case has an error (a 90-percent accuracy rate); under the decision element area, two cases have errors (an 80-percent accuracy rate); and under the notification area, one case has an error (a 90-percent accuracy rate). For this sample of 10 cases as a whole, the overall accuracy rate is the average of these three accuracy rates, or 86.6 percent.
For each case reviewed under STAR, however, VBA does not compute separate accuracy rates for the three areas of the claims adjudication process. If a case has any errors in any area of the claims adjudication process, the entire case is counted as incorrect for accuracy rate computation purposes. This approach tends to result in a lower accuracy rate than under the previous system. For example, in the hypothetical sample of 10 cases shown in table II.2, 3 cases would be counted as incorrect under STAR because they contain at least one processing error, and the resultant accuracy rate for the sample would be only 70 percent (7 out of 10 cases with no errors = 70-percent accuracy rate), compared with the overall accuracy rate of 86.6 percent calculated under the previous system.
section_title: Comments From the Department of Veterans Affairs
paragraphs: The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.