Background

Through its disability compensation program, VBA pays monthly benefits  to veterans with service-connected disabilities (injuries or diseases  incurred or aggravated while on active military duty) according to the  severity of the disability. VBA’s pension benefit program pays monthly  benefits to wartime veterans who have low incomes and are permanently  and totally disabled for reasons not service-connected. In addition, VBA  pays dependency and indemnity compensation to some deceased veterans’  spouses, children, and parents and to survivors of service members who  died on active duty. In fiscal year 2002, VBA paid over $22 billion in  disability compensation to an average of about 2.4 million veterans and  over 300,000 survivors. VBA also paid over $3 billion in pensions to an  average of about 580,000 veterans and survivors.
The amount of disability compensation largely depends on the degree to  which a veteran is disabled. VBA determines the degree to which veterans  are disabled in 10 percent increments on a scale of 0 to 100 percent. Basic  monthly payments range from $104 for 10 percent disability to $2,193 for  100 percent disability. About 65 percent of veterans receiving disability  compensation have disabilities rated at 30 percent and lower; about  8 percent have disabilities rated at 100 percent. The most common  impairments for veterans who began receiving compensation in fiscal year  2001 were tinnitus (ex., a ringing in the ear), auditory acuity impairment  rated at 0 percent (i.e., mild hearing difficulties), skeletal conditions,  arthritis due to trauma, post-traumatic stress disorder, and scars.  Eligibility and priority for other Department of Veterans Affairs (VA)  benefits and services such as health care and vocational rehabilitation are  affected by these VA disability ratings.
When a veteran submits a claim to any of VBA’s 57 regional offices, a  veterans service representative (VSR) is responsible for obtaining the  relevant evidence to evaluate the claim. Such evidence includes veterans’  military service records, medical examinations and treatment records  from VA medical facilities, and treatment records from private medical  service providers. Once a claim is developed (i.e., has all the necessary  evidence), a rating VSR, also called a rating specialist, evaluates the claim,  determines whether the claimant is eligible for benefits, and assigns a  rating based on degree of disability. Veterans with multiple disabilities  receive a single composite rating. For veterans claiming pension eligibility,  the regional office determines if the veteran served in a period of war, is  permanently and totally disabled for reasons not service-connected, and  meets the income thresholds for eligibility. A veteran who disagrees with  the regional office’s decision for either program can appeal sequentially to  VA’s Board of Veterans’ Appeals (BVA), the U.S. Court of Appeals for  Veterans Claims, and the U.S. Court of Appeals for the Federal Circuit.
In fiscal year 1999, VBA implemented a quality review system to improve  the measurement of the accuracy of its claims decisions. Under STAR,  VBA selects a random sample of completed claims decisions each month  from each of its 57 regional offices. STAR quality review staff review these  claims using a standard checklist. If a claim has any error, VBA counts the  entire claim as incorrect for accuracy rate computation purposes. STAR  then returns the case file and the results of the review to the regional  office that made the decision. If an error was found, the regional office is  required to either correct it or request reconsideration of the error by  STAR.
VBA made several significant changes to its STAR system in fiscal year  2002. First, in response to our March 1999 recommendation, VBA made the  STAR review staff organizationally independent of the regional offices to  ensure the independence of the reviews. Second, VBA more than doubled  the number of rating decisions it reviewed (from 3,209 in fiscal year 2001  to 6,646 in fiscal year 2002) in order to obtain more precise regional office  level accuracy scores. Third, VBA improved its key measure to focus more  on whether decisions to grant or deny benefits were correct. Previously,  VBA’s key measure also included decision documentation and notification  issues. Figure 1 shows the areas covered by the checklist used to measure  accuracy in fiscal year 2001. The revised key accuracy measure includes  only the first four areas of the checklist.

	Accuracy Declined in Fiscal Year 2002

From fiscal years 2001 to 2002, VBA’s accuracy of decision making in the  compensation and pension programs declined from 89 percent to 81  percent. We determined these accuracy rates by comparing substantially  similar questions for the 2 fiscal years that focused on whether the  decision to grant or deny benefits was correct. However, the agency  reported that it had improved its accuracy for that period. This  discrepancy occurred because VBA did not report comparable accuracy  data for the 2 fiscal years. VBA reported its fiscal year 2002 accuracy based  on a revised measure that excluded two areas of the checklist—reasons  and bases and notification. VBA then compared this to its fiscal year 2001  accuracy, which included these two areas of the checklist. VBA officials  attributed the decline in accuracy during this period to several factors. We  were not able to quantify the relative contribution of these factors. These  factors included headquarters emphasis on production, the specific  processing requirements of the Veterans Claims Assistance Act of 2000,  and the relative inexperience of VBA’s claims processing staff.
In fiscal year 2002, VBA made progress in increasing production of claims  decisions and reducing inventory. The number of claims decisions rose  substantially from about 481,000 in fiscal year 2001 to about 797,000 in  fiscal year 2002. During the same time period, the number of claims VBA  received increased from about 674,000 to 722,000. The higher level of  production permitted VBA to reduce its inventory of pending claims by  18 percent from about 421,000 to about 346,000.
Although accuracy actually declined, VBA reported in its performance  report to the Congress for fiscal year 2002 that the accuracy of its rating  decisions for compensation and pension benefits had improved slightly.  VBA reported that accuracy improved from 78 percent in fiscal 2001 to  80 percent in fiscal 2002. However, these rates were not comparable  because the new “benefit entitlement accuracy” rate for fiscal year 2002  was based on fewer areas of the checklist that focused on whether and to  what extent the claimant received a benefit. In its fiscal year 2002  performance report, VA made note of this change but did not recalculate  the fiscal 2001 rate to provide a correct comparison. When we recomputed  the agency’s accuracy rate for fiscal 2001 using its revised criteria, the  result was 89 percent, not 78 percent as reported. As a result, the correct  comparison showed a decline in accuracy from 89 percent to 81 percent.  The agency corroborated our finding after performing its own  computation at our request.
VBA officials we spoke with cited a number of factors that affected  accuracy over the fiscal years 2001 to 2002 period. A key factor cited was  VBA’s emphasis on production, such that accuracy became a lesser  priority. In 2001, the Secretary of Veterans Affairs set an ambitious goal to  reduce the agency’s average time to complete a rating decision from 173  days (in fiscal year 2000) to 100 days by the end of fiscal year 2003. Also,  the Secretary tasked VBA with reducing the inventory of rating claims to  250,000 by the end of September 2003. VBA managers responded by  emphasizing the completion of more rating cases each month. The result  has been a significant increase in production and a significant reduction in  the rating inventory since the end of fiscal year 2001.
In fiscal year 2002, VBA incorporated its new production targets into its  measures for regional office performance. Regional office directors  became accountable for specific targets for production, inventory  reduction, and timeliness improvement. The agency also established  regional office performance award criteria that gave more weight to  efficiency than accuracy. Three of the four criteria for cash awards were  based on production and timeliness and one on accuracy; bonuses could  be received without meeting the accuracy criteria.
VBA officials also attributed accuracy problems to the addition of more  specific processing requirements under the Veterans Claims Assistance  Act (VCAA) of 2000. The act broadened VBA’s responsibility to assist  veterans — such as notifying them of needed evidence and helping them  develop that evidence for their claims. VBA incorporated VCAA  requirements into its STAR rating accuracy checklist in fiscal year 2002. In  the first 4 months of fiscal year 2002, VCAA errors accounted for almost  half of all errors identified by STAR rating reviewers. In April 2002, VBA  required each regional office to re-train its claims processing staff on the  new requirements. Over the last 8 months of fiscal year 2002, the  proportion of VCAA-related errors declined to about one-third of all errors.
VBA officials also cited the relative inexperience of VBA’s claims  processing staff. Regional office officials noted that they had many VSRs  and rating specialists with insufficient training and experience to be fully  proficient at developing and making decisions on claims. For example,  officials at two offices we visited noted that their accuracy was affected by  large numbers of inexperienced staff, due to a large increase in staff size  or a relatively high rate of turnover. VBA has found that attrition rates for  new claims processing staff hired over a 3-year period ranged from  0 percent to 49 percent at some regional offices.

	VBA Could Have Done More to Hold Regional Offices Accountable for Accuracy

While VBA has established accuracy standards for its regional offices, it  has not made the best use of its accuracy data on regional offices. VBA  collected regional office accuracy data in fiscal year 2002 but has not fully  used the information to evaluate regional office performance, correct  errors, and identify needed training that could reduce errors. Unless VBA  better uses these data to hold regional offices accountable for accuracy, it  cannot ensure that as decisions become timelier, they also become more  accurate. VBA is developing an agencywide system for evaluating  individual claims processing employees’ performance against VBA’s  individual accuracy standards.
In fiscal year 2002, VBA improved its STAR review system to provide  independent review of decisions and more precise accuracy data at the  regional office level. Regional accuracy scores are estimates based on  samples of cases and revealed high and low performers relative to the  agency’s fiscal year 2002 accuracy goal of 85 percent. Eleven regional  offices clearly failed to meet the goal while 3 clearly met or exceeded the  goal. However, VBA managers did not fully use the regional office- specific information generated by the STAR unit as a basis for improving  accuracy at the regional office level. For example, VBA did not require  regional offices that failed to meet the national accuracy goal of 85 percent  to prepare strategies for improvement. In contrast, in fiscal year 2002,  13 regional offices that did not meet established targets for production,  inventory, and timeliness were required to develop corrective action plans.  Also, as noted earlier, VBA did not make rating accuracy a prerequisite for  receiving a performance award. Three of the four criteria for cash awards  were based on production and timeliness and one on accuracy; a bonus  could be received without meeting the accuracy criteria. In fiscal year  2002, 2 offices received performance awards despite clearly not meeting  VBA’s accuracy goal. These offices had accuracy rates of 71 percent and  75 percent, respectively.
VBA’s regional offices did not always properly address STAR errors that  were returned to them, a violation of VBA policy that regional offices  either correct errors or formally request that the STAR unit reconsider its  error call. VBA survey teams, which review judgmental samples of STAR  cases with errors, found that the 18 regional offices they visited from  October 2001 through December 2002 had failed to respond properly to  over 40 percent of the errors reviewed. At 14 of the 18 offices, the survey  teams found problems significant enough to recommend improvements in  STAR error handling. For example, teams recommended that several  offices establish management controls to ensure that they properly  address all errors identified by the STAR unit. Effective April 1, 2003, VBA  established a new tracking system that required regional offices to report  to headquarters on actions taken to address all errors identified by the  STAR unit.
In addition to not correcting all errors, several of the regional offices  visited by the survey teams were not making adequate use of STAR errors  to identify trends that could be used to provide feedback and training to  claims processing staff. Officials at regional offices we visited commented  that this information is useful for identifying trends for feedback and  training purposes. For example, in fiscal year 2002, the results of STAR  reviews helped to alert management of the Boston Regional Office that  many of its claims processors were having difficulties with the new VCAA  legislation, permitting management to provide special training for the  Boston staff. The agency survey teams recommended that 7 of the regional  offices reviewed between October 2001 and December 2002 take steps to  use STAR reviews for feedback and training.
In addition to regional office-level accuracy standards, VBA has also  established quantifiable accuracy standards for its experienced claims  processing employees, as part of these employees’ annual performance  evaluations.The accuracy standards for experienced rating specialists  require that they correctly decide 85 percent of 60 randomly selected  rating decisions per year (5 decisions per month). VBA is developing an  agencywide review process, the Systematic Individual Performance  Assessment (SIPA), to assess compliance with the individual accuracy  standards. VBA developed a uniform checklist, similar to the STAR  checklist for reviewing rating decisions, which offices could use to assess  rating specialists’ accuracy. Unlike the STAR system, reviews are  conducted before decisions are finalized.
In April 2003, VBA surveyed regional offices to determine the status of  their efforts to measure individual accuracy and assess the staff resources  needed to implement SIPA VBA-wide. Individual reviews were being  conducted by supervisors, such as team coaches, and nonsupervisory  staff, such as Decision Review Officers, who are responsible for handling  appeals. At the time of our review, VBA was in the process of deciding  who would perform the reviews. In its efforts to develop SIPA as a  consistent review process, VBA is considering issues such as ensuring that  consistent feedback is provided to the employees and all regional offices  are using the same review checklists.

	Conclusions

While VBA’s emphasis on production succeeded in increasing the number  of claims decisions made during fiscal year 2002, the accuracy of those  decisions declined. Because VBA did not report comparable accuracy  rates for fiscal years 2001 and 2002 in its performance and accountability  report, Members of Congress, Department of Veterans Affairs  management, and claimants did not know of this decline in accuracy. The  lack of accurate performance data hampers stakeholders’ ability to  monitor VBA’s performance in serving the nation’s veterans.
In addition, VBA could better use the accuracy data it has to hold its  regional offices more accountable for accuracy. While VBA has made  progress in measuring regional office accuracy, it has not made full use of  the results to reward high performing offices or to improve poorly  performing offices. VBA has made significant progress in getting its  compensation and pension claims workload under control to improve  timeliness for veterans waiting for decisions on their claims. However,  VBA’s accuracy has declined. Unless VBA balances its emphasis on  producing more and faster decisions with an increased emphasis on  accuracy, it will be difficult to ensure that veterans are getting the benefits  to which they are entitled in a timely manner.

	Recommendations

We recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Benefits to  report the accuracy of VBA disability compensation and pension claims  decisions to the Congress and other stakeholders in a manner that allows  for valid comparisons of accuracy across fiscal years and  better hold regional offices accountable for the accuracy of their claims  decisions, by increasing the use of its regional office accuracy data, while  at the same time maintaining an appropriate emphasis on production.

	Agency Comments and Our Response

In its written comments on a draft of this report (see app. I), VBA  concurred with our recommendations. However, VBA suggested that more  effective use of STAR had been made to improve performance than our  report concludes. We did note that VBA used STAR data to improve  accuracy.  Specifically, we noted in our report that VCAA-related errors  declined to about one-third of all errors after VBA identified the errors  through STAR and required retraining. We support VBA’s continued efforts  to emphasize accuracy and timeliness as critical components of quality.
We will send copies of this report to the Secretary of Veterans Affairs,  appropriate congressional committees, and other interested parties. We  will also make copies of this report available to others on request. The  report will also be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staffs have any questions regarding this report, please call  me at (202) 512-7215 or Irene Chu, Assistant Director, at (202) 512-7102. In  addition to those named, Susan Bernstein, Kristine Braaten, Kevin  Jackson, Joseph Natalicchio, Martin Scire, and Greg Whitney made key  contributions to this report.

Summary:
Why GAO Did This Study

The Veterans Benefits Administration (VBA) has a large inventory of claims for benefits under its compensation and pension programs. The Secretary of the Department of Veterans Affairs has pledged to substantially reduce this inventory in order to improve timeliness. In response, VBA emphasized producing more claims decisions per year. GAO was asked to ascertain how accuracy has changed since VBA increased its emphasis on production and to report on the agency's efforts to ensure the accuracy of its decisions.

What GAO Found

From fiscal years 2001 to 2002, VBA's accuracy of decision-making in the disability compensation and pension benefit programs declined from 89 percent to 81 percent. The agency had reported a slight improvement in accuracy between fiscal years 2001 and 2002--from 78 percent to 80 percent. However, we found that these two annual figures were not comparable because the agency had substantially changed the way it measured accuracy for fiscal year 2002. Although VBA acknowledged a change in its accuracy measure in its annual report to the Congress, the agency did not revise its 2001 figure to allow for an appropriate comparison with 2002. VBA officials GAO spoke with suggested several factors that may have contributed to the decline in accuracy. We were not able to quantify the relative contribution of these factors. These factors included VBA's emphasis on increasing claims decisions, the specific processing requirements of the Veterans Claims Assistance Act (VCAA) of 2000, and the relative inexperience of VBA's claims processing staff. To help ensure accountability for accuracy, VBA set accuracy standards for its regional offices. Although VBA has regional office-level accuracy data, it has not made full use of this information to encourage better performance from regional offices with low accuracy scores. For example, in fiscal year 2002, VBA did not require offices with poor accuracy to prepare improvement plans and gave performance awards to two offices that clearly failed to meet VBA's accuracy goal.