VA DISABILITY CLAIMS

PROCESSING

Preliminary Observations 

on Accuracy Rates and 

Quality Assurance 

Activities 

Statement of Daniel Bertoni

Director, Education, Workforce and Income

Security Issues

Testimony 

Before the Committee on Veterans' 

Affairs, House of Representatives

For Release on Delivery 

Expected at 7:30 p.m ET

Monday, July 14, 2014

GAO-14-731T

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-14-731T, a testimony 

before the Committee on Veterans’ Affairs, 

House of Representatives.

July 14, 2014

VA DISABILITY CLAIMS PROCESSING

Preliminary Observations on Accuracy Rates and 

Quality Assurance Activities

Why GAO Did This Study

With a growing workload of disability 

claims due, in part, to recent wars, 

VBA faces difficulties in improving the 

accuracy and consistency of its claims 

decisions by VBA staff in 57 regional 

offices. To help achieve its goal of 98 

percent accuracy by fiscal year 2015,

VBA recently implemented a new way 

of measuring accuracy, and changed

several quality assurance activities to

assess the accuracy and consistency 

of claims decisions and to provide

feedback and training to claims 

processors. 

In this statement, GAO discusses initial 

observations from its ongoing review of 

VBA’s quality assurance efforts, 

addressing the extent to which (1) VBA 

effectively measures and reports the

accuracy of its compensation claim 

decisions; and (2) VBA’s other quality 

assurance activities are 

complementary and coordinated. For 

this work, GAO analyzed STAR 

accuracy and VBA claims data from

fiscal year 2013 (the most recent fiscal 

year for which complete data are 

available); reviewed relevant federal 

laws, regulations, guidance and other 

documents relevant to quality 

assurance activities; and interviewed 

VBA staff from headquarters and four

VBA regional offices (selected to 

achieve variety in geography, size of 

workload and reported accuracy rates),

and Veterans Service Organization 

officials. GAO has no 

recommendations at this time. GAO 

plans to issue its final report later in 

2014, along with any related 

recommendations.

What GAO Found

The Veterans Benefit Administration (VBA) within the Department of Veterans 

Affairs (VA) now measures and reports the accuracy of its claim decisions in two 

ways—by claim and by medical issue—but its approach has limitations. When

calculating accuracy rates for either measure through its Systematic Technical 

Accuracy Review (STAR) program, VBA does not always follow generally 

accepted statistical practices. For example, VBA does not adjust its accuracy 

estimates to reflect that it samples the same number of claims for review from

each regional office despite their varying workloads, and thus, produces 

imprecise estimates of national and regional accuracy. Further, because VBA 

does not clearly explain differences in how its two accuracy measures are 

calculated or their associated limitations, reported information about accuracy 

performance lacks clarity and may be confusing. Finally, VBA reviews 39 percent

(over 5,000) more claims nationwide than necessary to achieve its desired 

precision in reported accuracy rates, thereby diverting limited resources from 

other important quality assurance activities, such as targeted reviews of errorprone cases. VBA could achieve its desired precision by reviewing fewer claims 

for offices with smaller workloads or higher performance levels. 

Overall, VBA has made enhancements to its other quality assurance activities, 

although GAO identified implementation shortcomings that may detract from their

effectiveness. To improve local quality, VBA recently created quality review 

teams (QRTs) comprised of certified staff in each regional office. QRTs review a 

sample of claims before they are finalized to identify specific error types and help 

prevent inaccurate decisions. QRTs also assess individual staff performance; 

however, claims processed during overtime were excluded from such reviews in 

three of the four offices GAO visited, which may undermine regional quality for a 

portion of claims. VBA officials told GAO they do not know the extent to which 

this occurs nationally. Recent VBA data shows that about 10 percent of claims 

were processed during overtime. Also, to help ensure claims processors make

consistent decisions when presented with the same evidence, VBA began using 

a questionnaire approach to test for consistency. Although these questionnaires 

allow VBA to reach more staff and require fewer resources to administer, VBA 

did not pre-test them to ensure the clarity of questions or validity of the expected 

results. For each enhancement, their effectiveness is unclear because VBA has 

done little to date to assess their impact on improving accuracy.

VBA has also taken steps to coordinate its quality assurance efforts, but has not 

maintained centralized guidance and other key supports that might help prevent 

future errors. VBA coordinates quality assurance efforts by disseminating

national accuracy and consistency results, trends and related guidance to 

regional offices, which use this information to train staff. Further, VBA uses the 

results of STAR reviews for other quality assurance activities, such as focusing

QRT in-process reviews on commonly made errors. However, regional office 

staff told GAO that there are multiple sources of guidance, that searching these 

sources is time-consuming and difficult, and that VBA’s policy manual and 

national training are not sufficiently updated to help claims processors avoid 

future errors. In addition, regional office data systems do not allow managers to 

readily track error trends, resulting in ad-hoc and inefficient work-arounds.

View GAO-14-731T. For more information, 

contact Daniel Bertoni at (202) 512-7215 or 

bertonid@gao.gov.
Page 1 GAO-14-731T 

Chairman Miller, Ranking Member Michaud, Members of the Committee,

I am pleased to discuss the Department of Veterans’ Affairs (VA) quality 

assurance activities for ensuring accurate and consistent decisions on 

veterans’ disability benefit claims. Through its disability compensation 

program, VA provides cash compensation to veterans for disabling 

conditions incurred or aggravated while in military service. In fiscal year 

2013, VA paid $53.6 billion in disability compensation to 3.6 million 

veterans. For this same time period, VA reported a 96-percent issuebased accuracy rate and a 90-percent claim-based accuracy rate, while 

setting a goal of 98 percent for both measures in fiscal year 2015. 

However, the Veterans Benefit Administration (VBA) within VA has faced 

difficulties in improving the accuracy and consistency of its claims 

decisions. Accurate initial claims decisions can help ensure that VA is 

paying disability benefits only to those entitled to such benefits, and in the 

correct amounts. Meanwhile, consistent decisions help ensure that 

veterans’ claims receive comparable treatment, regardless of which VBA 

staff member or regional office processes the claim. GAO and VA’s Office 

of Inspector General (OIG) have previously reported on shortcomings in 

VBA’s quality assurance activities.1

To inform our work, we reviewed STAR guidance, reports, and methods 

for sampling and estimating accuracy of claims decisions, and analyzed 

STAR and VBA claims data for claims processed in fiscal year 2013, the 

Concerns also have been raised 

about the lack of transparency related to recent changes in the calculation 

of VBA’s national accuracy rate for compensation claims—which is based 

on its national Systematic Technical Accuracy Review (STAR)—and 

whether recent changes reflect reliable measures of accuracy. My 

remarks today are based on ongoing work requested by this committee. 

Specifically, we examined: (1) the extent to which VBA effectively 

measures and reports the accuracy of compensation claim decisionmaking, and (2) the extent to which VBA’s other quality assurance 

activities are complementary and coordinated.

 1 Department of Veterans Affairs, Office of Inspector General, Systemic Issues Reported 

During Inspections at VA Regional Offices, 11-00510-167 (Washington, D.C.: May 18, 

2011)

GAO, Veterans’ Disability Benefits: VA Has Improved Its Programs for Measuring 

Accuracy and Consistency, but Challenges Remain, GAO-10-530T (Washington, D.C.: 

March 24, 2010)
Page 2 GAO-14-731T 

most recent year for which complete data are available.2 We also 

assessed VBA’s methods against accepted statistical practices. In 

addition, we assessed the reliability of STAR and VBA claims data used 

for all our analyses and determined that they were sufficiently reliable for 

the purposes of providing information on trends in claims decisions. We 

also reviewed relevant VBA reports and practices for reporting accuracy 

and compared these against requirements for agency performance 

reporting and related GAO work. To determine the extent to which STAR 

and other key quality assurance activities are complementary and 

coordinated, we reviewed relevant guidance and policy documents, 

interviewed cognizant VBA officials, and visited four regional offices to 

gain a range of perspectives on how quality assurance activities are 

implemented at the regional office level,3

Our work was conducted in accordance with generally accepted 

government auditing standards. Those standards require that we plan 

and perform the audit to obtain sufficient, appropriate evidence to provide 

a reasonable basis for our findings and conclusions based on our audit 

objectives. We believe that the evidence obtained provides a reasonable 

basis for our findings and conclusions.

as well as how information is 

shared across quality assurance activities. We reviewed VBA’s methods 

for designing and implementing its consistency reviews against generally 

accepted practices in survey and questionnaire development. We also 

compared VBA’s quality assurance practices against its internal guidance 

and Standards for Internal Control in the Federal Government (AIMD-00-

21.3.1). We intend to produce a report later this year that will provide our 

final results.

 2

We focused on claims VBA identifies as rating-related, which require decisions on 

claimants’ entitlement to disability compensation, and the amounts of monthly benefits. 

We did not review authorization decisions, where entitlement to compensation is not an 

issue; for example, cases where beneficiaries have died, and VA is required to terminate 

compensation. Our work does not include a review of quality assurance activities 

associated with VBA pension claims or appealed claims. 

3

We conducted site visits with the Newark, Oakland, Nashville and Waco VA regional 

offices. We selected these offices based on a range of criteria, including: (1) number of 

claims processed annually, (2) geography (at least one regional office in each of VA’s four 

areas), and (4) accuracy rates. At each office, we spoke with service center managers and 

quality assurance staff, as well as representatives of local veteran service organizations.
Page 3 GAO-14-731T 

VA pays monthly disability compensation to veterans with serviceconnected disabilities (i.e., injuries or diseases incurred or aggravated 

while on active military duty) according to the severity of the disability.4

VBA staff in 57 regional offices process disability compensation claims.5

Since fiscal year 1999, VBA has used the STAR program to review the 

decisional accuracy of disability compensation claims. Under the STAR 

program, VBA reviews a stratified random sample of completed claims, 

and certified reviewers use a checklist to assess specific aspects of each 

claim.

These claims processors include Veterans Service Representatives who 

gather evidence needed to determine entitlement, including VA and 

military medical records, and Rating Veterans Service Representatives 

who decide entitlement and the rating percentage. Each claim requires a 

determination on one or more medical conditions. In fiscal year 2013, 

VBA decided more than 1 million compensation claims.

6 Specifically, for each of the 57 regional offices, completed claims 

are randomly sampled each month and the data are used to produce 

estimates of the accuracy of all completed claims. VA reports national 

estimates of accuracy from its STAR program to Congress and the public 

through its annual performance and accountability report and annual 

budget submission. VBA also produces regional office accuracy 

estimates, which it uses to manage the program. These and its national 

accuracy rates are reported in a publicly-available performance database, 

the ASPIRE dashboard.7

 4

38 U.S.C. §1101 et seq. VA’s ratings are awarded in 10-percent increments, from 0 to 

100 percent. Generally, VA does not pay disability compensation for disabilities rated at 0 

percent. As of December 2013, basic monthly payments ranged from $130.94 for a 

veteran with 10-percent disability rating and no dependents, and to $3,134 for a veteran 

with a 100-percent disability rating, a spouse and one child. 

5

For quality assurance purposes, VBA counts one of its sub-offices as a separate regional 

office, in addition to its 56 regional offices. Thus, for reporting purposes, we refer to 57 

offices.

6

The STAR review has two major components. The benefit entitlement review assesses 

whether the correct steps were followed in addressing all issues in the claim, collecting 

appropriate evidence, and whether the resulting decision was correct, including effective 

dates and payment rates. Accuracy performance measures are calculated based on the 

results of the benefit entitlement review. The STAR review also assesses whether claims 

processors appropriately documented the decision and notified claimants. 

7 The ASPIRE dashboard is an online report of VBA’s performance by program. Data are 

updated monthly and available by regional office and nationally. See 

http://www.benefits.va.gov/REPORTS/aspire_dashboard.asp.

Background
Page 4 GAO-14-731T 

Prior to October 2012, VBA’s estimates of accuracy were claim-based; 

that is, claims with zero errors were considered accurate and, conversely, 

claims with one or more errors were considered inaccurate. Beginning in 

October 2012, VBA also began using STAR data to produce issue-based 

estimates of accuracy that measure the accuracy of decisions on the 

individual medical conditions within each claim. For example, a veteran 

could submit one claim seeking disability compensation for five disabling 

medical conditions. If VA made an incorrect decision on one of those 

conditions, the claim would be counted as 80 percent accurate under the 

issue-based measure. By comparison, under the claim-based measure, 

the claim would be counted as 0 percent accurate. In March 2014, VBA 

reported a national estimate of issue-based accuracy in its fiscal year 

2015 annual budget submission and plans to update this estimate in VA’s 

next performance and accountability report. VBA also produces issuebased estimates by regional office, and reports them in the ASPIRE 

dashboard. For fiscal year 2013, the regional office claim-based accuracy 

rates ranged from 78.4 percent to 96.8 percent, and the issue-based 

accuracy rates ranged from 87.0 percent to 98.7 percent.

Beyond STAR, VBA has programs for conducting regional office quality 

reviews and for measuring the consistency of decisions. In March 2012, 

VBA implemented a quality review team (QRT) at each regional office, 

consisting of staff dedicated to conducting quality reviews. A QRT 

conducts individual quality reviews of claims processing staff members’

work, for performance assessment purposes. The QRT also conducts inprocess reviews before claims are finalized to help prevent inaccurate 

decisions by identifying specific types of common errors. Such reviews 

also serve as learning experiences for staff members. Since fiscal year 

2008, VBA has also conducted studies to assess the consistency of 

disability claims decisions across regional offices. Initially, these studies 

have included inter-rater reliability studies that assess the extent to which 

a cross section of claims processors from all regional offices agree on an 

eligibility determination when reviewing the entire body of evidence from 

the same claim. In 2013, VBA moved beyond the inter-rater reliability 

studies and introduced consistency questionnaires as its primary means 

for assessing consistency. A questionnaire includes a brief scenario on a 

specific medical condition for which claims processors must correctly 

answer several multiple-choice questions.
Page 5 GAO-14-731T 

When calculating accuracy rates, VBA does not follow generally accepted 

statistical practices. For example, VBA does not calculate the margin of 

error associated with each estimate that it generates, which prevents a 

complete understanding of trends over time and comparisons among 

offices.8 In addition, VBA does not weight the results of its STAR reviews 

to reflect its approach to selecting claims by regional office, which can 

affect the accuracy of estimates.9 According to our analysis of VBA data, 

weighting would have resulted in a small change to VBA’s nationwide 

claim-based accuracy rate for fiscal year 2013: from 89.5 percent to 89.1 

percent. However, 29 of the 57 regional offices’ would have experienced 

a somewhat greater increase or decrease in their accuracy rates.10

According to VBA officials we interviewed, although STAR management 

used a statistician initially to help develop the way in which they measure 

accuracy, it currently does not use a statistician to, for example, weight 

STAR results and calculate margins of error for accuracy estimates. 

Further, VBA officials said they did not consult a statistician when 

developing the new issue-based accuracy measure, but rather relied on 

the same sampling methodology and approach for estimating accuracy as 

for the claim-based measure. We have previously reported that to be 

useful, performance information must meet users’ needs for 

completeness, accuracy, consistency, and validity among other factors.11

In addition, VBA’s accuracy reporting lacks methodological details that 

would help users understand the distinction between its two accuracy 

 8 STAR accuracy estimates are derived from sample data and have sampling error 

associated with them. The confidence interval is a range of values around the estimate 

which is likely to include the actual population value, and helps determine whether 

different estimates are significantly different from a statistical perspective.

9 VBA samples about the same number of claims from each regional office regardless of 

the offices’ varying sizes. Smaller regional offices are disproportionately represented. 

Thus, the set of all claims reviewed nationally does not comprise a random sample of all 

claims. Weighting accounts for this fact and yields more correct estimates.

10 In comparing the weighted accuracy estimates that we computed to the unweighted 

estimates that VBA reported for regional offices in fiscal year 2013, we found that 

weighting would increase the accuracy rate more than .4 percent for 17 offices and 

decrease the accuracy rate more than .4 percent for 12 offices. Weighting would increase 

the accuracy estimates for regional offices by as much as 2.1 percent and decrease the 

estimates by as much 3.6 percent.

11 GAO, Managing For Results: GPRA Modernization Act Implementation Provides 

Important Opportunities to Address Government Challenges, GAO-11-617T (Washington, 

D.C.: May 10, 2011).

VBA’s Approach

to Measuring and 

Reporting Accuracy

of Claim Decisions 

Has Limitations
Page 6 GAO-14-731T 

measures and their associated limitations. While VBA’s new issue-based 

measure provides additional perspective on quality of claims decisions, to 

date VBA has not fully explained in its public reports how the issue-based 

and claim-based measures differ. VBA began reporting the issue-based 

measure in its ASPIRE dashboard in 2013. The issue-based measure 

tends to be higher than the claim-based measure because the former 

allows for claims to be considered partially correct, whereas the claimbased measure does not. According to VBA officials, the issue-based 

estimate provides a better measure of quality because veterans’ claims 

have increasingly included multiple medical issues. Our analysis of STAR 

data confirms that, as the number of issues per claim increases, the 

chance of at least one issue being decided incorrectly within a single 

claim increases because there are more opportunities for error (see fig. 

1). However, VA did not report in its fiscal year 2015 budget request how 

these measures are calculated and why the issue-based measure might 

be higher than the claim-based measure.

Figure 1: Claim-Based and Issue-Based Accuracy Rates by Number of Issues Claimed, Fiscal Year 2013

Further, VA has not explained in public reports that its accuracy 

measures are estimates which have distinct margins of error and 

limitations. These margins of error are necessary for users to make 
Page 7 GAO-14-731T 

meaningful comparisons, for example, between the two measures or over 

time for the same measure. Further, each accuracy measure has distinct 

limitations, but VA does not report this information. For example, the 

claim-based measure does not provide a sense of the proportion of 

issues that the agency decides correctly because the measure counts an 

entire claim as incorrect if any error is found. On the other hand, the 

issue-based measure does not provide a sense of the proportion of 

claims that the agency decides with no errors. In prior work, we identified 

clarity as a key attribute to a successful performance measure, meaning 

that the measure is clearly stated and the associated methodology for the 

measure is identified.12 Measures that lack clarity may confuse or mislead 

users. We have also reported on best practices in implementing related 

federal performance reporting requirements, such as those in the GPRA 

Modernization Act of 2010.13

Finally, VBA’s approach to measuring accuracy is inefficient because it 

reviews more claims than needed to estimate accuracy. VBA randomly 

selects about 21 claims per month from each of its regional offices for 

STAR review, regardless of the offices’ varying workloads and historical 

accuracy rates. According to VBA, this uniform approach allows the 

agency to achieve a desired level of precision of its accuracy estimates 

for each regional office.

Specifically, agencies must disclose 

information about the accuracy and validity of their performance 

information in their performance plans, including the sources for their data 

and actions to address any limitations.

14

 12 GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season 

Performance Measures, 

However, this approach leads VBA to select 

more claims for review than are needed at regional offices where the 

number of claims processed has been relatively small or accuracy has 

been high. According to our analysis of fiscal year 2013 regional office 

workload and accuracy results, VBA could reduce the overall number of 

claims it reviewed annually by about 39 percent (over 5,000 claims) and 

still achieve its desired precision for its regional office accuracy estimates. 

GAO-03-143 (Washington, D.C.: Nov. 22, 2002).

13 GAO, Managing For Results: GPRA Modernization Act Implementation Provides 

Important Opportunities to Address Government Challenges, GAO-11-617T (Washington, 

D.C.: May 10, 2011).

14 VBA arrived at its sample size—246 rating claims per regional office per year—based 

on an assumed accuracy of rate of 80 percent for each regional office, and a desired 

precision that reflects sampling error of plus or minus 5 percentage points at the 95 

percent level of confidence in accuracy estimates for each regional office. 
Page 8 GAO-14-731T 

Only for one regional office did we find that VBA would need to increase 

the number of claims currently reviewed to achieve its confidence level 

goal for that office. More efficient sampling could allow VBA to select 

fewer cases for review and free up limited resources for other important 

quality assurance activities, such as additional targeted accuracy reviews 

on specific types of error-prone or complex claims. Specifically, reviewing 

about 5,000 fewer claims could free up about 1,000 staff days because, 

according to VBA officials, STAR staff review at least 5 claims per day.

Our review of STAR is ongoing. Specifically, we plan to analyze STAR 

data and processes to assess whether any types of claims are 

systematically underrepresented in accuracy reviews. In addition, we are 

further considering the implications of a more efficient sampling 

methodology on developing issue-based accuracy estimates.

In addition to its STAR reviews, VBA’s quality assurance framework 

includes other complementary activities, some of which have been 

enhanced to help meet its 98-percent goal for fiscal year 2015. For 

example, VBA established QRTs in regional offices as a means of 

strengthening its focus on quality at regional offices, where claims are 

processed. QRT personnel—like STAR reviewers—are required to pass 

an annual skills certification test. In addition to conducting individual 

quality reviews to determine whether claims processors are achieving 

individual accuracy targets,15

 15 QRT reviewers review an average of 5 randomly-selected claims per claims processing 

staff member per month. For claims processing staff members found in need of accuracy 

improvement, 10 reviews per claims processing staff member per month may be 

performed.

QRT personnel are charged with conducting 

in-process reviews of claims not yet finalized, looking for specific types of 

VBA Has Enhanced 

and Coordinated Its 

Quality Assurance

Activities, Although 

Gaps in Effectiveness 

Exist

VBA Enhanced Other 

Quality Assurance 

Activities, but 

Shortcomings Exist and 

Effectiveness is Unclear
Page 9 GAO-14-731T 

errors commonly made. Quality reviewers are also responsible for 

providing feedback to claims processors on the results of their quality 

reviews, including formal feedback from the results of individual quality 

reviews and more informal feedback from the results of in-process 

reviews. Typically, feedback on quality reviews is provided as reviews are 

completed. In addition, at the four offices we contacted, quality reviewers 

are available to answer questions and provide guidance to claims 

processors as needed.

During our site visits, however, we identified shortcomings in QRT 

practices and implementation that could reduce their effectiveness. With 

respect to assessing individual performance, we learned that three of the 

four offices we contacted have agreements with their local unions that 

prevent QRT personnel from reviewing claims processed during 

overtime.16 As a result, regional offices would be limited in their ability to 

address issues with the quality of work performed during overtime. VBA 

officials told GAO they do not know how many regional offices include or 

exclude claims processed during overtime, or the extent to which 

excluding cases worked during overtime occurs nationally. According to 

VBA data, claims processed on overtime represented about 10 percent of 

rating-related claims completed nationally in fiscal year 2013. In our 

ongoing work, we plan to review the extent to which this practice is 

followed at other regional offices and how claims processed during 

overtime are identified. In addition, regional offices we contacted told us 

that they face a challenge in conducting in-process reviews as required17

because VBA’s Veterans Benefits Management System lacks the 

capability to pause the process and prevent claims from being completed 

while a quality review is still underway.18

 16To help reduce its claims backlog, VBA has required claims processors to work 20 hours 

per month of mandatory overtime during portions of fiscal years 2013 and 2014. 

In our ongoing work, we will 

continue to review the extent to which claims are decided before inprocess reviews are finalized and whether this causes rework or other 

problems.

17A regional office is expected to perform in-process reviews equivalent to 10 percent of 

their expected claims decisions per month, according to VBA guidance.

18VBA’s Veterans Benefits Management System is intended to help streamline the claims 

process by allowing for paperless claims processing, including electronic claims files.
Page 10 GAO-14-731T 

VBA’s efforts to assess consistency of claims decisions have also 

expanded in recent years. The inter-rater reliability reviews that VBA 

largely relied on to assess consistency since 2007 were time consuming 

in that claims processors required about 4 hours to review an entire claim, 

the process was administered by proctors in the regional offices, and the 

results were hand-graded by national VBA staff. Given the resources 

involved, IRRs were typically limited to 300-500 (about 25-30 percent) 

claims processors, randomly selected from each regional office. Since VA 

expanded its consistency program in 2009 to include consistency 

questionnaires, it now relies more heavily on this streamlined approach to 

assess consistency. The questionnaires require less staff time to 

complete because, in addition to a brief scenario on a specific condition, 

participants have 10 or fewer multiple choice questions to answer. The 

questionnaires are administered electronically through the VA Talent 

Management System, which has allowed VBA to increase employee 

participation and administer the studies more frequently. For example, a 

recent consistency questionnaire was taken by about 3,000 claims 

processing employees—representing all employees responsible for rating 

claims.

Although VBA has enhanced its approach to measuring consistency, VA 

officials told us that consistency questionnaires to date have been 

developed and implemented without any prior pre-testing which would 

allow VA to examine the clarity of questions or the validity of the expected 

questionnaire results. Pre-testing is a generally accepted practice in 

sound survey/questionnaire development and would help determine 

whether the test questions are appropriate for field staff and are 

accurately measuring consistency. A quality assurance official told us that 

by July 2014, VBA plans to begin pre-testing consistency questionnaires 

with national quality assurance staff that have claims processing 

experience. However, VBA has not yet developed a concrete plan or 

written policies for pre-testing the questionnaires.

VBA efforts to evaluate the effectiveness of its efforts have been limited, 

although VBA told us that they have seen some improvements in 

accuracy following certain quality assurance enhancements. Specifically, 

VBA officials told us that, although they have not seen an increase in the 

national accuracy rate in the current fiscal year, the number of errors 

related to claim development—specifically, to ensuring the claim reflected 
Page 11 GAO-14-731T 

sufficient / appropriate medical examinations)19—has declined, showing 

the success of QRT efforts in targeting these errors through in-process 

reviews and providing related training. On the other hand, VBA central 

office has only begun to receive data from regional office IQR reviews 

and expects to begin receiving additional data and identifying accuracy 

trends in the summer of 2014. With respect to consistency studies, VBA 

has not evaluated and lacks plans to evaluate the efficacy of its new 

approach for conducting consistency reviews to determine its relative 

effectiveness. Evaluation can help to determine the “value added” of the 

expenditure of federal resources or to learn how to improve 

performance—or both. It can also play a key role in strategic planning and 

in program management, informing both program design and execution.20

In our ongoing work, we plan to further review how VBA uses the results 

of its consistency studies to improve quality and whether and how VBA 

could further leverage its vast amount of claims data to identify error 

trends and opportunities for improvement.

VA has taken steps to coordinate its quality assurance efforts, in part, by 

systematically disseminating information on national accuracy and 

consistency trends to regional office management and QRTs, which in 

turn used this information to provide feedback. With respect to STAR, 

regional offices receive quarterly reports on their STAR accuracy 

performance, and QRT reviewers receive periodic conference call 

updates from STAR staff to discuss error trend information. Managers or 

QRT members at each of the regional offices we contacted noted that 

they also share STAR data with claims processors through periodic 

training focused on STAR error trends. With respect to consistency 

studies, regional offices receive national results; regional office-specific 

results; and, since February 2014, individual staff results. Officials at each 

of the four regional offices we visited told us they discuss the results of 

consistency studies and inform claims processors of the correct answers 

to the questions.

 19 The STAR review assesses whether adequate evidence was developed to support the 

rating decision. Possible development errors include failure to obtain sufficient medical 

records, including a medical examination or opinion.

20GAO, Designing Evaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: January 

2012).

VBA Has Taken Steps

to Coordinate Quality 

Assurance Activities,

but Key Supports Have 

Not Been Updated
Page 12 GAO-14-731T 

Based on error trends identified through STAR and other quality 

assurance activities, QRT personnel are also expected to disseminate 

guidance and provide input into, and sometimes conduct, regional office 

training. For example, two of the four offices we contacted cited instances 

where they have used consistency study results for training purposes. In 

general, at each of the four offices, reviewers conduct, or work with 

regional office training coordinators to conduct, periodic training forums 

for claims processors. Regional offices we contacted also supplement 

training with other communications informed by quality review results. For 

example, QRTs at three of the four regional offices we contacted produce 

periodic newsletters for regional office claims processors, including 

guidance based on errors found in all types of reviews.

In addition to sharing STAR results for regional office training purposes, 

VA uses STAR to guide other quality assurance efforts. For example, 

according to VBA officials, the agency has used STAR data to identify 

error trends associated with specific medical issues, which in turn was 

used to target efforts to assess consistency of decisionmaking related to 

those issues. Recent examples are the August 2013 inter-rater reliability 

study, which examined rating percentages and effective dates assigned 

for diabetes mellitus (including peripheral neuropathy); and a February 

2014 study on obtaining correct disability evaluations on certain 

musculoskeletal and respiratory conditions. In addition, according to VBA, 

the focus of in-process reviews has been guided by STAR error trend 

data. VBA established in-process reviews to help identify and prevent 

claim development errors based on medical examination and opinionrelated errors, which it described as the most common error type. More 

recently, VBA has added two more common error types—incorrect rating 

percentages and incorrect effective benefit dates—to its in-process 

review efforts. VBA officials stated that they may add other common error 

types based on future STAR error analyses.

While regional offices receive a lot of information on accuracy and error 

trends, some quality review staff expressed concern that there was too 

much information. At the same time, staff in all four offices said that key 

supports were not sufficiently updated to help quality review staff and 

claims processors efficiently and effectively do their jobs. Staff at these 

offices consistently described problems with data systems, central 

guidance and training.

• Regarding data systems, at the regional office level quality assurance 

information is input into three different systems. Staff at all four offices 

we contacted said that these systems lack functionality to create 
Page 13 GAO-14-731T 

reports on error trends, so they maintain spreadsheets to track 

additional information to allow them to assess error trends from their 

office’s quality reviews. At the national level, VA central office has 

made improvements in reporting capabilities to obtain information on 

errors by diagnostic code, but has not made this available to regional 

office quality staff.

• Regarding guidance, regional office quality review staff also said they 

face challenges locating the most current guidance from all of the 

information they are provided. Managers or staff at each of the 

regional offices we contacted said that VBA’s policy manuals are 

outdated. As a result, staff must search numerous sources of 

guidance to locate current policy, which is time-consuming and 

difficult. VBA officials acknowledged that there are several ways it 

provides guidance to regional offices—such as guidance letters; 

periodic quality calls (and notes from those calls); various bulletins; 

and training materials maintained on VBA’s Intranet site—and that this 

could be confusing to staff. VBA officials also noted that they face 

challenges in updating the policy manual to ensure it is as current as 

possible.

• Regarding training, staff in the offices we contacted also said that in 

some cases national training has not been updated to reflect the most 

current guidance. This makes it difficult to provide claims processors 

with the information they need to avoid future errors. For example, 

staff from one regional office noted that training modules on one error 

prone issue—Individual Unemployability21

 21Individual Unemployability is a part of VA’s disability compensation program that allows 

VA to pay benefits at the 100 percent level to veterans whose service-connected 

disabilities prevent them from maintaining substantial gainful employment. 

and related effective dates 

of benefits— had not been updated to reflect all new guidance, the 

sources of which included conference calls, guidance letters, and 

Frequently Asked Questions compiled by VBA’s central office. VBA 

officials stated that they are continually updating national training to 

reflect new guidance, but that it takes time. Further, according to 

officials at the regional offices we contacted, VBA restricts regional 

offices’ flexibility to tailor course materials to address office-specific 

error trends. We will follow up with VBA’s national training office to 

discuss challenges to updating training and reasons behind 

restrictions on regional offices’ ability to modify training materials.
Page 14 GAO-14-731T 

In conclusion, VBA has made important enhancements to its quality 

assurance program, but has missed key opportunities to fully cement its 

commitment to quality. Although VBA’s dual approach for measuring 

accuracy provides additional information on error trends, which can allow 

VBA to better target quality improvement efforts, VBA is producing 

imprecise estimates of accuracy that, while not completely reliable, are 

being used by program managers to guide improvement efforts. VBA also 

missed an opportunity to win the public’s trust when it introduced a new 

accuracy measure without full explanation of its meaning and limitations. 

At the same time, VBA is expending more resources than needed to 

produce its accuracy estimates—resources that could be better used to 

achieve more precise estimates or drill down on error trends to guide 

improvement efforts. VBA has bolstered regional attention to quality and 

national consistency efforts, but several shortcomings—such as excluding 

claims worked during overtime from regional office quality reviews, or the 

lack of pretesting of consistency questionnaires—may detract from their 

overall effectiveness. Moreover, VBA does not systematically track the 

impact of these efforts on accuracy rates, again creating a gap in 

information that could help focus future quality assurance efforts. Finally, 

while VBA has made a concerted effort to leverage and share information 

resulting from its national STAR and consistency studies and to 

disseminate relevant guidance, VBA has not taken the final step of 

centralizing its guidance or promptly updating national training. Claims 

decisions are highly complex and subjective, and staff who need to sift 

through a patchwork of guidance and training to make those decisions 

may be vulnerable to repeating prior errors. As we complete our ongoing 

work, we will consider any recommendations needed to address these 

issues.

Chairman Miller, Ranking Member Michaud, and Members of the 

Committee, this concludes my prepared statement. I will be pleased to 

answer any questions that you or other members of the subcommittee 

may have.

For future contact regarding this testimony, please contact Daniel Bertoni 

at (202) 512-7215 or bertonid@gao.gov. Key contributors to this 

testimony were Michele Grgich (Assistant Director), Carl Barden, David 

Chrisinger, Alexander Galuten, Joel Green, Dana Hopings, Vernette 

Shaw, Walter Vance, Greg Whitney and Amber Yancey-Carroll.

GAO Contact and 

Staff 

Acknowledgments

(131319)
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.