Department of Veterans Affairs 

Office of Inspector General 

Audit of Veterans Benefits Administration 

Compensation Rating Accuracy and 

Consistency Reviews 

Report No. 08-02073-96 March 12, 2009

VA Office of Inspector General 

Washington, DC 20420 
To Report Suspected Wrongdoing in VA Programs and Operations 

Telephone: 1-800-488-8244 between 8:30AM and 4PM Eastern Time, 

 Monday through Friday, excluding Federal holidays 

 E-Mail: vaoighotline@va.gov 

 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Contents 

Page 

Executive Summary............................................................................................................ i

Introduction ....................................................................................................................... 1

Purpose............................................................................................................................. 1

Background...................................................................................................................... 1

Scope and Methodology .................................................................................................. 4

Results and Conclusions.................................................................................................... 6

VBA’s STAR Program Does Not Provide a Complete Assessment of Rating 

Accuracy .......................................................................................................................... 6

VBA Has Not Fully Implemented Plans To Ensure Veterans Receive Consistent 

Ratings ........................................................................................................................... 12

Appendixes....................................................................................................................... 17

A. Universe and Sample Review of Claims Folders ..................................................... 17

B. Under Secretary for Benefits Comments .................................................................. 21

C. OIG Contact and Staff Acknowledgments................................................................ 25

D. Report Distribution ................................................................................................... 26

VA Office of Inspector General 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Executive Summary 

Results in Brief 

The Office of Inspector General (OIG) conducted an audit to evaluate selected aspects of 

the Veterans Benefits Administration’s (VBA) quality assurance program. The 

objectives for this audit were to determine: (1) if the Systematic Technical Accuracy 

Review (STAR) process effectively identified and reported errors in compensation rating 

decisions and (2) if VBA’s rating consistency reviews ensured that rating claims for 

similar conditions received the same evaluations and benefits, regardless of which VA 

regional office’s staff completed the rating decision. 

VBA’s STAR process did not effectively identify and report errors in compensation 

claim rating decisions. VBA identified a national compensation claim rating accuracy of 

87 percent for the 12-month period ending February 2008. We projected that VBA 

officials understated the error rate by about 10 percent, which equates to approximately 

88,000 additional claims where veterans’ monthly benefits may be incorrect. In total, we 

projected about 77 percent (approximately 679,000) of the almost 882,000 claims 

completed were accurate for the 12-month period ending February 2008. The 87 percent 

rate is not comparable to the accuracy rate VBA reports in the Performance and 

Accountability Report because that rate includes pension claims. Further, this accuracy 

rate only included errors that affected a veteran’s benefits. STAR identifies, but does not 

report, other errors related to the rating decision’s documentation, notification, and 

administration. Since these errors did not affect veteran benefits, we did not review the 

validity of these errors. 

VBA officials developed an adequate rating consistency review plan and implemented 

metrics to monitor rating consistency. However, they did not fully implement the review 

plans designed to ensure that compensation claims for similar conditions received the 

same evaluations and benefits. For fiscal year (FY) 2008, VBA officials identified 

61 diagnostic code reviews with inconsistencies in either a regional office’s grant/denial 

rate for a specific diagnostic code or a regional office’s evaluation (the percentage of 

compensation granted to a claimant) of a specific diagnostic code. Of the 61 identified 

reviews, VBA officials planned to conduct 22 reviews in FY 2008 consisting of 

20 grant/denial rate and 2 evaluation reviews. However, they only initiated two 

grant/denial rate reviews and these were not completed until December 2008. 

Furthermore, VBA officials did not initiate either of the two planned evaluation reviews 

to analyze and improve the consistency of disability compensation ratings and to reduce 

the variances between states. Although data security and integrity issues impacted STAR 

management’s ability to complete the planned reviews, VBA officials could have 

mitigated these problems by increasing efforts to correct data security issues and by 

assigning a sufficient number of staff to conduct the consistency reviews. 

VA Office of Inspector General i 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Quality assurance programs, such as STAR, are instrumental in identifying program 

deficiencies and alerting managers of the need to take corrective actions. Without an 

effective and reliable quality assurance program, VBA leadership cannot adequately 

monitor performance to make necessary program improvements and ensure veterans 

receive accurate ratings. Further, without implementing an effective rating consistency 

program, VBA officials cannot successfully assess or prioritize the improvements needed 

for claim rating consistency. 

Background 

VBA administers a multi-faceted quality assurance program to ensure veterans and 

beneficiaries receive accurate and consistent compensation and pension (C&P) benefits. 

STAR is a key mechanism for evaluating regional office performance in processing 

accurate benefit claims for veterans and beneficiaries. According to a STAR manager, 

improving the accuracy and consistency of disability compensation claim rating decisions 

is among VBA’s highest priorities. VBA’s FY 2008 and 2009 goal for compensation 

rating claims was a 90 percent accuracy rate. STAR management is also responsible for 

conducting reviews of rating consistency. These reviews assess inconsistencies identified 

in either a regional office’s grant/denial rate for a specific diagnostic code or a regional 

office’s evaluation of a specific diagnostic code. 

Findings 

VBA’s STAR Program Does Not Provide a Complete Assessment of Rating 

Accuracy 

VBA’s STAR program did not provide a complete assessment of compensation claim 

rating accuracy. During the 12-month period ending February 2008, VBA staff 

completed approximately 882,000 compensation claim ratings and STAR reviewers 

found that VBA staff accurately rated about 87 percent (approximately 767,000 claims). 

We reviewed a random sample of the claims reviewed by STAR reviewers to support the 

87 percent accuracy rate and found additional errors affecting a veteran’s benefits that 

STAR reviewers did not identify. As a result, we projected that VBA’s accuracy rate for 

the claims reviewed by STAR was only 78 percent. 

In addition, VBA officials excluded brokered claims from STAR reviews. We reviewed 

a sample of brokered claims and found an accuracy rate of 69 percent. When we 

combined the results of our review of brokered claims with our review of STAR 

reviewed claims, we projected that about 77 percent (approximately 679,000 claims) of 

compensation claims were accurate and VBA’s reported error rate was understated by 

approximately 10 percent. This equates to approximately 88,000 additional, or about 

203,000 total, claims where veterans’ monthly benefits may be incorrect. This difference 

occurred because STAR reviewers did not identify all errors, and VBA officials excluded 

VA Office of Inspector General ii 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

a significant number of compensation claim ratings from review. Additionally, VBA 

officials had not implemented an effective formal training program for the STAR 

reviewers. 

STAR reviewers can identify three types of errors—benefit entitlement, decision 

documentation/notification, and administrative. Additionally, they can record a 

comment, instead of an error, if the reviewers identify issues unrelated to the 

compensation claim ratings selected for review. VBA officials use benefit entitlement 

errors to calculate the compensation claim rating accuracy for regional offices because 

these errors affect the veterans’ benefits. 

STAR reviewers did not identify some of the missed errors because they either did not 

thoroughly review available medical and non-medical evidence (or identify the absence 

of necessary medical information), or they inappropriately misclassified benefit 

entitlement errors as comments. VBA officials do not count comments against the 

regional office’s accuracy rate. The STAR manager stated that reviewers might identify 

errors as comments if they did not know whether the errors affected the veterans’ benefits 

because required documentation was missing from the file. However, this conflicts with 

VBA guidance that requires STAR reviewers to identify errors that could result in a 

remand from the Board of Veterans’ Appeals (BVA). Although STAR staff reviews each 

sample compensation claim rating twice, the second review is cursory. 

While STAR management required regional offices to report quarterly on actions taken to 

correct benefit entitlement errors, they did not require or follow up to ensure regional 

offices took corrective actions on comments made by STAR reviewers. Ultimately, they 

rely on the regional office to take corrective action on all issues identified whether the 

STAR reviewer identified an error or a comment. We did not evaluate STAR follow up 

procedures to determine if corrective actions for the errors were appropriate. However, 

during VBA site visits to regional offices, team members review the corrective actions 

taken on issues identified during STAR for randomly selected cases from the most recent 

12-month period. 

From our sample, we identified 33 compensation claim ratings where STAR reviewers 

made comments instead of reporting issues as errors. At least six of the comments 

related to issues that could affect the veterans’ benefits. We found that regional office 

staff had not corrected any of the six comments potentially affecting the veteran’s 

benefits by the time of our review. Additionally, STAR reviewers should have recorded 

three of the six comments as benefit entitlement errors and we included them in our 

projections. Because the other three comments did not directly affect the specific claim 

under review, the reviewer correctly commented on the issues instead of recording them 

as errors. STAR management told us STAR reviewers conducted cursory secondary 

reviews of the sample compensation claim ratings and followed up on all errors (but not 

comments) to ensure regional offices took corrective action. The accuracy of the 

VA Office of Inspector General iii 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

veterans’ compensation claim ratings is dependent on ensuring regional office staff take 

corrective actions even if STAR reviewers identified an error that was not associated with 

the sample claim. VBA took action to correct this condition and starting 1st Quarter 

FY 2009 now require regional offices to report quarterly on identified comments as well 

as errors. 

VBA officials excluded brokered claims from STAR reviews. Brokered compensation 

claim ratings are those assigned to one regional office but sent to another office to be 

rated. VBA officials told us STAR reviewers do not review brokered claims because the 

STAR program’s primary focus is on assessing and reporting rating accuracy for each of 

the regional offices. Since two regional offices are involved in brokered work, VBA 

officials stated it would be difficult to assign responsibility for the rating accuracy to one 

specific regional office. STAR management replaces brokered claims selected for review 

with a non-brokered claim. We reviewed a sample of brokered claims selected for STAR 

that were replaced with a different claim. We projected for the 12-month period ending 

February 2008, that about 31 percent (approximately 39,000) of the 126,000 claims 

brokered to other regional offices were not accurate. The 69 percent accuracy rate was 

significantly lower than both VBA’s reported national accuracy rate of 87 percent and 

our projected accuracy rate of 78 percent for the STAR reviewed claims. VBA officials 

declined to speculate why the accuracy rate was lower for brokered claim ratings than for 

STAR reviewed compensation claim ratings. 

The STAR reviewers did not ensure regional offices submitted all of the selected 

compensation claim ratings to the STAR program for review. Regional offices did not 

submit about 600 (7 percent) of the approximately 9,000 requested claim ratings for the 

12-month period ending February 2008. We reviewed 54 of the approximately 600 

pending requests and identified 12 (22 percent) benefit entitlement errors. STAR 

management relies on regional office staff to submit the requested claims and only follow 

up with the regional offices that do not submit any requested claims for a given month. A 

STAR manager stated they did not have the resources to follow up on individual claims 

that regional office staff do not submit. Therefore, regional office staff can choose not to 

submit a claim since STAR reviewers do not reconcile the claim requests. This control 

weakness provided opportunities for regional office staff to inappropriately withhold a 

claim from the STAR reviewers if they suspect the claim to have errors. 

STAR reviewers are not required to complete formal training on an annual basis. 

According to a STAR manager, the reviewers met infrequently to discuss issues, and had 

no set formal training schedule or requirements. Regional office staffs that prepare and 

complete ratings and awards for compensation claims are required to achieve 80 hours of 

training per year to stay competent on laws, policies, and processes on 

rating-related issues. While the STAR manager agreed the reviewers could benefit from 

formal training, the manager stated workload requirements currently do not allow for the 

amount of training time necessary. 

VA Office of Inspector General iv 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

VBA Has Not Fully Implemented Plans To Ensure Veterans Receive Consistent 

Ratings 

VBA officials developed an adequate rating consistency review plan and implemented 

metrics to monitor rating consistency. The Deputy Under Secretary for Benefits testified 

before the House Veterans’ Affairs Subcommittee on Oversight and Investigations in 

October 2007 that VBA had established an aggressive, comprehensive program of quality 

assurance and oversight to assess compliance with VBA claims processing policies and 

procedures. Further, VBA would identify unusual patterns of variance in claims, and 

then review selected variances to assess the level of decision consistency among and 

between regional offices. However, VBA had not fully implemented its rating 

consistency review plan as of March 2009. 

To address variances in compensation ratings, VBA developed an adequate rating 

consistency review plan that included metrics to monitor rating consistency and a method 

to identify variances in compensation claim ratings. In FY 2008, VBA officials 

identified 61 diagnostic code reviews with inconsistencies in the grant/denial rate for a 

specific diagnostic code or an office’s evaluation of a specific diagnostic code. VBA 

officials planned to conduct 22 of the 61 reviews in FY 2008 consisting of 

20 grant/denial rate and 2 evaluation reviews. However, they only initiated two 

grant/denial rate reviews and did not complete either review until December 2008. 

Additionally, VBA did not initiate either of the evaluation reviews designed to reduce 

variances in compensation claim ratings. 

From April through May 2008, VBA officials encountered a delay in completing planned 

consistency reviews because the universe of claims to be reviewed included completed 

claims with diagnostic codes outside the scope for the requested period. VBA officials 

notified the Office of Performance, Accountability, and Integrity (PA&I) of the data 

integrity issue and worked with C&P service staff to identify an appropriate universe in 

May 2008. Then, in July 2008, VBA officials stopped all consistency reviews until they 

could take action to secure personally identifiable information (PII) in STAR’s electronic 

records used to capture and review data and analyze the results. VBA officials took the 

necessary actions to correct this condition and secured the database in December 2008. 

Although data security and integrity issues impacted STAR management’s ability to 

complete the planned reviews, VBA officials could have mitigated these problems by 

increasing efforts to correct data security issues and by assigning a sufficient number of 

staff to accomplish consistency reviews. The STAR program office was authorized 

26 reviewers for rating accuracy and consistency reviews, and had 18 reviewers on board. 

However, only 8 of the 18 reviewers conducted consistency reviews. The eight reviewers 

tasked with completing reviews of consistency were not sufficient to complete the 

assigned work. A STAR manager estimated that approximately 12 reviewers were 

VA Office of Inspector General v 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

needed to complete the 22 planned reviews. However, in addition to the consistency 

reviews, STAR management also assigned these same eight reviewers to conduct at least 

seven special focus reviews involving thousands of compensation claim ratings. 

As part of future rating consistency review plans, inter-rater reliability reviews (IRRR) 

should be included, along with the annual rating consistency reviews. In July 2008, C&P 

service staff conducted an IRRR and found that 76 (31 percent) of the 246 participants 

incorrectly rated a relatively simple back strain claim. In August 2008, C&P service staff 

conducted another IRRR and found that 30 (13 percent) of the 247 participants 

incorrectly rated a Post Traumatic Stress Disorder (PTSD) compensation claim. 

According to VBA officials, the most common errors were incorrect evaluations due to 

misinterpretation of the appropriate facts and criteria. C&P service managers used the 

results of the IRRRs to plan focused training efforts, and they plan to conduct follow up 

IRRRs to evaluate the effectiveness of the training. The IRRRs allow VBA officials to 

target a single rating issue to ensure the consistent application of policies and procedures 

nationally. While IRRRs are an effective tool for evaluating raters’ ability to rate specific 

rating issues consistently, the annual consistency reviews focus on regional offices’ 

ratings that are not consistent with average national ratings for specific issues. 

Conclusion 

VBA needs to make further improvements to the STAR program to better identify and 

report rating errors, and fully implement plans to review and improve rating consistency. 

STAR reviewers did not identify all rating errors. Rating accuracy reviews need to 

include a sample of brokered claim ratings. In addition, the program needs to establish 

formal STAR reviewer training. The accuracy of the veterans’ compensation claim 

ratings are dependent on ensuring regional office staff take corrective action regardless of 

how the errors affect VBA’s accuracy rate. 

While a rating review plan was developed and metrics to monitor rating consistency were 

implemented, limited grant/denial reviews were initiated during the year and evaluation 

reviews to reduce the variances between states did not occur. VBA officials needed to 

increase efforts and assign additional staff to complete necessary rating consistency 

reviews. VBA’s quality assurance program cannot meet its mission until they implement 

planned rating consistency reviews. An effective STAR and rating consistency review 

program will assist VBA’s efforts to increase compensation rating accuracy and 

consistency, and ultimately ensure veterans receive their entitled benefits. Without an 

effective and reliable quality assurance program, VBA leadership cannot adequately 

monitor performance to make necessary program improvements and ensure veterans 

receive accurate ratings. Further, without implementing an effective rating consistency 

program, VBA officials cannot successfully assess or prioritize the improvements needed 

for claim rating consistency. 

VA Office of Inspector General vi 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

VA Office of Inspector General vii 

Recommendations 

1. We recommended the Under Secretary for Benefits establish a mechanism to ensure 

STAR reviewers evaluate all documentation related to the claim selected for review. 

2. We recommended the Under Secretary for Benefits establish a requirement that all 

STAR reviewer comments receive a second review to make sure the reviewer 

appropriately recorded the comment instead of a benefit entitlement error. 

3. We recommended the Under Secretary for Benefits establish procedures to review 

brokered claims as part of the STAR program. 

4. We recommended the Under Secretary for Benefits enforce procedures requiring 

regional offices to submit all requested claims to the STAR program office for their 

review or submit written justification to the STAR program’s office requesting to 

exclude the claim from the review. 

5. We recommended the Under Secretary for Benefits establish minimum annual 

training requirements for each STAR reviewer that are comparable to regional office 

rating staff training requirements. 

6. We recommended the Under Secretary for Benefits develop an annual rating 

consistency review schedule and complete all planned reviews as scheduled. 

7. We recommended the Under Secretary for Benefits dedicate sufficient staff to 

conduct consistency reviews in order to complete planned workload and reviews. 

8. We recommended the Under Secretary for Benefits include inter-rater reliability 

reviews as a permanent component of their consistency review program. 

Comments 

The Under Secretary for Benefits agreed with our findings and recommendations. The 

Under Secretary took appropriate actions to implement recommendations 2, 3, 4, 5, and 8 

so we consider these recommendations closed. The Under Secretary’s planned actions 

for recommendations 1, 6, and 7 were acceptable and we will follow up on their 

implementation. See Appendix B for the full text of the Under Secretary’s comments.

 (original signed by:)

 BELINDA J. FINN 

Assistant Inspector General 

for Auditing 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Introduction 

Purpose 

The purpose of the audit was to evaluate selected aspects of VBA’s quality assurance 

program. The objectives for the audit were to determine if: 

• The STAR process effectively identified and reported errors in rating decisions. 

• VBA’s rating consistency reviews ensured that rating claims for similar conditions 

received the same evaluations and benefits, regardless of which VA regional office’s 

staff completed the rating decision.

Background 

VBA administers a multi-faceted quality assurance program to ensure regional office 

staffs provide C&P benefits in an accurate and consistent manner. The four components 

of the quality assurance program are STAR, rating consistency reviews, special focus 

reviews, and VA regional office compliance oversight visits. This audit focused on the 

STAR program rating accuracy and rating consistency reviews. 

STAR Program Rating Accuracy Review

STAR is a key mechanism for evaluating regional office performance in processing 

accurate benefit claims for veterans and beneficiaries. The STAR process provides a 

comprehensive review and analysis of compensation rating processing associated with 

specific claims or issues. According to the STAR manager, improving the quality, which 

includes accuracy and consistency, of disability compensation claim rating decisions is 

among VBA’s highest priorities. VBA’s FY 2008 and 2009 goal for compensation claim 

ratings was a 90 percent accuracy rate. STAR reviewers use a checklist designed to 

facilitate a consistent, structured review and classify errors into three categories: 

• Benefit entitlement errors affect the veterans’ entitlement to benefits and the STAR 

program uses these errors to assess regional office accuracy and performance. For the 

12-month period ending February 2008, VBA identified a compensation claim 

accuracy rate of 87 percent. This rate is not comparable to the accuracy rate VBA 

reports in the Performance and Accountability Report because that rate includes 

pension claims. 

• Decision documentation/notification errors affect procedural processes because of 

missing or incomplete claim documents. For the 12-month period ending 

February 2008, VBA reported that 7 percent of rating claims had decision/notification 

errors. These errors do not affect veteran benefit entitlements and are not included in 

VA Office of Inspector General 1 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

VA Office of Inspector General 2 

the reported compensation claim accuracy rates. Therefore, we did not report or 

include these types of errors in our projected accuracy rate. 

• Administrative errors affect procedural processes because of missing approvals and 

signatures. For the 12-month period ending February 2008, VBA reported that 

1 percent of rating claims had administrative errors. These errors do not affect 

veteran benefit entitlements and are not included in the reported compensation claim 

accuracy rates. Therefore, we did not report or include these types of errors in our 

projected accuracy rate. 

VBA uses end product codes as identifiers to monitor workload and productivity of the 

regional offices. Errors unrelated to the selected end products for review are not recorded 

as errors, but are documented as comments instead. When a veteran submits a claim for 

benefits, the claim is given an end product code that identifies the type of claim filed. 

Claims that require ratings must be developed (obtain all necessary evidence of the 

veteran’s service and disability to support the claim) and given to a Rating Veterans 

Service Representative (RVSR) for a decision. RVSRs determine if a veteran’s military 

service caused or aggravated the veteran’s condition and, if so, assign a percentage of 

disability based on the documented evidence. For reopened claims, the veteran’s 

previously assigned diagnostic codes with evaluations also affect the veteran’s overall 

combined percentage of disability. Regional office staffs use previously assigned 

diagnostic codes, along with their percentages, in combination with current assigned 

diagnostic code percentages to determine the combined level of disability. Once a claim 

is completed, the end product is cleared, and work credit is given to the regional office. 

A completed claim and corresponding cleared end product is then subject to review by 

STAR reviewers based on a statistical sample of all completed rating end products. Each 

month, PA&I send the STAR program a list of randomly selected completed end 

products for review. In order to review a selected end product, the veteran’s entire claims 

folder must be submitted to STAR. According to VBA criteria, regional offices have 

14 days to comply with the request or send a status update explaining why they did not 

send a folder. For example, the request status could be pending1

, brokered2, or file could 

not be located3

. 

During the 12-month period ending February 2008, STAR management requested 

approximately 700 sample claims each month to evaluate the accuracy of compensation 

claim ratings. In April 2008, VBA officials increased the number of claims selected for 

review monthly to approximately 1,200 claims nationwide. According to VBA officials, 

the additional sample claims allowed them to increase the confidence interval from 90 to 

95 percent for each regional office. 

 

1

 Regional office will comply later. 

2

 STAR program selects a different file. 

3

 STAR program deselects the sample claims and requests a replacement from the regional office. 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Rating Consistency Review

On May 19, 2005, the VA Office of Inspector General issued Review of State Variances 

in VA Disability Compensation Payments (Report No. 05-00765-137). The results 

showed that variances in VA disability compensation ratings continue to exist. The 

report recommended the Under Secretary for Benefits pursue a scientific study to further 

understand the influences on disability compensation payments. 

VBA officials contracted with the Institute for Defense Analyses’ (IDA) to further 

analyze these disability compensation variances among the different regional offices. 

The IDA report issued in December 2006, Analysis of Differences in Disability 

Compensation in the Department of Veterans Affairs, showed insufficient processes to 

ensure rating consistency among regional offices and recommended VA standardize 

initial and ongoing training for rating specialists, increase oversight and review of rating 

decisions, and develop and implement metrics to monitor consistency in adjudication 

results. 

VBA’s Deputy Under Secretary for Benefits testified before the House Veterans’ Affairs 

Subcommittee on Oversight and Investigations in October 2007 to discuss VBA’s 

response to the IDA report recommendations. The Deputy Under Secretary testified that 

VBA had established an aggressive, comprehensive program of quality assurance and 

oversight to assess compliance with VBA claims processing policy and procedures, 

assuring consistent application. VBA would conduct analyses to identify unusual 

patterns of variance in claims, and then review selected variances to assess the level of 

decision consistency among and between regional offices. VBA would use these studies 

to identify where regional office staffs require additional guidance and training to 

improve consistency and accuracy, as well as to drive procedural or regulatory changes. 

STAR management is also responsible for conducting reviews of rating consistency. To 

review consistency, VBA developed metrics to monitor regional office consistency 

among diagnostic codes. The central metrics were grant/denial rates for each diagnostic 

code and evaluation of granted service-connected percentages. 

• Grant reviews compared the regional offices’ grant of each diagnostic code to the 

national average. VBA officials define any regional office grant more than two 

standard deviations away from the national rate as an outlier. 

• Evaluation mode reviews compared the regional offices’ most commonly granted 

service-connected percentage (mode) for each diagnostic code to the most commonly 

granted national mode. VBA officials define any regional office mode that was equal 

to two or more evaluation levels away from the national mode as an outlier. 

VA Office of Inspector General 3 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

The goal of VBA’s consistency reviews is to monitor and ensure that veterans with 

similar conditions receive the same evaluations or benefits regardless of which regional 

office staff completed the rating decisions. To conduct the consistency review, STAR 

management identifies a diagnostic code for a regional office that was identified as an 

outlier from data PA&I obtained from multiple VBA systems using the STAR 

management’s defined metrics. The metrics focus on VA assigned diagnostic codes that 

account for 90 percent of the human body systems, such as cardiovascular and 

respiratory. The grant rate metric measures the percentage of diagnostic codes 

granted/denied to the national average granted/denied for the same diagnostic code. The 

percent evaluation metric is the measure of the most frequently assigned evaluation for 

each diagnostic code compared to the national average assigned evaluation for the same 

diagnostic code. STAR management submits a request to PA&I staff to select a sample 

of claims for a specific regional office and diagnostic code. PA&I staff obtain the list of 

sample claims and provides the information to STAR management who requests the 

sample claims for review from the regional office staff. When the regional office 

personnel submit the claims for review, the STAR reviewers record the consistency 

review data that VBA officials use to analyze the sample results. 

Scope and Methodology 

We conducted our work from June 2008 to February 2009. We reviewed compensation 

claim ratings selected for STAR review from March 2007 through February 2008. We 

evaluated VBA’s STAR and rating consistency program plans and methodologies for 

FY 2008 and performed site reviews to evaluate program and local policies and 

procedures. 

We reviewed applicable laws, regulations, policies, procedures, and guidelines that 

pertained to VA compensation claim ratings and VBA’s quality assurance program. 

Additionally, we interviewed managers and staff from VBA’s quality assurance program 

offices in Washington, D.C. and Nashville, TN, and the following regional offices: 

Denver, CO; Milwaukee, WI; Nashville, TN; Phoenix, AZ; and Salt Lake City, UT. 

During the 12-month period ending February 2008, STAR management requested 

approximately 700 sample claims each month to evaluate the accuracy of compensation 

claim ratings. In April 2008, VBA officials increased the number of claims selected each 

month for review to approximately 1,200 claims nationwide. According to VBA 

officials, the additional sample claims allowed them to increase the confidence interval 

from 90 to 95 percent for each regional office. We projected the compensation claim 

ratings accuracy from a random statistical sample of claims requested by the STAR 

reviewers during the 12-month period ending February 2008. We followed the same 

process used by the STAR review staff and discussed all identified errors with STAR 

review staff and VBA officials. 

VA Office of Inspector General 4 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

To evaluate if the STAR rating review process effectively identified and reported errors, 

we reviewed two stratified statistically random samples of 167 total compensation claim 

ratings. The samples included 115 compensation claims reviewed by STAR reviewers 

for accuracy and 52 brokered claims that STAR reviewers had not reviewed. 

Additionally, we reviewed a statistical random sample of 54 pending compensation claim 

ratings that were selected for review, but regional offices did not submit for review. We 

used the STAR program’s rating review checklist during our claims folder review and 

recorded any benefit entitlement errors. STAR reviewers count each claim with errors as 

one error, regardless of the number of errors found in the claim. We discussed the results 

of our review with VBA officials, and where appropriate, we made changes to our results 

based on explanations and documentation VBA officials provided us. 

To determine the effectiveness of the rating consistency reviews, we evaluated VBA’s 

FY 2008 rating consistency review plan, standard operating procedures, and business 

rules used for the data analysis. We examined two rating consistency reports for 

grant/denial reviews issued prior to FY 2008 and the results of IRRRs conducted in July 

and August 2008. The Secretary of Veterans Affairs requested the IRRRs to determine 

if consistency of ratings was still an issue. The IRRR allows VBA to target a single 

rating issue to evaluate the consistent application of policies and procedures nationally.

We assessed the reliability of computerized data by comparing selected data elements— 

veteran identifier information, end product codes, and disposition dates—to 

documentation in the claims folders. We concluded that the data used to accomplish the 

audit objectives was sufficiently reliable. 

We conducted this performance audit in accordance with generally accepted government 

auditing standards. Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence obtained 

provides a reasonable basis for our findings and conclusions based on our audit 

objectives. 

VA Office of Inspector General 5 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Results and Conclusions 

VBA’s STAR Program Does Not Provide a Complete 

Assessment of Rating Accuracy 

Findings 

VBA officials needed to implement additional improvements to ensure managers have a 

true assessment of the accuracy of compensation claim ratings. We determined the error 

rate STAR program officials found was understated by approximately 10 percent 

because: 

• STAR reviewers did not identify errors affecting veterans’ entitlement to benefits. 

• VBA officials excluded a significant number of compensation claim ratings from their 

review. 

Additionally, VBA officials had not implemented an effective training program to ensure 

STAR reviewers were current on rating policies and procedures. Quality assurance 

programs, such as STAR, are instrumental in identifying program deficiencies and 

alerting managers of the need to take corrective actions. Without an effective and 

reliable quality assurance program, VBA leadership cannot adequately monitor 

performance to make necessary program improvements and ensure veterans receive 

accurate ratings. 

1. VBA Officials Understated their National Error Rate by About 10 Percent 

During the 12-month period ending February 2008, VBA staff completed approximately 

882,000 compensation claim ratings and STAR reviewers found that VBA staff 

accurately rated about 87 percent (approximately 767,000 claims). We reviewed a 

random sample of the claims reviewed by STAR reviewers to support the 87 percent 

accuracy rate and found additional errors affecting a veteran’s benefits that STAR 

reviewers did not identify. As a result, we projected that VBA’s accuracy rate for the 

claims reviewed by STAR was only about 78 percent. 

In addition, VBA officials excluded brokered claims from STAR reviews. We reviewed 

a sample of brokered claims and found an accuracy rate of about 69 percent. When we 

combined the results of our review of brokered claims with our review of STAR 

reviewed claims, we projected that only about 77 percent (approximately 

679,000 claims) of compensation claims were accurate and VBA’s reported error rate 

was understated by approximately 10 percent. This equates to approximately 

88,000 additional, or about 203,000 total, claims where veterans’ monthly benefits may 

be incorrect. 

VA Office of Inspector General 6 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

STAR Reviewers Did Not Identify Errors Affecting Veterans’ Entitled Benefits

A STAR manager stated they used VBA guidance to review and evaluate the accuracy of 

compensation claim ratings. The reviewers can identify three types of errors—benefit 

entitlement, decision documentation/notification, and administrative. Additionally, 

reviewers can record a comment instead of an error if reviewers identify issues unrelated 

to the sample rating selected for review. VBA officials use benefits entitlement errors to 

calculate the accuracy rate for regional offices because they affect the veterans’ benefits. 

VBA does not use decision documentation/notification and administrative errors in 

calculating the regional office’s accuracy rate. 

VBA guidance requires STAR reviewers to report an error when a claim is not fully 

developed or a rating action conflicts with current regulations or directives. Fully 

developing the claim means to obtain all necessary documentation such as VA exams, 

service treatment records, discharge exams, and current medical records. Without the 

required information, VBA personnel could assign an inappropriate rating to a veteran 

which could affect the veteran’s entitlement to benefits and medical care. 

We projected that STAR reviewers did not identify an estimated 88,000 additional errors 

affecting veterans’ entitled benefits. STAR reviewers did not identify some of the errors 

because reviewers: 

• Did not thoroughly review claims folders to detect the presence or omission of key 

documentation, which was critical for the regional office staff to make an accurate 

rating decision. 

• Classified some errors as comments instead of benefit entitlement errors. VBA 

officials do not count comments against a regional office’s accuracy rate. 

The STAR manager stated that reviewers might identify errors as comments if they did 

not immediately know whether the errors affected the veterans’ benefits because required 

documentation was missing from the file. However, this conflicts with VBA guidance 

that requires STAR reviewers to identify errors that could result in a remand from BVA. 

A remand is an order from BVA to VBA to complete specified tasks, such as to order an 

exam, request missing records, or contact the veteran for more information. 

For example, in May 2007, a veteran filed a claim and informed the regional office that 

treatment records for his claimed condition were located at a VA medical center. The 

effective date for service-connection was the date of claim. The STAR reviewer’s 

comment stated that efforts should be made to obtain the information from the medical 

center to determine if the veteran is entitled to an earlier effective date. However, for the 

claimed condition, treatment records from 1998 in the claims folder already indicated the 

condition was probably due to his service-connected condition of diabetes mellitus. 

VA Office of Inspector General 7 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Therefore, the veteran was entitled to the earlier effective date. VBA officials agreed the 

STAR reviewer should have categorized this as an error instead of a comment. 

A STAR manager stated that STAR reviewers conduct a cursory second review for each 

claim reviewed. However, the secondary reviews focused on claims with benefit 

entitlement errors. STAR reviewers primarily checked for spelling and to ensure the 

reviewer answered checklist questions when they did not identify errors. A more 

thorough review of claims with reviewer comments would help ensure reviewers identify 

all benefit entitlement errors. 

VBA Officials Excluded a Significant Number of Ratings from Their Review

For the 12-month period ending February 2008, VBA brokered to other offices about 

126,000 of the approximately 882,000 completed ratings. VBA officials stated that 

STAR reviewers did not review brokered claims because the STAR program’s primary 

focus was on assessing and reporting rating accuracy for each of the regional offices. 

Since two regional offices are involved in brokered work, VBA officials stated it would 

be difficult to assign responsibility for the rating accuracy to one specific office. STAR 

management replaced brokered claims selected for review with other sample claims. Our 

review projected that of the 126,000 brokered claims not reviewed by STAR reviewers; 

about 31 percent (approximately 39,000) included errors affecting veterans' benefits, 

which resulted in about a 69 percent accuracy rate. The estimated 69 percent accuracy 

rate is significantly lower than both VBA’s national rate of 87 percent and our estimated 

result of 78 percent for the STAR reviewed claims. VBA officials would not speculate 

on why the error rates for brokered claims were significantly higher than non-brokered 

claims. VBA officials agreed with our results and stated they planned to review brokered 

claims in the future. 

VBA Officials Did Not Make Sure Selected Claims Were Submitted

VBA officials did not make sure regional offices submitted requested claims for review. 

For the 12-month period ending February 2008, regional offices did not submit about 

600 (approximately 7 percent) of the almost 9,000 requested claims and were not 

required to provide an explanation as to why the claims were not submitted. We 

reviewed 54 of the 600 pending requests and identified 12 (approximately 22 percent) 

benefit entitlement errors. Regional office staff could not determine a reason they did not 

submit certain claims folders when requested, but some of the reasons provided were 

because the claims or folders were: 

• Brokered claims or lost folders that STAR reviewers would have replaced with 

another claim. 

VA Office of Inspector General 8 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

• Unavailable at the time of the request because they could not be located or were 

actively being used by staff at regional offices, medical centers, or BVA. 

These errors were not included in our projected error rates since it was not possible to 

identify a universe of selected claims the regional offices would not submit for review. 

The STAR manager stated they rely on the regional office staff to submit the requested 

claims and only follow up with the regional offices that do not submit any requested 

claims for a given month. This was because the STAR program did not have the 

resources to follow up on individual claims that regional office staff did not submit. As a 

result, regional office staff may choose not to submit a claim since STAR reviewers do 

not follow-up or reconcile the request. This control weakness provided opportunities for 

regional office staff to inappropriately withhold a claim from the STAR reviewers if they 

suspect the claim to have errors. 

The STAR Program Office Did Not Follow Up on STAR Comments 

While STAR management required regional offices to report quarterly on actions taken 

for benefit entitlement errors, they did not require or follow up to make sure regional 

offices took corrective actions on comments made by STAR reviewers. Ultimately, they 

rely on the regional office to take corrective action on all issues identified whether the 

STAR reviewer identified an error or a comment. We did not evaluate STAR follow up 

procedures to determine if corrective actions for the errors were appropriate. However, 

we determined cyclic VBA site visits review the corrective actions taken on issues 

identified by STAR for randomly selected cases from the most recent 12-month period. 

From our sample, we identified 33 compensation claim ratings where STAR reviewers 

made comments instead of reporting issues as errors. At least six of the comments 

related to issues that could affect the veterans’ benefits. We found that regional office 

staff had not corrected any of the six comments potentially affecting the veteran’s 

benefits by the time of our review. Additionally, STAR reviewers should have recorded 

three of the six comments as benefit entitlement errors so we included them in our 

projections. Because the other three comments did not directly affect the specific claim 

under review, the reviewer correctly commented on the issues instead of recording them 

as errors. These three comments were not included in the national accuracy rate for our 

projected errors. STAR management told us the STAR reviewers conducted cursory 

secondary reviews of the sample compensation claim ratings and followed up on all 

errors (but not comments) to ensure regional offices took corrective action. 

The accuracy of the veterans’ compensation claim ratings is dependent on ensuring 

regional office staff take corrective actions even if STAR reviewers identified an error 

that was not associated with the sample claim. For example, a STAR reviewer made a 

comment on a benefit entitlement error that VBA identified on a prior rating decision. 

Three years ago, BVA stated that regional office staff completing the rating did not 

VA Office of Inspector General 9 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

address two claimed benefits. Because the benefit entitlement errors did not relate to the 

compensation claim ratings under review, the errors did not count toward the rating 

accuracy rate. Further, the regional office did not act on the STAR reviewer’s comment 

causing further delays in providing the veteran a decision on the two claimed conditions. 

At the time of our review, we found that regional office staff had not corrected any of the 

six comments potentially affecting the veteran’s benefits. Regional offices were not 

required to address comments. However, VBA took action to correct this condition and 

starting 1st Quarter FY 2009 now requires regional offices to report quarterly on 

identified comments as well as errors. 

2. VBA Needs To Establish a Training Program for the STAR Reviewers 

Formal training requirements had not been established to ensure STAR reviewers remain 

current in the competencies to review the work of regional office staff. Regional office 

staffs that prepare and complete ratings and awards for compensation claims are required 

to complete 80 hours of formal training per year. The requirements specify that 60 of the 

required 80 hours should cover rating-specific issues such as post traumatic stress 

disorder, effective dates, ancillary benefits, and various physically disabling conditions. 

Regional office managers will select the training topics for the remaining 20 hours at 

their discretion. 

While the STAR manager agreed STAR reviewers could benefit from formal training, the 

manager stated workload requirements currently do not allow for training time. VBA 

constantly adjusts the way regional office staff handle claims due to changes in law, 

Office of General Counsel Precedent Opinions, and court decisions. Therefore, regional 

offices conduct regular refresher training on development and rating issues, as necessary, 

as well as regularly scheduled training for RVSRs and Decision Review Officers to stay 

up to date on current policies and procedures. For one of the errors we identified, the 

STAR manager stated the experienced STAR reviewer would have been more likely to 

miss this error than a newer STAR reviewer. The experienced STAR reviewer had been 

away from rating claims for several years, and the changes in policy and procedures for 

this particular issue have been substantial. Formal training requirements would help the 

STAR reviewers remain current on rating issues such as this. 

Conclusion 

Rating accuracy assessments and reporting could be strengthened to better identify rating 

errors and improve the accuracy of the compensation claim ratings. VBA had more than 

doubled the number of STAR reviewers and increased the monthly sample size for each 

regional office, but additional improvements are needed to accurately evaluate 

compensation claim ratings. Rating accuracy reviews need to include a sample of 

brokered claim ratings. In addition, the program needs to establish STAR reviewer 

VA Office of Inspector General 10 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

training requirements. An effective STAR program will assist VBA’s efforts to increase 

compensation rating accuracy and ensure veterans receive their entitled benefits. 

Recommendations 

1. We recommended the Under Secretary for Benefits establish a mechanism to ensure 

STAR reviewers evaluate all documentation related to the claim selected for review. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated VBA will amend the STAR rating checklist to 

include automated reminders of salient points of the claim review process to enhance 

the quality review process. VBA anticipates programming modifications will be 

completed by mid-summer 2009. 

2. We recommended the Under Secretary for Benefits establish a requirement that all 

STAR reviewer comments receive a second review to make sure the reviewer 

appropriately recorded the comment instead of a benefit entitlement error. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated quality assurance management now requires a 

second-level review of STAR comments. 

3. We recommended the Under Secretary for Benefits establish procedures to review 

brokered claims as part of the STAR program. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated system enhancements completed in May 2008, 

will allow electronic sampling of work completed at the rating resource centers and 

additional staff are being hired to handle the increased workload demands. Quality 

assurance management plans to begin sampling completed work from the 

10 designated brokered sites during the third quarter FY 2009. 

4. We recommended the Under Secretary for Benefits enforce procedures requiring 

regional offices to submit all requested claims to the STAR program office for their 

review or submit written justification to the STAR program’s office requesting to 

exclude the claim from the review. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated follow-up procedures have been implemented 

that will include the Office of Field Operations working through the area offices to 

ensure all STAR cases are submitted for review. 

VA Office of Inspector General 11 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

5. We recommended the Under Secretary for Benefits establish minimum annual 

training requirements for each STAR reviewer that are comparable to regional office 

rating staff training requirements. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated the quality assurance management developed a 

training plan for FY 2009 and requires STAR staff to complete 80 hours of training 

per year. 

Office of Inspector General Comments

The Under Secretary took appropriate actions to implement recommendations 2, 3, 4 and 

5 so we consider these recommendations closed. The planned actions for 

recommendation 1 are acceptable and we will follow up until they are implemented. 

VBA Has Not Fully Implemented Plans To Ensure Veterans 

Receive Consistent Ratings 

Findings 

VBA officials developed an adequate rating consistency review plan and implemented 

metrics to monitor rating consistency. However, VBA had not fully implemented its plan 

to review consistency of compensation claim ratings and meet its program mission. 

STAR reviewers did not complete planned consistency reviews due to insufficient staff 

resources, delays caused by data integrity issues, and PII security issues that caused 

program managers to halt the rating consistency reviews. As a result, as the IRRR 

showed and VBA officials acknowledged, variances in compensation ratings for similar 

disabilities still exist and ratings may not be consistent. 

1. Rating Consistency Review Plan Has Not Been Fully Implemented 

The Deputy Under Secretary for Benefits testified before the House Veterans’ Affairs 

Subcommittee on Oversight and Investigations in October 2007 that VBA had 

established an aggressive, comprehensive program of quality assurance and oversight to 

assess compliance with VBA claims processing policy and procedures, assuring 

consistent application. Further, VBA would identify unusual patterns of variance in 

claims, and then review selected variances to assess the level of decision consistency 

among and between regional offices. To address variances in compensation ratings, 

VBA developed an adequate rating consistency review plan. This plan included 

implementing metrics to monitor rating consistency and identifying outliers to evaluate 

variances in compensation claim ratings. However, VBA had not fully implemented its 

rating consistency plan as of March 2009. 

VA Office of Inspector General 12 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

From the universe of assigned diagnostic codes, VBA identified 61 outliers for potential 

consistency reviews—comprised of 22 grant reviews and 39 evaluation reviews. For 

these 61 outliers, VBA officials scheduled 22 reviews—20 grant/denial reviews and 

2 evaluation reviews. However, STAR management only initiated two grant/denial 

reviews and these reviews did not begin until March 2008. These two reviews were not 

completed until December 2008. Additionally, evaluation reviews to analyze and 

improve disability compensation rating consistency have not been initiated. Thus, VBA 

did not reduce variances in compensation claim ratings. 

2. Rating Consistency Review Team Was Not Fully Staffed 

VBA officials did not fully implement the rating consistency plan because STAR staffing 

resources were insufficient to accomplish both of the rating consistency reviews and 

other special projects assigned. The STAR program office was authorized 26 reviewers 

for rating accuracy and consistency reviews, and had 18 reviewers on board. VBA 

officials expect to increase staff levels now that an office expansion project in Nashville, 

TN, was completed in December 2008. STAR program officials hired 10 of the 

18 reviewers in February 2008 to conduct the STAR reviews. STAR management 

assigned the experienced 8 of these 18 reviewers to conduct consistency reviews and/or 

special projects. Based on the STAR manager’s estimate, approximately 12 reviewers 

were needed just to complete the 22 planned reviews. However, in addition to the 

consistency reviews, these same eight reviewers also conducted at least seven special 

focus reviews involving thousands of compensation claim ratings. These included 

recurring and one-time special reviews. 

Recurring Reviews

• Review of all awards over $250,000 or retroactive 8 years or more—VBA officials 

permanently transferred this review to the STAR program in February 2008. 

• C&P Exam Program Review—reviewers completed 7,600 reviews during FY 2008. 

• IRRR—completed one each in July and August 2008. 

One-Time Special Reviews

• Radiation claim review. 

• Reviews of post-traumatic stress disorder claims. 

• Quality review of canceled disability exams (future disability rating exams). 

VA Office of Inspector General 13 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

• Special review of Veterans Health Administration contract disability rating exam 

quality. 

3. Data Integrity Issues Delayed Consistency Reviews 

VBA officials delayed the consistency reviews from April through May 2008 because 

claims selected for sampling were outside the scope for the requested period. 

Consistency reviews focus on a diagnostic code from a completed claim during a specific 

time. C&P Service staff prepared and submitted the data request to PA&I in 

March 2008. PA&I staff extracted the requested data, which C&P Service staff used to 

request sample claims for consistency reviews. However, in April 2008, C&P Service 

staff determined that the March sample failed to limit the data to claims with only 

diagnostic codes for the requested period. VBA officials notified PA&I of the integrity 

issue and worked with C&P Service staff to identify an appropriate universe in 

May 2008. 

4. Consistency Reviews Stopped Until Data is Secured 

From May through July 2008, STAR reviewers evaluated claims from the two 

consistency reviews. However, in July 2008, VBA officials stopped all consistency 

reviews until they could take action to secure PII in STAR’s electronic records used to 

capture and review data and analyze the results. To conduct a consistency review, STAR 

reviewers uploaded the results of each review electronically into a database. The 

reviewers used the database for analysis to identify any trends or deficiencies that needed 

correction. However, in July 2008, VBA officials determined that the database did not 

meet security standards for PII and halted the consistency reviews until they could correct 

the data protection issue. VBA officials took action to correct this condition and secured 

the database in December 2008. 

5. Consistency of Ratings Remained a Problem 

At the request of the Secretary of Veterans Affairs, C&P service officials conducted an 

IRRR in July 2008 to determine if consistency of ratings was still an issue. The IRRR 

complements the consistency review methodology by allowing VBA officials to target a 

single rating issue to ensure the consistent application of policies and procedures. To 

conduct an IRRR, VBA officials provided selected regional office rating staff a 

compensation claim ratings scenario that had only one correct decision. Participants 

from the regional offices made a rating decision and included their justification on the 

provided compensation claim scenario. 

For the July 2008 IRRR, VBA officials designed a back strain compensation claim to be 

simple and straightforward. The straightforward design was intended to obtain consistent 

rating results that C&P service staff could easily evaluate. The results showed that only 

VA Office of Inspector General 14 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

69 percent of the participants correctly rated the claim. According to VBA officials, the 

most significant errors were the range of evaluation percentages assigned to the 

diagnostic codes, which accounted for 88 percent of the total errors. 

C&P service managers were able to analyze the results and the rater’s individual 

responses to effectively plan focused training efforts regarding specific issues identified 

through the review. From the July review, VBA officials coordinated with the VBA 

policy and training offices and conducted a training session for all regional offices on 

August 28, 2008. However, VBA officials will not know the effectiveness of the training 

until another review of the back strain rating issue is completed. According to VBA 

officials, they plan to conduct a follow up in 6 months to evaluate the effectiveness of the 

training. 

VBA officials conducted a second IRRR in August 2008 with a PTSD compensation 

claim. The results showed that 87 percent of the participants correctly rated the claim. 

Again, the most significant error was the range of evaluation percentages assigned to the 

diagnostic codes which accounted for 97 percent of the total errors. The participants 

assigned a higher and lower evaluation of 50 percent or 10 percent instead of the correct 

answer of 30 percent. This occurred due to misinterpretation of the severity of 

symptoms. 

As part of the future rating consistency review plans, IRRRs should be continued along 

with the already planned annual rating consistency reviews. While IRRRs are an 

effective tool for evaluating rating consistency nationally for specific rating issues, the 

annual consistency reviews focus on regional offices’ ratings that are not consistent with 

average national ratings for specific issues. 

Conclusion 

During FY 2008, efforts to improve rating consistency were too limited to effectively 

address the number of consistency issues impacting ratings. While VBA officials 

developed a rating consistency review plan and implemented metrics to monitor rating 

consistency, the implementation of the planned reviews was limited. Although data 

security and integrity issues disrupted the consistency reviews, the reviews were not a 

high priority. VBA officials needed to increase efforts and assign additional staff to 

complete necessary rating consistency reviews. The IRRRs showed that variances in 

compensation claim ratings for similar disabilities still exist and VBA officials need to 

make improving rating consistency and its assessment more of a priority. VBA’s quality 

assurance program cannot meet its mission until they implement planned rating 

consistency reviews. Further, without implementing an effective rating consistency 

program, VBA officials cannot successfully assess or prioritize the improvements needed 

for claim rating consistency. 

VA Office of Inspector General 15 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

VA Office of Inspector General 16 

Recommendations 

6. We recommended the Under Secretary for Benefits develop an annual rating 

consistency review schedule and complete all planned reviews as scheduled. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated VBA developed a schedule for conducting 

rating consistency-focused case reviews for FY 2009. The target completion date for 

this action is September 30, 2009. 

7. We recommended the Under Secretary for Benefits dedicate sufficient staff to 

conduct consistency reviews in order to complete planned workload and reviews. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated three additional reviewers have been hired and 

will begin in May 2009, and VBA plans to continue to aggressively recruit to fill 

authorized vacancies. 

8. We recommended the Under Secretary for Benefits include inter-rater reliability 

reviews as a permanent component of their consistency review program. 

Under Secretary for Benefits Comments 

The Under Secretary agreed and stated VBA developed an Inter-Rater Reliability 

Study schedule for FY 2009. 

Office of Inspector General Comments

The Under Secretary took appropriate actions to implement recommendation 8 so we 

consider this recommendation closed. The planned actions for recommendations 6 and 7 

are acceptable and we will follow up until they are implemented. 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix A 

Universe and Sample Review of Claims Folders 

Our review of the STAR rating accuracy program covered the 12-month period ending 

February 2008. During this period, VBA completed 881,644 compensation rating end 

products. Of these completed end products, PA&I stated 125,968 were brokered. STAR 

reviewers did not review brokered claims selected for review by the STAR program. 

STAR reviewers only reviewed completed, non-brokered claims. 

Table 1. Compensation Ratings Completed by VBA 

March 2007 through February 2008 

Non-Brokered 

Ratings 

Brokered 

Ratings 

Total Compensation 

Ratings 

755,676 125,968 881,644 

Universe

The sampling universe for evaluating the STAR rating accuracy program consisted of 

6,257 compensation claim ratings reviewed by STAR reviewers and 1,514 claims that 

were not reviewed because the claims were brokered. The timeframe selected for the 

claims was end products that were cleared (completed) during the 12-month period 

ending February 2008. The STAR rating accuracy review includes the following seven 

compensation end products. 

Table 2. End Products Used for Compensation Rating Accuracy 

Description Code 

Initial Disability Compensation—8 or More Issues 010 

Reopened Compensation 020 

Appeal Certifications and Supplemental 

Statements of the Case 070 

Initial Disability Compensation—1 to 7 Issues 110 

Initial Death Compensation & Dependency and 

Indemnity Compensation 140 

Statements of the Case 172 

C&P Hearings—Decision Review Officer 174 

VA Office of Inspector General 17 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix A 

End Products

The STAR management requested compensation claim rating end products for the STAR 

rating accuracy review from a list of randomly selected end products provided by PA&I. 

We identified four categories from the claims selected for STAR rating review, which 

were “reviewed,” “brokered,” “pending,” and “deselected.” The STAR staff only 

reviewed the non-brokered claims selected for review. We reviewed a sample of the nonbrokered and brokered claims, as well as a sample of pending claims. 

Table 3. Compensation End Products Requested for Review by the 

 STAR program office, March 2007 through February 2008 

Strata End Products 

Percentage of Total 

Requested End Products 

Non-Brokered (STAR-Reviewed) 6,257 69

Brokered 1,514 17

Pending 623 7

Deselected 613 7

Total Requested 9,007 100

Sample Design

We used a stratified simple random statistical model with two sample strata. The sample 

sizes were determined using a 90 percent confidence interval with a 10 percent precision 

rate. For reviewed and brokered claims, we reviewed 115 and 52 compensation rating 

end products respectively. 

We also selected a simple random statistical sample and reviewed 54 pending 

compensation rating claims using a 90 percent confidence interval with a 10 percent 

precision rate. Pending compensation rating claims are claims requested by STAR for 

review where the regional office did not provide the claims folder for review. 

Sample Results

For the STAR rating accuracy review, STAR reviewers evaluated regional offices on the 

outcome of cleared end products. STAR reviewers looked for all errors made in a cleared 

end product, but only reported benefit entitlement errors. Benefit entitlement errors are 

not limited to, but can result in overpayments, underpayments, and BVA remands. BVA 

remands a claim when development fails to provide necessary evidence to accurately rate 

an end product. 

VA Office of Inspector General 18 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix A 

To evaluate the reported accuracy rates, we reviewed end products using the same 

methodology used by STAR to review files. The checklist questions are grouped into 

three major error question types—A through D are benefit entitlement errors, E and F are 

decision documentation errors, and G and H are administrative errors. Question type J is 

for a comment which does not relate to recorded errors. 

We reviewed 115 sample compensation claim ratings previously reviewed by STAR 

reviewers and identified 25 with errors. Additionally, we identified 16 errors in the 

52 sample compensation claim ratings not reviewed by STAR reviewers because one 

regional office developed the claim and another regional office brokered the claim for 

rating purposes. 

Table 4. OIG Projection of STAR Reviewed and Brokered Sample Results 

Strata 

Sample 

Size Error 

Strata 

Size 

Projected 

Accurate 

Claims 

Margin 

of Error

Accuracy 

Percent 

Margin 

of Error 

Percent 

Reviewed 115 25 6,257 4,897 400 78 6.4

Brokered 52 16 1,514 1,048 162 69 10.7

Weighted 7,771 5,945 431 77 5.6

Based on the results of our review we projected that about 77 percent of the 

881,644 compensation claim ratings were accurate. This was approximately 10 percent 

lower than VBA’s reported 87 percent accuracy rate. 

Table 5. OIG Projection of VBA Compensation Rating Accuracy 

March 2007 through February 2008

Sample 

Accuracy 

Percent 

Margin 

of Error 

Percent 

Accurate 

Claims 

VBA 87 .95 767,030 

OIG 77 5.60 678,866 

Additional Inaccurate Claims 88,164 

The difference between VBA and OIG’s margin of error was primarily attributed to the 

different sample sizes used by both organizations. Although larger, our margin of error 

was statistically acceptable for projection and comparison purposes. 

VA Office of Inspector General 19 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix A 

VA Office of Inspector General 20 

For pending claims, we used the same review techniques as the other two reviews. We 

identified 12 (22 percent) of the 54 claims with benefit entitlement errors, which equates 

to an estimated 78 percent accuracy rate. The defining attribute of a pending end product 

is the lack of compliance by regional offices to satisfy the sample request. We 

determined that identifying a universe of claims that regional offices would not submit 

was not attainable. Therefore, we did not project these results. 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix B 

Department of 

Veterans Affairs Memorandum

Date: March 3, 2009 

From: Under Secretary for Benefits (20) 

Subject: Audit of Veterans Benefits Administration Compensation 

Rating Accuracy and Consistency Reviews

To: Assistant Inspector General for Audit (52) 

1. This is in response to your request for VBA’s review of OIG Draft Report: 

Audit of Veterans Benefits Administration Compensation Rating Accuracy and 

Consistency Reviews. Attached are VBA’s comments. 

2. Questions may be referred to Alyce Williams, Program Analyst, at 

(202) 461-9364. 

(original signed by:) 

 P. W. Dunne 

Attachments 

VA Office of Inspector General 21 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix B 

Under Secretary for Benefits Comments 

to Office of Inspector General’s Report 

The following Under Secretary for Benefits comments are submitted in response to the 

recommendations in the OIG’s Report: 

Recommendation 1: We recommend the Under Secretary for Benefits establish a 

mechanism to ensure STAR reviewers evaluate all documentation related to the claim 

selected for review. 

VBA Response: Concur. The STAR program reviews rating accuracy, authorization 

accuracy, and fiduciary program accuracy. To help ensure that STAR reviewers conduct 

a full and thorough review of the selected claim, VBA will amend the STAR rating 

checklist to include automated reminders of salient points of the claim review process. 

We believe the addition of focused reminders aimed at certain key decision points will 

enhance our current quality review process and result in a stronger national accuracy 

review program. 

A workgroup consisting of several members of the STAR staff has been formed to 

develop an assessment document that will be programmed into the STAR database. This 

additional step in the review process will help eliminate errors and omissions by 

reviewers. We anticipate programming modifications to be complete by mid-summer 

2009. 

Target Completion Date: Mid-summer 2009 

Recommendation 2: We recommend the Under Secretary for Benefits establish a 

requirement that all STAR reviewer comments receive a second review to make sure the 

reviewer appropriately recorded the comment instead of a benefit entitlement error. 

VBA Response: Concur. Management of the Quality Assurance Staff now requires a 

second-level review of STAR comments. 

Target Completion Date: Action completed. 

Recommendation 3: We recommend the Under Secretary for Benefits establish 

procedures to review brokered claims as part of the STAR program. 

VBA Response: Concur. In 2007, VBA identified the need to separately sample 

brokered work apart from work sampled for national quality review from the regional 

offices. System and procedural enhancements were needed to accommodate such a 

review. VBA also evaluated the additional resources needed to support the expansion. 

VA Office of Inspector General 22 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix B 

System enhancements to allow electronic sampling of work completed at the rating 

resource centers and exclude such work from samples pulled for regional office accuracy 

reviews were completed in May 2008. The space acquisition and renovation plan to 

accommodate the expanded quality assurance staff was completed in December 2008. 

The quality assurance staff is now hiring to handle the increased review demands. 

VBA will begin sampling the work completed from 10 designated brokered sites during 

the third quarter of FY 2009. Current STAR rating-related sampling and review 

procedures will be employed with the brokered work. Brokered work accuracy reports 

will be generated in addition to the existing STAR accuracy reports. 

Target Completion Date: Action completed. 

Recommendation 4: We recommend the Under Secretary for Benefits enforce 

procedures requiring regional offices to submit all requested claims to the STAR program 

office for their review or submit written justification to the STAR program’s office 

requesting to exclude the claim from the review. 

VBA Response: Concur. Follow-up procedures have been implemented for all STAR 

cases not received monthly. Revised procedures now require the quality assurance staff 

to provide monthly lists to the Office of Field Operations (OFO) of cases requested but 

not received for quality review. OFO will work through the area offices to ensure that all 

STAR cases are submitted for review. 

Target Completion Date: Action completed. 

Recommendation 5: We recommend the Under Secretary for Benefits establish 

minimum annual training requirements for each STAR reviewer that are comparable to 

regional office rating staff training requirements. 

VBA Response: Concur. The quality assurance staff developed a training plan for 

FY 2009. Under the plan, members of the STAR staff are required to complete 80 hours 

of training per year. All training will be recorded in VA’s Learning Management 

System. Training plans and accomplishments will be provided to the C&P Service 

Director annually. 

Target Completion Date: Action completed. 

Recommendation 6: We recommend the Under Secretary for Benefits develop an 

annual rating consistency review schedule and complete all planned reviews as 

scheduled. 

VBA Response: Concur. VBA developed a schedule for conducting rating consistencyfocused case reviews for FY 2009. 

VA Office of Inspector General 23 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix B 

VA Office of Inspector General 24 

Target Completion Date: September 30, 2009 

Recommendation 7: We recommend the Under Secretary for Benefits dedicate 

sufficient staff to conduct consistency reviews in order to complete planned workload and 

reviews. 

VBA Response: Concur. Three additional quality reviewers are expected to be on board 

by May 2009. We will continue to aggressively recruit to fill authorized vacancies. 

Target Completion Date: September 30, 2009 

Recommendation 8: We recommend the Under Secretary for Benefits include interrater reliability reviews as a permanent component of their consistency review program. 

VBA Response: Concur. VBA initially conducted two inter-rater reliability studies in 

July and August 2008. VBA developed an Inter-Rater Reliability Study schedule for 

FY 2009. 

Target Completion Date: Action completed. 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix B 

OIG Contact and Staff Acknowledgments 

OIG Contact Larry Reinkemeyer (816) 997-6940 

Acknowledgments Lee Giesbrecht 

Timothy Halpin 

Carol Hickman 

Patricia Hudon 

Lance Kramer 

Brad Lewis 

Russ Lewis 

Ken Myers 

Dao Pham 

Jason Schuenemann 

Marcia Schumacher 

Brenda Uptain 

VA Office of Inspector General 25 
Audit of Veterans Benefits Administration Compensation Rating Accuracy and Consistency Reviews 

Appendix C 

Report Distribution 

VA Distribution

Office of the Secretary 

Veterans Health Administration 

Veterans Benefits Administration 

National Cemetery Administration 

Assistant Secretaries 

Office of General Counsel 

Non-VA Distribution

House Committee on Veterans’ Affairs 

House Appropriations Subcommittee on Military Construction, Veterans Affairs, 

 and Related Agencies 

House Committee on Oversight and Government Reform 

Senate Committee on Veterans’ Affairs 

Senate Appropriations Subcommittee on Military Construction, Veterans Affairs, 

 and Related Agencies 

Senate Committee on Homeland Security and Governmental Affairs 

National Veterans Service Organizations 

Government Accountability Office 

Office of Management and Budget 

This report will be available in the near future on the OIG’s Web site at 

http://www.va.gov/oig/publications/reports-list.asp. This report will remain on the OIG’s 

Web site for at least 2 FYs after it is issued. 

VA Office of Inspector General 26