Veterans' Disability Benefits: Improvements Could Further Enhance Quality Assurance Efforts
Released: 2014-11-19T00:00:00

Letter

Background

VA pays monthly disability compensation to veterans with service- connected disabilities (i.e., injuries or diseases incurred or aggravated while on active military duty) according to the severity of the disability. VBA staff in 57 regional offices process disability compensation claims.These claims processors include Veterans Service Representatives (VSR) who gather evidence needed to determine entitlement, and Rating Veterans Service Representatives (RVSR) who decide entitlement and the rating percentage. Veterans may claim more than one medical condition, and a rating percentage is assigned for each claimed medical condition, as well as for the claim overall.decided more than 1 million compensation claims.

In fiscal year 2013, VBA Since fiscal year 1999, VBA has used STAR to measure the decisional accuracy of disability compensation claims. Through the STAR process, VBA reviews a stratified random sample of completed claims, and certified reviewers use a checklist to assess specific aspects of each claim. are randomly sampled each month and the data are used to produce estimates of the accuracy of all completed claims. VA reports national estimates of accuracy from its STAR reviews to Congress and the public through its annual performance and accountability report and annual budget submission. VBA also produces regional office accuracy estimates, which it uses to manage the program. Regional office and national accuracy rates are reported in a publicly available performance database, the Aspire dashboard.

The STAR review has two major components. The benefit entitlement review assesses whether the correct steps were followed in addressing all issues in the claim, collecting appropriate evidence, and whether the resulting decision was correct, including effective dates and payment rates. Accuracy performance measures are calculated based on the results of the benefit entitlement review. The STAR review also assesses whether claims processors appropriately documented the decision and notified claimants. new issue-based measure. By comparison, under the existing claim- based measure, the claim would be counted as 0 percent accurate unless the error did not affect benefits when considered in the context of the whole claim. In March 2014, VBA reported a national estimate of issue- based accuracy in its fiscal year 2015 annual budget submission and plans to update this estimate in VA’s next performance and accountability report. VBA also produces issue-based estimates by regional office, and reports them in the Aspire dashboard. For fiscal year 2013, the regional office claim-based accuracy rates ranged from an estimated 78.4 to 96.8 percent, and the issue-based accuracy rates ranged from an estimated 87.0 to 98.7 percent.

Beyond STAR, VBA has programs for conducting regional office quality reviews and for measuring the consistency of decisions. In March 2012, VBA established quality review teams (QRT) with one at each regional office. A QRT conducts individual quality reviews of claims processors’ work for performance assessment purposes. The QRT also conducts in- process reviews before claims are finalized to help prevent inaccurate decisions by identifying specific types of common errors. Such reviews also serve as learning experiences for staff members. Since fiscal year 2008, VBA has also conducted studies to assess the consistency of disability claims decisions across regional offices. Initially, this initiative used inter-rater reliability (IRR) studies to assess the extent to which a cross-section of claims processors from all regional offices agree on an eligibility determination when reviewing the entire body of evidence from the same claim. In 2013, VBA revised its approach and began using questionnaires as its primary means for assessing consistency. A questionnaire includes a brief scenario on a specific medical condition for which claims processors must correctly answer several multiple-choice questions.

VBA’s Approach to Measuring and Reporting Accuracy of Claim Decisions Has Limitations

VBA Does Not Follow Accepted Statistical Practices and Thus Generates Imprecise Accuracy Data

When calculating accuracy rates, VBA does not always follow generally accepted statistical practices. For example, VBA does not weight the results of its STAR reviews to reflect its approach to selecting claims by regional office, which can affect the accuracy of estimates. According to our analysis of VBA data, weighting would have resulted in a small change to VBA’s nationwide claim-based accuracy rate for fiscal year 2013—from 89.5 to 89.1 percent. At the regional level, 29 of the 57 offices would have experienced a somewhat greater increase or decrease in their accuracy rates. Without taking weighting into consideration, regional office accuracy performance may be misleading and VBA management may focus corrective action or positive recognition on the wrong offices. For example, by taking weighting into account for the 57 regional offices in fiscal year 2013, the Reno regional office would have improved in relative accuracy by 12 places (from 34th to 22nd place), whereas the Los Angeles office would have declined in relative accuracy by 10 places (from 46th to 56th place) (see fig. 1).

VBA also does not calculate the confidence intervals associated with the accuracy estimates that it generates, which prevents a complete understanding of trends over time and comparisons among offices. Accuracy estimates for different regional offices, or for the same office over time, are considered statistically different from each other when their confidence intervals do not overlap. As such, meaningful comparisons could be made on the basis of our analysis between, for example, Fort Harrison’s estimated claim-based accuracy rate (ranked #1) and New York’s estimated claim-based accuracy rate (ranked #36) because their confidence intervals did not overlap in fiscal year 2013 (see fig. 2). Conversely, comparisons between Fort Harrison’s and Milwaukee’s or Pittsburgh’s estimated claim-based accuracy rates (ranked #2 and #35 respectively)—which had overlapping confidence intervals in fiscal year 2013—require a statistical test to determine if their differences are statistically meaningful. In effect, the claim-based accuracy rate of Fort Harrison and those of the regional offices with the next 34 highest reported accuracy rates may not be meaningfully different despite being ranked 1 through 35 of 57. Similarly, according to agency officials, VBA also does not calculate the confidence intervals associated with its newer issue-based accuracy estimates, which prevents meaningful comparisons between those estimates as well. Because VBA produces issue-based estimates using the same sample drawn to produce claim-based estimates, it would have to take extra steps to calculate the associated confidence intervals.computing the confidence intervals associated with issue-based As with the claim-based accuracy estimates, not estimates limits VBA’s ability to monitor its regional offices’ relative performance and its overall performance over time.

VBA’s approach to measuring accuracy is also inefficient because it reviews more claims than are statistically required to estimate accuracy. VBA randomly selects about 21 claims per month from each of its regional offices for STAR review, regardless of the offices’ varying workloads and historical accuracy rates. According to VBA, this uniform approach allows the agency to achieve a desired level of precision of its accuracy estimates for each regional office. However, accepted statistical practices would allow for fewer cases to be reviewed at regional offices where the number of claims processed has been relatively small or accuracy has been high. According to our analysis of fiscal year 2013 regional office workload and accuracy results, VBA could reduce the overall number of claims it reviews annually by about 39 percent (over 5,000 claims) and still achieve its desired precision for its regional office accuracy estimates. More efficient sampling could allow VBA to select fewer cases for review and free up limited resources for other important quality assurance activities, such as additional targeted accuracy reviews on specific types of error-prone or complex claims. Specifically, reviewing about 5,000 fewer claims could free up about 1,000 staff days because, according to VBA officials, STAR staff review at least 5 claims per day.

Calculating weighted estimates and confidence intervals, and adjusting sampling according to shifting workloads and accuracy rates, requires use of statistical methodology. According to VBA officials we interviewed, although STAR management used a statistician to help develop the way in which they measure accuracy, it currently does not use a statistician to, for example, weight STAR results and calculate confidence intervals for accuracy estimates. Further, VBA officials said they did not consult a statistician when developing the new issue-based accuracy measure, but rather relied on the same sampling methodology and approach for estimating accuracy as for the claim-based measure. We have previously reported that to be useful, performance information must meet users’ needs for completeness, accuracy, consistency, and validity, among other factors. In response to our draft July 2014 testimony based on preliminary work, VBA officials stated they are exploring alternatives to their current methodology for estimating accuracy.

Beyond not following generally accepted statistical practices, VBA’s STAR review systematically excludes certain claims, which may inflate accuracy rate estimates. Specifically, according to VBA officials, when a claim moves from one regional office to another, because a veteran has moved or workloads are redistributed, the database VBA uses to select claims for STAR review does not always reflect the office responsible for As a result, STAR staff making the final determination for the claim. often select for review, then subsequently de-select, claims that have changed regional office jurisdiction. Of the 14,286 rating claims randomly selected initially by VBA for review in fiscal year 2013, about 10 percent were de-selected because of a change in jurisdiction and replaced with other randomly selected claims. Those de-selected claims are not eligible for STAR review for the regional office that was ultimately responsible for the claim, thereby causing an underrepresentation of these claims in the STAR sample. Such underrepresentation may inflate VBA’s reported accuracy rate because redistributed claims have historically had lower accuracy rates than non-redistributed claims. In responding to our draft report, VBA indicated it is revising its procedures to ensure that claims selected for STAR review are included in the accuracy rate of the responsible regional office regardless of whether a change of jurisdiction occurred.

VBA Does Not Report Key Information to Help Users Understand Accuracy Metrics

Federal agencies should report clear performance information to the Congress and the public to ensure that the information is useful for decision making. In prior work, we identified clarity as a key attribute to a successful performance measure, meaning that the measure is clearly stated and its associated methodology is identified. Measures that lack clarity may confuse or mislead users, and not provide a good picture of how well the agency is performing. We have also reported on best practices in implementing related federal performance reporting requirements, such as those in the GPRA Modernization Act of 2010. Specifically, agencies must disclose information about the accuracy and validity of their performance information in their performance plans, including the sources for their data and actions to address any limitations.

VBA’s accuracy reporting lacks methodological details that would help users understand the distinction between its two accuracy measures and their associated limitations. While VBA’s new issue-based measure provides some additional perspective on the quality of claim decisions to date, VBA has not fully explained in its public reports how the issue- based and claim-based measures differ. For example, the issue-based measure tends to be higher than the claim-based measure because the former allows for claims to be considered partially correct, whereas the claim-based measure does not. According to VBA officials, the issue- based estimate provides a better measure of quality because veterans’ Our analysis claims have increasingly included multiple medical issues.of STAR data confirms that as the number of issues per claim increases, the chance of at least one issue being decided incorrectly within a single claim increases because there are more opportunities for error (see fig. 3). However, VA did not report in its fiscal year 2015 budget request how these measures are calculated and why the issue-based measure might be higher than the claim-based measure. VA has also not reported these distinctions in its Aspire dashboard.

VBA also counts claims processing errors differently under its claim- based measure than it does under its issue-based measure but does not report these distinctions, which raises questions about the transparency and consistency of VBA’s accuracy measures. For both measures, VBA differentiates between benefit entitlement errors that may financially affect the veteran and other errors, such as documentation and administrative errors that do not financially affect the veteran. For claim-based accuracy, VBA counts errors that financially affect the veteran now, but does not count errors that may financially affect the veteran in the future, although it works to correct both types of errors. For example, if one of several claimed medical conditions was rated incorrectly (e.g., 10 percent instead of 20 percent), but this error did not immediately affect the overall rating of the claim, VBA would not consider the claim in error because it did not affect the benefits that the veteran would receive. For the issue-based accuracy measure, however, VBA would count this as an error even if the error did not immediately affect the veteran’s benefits. Unlike claim-based accuracy, issue-based accuracy may also include errors that would never affect future payments. For example, an incorrect effective date that is within the same month as the correct effective date does not affect benefits, but is counted as an error in VBA’s issue-based accuracy measure. Conversely, according to VBA officials, this is not counted as an error in its claim-based measure. According to our analysis of STAR data, up to 6.9 percent of reviewed claims in fiscal year 2013 had these types of errors (i.e., benefit entitlement errors that do not immediately and may never affect benefits), and if they were all counted as errors, VBA’s unweighted claim-based accuracy rate would have decreased by about 2 percent.

Further, VA has not explained in public reports that its accuracy measures are estimates that have distinct confidence intervals and limitations. Users should be aware of these confidence intervals to make meaningful comparisons, for example, between the two measures or over time for the same measure. In terms of each accuracy measure’s limitations, the claim-based measure does not provide a sense of the proportion of issues that the agency decides correctly because the measure counts an entire claim as incorrect if any error is found. On the other hand, the issue-based measure does not provide a sense of the proportion of claims that the agency decides with no errors.

VBA Has Enhanced and Coordinated Its Quality Assurance Activities, Though Gaps in Implementation May Limit Their Effectiveness

VBA Has Taken Steps to Enhance and Coordinate Key Quality Assurance Activities

In addition to its STAR reviews, VBA’s quality assurance framework includes other complementary activities, which have been enhanced to help meet its goal of 98 percent accuracy in fiscal year 2015. Specifically, VBA (1) established quality review teams (QRT) in March 2012 in regional offices as a means of strengthening its focus on quality where claims are processed, and (2) enhanced efforts to assess the consistency of decisions.

Although regional offices were previously responsible for assessing individual performance, QRTs represent a departure from the past because QRT personnel are dedicated primarily to performing these and other local quality reviews. In addition, VBA requires QRT staff to pass a skills certification test annually—similar to VBA requirements for STAR staff and in contrast to requirements for claims processors who must pass a test every 2 years. In July 2013, VBA issued national guidance to ensure consistent QRT roles and practices across regional offices. For example, it included guidance on selecting individual quality review claim samples and conducting additional reviews for claims processors who do not meet their accuracy goals. In addition to conducting individual quality reviews, QRT personnel are charged with conducting in-process reviews of claims that are not yet finalized, looking for specific types of common errors. Quality reviewers are also responsible for providing feedback to claims processors on the results of their quality reviews, typically as reviews are completed, including formal feedback from the results of individual quality reviews and more informal feedback from the results of in-process reviews. In addition, at the four offices we contacted, quality reviewers are available to answer questions and provide guidance to claims processors as needed.

VBA’s efforts to assess consistency of claims decisions have also expanded in recent years. Up until 2013, VBA largely relied on inter-rater reliability (IRR) studies to assess consistency, which to date have been time consuming and resource intensive. Claims processors typically required about 4 hours to review an entire claim. The process was administered by proctors in the regional offices and the results were hand-graded by national VBA staff. Given the resources involved, IRR studies have been typically limited to 300-500 (about 25-30 percent) claims processors, randomly selected from the regional offices. In 2009, VBA expanded its consistency program to include questionnaires, which it now relies on more heavily to assess consistency. The more streamlined consistency questionnaires require less staff time to complete because, in addition to a brief scenario on a specific condition, participants have 10 or fewer multiple-choice questions to answer. The questionnaires are administered electronically through the VA Talent Management System, removing the need to proctor or hand-grade the tests, which has allowed VBA to significantly increase employee participation. A recent consistency questionnaire was taken by about 3,000 claims processing employees— representing all employees responsible for rating claims. Further, VBA now administers these studies more frequently, from about 3 to 24 per year. According to VBA officials, they plan to further expand the use of consistency studies from two questionnaires per month to six to eight per month, pending approval of additional quality assurance staff.

VBA also has taken steps to coordinate its quality assurance efforts in several ways, such as systematically disseminating information on national accuracy and consistency results and trends to regional office management and QRTs, which in turn share this information with claims processing staff. With respect to STAR, in addition to receiving monthly updates on overall accuracy performance, regional offices receive quarterly reports with analyses of accuracy performance including information by error type. QRT reviewers also participate in monthly conference calls with STAR staff during which they discuss error trend information. While claims processing staff learn about errors they made on claims directly from STAR, managers or QRT members at each of the regional offices we contacted noted that they also share STAR trend data with claims processors during periodic training focused on STAR error trends. With respect to consistency studies, regional offices receive national results; regional office-specific results; and, since February 2014, individual staff results. Officials at each of the four regional offices we visited told us QRT staff share the results of consistency studies with staff and inform claims processors of the correct answers to the questions.

Coordination also occurs when QRT personnel disseminate guidance and support regional office training based on error trends identified through STAR and other quality assurance activities. Two of the four offices we contacted cited instances where they have used consistency study results for training purposes. At one office, the results from a consistency study were used to provide training on when to request an exam for certain conditions, such as tinnitus. In general, at each of the four offices, officials told us that QRT reviewers conduct, or work with regional office training coordinators to conduct, periodic training forums for claims processors. Regional offices we contacted also supplement training with other communications informed by quality review results. For example, QRTs at three of the four regional offices we contacted produce periodic newsletters for regional office claims processors, which include guidance based on errors found in all types of reviews. Specifically, at one office, a newsletter was used to disseminate guidance on ensuring that a rating decision addresses all issues in a claim. The need for this guidance was identified on the basis of STAR and local quality review results.

Lastly, VBA coordinates its quality assurance activities by using STAR results to guide other quality assurance efforts. According to VBA officials, the agency has used STAR data to identify error trends associated with specific medical issues, which in turn were used to target efforts to assess consistency of decision-making related to those issues. Recent examples are (1) the August 2013 IRR study, which examined rating percentages and effective dates assigned for diabetes mellitus (including peripheral neuropathy); and (2) a February 2014 study on obtaining correct disability evaluations on certain musculoskeletal and respiratory conditions. In addition, according to VBA, the focus of in- process reviews performed by QRTs has been guided by STAR error trend data. VBA established in-process reviews in March 2012 to help the QRTs identify and prevent claim development errors related to medical examinations and opinions, which it described as the most common error type. More recently, VBA has added two more common error types— incorrect rating percentages and incorrect effective benefit dates—to its in-process review efforts. VBA officials stated that they may add other common error types based on future STAR error analyses.

Some Gaps in Implementation Persist and the Effectiveness of Quality Assurance Activities Is Unclear

While QRTs reflect VBA’s increased focus on quality, during our site visits we identified shortcomings in QRT practices and implementation that could reduce their effectiveness. Specifically, we identified the following shortcomings: (1) the exclusion of claims processed during overtime to assess individual performance; (2) the inability to correct errors identified before a claim is finalized in certain situations; and (3) a lack of pre- testing of consistency questionnaires.

Regarding the first shortcoming, we learned that three of the four offices we contacted had agreements with their local unions that prevented QRT personnel from reviewing claims processed during overtime to assess individual performance. As a result, those regional offices were limited in their ability to address issues with the quality of work performed during overtime. Centrally, VBA officials did not know which or how many regional offices excluded claims processed during overtime, or the extent to which excluding cases worked during overtime occurred nationally. According to VBA data, claims processed on overtime represented about 10 percent of rating-related claims completed nationally in fiscal year 2013. After we reported this finding, VBA issued guidance in August 2014 to regional offices stipulating inclusion of claims processed on overtime, and that the regional offices work with their local unions to rescind any agreements that exclude such claims from review.

Second, officials at four regional offices we contacted told us that they face a challenge in conducting individual quality and in-process reviews as expected because VBA’s Veterans Benefits Management System lacks the capability to briefly pause the process and prevent claims from being completed while a review is still underway. VBA officials acknowledged that this was a problem for regional offices in completing reviews, based on anecdotal information from regional offices, but did not have information on the extent to which this occurred. VBA officials noted that reviews could be performed after a claim is completed; however, if an error is found, the regional office might need to rework the claim and provide the veteran with a revised decision. The officials also noted that VBA is working toward modifying its Veterans Benefits Management System to address this issue, but is at the initial planning stage of gathering requirements and could not provide a time frame for completion.

Thirdly, although VBA has developed a more streamlined approach to measuring consistency, VBA officials told us that consistency questionnaires were developed and implemented without any pre-testing, which would have helped the agency determine whether the test questions were appropriate for field staff and were accurately measuring consistency. Pre-testing is a generally accepted practice in sound questionnaire development for examining the clarity of questions or the validity of the questionnaire results. In the course of our review, VBA quality assurance officials noted that they plan to begin pre-testing consistency questionnaires as a part of a new development process. Specifically, after each questionnaire has been developed, two to three quality assurance staff who have claims processing experience, but were not involved in the questionnaire’s development, would be targeted to pre-test it. Quality assurance staff responsible for the consistency studies would then adjust the questionnaire if necessary before it is administered widely. While initially slated to occur in July 2014, VBA quality assurance staff now anticipate pre-testing to begin in September 2014.

Beyond these implementation shortcomings, staff in each of the four offices we contacted said that several key supports were not sufficiently updated to help quality review staff and claims processors do their jobs efficiently and effectively. Staff at these offices consistently described persistent problems with central guidance, training, and data systems.

Guidance: Federal internal control standards highlight the need for pertinent information being captured and distributed in a form that allows people to perform their duties efficiently. However, regional office quality review staff said they face challenges locating the most current guidance among all of the information they are provided. Managers or staff at each of the regional offices we contacted said that VBA’s policy manuals are outdated. As a result, staff must search numerous sources of guidance to locate current policy, which is time- consuming and difficult. This, in turn, could affect the accuracy with which they decide claims. One office established a spreadsheet to consolidate guidance because the sources were not readily available to claims processors. VBA officials acknowledged that there are several ways it provides guidance to regional offices. In addition to the existence of relevant regulations and VBA’s policy and procedures manual, VBA provides guidance to claims processors through policy and procedures letters, monthly quality calls and notes from these calls, various bulletins, and training letters and other materials maintained on VBA’s intranet site. While agreeing that having multiple sources of guidance could be confusing to staff, VBA officials noted they face challenges in updating the policy manual and other available guidance materials to ensure that they are as current as possible. After we reported on this issue, VBA officials noted that they are considering streamlining the types of guidance provided. They also plan to develop a system of consolidated links to guidance documents by alphabetized topic to help claims processors access the information more efficiently; however, VBA officials acknowledge that developing a single repository will be a challenging project and have not yet dedicated adequate resources for this effort.

Training: Staff in the offices we contacted also said that in some cases national training has not been updated to reflect the most current guidance, which in turn makes it difficult to provide claims processors with the information they need to avoid future errors. For example, staff from one regional office noted that training modules on an error-prone issue—Individual Unemployability and related effective dates of benefits—had not been updated to reflect all new guidance, the sources of which included conference calls, guidance letters, and frequently asked questions compiled by VBA’s central office. Further, officials at regional offices we contacted expressed concern that VBA limits their flexibility to update out-of-date course materials. In response to these concerns, VBA training officials explained that that they are continually updating national training to reflect new guidance, but how long it takes is a function of the extent of the policy change. These officials noted that updating the Individual Unemployability training was particularly delayed because of numerous, unanticipated changes in policy and related guidance that resulted in their setting aside previously updated course materials and starting over. VBA training officials also explained that while VBA does not allow changes to the contents of courses in its catalog, regional offices can propose courses for the catalog, based on their needs identified through quality reviews.

Data systems: Regional office quality review staff also told us that they are required to log errors into three systems or databases that do not “speak to one another” and two lack the capability to fully track errors trends, thereby limiting their ability to take corrective actions. At the regional office level, quality assurance information is entered into three different databases or systems. Staff at each of the four offices we contacted said that the Automated Standardized Performance Elements Nationwide system used for tracking individual accuracy for performance management purposes lacks functionality to create reports on error trends by claimed medical issue or reasons for specific types of errors. As a result, three offices maintain separate spreadsheets to identify error trends related to individual accuracy. Regional office staff also noted that one of the two systems used to track in-process reviews does not help track error trends, for example, by employee, resulting in two offices maintaining additional spreadsheets to track this information. At the national level, VBA central office has made some improvements in reporting and now has the ability to analyze regional office information on errors by medical issue. According to VBA officials, they share this information with regional office managers and quality staff during training calls. VBA officials stated that a planned replacement for its Automated Standardized Performance Elements Nationwide system would have addressed reporting limitations at the local level, but was halted. As of September 2014, VBA did not have a timeframe for restarting the process for acquiring a new system.

Finally, VBA’s efforts to evaluate the effectiveness of its quality assurance activities have been limited. Specifically, VBA officials told us that although they have not seen an increase in the national accuracy rate in fiscal year 2014, the number of errors related to claim development has declined, demonstrating the success of QRT reviews and training in targeting these errors. Also, VBA identified 13 regional offices whose issue-based accuracy rates improved between the first and third quarters of fiscal year 2014, attributing these improvements to actions taken by However, it was not clear quality assurance staff in fiscal year 2014.from the documentation VBA provided whether and how it monitored the effectiveness of these actions for all regional offices. With respect to consistency studies, VBA also has not evaluated—and lacks plans to evaluate—the efficacy of using consistency questionnaires relative to the more resource-intensive IRR studies. According to a VBA official, the consistency questionnaires have helped identify regional offices and individuals in need of further training on the basis of the percentage of incorrect answers, as well as the need for national training. However, officials could not provide data or evaluations indicating that consistency questionnaires have improved accuracy rates in the areas studied.

VBA officials noted that they are considering a new data system that would combine all local and national quality assurance data—including STAR, in-process reviews, and individual quality reviews—and allow for more robust analyses of root causes of errors. Specifically, they expect the system will show relationships across the results of various quality assurance reviews to determine employee competence with various aspects of claims processing. According to VBA officials, this system would also enable them to more easily evaluate the effectiveness of specific quality assurance efforts. Evaluation can help to determine the “value added” of the expenditure of federal resources or to learn how to improve performance—or both. It can also play a key role in strategic planning and in program management, informing both program design and execution. Continuous monitoring also helps to ensure that progress is sustained over time. However, VBA officials indicated that this proposal is still in the conceptual phase and requires final approval for funding and resources.

Conclusions

VBA’s dual approach for measuring accuracy is designed to provide additional information to better target quality improvement efforts, but its methods and practices lack rigor and transparency, thereby undermining the usefulness and credibility of its measures. By not leveraging a statistician or otherwise following statistical practices in developing accuracy estimates, VBA is producing and relying on inaccurate estimates to make important internal management decisions. Similarly, by using a one-size sampling methodology, VBA is unnecessarily expending limited resources that could be used elsewhere. The systematic exclusion of redistributed claims and those moved between offices further calls into question the rigor of its accuracy estimates. Lastly, VBA’s reporting of its two accuracy metrics lacks sufficient transparency to help members of Congress and other stakeholders fully understand the differences and limitations of each, and thus may undermine their trust in VBA’s reported performance.

VBA has enhanced and coordinated other aspects of its quality assurance framework, but shortcomings in implementation and evaluation detract from their overall effectiveness. For example, although VBA is disseminating the results of national STAR reviews and consistency studies, and local QRTs are using those results to focus related training or guidance to claims processing staff, until centralized guidance is consolidated and streamlined, staff lack ready access to information that will help them prevent errors. Moreover, absent adequate system capabilities to support local quality reviews, QRTs are unable to stop incorrect decisions from being finalized, and may not be aware of error trends that could be mitigated through training or other corrective action. Finally, although some of its quality assurance activities are relatively new, VBA lacks specific plans to evaluate their effectiveness and may miss opportunities to further improve or target these activities to more error-prone areas. In general, unless VBA takes steps to improve the rigor of all its quality assurance methods and practices, VBA may find progress toward achieving its goal of 98 percent accuracy in fiscal year 2015 illusive—especially in the face of challenging workloads, limited resources, and expectations of timely claim decisions.

Recommendations for Executive Action

To help improve the quality of VBA’s disability compensation claim decisions, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits to:

Leverage appropriate expertise to help VBA do each of the following: weight its accuracy estimates to reflect the sample design for determine and report the confidence intervals associated with its reported accuracy estimates; and re-examine its approach to calculating the regional office sample size for STAR.

Take steps to ensure that redistributed claims and those moved between regional offices are not underrepresented in the STAR sample.

Increase transparency in explaining how the claim-based and issue- based accuracy rates are calculated as well as their key limitations when publicly reporting these metrics.

Review the multiple sources of policy guidance VBA provides to determine ways to consolidate them or otherwise improve their availability and accessibility for use by staff in regional offices.

Take steps to ensure that any future upgrades to local data systems allow QRTs to pause the claims process when errors are detected and enable QRTs to better track error trends.

Take additional steps to evaluate the effectiveness of quality assurance activities to identify opportunities to improve or better target these activities.

Agency Comments and Our Evaluation

We provided a draft of this report to VA for review and comment, and its written comments are reproduced as appendix III in this report. VA generally agreed with our conclusions and concurred with all of our recommendations. The agency outlined how it plans to address our recommendations as follows:

Regarding our recommendations to leverage appropriate expertise to improve its measurement and reporting of accuracy, VA stated that a VBA statistician has begun developing a revised sampling methodology that takes into consideration output and claims processing accuracy at each regional office to determine sample sizes. VBA also plans to appropriately weight accuracy estimates and calculate the margins of error based on the revised sampling methodology. VBA intends to report results based on this new methodology beginning in March 2015.

Regarding our recommendation to take steps to ensure that redistributed claims and those moved between regional offices are not underrepresented in the STAR sample, VA stated that VBA’s revised sampling methodology will be based on the office completing the claim, and that no claims will be excluded from samples due to changes in jurisdiction. VBA intends to implement this revised sampling methodology by the end of March 2015.

Regarding our recommendation to increase transparency in explaining how the claim-based and issue-based accuracy rates are calculated, VA stated that VBA will describe its sampling, assessment criteria, calculation, and reporting methodologies for claim and issue- level accuracy as part of future performance documents and public reports. VBA anticipates implementing this recommendation by the end of March 2015.

Regarding our recommendation to review the multiple sources of policy guidance VBA provides to regional office staff, VA stated that in September 2014, VBA began improving the availability and accessibility of policy guidance, as well as consolidating references to this guidance. VBA anticipates completing this project by the end of April 2015.

Regarding our recommendation to take steps to ensure that any future upgrades to local data systems allow QRTs to pause the claims process when errors are detected and enable QRTs to better track error trends, VA stated that VBA is designing a new database that will incorporate all types of quality reviews (i.e., regional office reviews, STAR, and consistency studies) and provide VBA with more data analysis capabilities. Although VA did not outline specific steps VBA plans to take to upgrade local data systems so that QRTs may pause the claims process, VBA plans to implement this recommendation by the end of June 2015.

Regarding our recommendation to take additional steps to evaluate the effectiveness of quality assurance activities to identify opportunities to improve or better target these activities, VA stated that VBA’s new database will enable VBA to do so by the end of June 2015.

VA also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Veterans Affairs. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope and Methodology

The objectives of this report were to examine (1) the extent to which the Veterans Benefits Administration (VBA) effectively measures and reports the accuracy of compensation claim decision-making, and (2) whether VBA’s other quality assurance activities are coordinated and effective.

Review of Systematic Technical Accuracy Review (STAR)

To assess VBA’s measurement and reporting of the accuracy of compensation claim decision-making, we focused on the STAR process for reviewing disability compensation claims that VBA identifies as rating- related—that is, requiring a decision on the claimant’s eligibility for benefits and the monthly benefit amount. We did not review quality assurance over disability compensation claims that did not involve a rating, including adjustments for additional dependents. We also did not review quality assurance efforts involving appealed cases, aspects of which fall under the Board of Veterans’ Appeals. Finally, we did not review pension claims, which represent a small portion of VBA’s disability benefits workload, because VBA is reviewing its approach to the accuracy assessment of pension claims.

To determine the extent to which STAR appropriately reflects the accuracy of claims, we reviewed VBA policy manuals, the STAR checklist, and other tools used in VBA’s STAR review. We interviewed VBA and Office of Inspector General (OIG) officials to learn whether there are claim types that are omitted from STAR review and, if so, the reasons for these omissions. To determine how errors are identified and counted under STAR, we examined the ways in which the checklist and other STAR procedures are used to quantify errors. We visited VBA’s office in Nashville, Tennessee, where the STAR reviews are conducted to observe the review process and program methodology in action. We reviewed checklists used to assess accuracy of claims and identified information VBA uses on the basis of these checklists to calculate accuracy rates.

To assess the extent to which VBA uses generally accepted statistical practices to generate accuracy rates, we analyzed VBA data on claims processed and reviewed from October 2012 through September 2013. In analyzing STAR data, we calculated the weighted claim-based annual accuracy rate for each regional office and nationwide. We then calculated the 95 percent confidence intervals associated with these estimated accuracy rates. We applied a statistical sample size formula suitable for use in a stratified random sample and analyzed the differences this approach produced compared to VBA’s sample size estimation methodology for regional offices.

We assessed the reliability of VBA’s STAR data by performing electronic data testing, reviewing related documentation, and interviewing knowledgeable agency officials. We also assessed the reliability of VBA’s claim processing data by interviewing knowledgeable agency officials about the data. To electronically assess the reliability of the STAR data, we tested for duplicate benefit records, tested the claim disposition date field to ensure we only analyzed STAR claims from fiscal year 2013, checked the benefit claim end product code to ensure we only included benefit claims with end product codes eligible for inclusion in the STAR accuracy sample, checked for missing data in key analysis variables, and examined the range of values in key variables to check for outliers. We determined that the data were sufficiently reliable for our purposes.

To assess how VBA reports accuracy, we identified and reviewed relevant VBA performance reports, such as VA’s Performance and Accountability Report and Aspire dashboard data. We also interviewed VBA officials about the rationale for creating the issue-based accuracy measure, and the agency’s plans for reporting its performance on accuracy and consistency. We compared VBA practices with legal requirements for agency performance reporting such as the GPRA Modernization Act of 2010 and related GAO work (e.g., GAO, Managing For Results: GPRA Modernization Act Implementation Provides Important Opportunities to Address Government Challenges, GAO-11-617T, Washington, D.C.: May 10, 2011).

Coordination and Effectiveness of Quality Assurance Activities

To determine whether VBA’s quality assurance activities are coordinated and effective, we reviewed VBA quality assurance policies, reports, and guidance to identify key quality assurance activities. Based on this review, we focused on quality review teams (QRT), which are located in each regional office and responsible for local quality assurance, as well as on VBA’s consistency program that is administered by VBA’s centralized quality assurance staff. We then examined each activity’s function and process by reviewing relevant guidance and policy documents and interviewing central office officials. Specifically:

We reviewed VBA policy and procedure documents for quality review teams (QRT) to learn the purposes of, and the information generated by, these efforts. In addition, we interviewed VBA central office and regional office officials to gather their perspectives on any redundancy or gaps between quality assurance efforts. We compared the functions of and information yielded by quality assurance components with the framework laid out in VBA’s Quality Assurance Program Plan, as well as standards for internal control in the federal government (see GAO, Standards for Internal Control in The Federal Government, GAO/AIMD-00-21.3.1, Washington, D.C. : November 1999). In addition, we interviewed VBA regional office officials to learn about processes QRTs follow and how these procedures may vary across regional offices. We also reviewed and compared VBA criteria for QRT staff, STAR reviewer, and claims processor certification.

We reviewed documents and interviewed VBA officials to learn more about the recent changes to the agency’s approach to assessing consistency. More specifically, we explored the rationale for the change from using inter-rater reliability (IRR) studies to using consistency questionnaires. We assessed the development and implementation of the recent consistency questionnaires by, for example, examining VBA’s consideration of pre-testing the instruments using generally accepted survey procedures, and how pre-testing may affect the resulting measures of consistency. Finally, to further determine how consistency questionnaires are complementary with other quality assurance efforts, we reviewed VBA’s process for determining topics for consistency questionnaires. Specifically, we asked about the methods used to select and prioritize topics, including the extent to which officials use findings from QRTs and STAR.

To further determine what and how information is shared among quality assurance components and how this coordination helps to identify problem areas, we interviewed VBA regional office officials to gather their perspectives on how information is shared from STAR, QRT, consistency studies, and regional office compliance visits and how that information- sharing could be improved. We interviewed officials at the regional level to gain their perspectives on coordination and effectiveness of all of VBA’s quality assurance activities. At each office, we spoke with service center managers and quality assurance staff, as well as representatives of local veteran service organizations. The regional offices were selected to reflect a range of characteristics related to: (1) geography (at least one regional office in each of VA’s four areas), (2) number of claims processed annually, (3) claim-based accuracy rates, and (4) issue-based accuracy rates. We did not identify specific quality assurance pilots or initiatives being tested in regional offices. We selected 4 of VBA’s 57 regional offices for review. We visited the Oakland and Newark regional offices and conducted telephone interviews with Nashville and Waco regional office staff. Table 1 provides information about the regional offices we selected to visit.

Appendix II: Statistical Sampling Methodology

This appendix provides additional technical details on ratio estimation for producing issue-based accuracy rates, as well as the audit work we did to re-estimate the regional office Systematic Technical Accuracy Review (STAR) sample sizes using a formula for stratified random probability samples.

Ratio Estimation

Because STAR is designed to sample claims and produce an estimate of the claim-based accuracy rate and because the number of medical issues per claim varies, ratio estimation should be used to develop issue-based accuracy rates. Furthermore, during their review of sampled claims, STAR reviewers may find that one or more inferred issues were missed or, conversely, that the review process included one or more issues inappropriately. Thus, the STAR sample of claims must be used to estimate both the total number of issues as well as the number of issues that were processed correctly. With respect to STAR, ratio estimation takes the form shown below.

In the formula, the subscript i represents the regional office, the subscript j represents the month of the fiscal year, jin , represents the monthly sample size for regional office i in month j, jiW , represents the stratum sampling weight for regional office i in month j, kjia , number of issues adjudicated correctly on claim k in month j and regional office i and represents the total number of issues on claim k in kjim , month j and regional office i.

The ability to calculate a ratio estimate and its associated confidence interval are available in most statistical software applications.

Sample Size Re-Estimation

Each month the Veterans Benefits Administration (VBA) selects a random sample of benefit claims within each VA regional office to review under the STAR program. The measure of interest is the estimated percent of claims that were processed correctly by VBA regional office staff. The sample size formula used by VBA to derive the number of claims to select in each VBA regional office is shown below.

In the formula, Z = the quantile from the Normal distribution for the desired level of confidence. The desired margin of sampling error is denoted by E. The assumed percent of accuracy in the population is denoted by P, and Q is defined as Q = (1 – P). For their calculations, VBA uses the following values: .

When these values are plugged into the equation, n = 246. This is VBA’s target annual sample size for each VA regional office. With 57 regional offices, this translates into 14,022 claims selected nationally per fiscal year in the STAR sample. On a monthly basis, when divided by 12, 246/12 = 20.5 which rounds up to 21. Thus, VBA’s monthly sample size for each regional office is 21 claims. By definition, the sample frame for each month is the set of veteran benefit claims completed by the regional office within the previous month.

The standard statistical formula for the sample size calculation with a stratified random sample is shown below. We applied this formula to determine an annual total sample size for a regional office in the coming fiscal year using observed monthly accuracy rates and monthly number of claims completed from the previous fiscal year. .

In turn, this initial sample size is adjusted with the finite population correction factor. The formula for the adjusted sample size is shown below.

Appendix III: Comments from the Department of Veterans Affairs

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Michele Grgich (Assistant Director), Dana Hopings (Analyst-In-Charge), Carl Barden, James Bennett, David Chrisinger, Alexander Galuten, Joel Green, Avani Locke, Vernette Shaw, Almeta Spencer, Walter Vance, and Greg Whitney made key contributions to this report.