Veterans Health Administration: Past Performance System Recommendations Have Not Been Implemented
Released: 2019-04-30T00:00:00

Letter

Background

Medical Center Performance: SAIL

VHA began using the SAIL system in 2012 to measure, evaluate, and benchmark the quality, efficiency, and productivity of medical centers, and to highlight successful strategies of high-performing medical centers. SAIL includes 29 performance measures (27 quality measures and two measures of overall efficiency and capacity) in areas such as acute-care mortality, access to care, and employee satisfaction. (See appendix I for the full list of SAIL measures.) SAIL is a diagnostic tool that allows VHA to assess medical centers’ performance relative to their peers, and determine how much absolute improvement they have made in the past year based on relevant clinical data. VHA publishes SAIL results quarterly to provide information to network and medical center officials regarding improvement opportunities at each medical center. SAIL data are also available on VHA’s intranet site. VHA staff can view a wide range of detailed information about their medical center, compare performance to other medical centers, and (for those staff with medical-record-level access) view information on patients with a particular medical condition.

Network and Medical Center Director Performance Appraisal Process

VHA conducts annual performance appraisals for all network and medical center officials. The appraisal process begins when officials from VHA’s office of Workforce Management and Consulting transmit a performance plan template to the network directors. The template identifies performance priorities and expectations for the upcoming appraisal period and criteria to be used to measure performance outcomes and ratings for each performance element. Network directors use the template to develop performance plans that include targets and time frames—the schedule of when performance targets are to be achieved during the year—with each of the medical center directors in their network. According to VA policy, performance plans resulting from the template should be finalized within 30 days of the start of the appraisal period. After expectations have been set for a medical center director, the director, in turn, sets performance expectations for the department heads within the medical center.

VHA Primarily Uses SAIL and Its Associated Star Ratings to Assess and Manage Medical Center Performance

VHA Primarily Uses SAIL and Its Star Ratings to Assess Medical Center Performance

VHA officials told us they primarily use the SAIL system to assess the performance of medical centers. Specifically, VHA uses SAIL data to calculate and assign each medical center an annual star rating of 1 (lowest) to 5 (highest) stars as an assessment of overall quality. SAIL documentation states that the goal of the star ratings is for low-performing medical centers to learn from the best practices of high-performing ones, although all medical centers have the opportunity to improve. VHA applies a weighting and calculation methodology to each of SAIL’s 27 quality measures to determine a single composite score for each medical center. The scores are then ranked and grouped by percentile and the associated medical centers are assigned initial star ratings based on their relative ranking. For example, the lowest performing 10 percent of medical centers as determined by SAIL’s 27 quality measures are assigned a 1-star rating, and the next lowest performing 20 percent of medical centers are assigned a 2-star rating. (See fig. 1.)

After the initial star rating is determined by SAIL measures each year, VHA officials can make changes to the rating if a medical center meets certain conditions. For example, SAIL documentation states that a medical center that initially received a 5-star rating will be reduced to a 4- star rating if it has a high mortality rate. In addition, VHA officials told us they can decide to increase a 1-star medical center’s rating to a 2-star rating if the medical center outperforms the bottom 10 percent of U.S. hospitals in certain criteria as measured by external systems such as the Centers for Medicare & Medicaid Services’ Hospital Compare website.

We found that the percentage of medical centers that received a final 1- star rating ranged from 4 percent to 10 percent from fiscal years 2013 through 2018. VHA officials publish the final annual star ratings for each medical center both internally and externally. See figure 2 for the number of medical centers that received each final star rating for fiscal years 2013 through 2018.

Although the specific medical centers within each star-rating category could change from year to year, we found that the fiscal year 2018 star ratings for 110 of the 127 medical centers (87 percent) that received star ratings in fiscal year 2013 did not differ by more than 1 star from their fiscal year 2013 star rating. For example, eight of the 10 1-star medical centers in fiscal year 2013 received either a 1- or 2-star rating in fiscal year 2018. (See fig. 3.) In addition, 44 of the 127 medical centers had the same rating in fiscal year 2018 as they did in fiscal year 2013. At the end of the 6-year period of our review, only one medical center differed by more than 2 stars from its fiscal year 2013 star rating, decreasing from 5 stars to 2.

VHA Uses Tools from the SAIL System to Manage Medical Center Performance

VHA officials told us they use SAIL tools on VHA’s intranet when conducting site visits to medical centers and for other performance management efforts. The SAIL system includes several performance management tools that present data in greater detail than SAIL’s quarterly data release and enable officials to identify areas for improvement. VHA, network, and medical center officials we interviewed mentioned three in particular:

Opportunity matrix – This matrix shows how a medical center ranks compared to others on all SAIL performance measures based on quarterly data. Each performance measure is labeled by quintile, with the first quintile comprising the top 20 percent of medical centers and the fifth quintile comprising the bottom 20 percent. Officials told us they use this tool to focus improvement efforts by examining specific measures for which a medical center needs improvement.

Geometric control charts – These charts, referred to as G-Charts, allow officials to monitor on a daily basis what VHA considers to be rare occurrences. For example, one G-Chart allows VHA to monitor patient safety indicators that contain information on occurrences of specific medical conditions, such as cardiac arrest, pneumonia, and sepsis. Medical center officials can use these charts to examine the occurrence of events over time, analyze patient-level data, and quickly detect changes in the frequency of these events. Other events that the charts allow VHA to monitor include inpatient complications and deaths.

Symphony action triggers – Symphony is an online tool that tracks over 100 performance measures daily, related to medical center access, outcomes, and productivity, and includes an early warning system to notify network and medical center officials of results that require attention. Officials can use Symphony to view patient-level information to understand the details of particular events and determine solutions.

VHA officials also told us that they use these tools to manage medical center performance as part of their ongoing support of lower performing medical centers. Specifically, officials who oversee SAIL identify lower performing medical centers using SAIL and conduct site visits as part of VHA’s Strategic Action for Transformation initiative. This initiative utilizes a four-tiered, escalating approach based on the severity of concern at a medical center. In order of increasing severity, the four levels are watch, high-risk, critical, and VA receivership. One-star medical centers are automatically placed on the high-risk list, along with some 2-star medical centers with decreasing performance. If performance continues to decrease, medical centers are considered critical, and can be escalated to VA “receivership,” at which point VHA officials may step in to correct ongoing problems and replace network or medical center leadership officials. As of January 2019, VHA officials told us no medical center had entered VA receivership since the initiative began. VHA officials told us that they may also conduct site visits or hold calls with medical center leadership by request, although their focus is on lower performing medical centers.

In addition to the SAIL tools, which report data on performance measures across the entire medical center, VHA officials told us that they may also use other data sources as part of medical center performance management. For example, several program offices—such as primary care, mental health, and surgery—have dashboards that track performance and quality of care specific to those offices. In addition, VA’s Inpatient Evaluation Center focuses on mortality data, including estimates of expected patient mortality.

VHA’s Appraisal Process for Assessing Network and Medical Center Directors’ Performance Relies Heavily on Medical Center Performance Information

We found that VHA relies heavily on medical center performance information to assess the performance of its network and medical center directors. VA’s Senior Executive Service Part V. Performance Appraisal System handbook states that directors are assessed using five appraisal elements established by the Office of Personnel Management: (1) Results Driven, (2) Leading People, (3) Leading Change, (4) Business Acumen, and (5) Building Coalitions. The five elements are included in VHA’s performance plan template, which forms the basis for network and medical center directors’ performance plans. The handbook designates a relative weight for each element used to calculate a director’s rating. (See fig. 4.) The handbook states that a director is rated in each element on a scale of level 1 to level 5, with 5 being the highest level. Each rating is then multiplied by the weight for its corresponding element, and the results are added to generate a summary score. According to the handbook, the summary score is used to identify potential recipients of pay increases and monetary awards.

The most heavily weighted appraisal element in the handbook, Results Driven, represents 40 percent of a director’s overall performance and is based entirely on medical center performance information. Specifically, for fiscal year 2018, SAIL results comprised 25 percent of the overall rating and included measures such as patient mortality, length of stay, and readmissions. Other medical center performance information comprised the remaining 15 percent of the overall rating. (See fig. 5.)

Medical center performance information is also used when assessing directors’ performance across other appraisal elements. For example, in VHA’s fiscal year 2018 performance plan template, the Leading Change appraisal element included the implementation of suicide prevention initiatives, using medical center performance in the SAIL mental health domain as criteria. In addition, the Leading People element included performance information from VA’s All Employees’ Survey, which included medical center staff.

Although medical center performance information plays a prominent role in the performance assessment process, VHA officials told us that there are other considerations that may result in medical center directors receiving a rating that is higher than that indicated by the star rating of the medical center. For example, VHA officials told us that when calculating a medical center director’s rating for the Results Driven element, they consider whether the medical center’s overall performance improved or deteriorated compared to the previous year’s performance. These officials also stated that they take into consideration the length of a director’s tenure, such as cases where a director started at a low- performing medical center partway through the rating year and would not have had sufficient time to improve the medical center’s performance from the previous year.

In our review, we also found that the release of VHA’s performance plan template is often delayed, which can limit its effectiveness as a means of assessing directors’ performance. Specifically, in fiscal years 2016, 2017, and 2018, VHA released the performance plan template to network directors in November or December, close to the end of the first quarter of the performance appraisal period. Directors at two of the medical centers in our review expressed frustration with the delay and not having a full year to meet performance expectations, but directors at the two other medical centers stated that they find the process clear and are able to anticipate performance expectations. Officials from VHA’s office of Workforce Management and Consulting, which sends out the template, told us that they have been working in recent years to shorten the template’s development and review process within VHA; however, the delays may continue because of late changes from VA or the Office of Personnel Management. In our December 2016 review of human resource management practices at VHA, we also reported on delays in the release of VHA’s performance plan template. We reported that the delay limited medical center officials’ ability to use the template as a tool to align expectations and performance, which is inconsistent with leading practices on employee performance management. We recommended that VHA accelerate its efforts to develop a modern, credible, and effective performance management system, including the timely release of the performance plan template. VA partially concurred with our recommendation and has made limited progress in implementing it. As of December 2018, this recommendation remains open and we reiterate the need for VHA to implement it.

VHA Has Not Assessed for Implementation Previous Recommendations Made to Ensure SAIL’s Effectiveness in Assessing Medical Center Performance

Although SAIL is used in the assessment of both medical centers’ and directors’ performance, VHA officials have not assessed and implemented as appropriate the recommendations from previous evaluations of the SAIL system to ensure its effectiveness. This is inconsistent with federal standards for internal control, which state that management should remediate identified internal control deficiencies on a timely basis. This remediation may include assessing the results of reviews to determine appropriate actions, and, once decisions are made, completing and documenting corrective actions on a timely basis.

VHA officials told us that since it was established in 2012, there have been two evaluations of SAIL:

The first evaluation was an internal review, which VHA officials told us was completed in February 2014 and submitted to the director of VHA’s Office of Analytics and Business Intelligence and reviewed by the then Under Secretary for Health and Principle Deputy Under Secretary for Health. The internal review, which had 22 recommendations, found issues related to the validity and reliability of SAIL as a tool for measuring performance and fostering accountability. For example, it included a recommendation that VHA no longer use aggregate star ratings for accountability, or for presenting medical center quality and efficiency information to stakeholders. Rather, the recommendation called for VHA to work to identify valid and reliable approaches for presenting this information.

The second evaluation was an external review, which VHA officials told us was submitted to the Office of the Principal Deputy Under Secretary for Health in April 2015. The external review included 19 recommendations for short- and long-term improvements to SAIL, such as a recommendation to examine the potential for misclassifying medical centers—i.e., assigning star ratings that do not reflect medical centers’ pattern of performance on the underlying measures. The review noted two ways such misclassification could occur: (1) two medical centers with summary scores that are close together could receive different star ratings, or (2) two medical centers with widely different summary scores could receive the same star rating.

The findings of the previous SAIL evaluations are similar to concerns that officials from the four networks and four medical centers in our review expressed about SAIL’s effectiveness, including whether the star ratings were an accurate reflection of medical center performance. For example, officials from one medical center told us that, because the mortality measure has a higher weight relative to other SAIL measures, it can amplify the importance of a small difference between medical centers. As a result, a 1-star medical center may appear to be performing much more poorly on this measure than it is in practice; and officials from two medical centers told us that the length-of-stay measure may not be an accurate reflection of quality of care, as there are valid clinical reasons why some veterans need a longer length of stay that may not be reflected in the diagnostic and procedure codes for that stay. Therefore, the difference in performance on the length of stay measure between two medical centers may be the result of how data were entered into the medical record and coded, rather than actual differences in quality of care.

In addition, VHA officials also expressed concerns about SAIL and how it is currently being used to assess medical center performance. For example, VHA officials who oversee SAIL told us it was designed to be an internal performance improvement tool, but is now also being used as a performance accountability tool. The external review included a recommendation that VHA consider whether the primary purpose of SAIL is improvement or accountability, as SAIL would need to be redesigned to do both. One VHA official told us that SAIL is being used in punitive ways through the Strategic Action for Transformation initiative. For example, at one medical center, officials told us that they received a letter from VHA’s Executive in Charge about the medical center’s low performance only a few months after its star level increased from 1 to 2 stars. Officials said the letter warned them that medical center leadership could be removed if performance does not improve. Medical center officials described this as counterproductive for their improvement efforts, as it was demoralizing while not identifying any specific areas for improvement.

VHA officials confirmed that, other than their routine reviews to determine the need for annual adjustments to SAIL measures and other minor adjustments to the system, they have not assessed or implemented as appropriate the recommendations from the internal and external SAIL evaluations. In addition, although the Under Secretary for Health received a response to the internal review’s recommendations from an individual program office, VHA officials told us no action was taken on the response or to formally assess the recommendations from the internal review. Officials noted that two reasons for the lack of action taken to assess recommendations for implementation were leadership turnover and attention diverted to other issues, such as concerns about extended wait times for medical appointments at VHA medical facilities. In addition, officials stated that the evaluations were not widely distributed within VHA. As a result, officials we spoke with from several VHA offices were unaware that SAIL had ever been evaluated. To address the federal internal control standard for timely remediation of identified deficiencies, federal agencies assign responsibility and authority for carrying out and documenting corrective actions. VHA officials told us they did not formally assign responsibility to an office to assess recommendations from previous evaluations of SAIL. As a result, when the officials who received both evaluations left VHA, there were no other individuals or offices responsible for ensuring that recommendations were acted on.

VHA officials who oversee SAIL told us that they are planning to use the 2015 external review as part of their plans to make changes to SAIL and its measures. However, there is no documentation available describing the planned changes to SAIL or how those planned changes will incorporate the results of the external review. If changes made to SAIL run counter to the evidence, it could potentially diminish the integrity of the system to effectively evaluate performance.

Conclusions

VHA primarily uses the SAIL system to assess and compare the performance of medical centers. Veterans can also view SAIL data to compare medical center performance when making health care decisions. Officials from the networks and medical centers in our review expressed concerns about how SAIL is being used and whether star ratings are an accurate reflection of medical center quality. SAIL has been evaluated twice, and both evaluations have found similar concerns with SAIL. However, VHA has yet to use the results of those evaluations to address identified concerns and make evidence-based improvements to the SAIL system. Specifically, VHA has not taken action to ensure that officials assess the recommendations from SAIL evaluations, document their decisions, and implement recommendations as appropriate. If changes to SAIL are implemented without this assessment of existing evaluations, VHA may make changes that run counter to the evidence, potentially diminishing the integrity of the system to effectively evaluate performance.

Recommendations

We are making the following two recommendations to VA:

The Under Secretary for Health should assess recommendations from two previous evaluations of SAIL. This assessment should include the documentation of decisions about which recommendations to implement and assignment of officials or offices as responsible for implementing them. (Recommendation 1)

The Under Secretary for Health should implement, as appropriate, recommendations resulting from the assessment of the two previous SAIL evaluations. (Recommendation 2)

Agency Comments

We provided VA with a draft of this report for review and comment. VA provided written comments, which are reprinted in appendix II. In its written comments, VA concurred with both of the report’s recommendations, and identified actions it is taking to implement them.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, the Under Secretary for Health, and other interested parties. In addition, the report is available at no charge on the GAO web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: VHA Strategic Analytics for Improvement and Learning (SAIL) Performance Measures

Number of measures

2 Risk adjusted complication Index In‐Hospital risk adjusted mortality (SMR) Measure 30‐day risk adjusted mortality (SMR30)

Desired Direction of Measure Lower 3 Severity adjusted average length of stay (ALOS) %Acute admission reviews met InterQual criteria %Acute continued stay reviews met InterQual criteria Inpatient core measures mean percentage (ORYX)

HEDIS outpatient core measure mean percentage (chart abstract)

HEDIS outpatient core measure mean percentage (population based) 5 HCAHPS score (patient rating of hospital)

Rating of primary care provider Rating of specialty care provider Care Transition (inpatient)

Stress discussed (PCMH) 2 Best Places to Work score Hospital-wide all conditions 30-day readmission rate 5 Timely Appointment, Care and Information – PCMH Timely Appointment, Care and Information – SC Same Day Appointment When Needed – PCMH Call center speed in picking up calls 3 Mental health population coverage Mental health continuity of care Mental health experience of care 2 Stochastic frontier analysis (= 1/SFA)

The acronyms VHA used in the table are as follows: SMR=standard mortality ratio; HEDIS=Healthcare Effectiveness Data and Information Set; HCAHPS= Hospital Consumer Assessment of Healthcare Providers and Systems; PCMH=patient-centered medical home; ACSC=ambulatory care sensitive conditions; SC=specialty care.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Debra A. Draper, (202) 512-7114 or draperd@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Janina Austin, Assistant Director; Sarah Harvey and Malissa G. Winograd, Analysts-in-Charge; Jennie F. Apter; Frederick Caison; and Alexander Cattran made key contributions to this report. Also contributing were Vikki Porter and Jennifer Whitworth.

Related GAO Products

VA Health Care: Actions Needed to Improve Oversight of Community- Based Outpatient Clinics. GAO-18-375. Washington, D.C.: Apr. 12, 2018.

Veterans Health Care: Additional Actions Could Further Improve Policy Management. GAO-17-748. Washington, D.C.: Sept. 22, 2017.

Veterans Affairs: Improved Management Processes Are Necessary for IT Systems That Better Support Health Care. GAO-17-384. Washington, D.C.: June 21, 2017.

VA Health Care: Improvements Needed in Data and Monitoring of Clinical Productivity and Efficiency. GAO-17-480. Washington, D.C.: May 24, 2017.

Veterans Health Administration: Management Attention Is Needed to Address Systemic, Long-standing Human Capital Challenges. GAO-17-30. Washington D.C.: Dec. 23, 2016.

Veterans Health Care: Improvements Needed in Operationalizing Strategic Goals and Objectives. GAO-17-50. Washington D.C.: Oct. 21, 2016.

VA Health Care: Processes to Evaluate, Implement, and Monitor Organizational Structure Changes Needed. GAO-16-803. Washington D.C.: Sept. 27, 2016.