VA Dialysis Pilot: Documentation of Plans for Concluding the Pilot Needed to Improve Transparency and Accountability
Released: 2014-09-02T00:00:00

Letter

Background

VA set four goals for the Dialysis Pilot: (1) improve the quality of dialysis care veterans receive, (2) increase veterans’ access to dialysis care, (3) provide additional dialysis research opportunities, and (4) achieve cost savings for VA-funded dialysis treatments.

VA Dialysis Pilot Evaluation

VA is performing an internal evaluation of the performance of the four pilot locations by assessing the quality of care—including clinical quality, patient satisfaction, and access to care—and treatment costs at each pilot location. Results of this evaluation are reported in an annual performance review produced by VA each calendar year. To date, VA has completed reviews for calendar years 2012 and 2013.

Clinical quality: To evaluate the clinical quality at the pilot locations, VA selected 11 clinical quality measures that were derived from measures and associated performance standards established by the Centers for Medicare & Medicaid Services (CMS)—the federal entity with primary responsibility for evaluating the quality of dialysis care in the United States—and endorsed by the National Quality Forum, a leader in evaluating clinical performance measurement for a variety of chronic clinical conditions.

Patient satisfaction: To evaluate veterans’ satisfaction with the quality of care provided at the pilot locations, VA analyzed the CMS standard dialysis facility survey—the In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems—for the veterans treated at pilot locations.

Access to care: To evaluate veterans’ access to the pilot locations, VA is measuring the percentage of veterans whose actual travel distances are no more than 30 miles from their residence to the pilot location where they receive care.

Treatment cost: To evaluate the cost of care, VA is comparing the pilot locations’ average cost per treatment and the average cost per treatment for non-VA dialysis providers in each pilot location’s Veterans Integrated Service Network (VISN). To calculate each pilot location’s average cost per treatment, VA divided each pilot location’s total health care costs by its total number of treatments. To calculate non-VA dialysis providers’ average cost per treatment, VA created a composite figure using the average per treatment contract rate from the two largest non-VA dialysis providers located in each pilot location’s VISN, a per treatment VA claims processing fee, a per treatment VA physician cost for nephrology oversight, and a per treatment VA financial administrative cost.

Dialysis Pilot Evaluation Contractor Review

In September 2012, VA awarded a 3-year contract to an evaluation contractor to provide an independent evaluation of its Dialysis Pilot— including a comparison of the clinical quality provided by the four pilot locations with non-VA dialysis providers and a dialysis per treatment cost comparison between the pilot locations and the Non-VA Medical Care Program.

Clinical quality: To evaluate the clinical quality of care provided by the pilot locations, the evaluation contractor is assessing the results of several measures—such as mortality, hospitalization, and infection rates—for the pilot locations and comparing them to relevant state and national results. These results are presented for each pilot location in quarterly quality reviews and an annual dialysis facility report (DFR), a standard report produced by the evaluation contractor for CMS for each of the more than 6,000 private sector dialysis facilities nationwide.

Treatment cost: To evaluate the treatment costs for each pilot location, the evaluation contractor is comparing the pilot locations’ average cost per treatment to that of private sector dialysis providers, including certain contracted non-VA dialysis providers. Results are presented for each pilot location in quarterly cost reviews. The evaluation contractor used a similar methodology as VA to determine each pilot location’s average cost per treatment. To determine the non-VA dialysis provider average cost per treatment in calendar year 2012, the evaluation contractor created a composite figure that included (1) the contract per treatment average price for the three non-VA dialysis providers with the largest market share in the VISN responsible for the Raleigh and Fayetteville pilot locations, (2) VA’s per treatment travel-related costs, and (3) VA’s per treatment medical oversight costs.

VA Has Not Documented Criteria or Plans for Concluding the Dialysis Pilot

Even though it is 5 years into the Dialysis Pilot, VA Central Office has not yet set a timetable for completing the pilot or documented how it will determine the success of the four pilot locations. We have previously outlined best practices for project scheduling that specify project timelines are critical for managing and measuring an agency’s performance on projects and that it is necessary for the agency to have a document that details the rationale used in developing a project timeline. In addition, we have outlined best practices for designing evaluations that state that choosing and documenting well-regarded criteria that are used to make comparisons can lead to strong, defensible conclusions.

According to VA’s business analysis of the Dialysis Pilot conducted prior to opening the pilot locations, VA planned to conclude the Dialysis Pilot 5 years after the pilot locations opened and assumed all four pilot locations would begin operations at generally the same time. Due to delays in opening the pilot locations, the completion of this 5-year period would be in fiscal year 2018—5 years after the opening of the fourth pilot location. However, in March 2014, VA officials told us they are no longer operating under the 5-year timeline outlined in this plan, although they have not updated the timeline included in the Dialysis Pilot business analysis. VA officials told us that instead of ending the Dialysis Pilot based on the timeline outlined in VA’s business analysis, they believe the Dialysis Pilot will be successful and should conclude once the pilot locations have achieved the following:

Creation of a replicable model. According to VA officials, the first achievement that indicates success is the creation of a model for a VA-operated, free-standing dialysis clinic that can be replicated by other VAMCs. Such a model would include clear documentation of the necessary staff, equipment, time, and resources required to establish a new clinic and serve as a guide to other VAMCs seeking to establish similar clinics. VA officials did not specify a timeline for the creation of this model; however, they did state that they believe the pilot locations have already achieved this milestone.

Pilot location breakeven points. VA officials stated that the second achievement that indicates success is the confirmation of the time necessary for a pilot location to reach its breakeven point—which VA has defined as the point when a pilot location achieves cost savings in the Non-VA Medical Care Program for dialysis services for its sponsoring VAMC and repays its start-up funding to VA Central Office. Through the Dialysis Pilot, sponsoring VAMCs for the pilot locations were each provided about $2.5 million in start-up funding to establish a VA-operated, free-standing dialysis clinic and were to finish repaying this start-up funding to VA Central Office in fiscal year 2014. According to VA Central Office officials, all four pilot locations have repaid their start-up funding as of the beginning of fiscal year 2014 and two pilot locations—Raleigh and Fayetteville—have demonstrated that their cost per treatment is lower than comparable per treatment costs of non-VA dialysis providers in their areas. Achieving this breakeven point indicates that the pilot locations are no longer incurring additional costs for VA and are fulfilling VA’s goal of providing dialysis treatments to veterans at a lower cost than similar care provided through non-VA dialysis providers.

While VA officials indicated they could conclude the Dialysis Pilot once the pilot locations had created a replicable model and realized their breakeven points, VA has not formally communicated these achievements in writing as criteria for concluding the pilot. Moreover, these criteria were not included in the performance work statement for the evaluation contractor, and according to a VA official, have not been formally recorded in any official document. By not clearly communicating these milestones as criteria for concluding the Dialysis Pilot, the transparency of VA’s management decisions on pilot location outcomes is compromised and the Department lacks accountability for ensuring the success of the Dialysis Pilot.

Current Data on Quality of Care and Treatment Costs Are Limited Due to Ongoing Evaluations

VA and the evaluation contractor have been evaluating the Dialysis Pilot for 2 years; however, neither has concluded its evaluation of the Dialysis Pilot and data on two of the pilot locations—Philadelphia and Cleveland— is limited due to their delayed opening. Table 1 shows the data available for each pilot location as of June 2014.

To date, VA has produced two annual performance reviews on quality of care and treatment costs for the Dialysis Pilot—the first for calendar year 2012 and the second for calendar year 2013—that include the results of pilot locations’ clinical quality, patient satisfaction, access to care, and treatment cost performance. The VA calendar year 2012 annual performance review included results for the Raleigh and Fayetteville pilot locations only. VA included results for all four pilot locations in its calendar year 2013 annual performance review; however, this review does not include a full year of data for the Cleveland pilot location and VA noted operational challenges that affected the data for the Philadelphia pilot location. VA noted several contributing factors to the limited data available on the Cleveland and Philadelphia pilot locations, including the following:

The Cleveland pilot location experienced issues with laboratory data in veterans’ VA medical records that limited the results on one of the clinical quality measures in 2013.

For both the Cleveland and Philadelphia pilot locations, VA reported that the dialysis machines did not operate properly, which resulted in a lower than expected number of veterans being treated at the Philadelphia pilot location between January and June 2013 and delayed the opening of the Cleveland pilot location until July 2013.

As a result, VA recommended that another full 12 months of data from the Philadelphia and Cleveland pilot locations be analyzed before making conclusive comparisons of the treatment costs of these pilot locations with non-VA dialysis providers. In addition, the evaluation contractor has reviewed the Dialysis Pilot for 2 years and has one year remaining on its contract. At the time of our analysis, the evaluation contractor had completed 2 years of evaluations on the quality of care provided by the four pilot locations and 1 year of treatment cost evaluations for two of the pilot locations. For information on VA’s clinical quality review of pilot locations, see appendix I; VA’s patient satisfaction review for pilot locations, see appendix II; VA’s access to care review for pilot locations, see appendix III; the evaluation contractor’s clinical quality review of pilot locations, see appendix IV; and VA’s and the evaluation contractor’s reviews of pilot location average cost per treatment, see appendix V.

Conclusions

VA has not documented its plan for the conclusion of the Dialysis Pilot, despite beginning pilot development 5 years ago. Without a formally established project timeline, VA cannot effectively monitor performance against specified timeframes, validate and defend the timeline of the Dialysis Pilot to VA decisionmakers and other stakeholders, or ultimately hold VA decisionmakers accountable for future resource investment decisions. In addition, the transparency of the Dialysis Pilot is jeopardized by VA not clearly documenting the achievements stated by VA officials— the creation of a replicable model for additional VA-operated free- standing dialysis clinics and demonstrating a breakeven point—as criteria for concluding the Dialysis Pilot.

Recommendation

To improve the transparency and accountability of the Dialysis Pilot, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to document plans for concluding the Dialysis Pilot, including establishing an end date or documenting criteria for deciding what constitutes the successful completion of the pilot.

Agency Comments and Our Evaluation

VA provided written comments on a draft of this report, which we have reprinted in appendix VI. In its comments, VA generally agreed with our conclusions and concurred with our recommendation to improve the transparency and accountability of the Dialysis Pilot. VA also stated that in July 2015—2 years after the Cleveland pilot location became operational—it plans to review the outcomes from the four pilot locations and make recommendations for the Dialysis Pilot. VA did not provide any technical comments.

In its general comments, VA noted that the criteria for deciding what constitutes the successful completion of its Dialysis Pilot is included in the department’s document entitled, “Evaluation Plan: VA Free-Standing Dialysis Centers,” which VA enclosed with its comments. While this document describes the measures VA is using to evaluate the pilot, it does not delineate the amount of data or length of review that is needed in order to conclude the pilot. In addition, VA did not clearly state when recommendations will be made and whether the pilot will end at that time. We continue to believe that VA should clearly document its plan for concluding the Dialysis Pilot by clearly specifying an end date for the pilot or specific outcomes that must be met.

We are sending copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Appendix I: Results of Clinical Quality Review by the Department of Veterans Affairs

This appendix provides information on the Department of Veterans Affairs’ (VA) results from its review of Dialysis Pilot location clinical quality. To assess clinical quality, VA selected and evaluated Dialysis Pilot locations using 11 performance measures established by the Centers for Medicare & Medicaid Services (CMS). For calendar years 2012 and 2013, CMS defined the lowest achievement threshold for facility performance as the 15th percentile, the average performance standard as the 50th percentile, and the highest benchmark as the 90th percentile. Specifically, table 2 provides results of VA’s review of clinical quality at the four Dialysis Pilot locations—Raleigh and Fayetteville, North Carolina; Philadelphia, Pennsylvania; and Cleveland, Ohio.

Appendix II: Results of Patient Satisfaction Review by the Department of Veterans Affairs

This appendix provides results from the Department of Veterans Affairs’ (VA) review of patient satisfaction at its two more established Dialysis Pilot locations—Raleigh and Fayetteville, North Carolina. VA did not analyze patient satisfaction for the newer pilot locations—Philadelphia, Pennsylvania, and Cleveland, Ohio—because these pilot locations were either not open or not operating at full capacity during the periods of VA’s analysis.

Table 3 provides results for VA’s review of the Raleigh, North Carolina, Dialysis Pilot location’s patient satisfaction.

Table 4 provides results for VA’s review of the Fayetteville, North Carolina, Dialysis Pilot location’s patient satisfaction.

Appendix III: Results of Access to Care Review by the Department of Veterans Affairs

This appendix provides results from the Department of Veterans Affairs’ (VA) review of veterans’ access to care at its two more established Dialysis Pilot locations—Raleigh and Fayetteville, North Carolina. In calendar year 2012, VA did not analyze veterans’ access to care for the newer pilot locations—Philadelphia, Pennsylvania, and Cleveland, Ohio— because these pilot locations were either not open or not operating at full capacity during the periods of VA’s analysis.

Table 5 provides results for VA’s review of veterans’ access to care at the Raleigh and Fayetteville pilot locations for calendar year 2012.

Table 6 provides results for VA’s review of veterans’ access to care at all four pilot locations for calendar year 2013.

Appendix IV: Results of Clinical Quality Review by the Evaluation Contractor

This appendix provides results from the evaluation contractor’s review of clinical quality for three of the Department of Veterans Affairs’ (VA) Dialysis Pilot locations in Raleigh and Fayetteville, North Carolina, and Philadelphia, Pennsylvania. VA contracted with a leading university research center to evaluate the Dialysis Pilot. The evaluation contractor’s review of the Philadelphia Dialysis Pilot location was limited due to operational difficulties at the pilot location in calendar year 2012. The evaluation contractor did not complete a review of the Cleveland, Ohio, Dialysis Pilot location due to insufficient data resulting from the delayed opening of this pilot location.

Table 7 provides results for the evaluation contractor’s review of the Raleigh and Fayetteville, North Carolina, Dialysis Pilot locations’ clinical quality.

Table 8 provides results for the evaluation contractor’s review of the Philadelphia, Pennsylvania, Dialysis Pilot location’s clinical quality.

Appendix V: Results of Treatment Cost Reviews by the Department of Veterans Affairs and the Evaluation Contractor

Table 9 provides results from the Department of Veterans Affairs’ (VA) and the evaluation contractor’s reviews of the average cost per treatment for Dialysis Pilot locations in Raleigh and Fayetteville, North Carolina, Philadelphia, Pennsylvania, and Cleveland, Ohio, and non-VA dialysis providers for calendar year 2012 and calendar year 2013. VA contracted with a leading university research center to evaluate the Dialysis Pilot.

Appendix VI: Comments from the Department of Veterans Affairs

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Marcia A. Mann, Assistant Director; Cathleen Hamann; Katherine Nicole Laubacher; Said Sariolghalam; and Teresa Tam made key contributions to this report. Jacquelyn Hamilton provided legal support.