VA Dialysis Pilot: Documentation of Plans for Concluding the Pilot Needed to Improve Transparency and Accountability
Published: 2014-09-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Veterans with end-stage renal disease—a condition of permanent kidney failure—are one of the most resource-intensive patient populations served by the VA and are generally prescribed a life-saving medical procedure called dialysis. In 2009, VA began developing a pilot program at four VAMCs to provide dialysis to veterans in VA-operated, free-standing dialysis clinics largely in an effort to stem rising costs for providing such care in the private sector through the Non-VA Medical Care Program. In May 2012, GAO issued a report identifying several weaknesses in VA's execution of the planning and early implementation phases of the Dialysis Pilot.
GAO was asked to continue its evaluation of the Dialysis Pilot. GAO examined the extent to which VA documented plans for concluding the Dialysis Pilot and the status of data on the quality of care and treatment costs for the four pilot locations.
GAO reviewed relevant documents from VA and the evaluation contractor selected by VA to perform an independent analysis of the pilot locations. GAO also spoke with VA officials responsible for managing the Dialysis Pilot, representatives from all four pilot locations, and evaluation contractor officials responsible for reviewing the performance of the four pilot locations.
section_title: What GAO Found
paragraphs: Five years into the Dialysis Pilot, the Department of Veterans Affairs (VA) Central Office still has not set a timetable for completing the pilot or documented how it will determine the success of the pilot locations. GAO previously identified best practices that state that a project timeline is critical for managing and measuring an entity's performance on projects and that choosing and documenting well-regarded criteria that are used to make comparisons can lead to strong, defensible conclusions. Initially, VA planned to conclude the Dialysis Pilot after the pilot locations were all open for 5 years. However, in March 2014, VA officials told GAO they are no longer operating under this timeline but instead plan to conclude the pilot once the pilot locations achieve (1) the creation of a model for a VA-operated, free-standing dialysis clinic that can be replicated by other VA medical centers (VAMC) and (2) the confirmation of the time necessary for a pilot location to reach a “breakeven point.” VA considers that a pilot location has achieved a breakeven point when it repays its start-up funding and the VAMC realizes a cost savings because its treatment cost for dialysis at the pilot location is lower than purchasing care from non-VA dialysis providers. However, VA has not formally documented these pilot location achievements as criteria for concluding the Dialysis Pilot. By not doing so, the transparency of VA's management decisions on pilot location outcomes is compromised and the Department lacks accountability for ensuring the success of the Dialysis Pilot.
The Dialysis Pilot has been under evaluation for 2 years by VA and the contractor it selected to conduct an independent analysis of pilot location quality of care and treatment costs. However, neither has concluded its evaluation. Specifically, VA noted that the delayed openings and initial operational issues of two pilot locations—Philadelphia, Pennsylvania, and Cleveland, Ohio—led to limited data availability, and it recommended another 12 months of data be collected on these two pilot locations before drawing conclusions.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA document its plans for concluding the Dialysis Pilot. VA concurred with GAO's recommendation but did not clearly delineate its plans for pilot conclusion.

section_title: Letter
paragraphs: Veterans diagnosed with end-stage renal disease (ESRD)—a condition of  permanent kidney failure—are one of the most resource-intensive patient  populations served by the Department of Veterans Affairs (VA) Veterans  Health Administration (VHA). Veterans with ESRD are generally  prescribed dialysis, which is a life-saving medical procedure that removes  excess fluids and toxins from the bloodstream. Dialysis treatments are  time intensive. Specifically, veterans receiving dialysis treatments  typically must receive three outpatient treatments per week with each  treatment lasting about four hours. Veterans usually remain on dialysis for  the rest of their lives unless they receive a kidney transplant. Due to VA’s  limited dialysis capacity in its own facilities, VA most commonly provides  dialysis to veterans by referring them to non-VA dialysis providers in their  local communities through the Non-VA Medical Care Program (non-VA  medical care). In fiscal year 2013, VA referred about 14,000 veterans to  non-VA dialysis providers at a cost of about $425 million.
In 2009, VA began developing a pilot program at several VA medical  centers (VAMC) to provide dialysis to veterans in VA-operated dialysis  clinics in response to several issues—including an increasing number of  veterans needing dialysis, rising costs of providing dialysis through non- VA medical care, and unsuccessful efforts to lower these costs. The  resulting Dialysis Pilot was approved by the Secretary of VA in  September 2010. Ultimately, VA believes the Dialysis Pilot will lead to the  development of a business model with a mission of providing more  veterans’ dialysis treatments in VA facilities and less through non-VA  medical care. Through this pilot, VA established four VA-operated dialysis  clinics located near sponsoring VAMCs in Durham and Fayetteville, North  Carolina; Cleveland, Ohio; and Philadelphia, Pennsylvania. The Raleigh  and Fayetteville pilot locations opened in June 2011, the Philadelphia  pilot location opened in October 2012, and the Cleveland pilot location  opened in July 2013.
In May 2012, we issued a report on VA’s Dialysis Pilot that identified  several weaknesses in VA’s execution of the planning and early  implementation phases of the pilot. Specifically, we found VA did not  appropriately document its pilot location selection process, produce  consistent and comparable cost estimates for pilot locations, provide clear  and timely written guidance on how to repay start-up funding to VA  Central Office, and provide guidance on how to calculate cost savings  generated by the pilot locations.concern about issues raised in our prior report, you asked us to continue  our evaluation of VA’s Dialysis Pilot. In this report, we examine (1) the  extent to which VA has documented plans for concluding the Dialysis  Pilot and (2) the status of data on the quality of care and treatment costs  for the four pilot locations.
To examine the extent to which VA has documented plans for concluding  the Dialysis Pilot, we interviewed VA Central Office officials and VAMC  officials responsible for managing pilot locations regarding VA’s plans for  the completion of the Dialysis Pilot and reviewed relevant VA  documentation. These documents included a VA business analysis for  the Dialysis Pilot and VA’s contract with a leading university research  center (evaluation contractor) to review the Dialysis Pilot locations’ quality  of care and treatment costs.
To examine the status of data on the quality of care and treatment costs  for the four pilot locations, we interviewed VA Central Office officials and  officials from the evaluation contractor regarding the status and initial  findings of their evaluations. We analyzed documents that VA and the  evaluation contractor produced regarding the quality of care—including  clinical quality, patient satisfaction, and access to care—and treatment  costs at the pilot locations compared with non-VA dialysis providers.
We conducted this performance audit from January 2014 to July 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: VA set four goals for the Dialysis Pilot: (1) improve the quality of dialysis  care veterans receive, (2) increase veterans’ access to dialysis care,   (3) provide additional dialysis research opportunities, and (4) achieve cost  savings for VA-funded dialysis treatments.
subsections: section_title: VA Dialysis Pilot Evaluation
paragraphs: VA is performing an internal evaluation of the performance of the four  pilot locations by assessing the quality of care—including clinical quality,  patient satisfaction, and access to care—and treatment costs at each pilot  location. Results of this evaluation are reported in an annual performance  review produced by VA each calendar year. To date, VA has completed  reviews for calendar years 2012 and 2013.
Clinical quality: To evaluate the clinical quality at the pilot locations,  VA selected 11 clinical quality measures that were derived from  measures and associated performance standards established by the  Centers for Medicare & Medicaid Services (CMS)—the federal entity  with primary responsibility for evaluating the quality of dialysis care in  the United States—and endorsed by the National Quality Forum, a  leader in evaluating clinical performance measurement for a variety of  chronic clinical conditions.
Patient satisfaction: To evaluate veterans’ satisfaction with the  quality of care provided at the pilot locations, VA analyzed the CMS  standard dialysis facility survey—the In-Center Hemodialysis  Consumer Assessment of Healthcare Providers and Systems—for the  veterans treated at pilot locations.
Access to care: To evaluate veterans’ access to the pilot locations,  VA is measuring the percentage of veterans whose actual travel  distances are no more than 30 miles from their residence to the pilot  location where they receive care.
Treatment cost: To evaluate the cost of care, VA is comparing the  pilot locations’ average cost per treatment and the average cost per  treatment for non-VA dialysis providers in each pilot location’s  Veterans Integrated Service Network (VISN). To calculate each pilot  location’s average cost per treatment, VA divided each pilot location’s  total health care costs by its total number of treatments. To calculate  non-VA dialysis providers’ average cost per treatment, VA created a  composite figure using the average per treatment contract rate from  the two largest non-VA dialysis providers located in each pilot  location’s VISN, a per treatment VA claims processing fee, a per  treatment VA physician cost for nephrology oversight, and a per  treatment VA financial administrative cost.
section_title: Dialysis Pilot Evaluation Contractor Review
paragraphs: In September 2012, VA awarded a 3-year contract to an evaluation  contractor to provide an independent evaluation of its Dialysis Pilot— including a comparison of the clinical quality provided by the four pilot  locations with non-VA dialysis providers and a dialysis per treatment cost  comparison between the pilot locations and the Non-VA Medical Care  Program.
Clinical quality: To evaluate the clinical quality of care provided by  the pilot locations, the evaluation contractor is assessing the results of  several measures—such as mortality, hospitalization, and infection  rates—for the pilot locations and comparing them to relevant state  and national results. These results are presented for each pilot  location in quarterly quality reviews and an annual dialysis facility  report (DFR), a standard report produced by the evaluation contractor  for CMS for each of the more than 6,000 private sector dialysis  facilities nationwide.
Treatment cost: To evaluate the treatment costs for each pilot  location, the evaluation contractor is comparing the pilot locations’  average cost per treatment to that of private sector dialysis providers,  including certain contracted non-VA dialysis providers. Results are  presented for each pilot location in quarterly cost reviews. The  evaluation contractor used a similar methodology as VA to determine  each pilot location’s average cost per treatment. To determine the  non-VA dialysis provider average cost per treatment in calendar year  2012, the evaluation contractor created a composite figure that  included (1) the contract per treatment average price for the three  non-VA dialysis providers with the largest market share in the VISN  responsible for the Raleigh and Fayetteville pilot locations, (2) VA’s  per treatment travel-related costs, and (3) VA’s per treatment medical  oversight costs.
section_title: VA Has Not Documented Criteria or Plans for Concluding the Dialysis Pilot
paragraphs: Even though it is 5 years into the Dialysis Pilot, VA Central Office has not  yet set a timetable for completing the pilot or documented how it will  determine the success of the four pilot locations. We have previously  outlined best practices for project scheduling that specify project timelines  are critical for managing and measuring an agency’s performance on  projects and that it is necessary for the agency to have a document that  details the rationale used in developing a project timeline. In addition,  we have outlined best practices for designing evaluations that state that  choosing and documenting well-regarded criteria that are used to make  comparisons can lead to strong, defensible conclusions.
According to VA’s business analysis of the Dialysis Pilot conducted prior  to opening the pilot locations, VA planned to conclude the Dialysis Pilot   5 years after the pilot locations opened and assumed all four pilot  locations would begin operations at generally the same time. Due to  delays in opening the pilot locations, the completion of this 5-year period  would be in fiscal year 2018—5 years after the opening of the fourth pilot  location. However, in March 2014, VA officials told us they are no longer  operating under the 5-year timeline outlined in this plan, although they  have not updated the timeline included in the Dialysis Pilot business  analysis. VA officials told us that instead of ending the Dialysis Pilot  based on the timeline outlined in VA’s business analysis, they believe the  Dialysis Pilot will be successful and should conclude once the pilot  locations have achieved the following:
Creation of a replicable model. According to VA officials, the first  achievement that indicates success is the creation of a model for a  VA-operated, free-standing dialysis clinic that can be replicated by  other VAMCs. Such a model would include clear documentation of the  necessary staff, equipment, time, and resources required to establish  a new clinic and serve as a guide to other VAMCs seeking to  establish similar clinics. VA officials did not specify a timeline for the  creation of this model; however, they did state that they believe the  pilot locations have already achieved this milestone.
Pilot location breakeven points. VA officials stated that the second  achievement that indicates success is the confirmation of the time  necessary for a pilot location to reach its breakeven point—which VA  has defined as the point when a pilot location achieves cost savings in  the Non-VA Medical Care Program for dialysis services for its  sponsoring VAMC and repays its start-up funding to VA Central  Office. Through the Dialysis Pilot, sponsoring VAMCs for the pilot  locations were each provided about $2.5 million in start-up funding to  establish a VA-operated, free-standing dialysis clinic and were to  finish repaying this start-up funding to VA Central Office in fiscal year  2014. According to VA Central Office officials, all four pilot locations  have repaid their start-up funding as of the beginning of fiscal year  2014 and two pilot locations—Raleigh and Fayetteville—have  demonstrated that their cost per treatment is lower than comparable  per treatment costs of non-VA dialysis providers in their areas.  Achieving this breakeven point indicates that the pilot locations are no  longer incurring additional costs for VA and are fulfilling VA’s goal of  providing dialysis treatments to veterans at a lower cost than similar  care provided through non-VA dialysis providers.
While VA officials indicated they could conclude the Dialysis Pilot once  the pilot locations had created a replicable model and realized their  breakeven points, VA has not formally communicated these  achievements in writing as criteria for concluding the pilot. Moreover,  these criteria were not included in the performance work statement for the  evaluation contractor, and according to a VA official, have not been  formally recorded in any official document. By not clearly communicating  these milestones as criteria for concluding the Dialysis Pilot, the  transparency of VA’s management decisions on pilot location outcomes is  compromised and the Department lacks accountability for ensuring the  success of the Dialysis Pilot.
section_title: Current Data on Quality of Care and Treatment Costs Are Limited Due to Ongoing Evaluations
paragraphs: VA and the evaluation contractor have been evaluating the Dialysis Pilot  for 2 years; however, neither has concluded its evaluation of the Dialysis  Pilot and data on two of the pilot locations—Philadelphia and Cleveland— is limited due to their delayed opening. Table 1 shows the data available  for each pilot location as of June 2014.
To date, VA has produced two annual performance reviews on quality of  care and treatment costs for the Dialysis Pilot—the first for calendar year  2012 and the second for calendar year 2013—that include the results of  pilot locations’ clinical quality, patient satisfaction, access to care, and  treatment cost performance. The VA calendar year 2012 annual  performance review included results for the Raleigh and Fayetteville pilot  locations only. VA included results for all four pilot locations in its calendar  year 2013 annual performance review; however, this review does not  include a full year of data for the Cleveland pilot location and VA noted  operational challenges that affected the data for the Philadelphia pilot  location. VA noted several contributing factors to the limited data  available on the Cleveland and Philadelphia pilot locations, including the  following:
The Cleveland pilot location experienced issues with laboratory data  in veterans’ VA medical records that limited the results on one of the  clinical quality measures in 2013.
For both the Cleveland and Philadelphia pilot locations, VA reported  that the dialysis machines did not operate properly, which resulted in  a lower than expected number of veterans being treated at the  Philadelphia pilot location between January and June 2013 and  delayed the opening of the Cleveland pilot location until July 2013.
As a result, VA recommended that another full 12 months of data from the  Philadelphia and Cleveland pilot locations be analyzed before making  conclusive comparisons of the treatment costs of these pilot locations  with non-VA dialysis providers. In addition, the evaluation contractor has  reviewed the Dialysis Pilot for 2 years and has one year remaining on its  contract. At the time of our analysis, the evaluation contractor had  completed 2 years of evaluations on the quality of care provided by the  four pilot locations and 1 year of treatment cost evaluations for two of the  pilot locations. For information on VA’s clinical quality review of pilot  locations, see appendix I; VA’s patient satisfaction review for pilot  locations, see appendix II; VA’s access to care review for pilot locations,  see appendix III; the evaluation contractor’s clinical quality review of pilot  locations, see appendix IV; and VA’s and the evaluation contractor’s  reviews of pilot location average cost per treatment, see appendix V.
section_title: Conclusions
paragraphs: VA has not documented its plan for the conclusion of the Dialysis Pilot,  despite beginning pilot development 5 years ago. Without a formally  established project timeline, VA cannot effectively monitor performance  against specified timeframes, validate and defend the timeline of the  Dialysis Pilot to VA decisionmakers and other stakeholders, or ultimately  hold VA decisionmakers accountable for future resource investment  decisions. In addition, the transparency of the Dialysis Pilot is jeopardized  by VA not clearly documenting the achievements stated by VA officials— the creation of a replicable model for additional VA-operated free- standing dialysis clinics and demonstrating a breakeven point—as criteria  for concluding the Dialysis Pilot.
section_title: Recommendation
paragraphs: To improve the transparency and accountability of the Dialysis Pilot, we  recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Health to document plans for concluding the Dialysis Pilot,  including establishing an end date or documenting criteria for deciding  what constitutes the successful completion of the pilot.
section_title: Agency Comments and Our Evaluation
paragraphs: VA provided written comments on a draft of this report, which we have  reprinted in appendix VI. In its comments, VA generally agreed with our  conclusions and concurred with our recommendation to improve the  transparency and accountability of the Dialysis Pilot. VA also stated that  in July 2015—2 years after the Cleveland pilot location became  operational—it plans to review the outcomes from the four pilot locations  and make recommendations for the Dialysis Pilot. VA did not provide any  technical comments.
In its general comments, VA noted that the criteria for deciding what  constitutes the successful completion of its Dialysis Pilot is included in the  department’s document entitled, “Evaluation Plan: VA Free-Standing  Dialysis Centers,” which VA enclosed with its comments. While this  document describes the measures VA is using to evaluate the pilot, it  does not delineate the amount of data or length of review that is needed  in order to conclude the pilot. In addition, VA did not clearly state when  recommendations will be made and whether the pilot will end at that time.  We continue to believe that VA should clearly document its plan for  concluding the Dialysis Pilot by clearly specifying an end date for the pilot  or specific outcomes that must be met.
We are sending copies of this report to the Secretary of Veterans Affairs,  appropriate congressional committees, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VII.
section_title: Appendix I: Results of Clinical Quality Review by the Department of Veterans Affairs
paragraphs: This appendix provides information on the Department of Veterans  Affairs’ (VA) results from its review of Dialysis Pilot location clinical  quality. To assess clinical quality, VA selected and evaluated Dialysis  Pilot locations using 11 performance measures established by the  Centers for Medicare & Medicaid Services (CMS). For calendar years  2012 and 2013, CMS defined the lowest achievement threshold for facility  performance as the 15th percentile, the average performance standard as  the 50th percentile, and the highest benchmark as the 90th percentile.  Specifically, table 2 provides results of VA’s review of clinical quality at  the four Dialysis Pilot locations—Raleigh and Fayetteville, North Carolina;  Philadelphia, Pennsylvania; and Cleveland, Ohio.
section_title: Appendix II: Results of Patient Satisfaction Review by the Department of Veterans Affairs
paragraphs: This appendix provides results from the Department of Veterans Affairs’  (VA) review of patient satisfaction at its two more established Dialysis  Pilot locations—Raleigh and Fayetteville, North Carolina. VA did not  analyze patient satisfaction for the newer pilot locations—Philadelphia,  Pennsylvania, and Cleveland, Ohio—because these pilot locations were  either not open or not operating at full capacity during the periods of VA’s  analysis.
Table 3 provides results for VA’s review of the Raleigh, North  Carolina, Dialysis Pilot location’s patient satisfaction.
Table 4 provides results for VA’s review of the Fayetteville, North  Carolina, Dialysis Pilot location’s patient satisfaction.
section_title: Appendix III: Results of Access to Care Review by the Department of Veterans Affairs
paragraphs: This appendix provides results from the Department of Veterans Affairs’  (VA) review of veterans’ access to care at its two more established  Dialysis Pilot locations—Raleigh and Fayetteville, North Carolina. In  calendar year 2012, VA did not analyze veterans’ access to care for the  newer pilot locations—Philadelphia, Pennsylvania, and Cleveland, Ohio— because these pilot locations were either not open or not operating at full  capacity during the periods of VA’s analysis.
Table 5 provides results for VA’s review of veterans’ access to care at  the Raleigh and Fayetteville pilot locations for calendar year 2012.
Table 6 provides results for VA’s review of veterans’ access to care at  all four pilot locations for calendar year 2013.
section_title: Appendix IV: Results of Clinical Quality Review by the Evaluation Contractor
paragraphs: This appendix provides results from the evaluation contractor’s review of  clinical quality for three of the Department of Veterans Affairs’ (VA)  Dialysis Pilot locations in Raleigh and Fayetteville, North Carolina, and  Philadelphia, Pennsylvania. VA contracted with a leading university  research center to evaluate the Dialysis Pilot. The evaluation contractor’s  review of the Philadelphia Dialysis Pilot location was limited due to  operational difficulties at the pilot location in calendar year 2012. The  evaluation contractor did not complete a review of the Cleveland, Ohio,  Dialysis Pilot location due to insufficient data resulting from the delayed  opening of this pilot location.
Table 7 provides results for the evaluation contractor’s review of the  Raleigh and Fayetteville, North Carolina, Dialysis Pilot locations’  clinical quality.
Table 8 provides results for the evaluation contractor’s review of the  Philadelphia, Pennsylvania, Dialysis Pilot location’s clinical quality.
section_title: Appendix V: Results of Treatment Cost Reviews by the Department of Veterans Affairs and the Evaluation Contractor
paragraphs: Table 9 provides results from the Department of Veterans Affairs’ (VA)  and the evaluation contractor’s reviews of the average cost per treatment  for Dialysis Pilot locations in Raleigh and Fayetteville, North Carolina,  Philadelphia, Pennsylvania, and Cleveland, Ohio, and non-VA dialysis  providers for calendar year 2012 and calendar year 2013. VA contracted  with a leading university research center to evaluate the Dialysis Pilot.
section_title: Appendix VI: Comments from the Department of Veterans Affairs
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Marcia A. Mann, Assistant  Director; Cathleen Hamann; Katherine Nicole Laubacher;   Said Sariolghalam; and Teresa Tam made key contributions to   this report. Jacquelyn Hamilton provided legal support.