VA Dialysis Pilot: Increased Attention to Planning, Implementation, and Performance Measurement Needed to Help Achieve Goals
Published: 2012-05-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Veterans diagnosed with end-stage renal diseasea condition of permanent kidney failurerepresent one of the most resource-intensive patient populations at VA. These veterans are often prescribed dialysis, which is a life-saving and relatively expensive medical procedure that removes excess fluids and toxins from the bloodstream. VA began developing its Dialysis Pilot in 2009 with four goals: (1) improved quality of care, (2) increased veteran access, (3) additional medical research opportunities, and (4) cost savings. Through this pilot, VA will establish four VA-operated outpatient dialysis clinics in communities surrounding select VA medical centers by the end of fiscal year 2012 using start-up funding provided by VA Central Office. Pilot locations are expected to achieve cost savings and to repay their start-up funding. GAO examined VAs planning and early implementation efforts for the Dialysis Pilot, and how VA plans to evaluate the pilot.
GAO reviewed relevant VA documents, including those related to pilot location selection and cost estimation, and spoke with VA officials responsible for overseeing the Dialysis Pilot and representatives from all pilot locations.
section_title: What GAO Found
paragraphs: GAO found a number of weaknesses in the Department of Veterans Affairs (VA) execution of the planning and early implementation phases of the Dialysis Pilot. These weaknesses involved pilot location selection, cost estimation practices, and cost savings calculations that could collectively limit the achievement of the pilots goals. Specifically, VA did not do the following:
Appropriately document its pilot location selection process. VA did not maintain a clear and transparent pilot location selection process; it did not document how its criteria for pilot location selection were applied to all 153 VA medical centers (VAMC) or why substitutions in pilot locations were made. However, VA officials reported that several criteria, including dialysis patient prevalence and average treatment costs, were used to select the pilot locations in Durham and Fayetteville, North Carolina; Philadelphia, Pennsylvania; and Cleveland, Ohio.
Produce consistent and comparable cost estimates for pilot locations. VA did not complete consistent and comparable cost estimates for the four pilot locations. Specifically, GAO found several cases where pilot locations did not complete reliable cost estimates because they made changes to formulas and assumptions of the Dialysis Pilot cost estimation model, which was developed by VA systems redesign engineers.
Provide clear and timely guidance on start-up fund repayment and cost savings calculations. VA did not provide Veterans Integrated Service Network and VAMC officials with clear and timely written guidance or instructions on how to pay back start-up funds, or how to calculate cost savings generated by the pilot locations.
VA Central Office has not yet determined how it will achieve its goals for the Dialysis Pilot or created clear performance measures for the pilot locations. Previously, GAO found that leading public sector organizations take three steps to improve their performance and help their organizations become more effective: (1) define a clear mission and goals, (2) measure performance to gauge progress toward achieving goals, and (3) use performance information as a basis for decision making. While VA has defined a clear mission and goals for the Dialysis Pilot, it has only made limited progress in the remaining two steps. In March 2012, VA reported that it was in the early stages of establishing an agreement with a leading university research center to conduct an evaluation of the Dialysis Pilot; however, no target dates were provided for when this evaluation would begin or what aspects of the Dialysis Pilot it would evaluate. Because VA has not yet developed an evaluation plan or defined performance measures for pilot locations, it is not collecting consistent and reliable information on the performance of the pilot locations and thus may not have this information available when it is time to either make midcourse corrections to the Dialysis Pilot or decide whether and how to open additional VA-operated outpatient dialysis clinics. VA officials also told GAO they have developed a limited plan for expanding the Dialysis Pilot despite not having access to performance information on the existing four pilot locations.
section_title: What GAO Recommends
paragraphs: Among other actions, GAO recommends that VA improve its Dialysis Pilot by providing guidance for start-up fund repayment, as well as developing an evaluation plan that includes performance measures for the pilot locations. VA concurred with GAOs recommendations and provided an action plan to address them.

section_title: Letter
paragraphs: Veterans diagnosed with end-stage renal disease (ESRD)—a condition of  permanent kidney failure—are one of the most resource-intensive patient  populations the Department of Veterans Affairs’ (VA) health care system  serves. To treat the effects of ESRD, veterans with the disease are often  prescribed dialysis, which is a life-saving medical procedure that removes  excess fluids and toxins from the bloodstream. Veterans diagnosed with  ESRD must receive dialysis treatments for the rest of their lives unless  they receive a kidney transplant. These treatments are time intensive for  veterans and relatively expensive for VA. Dialysis typically requires  veterans to undergo three outpatient treatments per week that each last  about 4 hours. Due to VA’s limited dialysis capacity in its own facilities,  VA most commonly provides these dialysis treatments to veterans by  referring them to non-VA outpatient dialysis clinics in their local  communities through VA’s fee basis program. In fiscal year 2011, VA  referred about 9,600 veterans to fee basis providers for dialysis  treatments at a cost of about $314 million, up from about 6,900 veterans  at a cost of about $133 million in fiscal year 2008.
To help address these rising costs, VA began developing its Dialysis Pilot  in 2009. Through this pilot, VA will establish four VA-operated outpatient  dialysis clinics in communities surrounding select VA medical centers  (VAMC) by the end of fiscal year 2012. VA officials expect these new  clinics will reduce the number of veterans referred to fee basis providers  for dialysis treatments. Ultimately, VA believes the Dialysis Pilot will lead  to the development of a business model with a mission of providing more  veterans’ dialysis treatments in VA facilities rather than through the fee  basis program. To meet this mission, VA set four goals for the Dialysis  Pilot: (1) to improve the quality of dialysis care veterans receive, (2) to  increase veterans’ access to dialysis care, (3) to provide additional  dialysis research opportunities, and (4) to achieve cost savings for VA- funded dialysis treatments.
Congressional requesters expressed interest in the analysis VA  performed to determine the cost-effectiveness of providing more veterans’  dialysis treatments in VA-operated facilities. This report examines   (1) VA’s early planning and implementation efforts for the Dialysis Pilot  and (2) how VA plans to evaluate the Dialysis Pilot.
To examine VA’s early planning and implementation efforts for the  Dialysis Pilot, we reviewed relevant VA Central Office, Veterans  Integrated Service Network (VISN), and VAMC documents related to   (1) pilot location selection, (2) cost estimation, and (3) pilot location start- up funding. We also interviewed VA Central Office, VISN, and VAMC  officials involved with the Dialysis Pilot—including officials from each  VISN and VAMC operating a pilot location, the Veterans Health  Administration’s (VHA) Chief Business Office, the VHA Dialysis Steering  Committee, the Dialysis Workgroup, and VA systems redesign  engineers. In addition, we analyzed VA’s key decision-making tools— including the cost estimation models used to assess pilot costs—for each  Dialysis Pilot location and the pilot as a whole. We also analyzed VA’s  data on dialysis fee basis expenditures. We verified these data with VA  officials and found information derived from analysis of these expenditure  data to be sufficiently reliable for our purposes.
To examine VA’s plans to evaluate the Dialysis Pilot, we reviewed  relevant VA and VISN documentation of key pilot plans and meetings— including business plans, implementation plans, approval documents, and  meeting minutes. We also spoke with officials from VA Central Office and  all three VISNs and four VAMCs operating pilot locations regarding   (1) reporting responsibilities and expectations for the Dialysis Pilot and   (2) any ongoing or planned evaluations of the Dialysis Pilot.
We conducted this performance audit from September 2011 to May 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: VHA oversees VA’s health care system, which includes 153 VAMCs  organized into 21 VISNs. Each VISN is charged with the day-to-day  management of the VAMCs within its network. VA Central Office  decentralized its budgetary, planning, and decision-making functions to  the VISN offices in an effort to improve accountability and oversight of  daily facility operations. However, VA Central Office maintains  responsibility for monitoring and overseeing both VISN and VAMC  operations.
subsections: section_title: Providing Dialysis Treatments to Veterans
paragraphs: Veterans may elect to have their dialysis treatments through VA or  Medicare but cannot receive dialysis benefits from both simultaneously. In  2008, there were over 18,000 veterans enrolled in VA’s health care  system diagnosed with ESRD who required dialysis treatments. Of these  enrolled veterans, about two-thirds elected to receive their dialysis  treatments through VA. However, VA treated less than half of these  veterans in VAMC-based dialysis clinics because of capacity limitations  and other factors, such as long distances veterans may have to travel to  the VAMC.most of these veterans receiving their dialysis treatments through VA’s  fee basis program. Through VA’s fee basis program, veterans can select  any dialysis provider, as long as their chosen provider accepts VA’s  established payment rate for dialysis treatment. Currently, fee basis rates  differ by VISN; however, several VISNs are currently paying for dialysis  treatments through several multi-VISN-negotiated agreements with  dialysis providers. Under these agreements, per-treatment costs for these  VISNs currently range from $248 to $310. Veterans who elect to have  their dialysis treatments provided through VA—either in VAMCs or  through the fee basis program—may not incur any out-of-pocket  expenses.
This limited internal capacity and other factors resulted in  The remaining one-third of all veterans enrolled in VA’s health care  system who were diagnosed with ESRD, in fiscal year 2008, elected to  have their dialysis treatments paid for by Medicare.  The veteran can  select any Medicare-certified dialysis provider that accepts Medicare  payment and there are few, if any, restrictions on this choice since all  major dialysis providers accept individuals covered by Medicare.  Medicare reimburses dialysis providers 80 percent of a specified per- treatment base bundled rate—about $230 in 2011—and beneficiaries   or private insurance companies are responsible for the remaining   20 percent.for by Medicare, the remaining 20 percent may be an out-of-pocket  expense that was about $7,600 per year in 2008, because VA is not  authorized to pay these out-of-pocket expenses incurred by veterans  covered by Medicare.
section_title: Development of the Dialysis Pilot
paragraphs: In 2009, VA began developing the Dialysis Pilot to build its in-house  capacity to provide dialysis treatments to veterans in response to several  issues, including the following:
Rising numbers of veterans needing dialysis. A VA-funded study  found that the number of veterans requiring dialysis treatments was  projected to increase 6 percent from fiscal year 2008 to fiscal year  2015 and the number of veterans receiving these dialysis treatments  from community providers through the fee basis program was  projected to increase 16 percent.
Rising costs of providing dialysis through the fee basis program.
The same VA-funded study also found that VA’s fee basis per- treatment costs were projected to increase about 59 percent from  $337 per treatment in fiscal year 2008 to $535 per treatment in fiscal  year 2015.
Unsuccessful efforts to achieve lower reimbursement rates with  fee basis dialysis providers. Another VA-funded study found that if  VA adopted Medicare rates, set by the Centers for Medicare &  Medicaid Services (CMS), for outpatient dialysis treatments, it would  reduce its dialysis fee basis expenditures by 39 percent resulting in  projected cost reductions from fiscal year 2011 through fiscal year  2020 of about $2 billion. As a result of this study, VA Central Office  instructed VISN directors to begin using Medicare rates as the  prevailing reimbursement rate for fee basis dialysis treatments in  2009. However, the major fee basis dialysis providers did not agree to  provide dialysis treatments to veterans through VA’s fee basis  program at these reduced rates. This resulted in VA continuing to  pay for these treatments according to previously established rates,  which are typically higher than Medicare rates.
In response to these issues, VA Central Office charged VISN 6, a VISN  with a significant volume of veterans who require dialysis treatments, with  establishing a dialysis workgroup to evaluate VA’s options for dialysis  treatment delivery. The Dialysis Workgroup—led by officials from   VISN 6, with representatives from VA Central Office and the VHA Chief  Business Office and others with financial and clinical expertise—met in  2009 to discuss various options for providing dialysis care for veterans.  This workgroup identified several options VA could take to build its  internal capacity to provide dialysis treatments to veterans and identified  several ways VA could address the rising costs of dialysis care provided  through the fee basis program, including (1) building dialysis units in  leased space in communities surrounding select VAMCs, (2) purchasing  modular dialysis units, (3) modifying existing space in selected VAMCs to  expand or build dialysis units, and (4) negotiating pricing agreements with  select dialysis providers to allow VAMCs to pay a lower rate for fee basis  dialysis treatments.
In March 2010, after discussing these options and exploring potential  solutions, the Dialysis Workgroup began designing the Dialysis Pilot as  an effort to build VA’s capacity to provide dialysis treatments to veterans  in VA-operated facilities and reduce fee basis costs. The Dialysis  Workgroup projected that the Dialysis Pilot would result in a 5-year cost  savings of about $33 million by operating four outpatient dialysis clinics  that could each treat 48 veterans a week. The Dialysis Pilot was  approved by the Under Secretary for Health in August 2010 and by the  Secretary of Veterans Affairs in September 2010. The final four pilot  locations selected by the Dialysis Workgroup were Durham and  Fayetteville, North Carolina; Philadelphia, Pennsylvania; and Cleveland,  Ohio. Each of these locations was provided approximately $2.5 million in  start-up funding by VA Central Office to establish an outpatient dialysis  clinic with 12 dialysis stations that could treat 48 veterans per week. VA  Central Office expected this start-up funding would be repaid by the pilot  locations. (See app. I for more information on the current status of each  pilot location.)
The pilot locations in Durham and Fayetteville, North Carolina, began  treating veterans in June 2011. The Philadelphia, Pennsylvania, pilot  location is scheduled to open in May 2012, and the Cleveland, Ohio, pilot  location is scheduled to open in September 2012.
section_title: Planning and Early Implementation Phases of the Dialysis Pilot Had Significant Weaknesses
paragraphs: There were a number of weaknesses in VA’s execution of the planning  and early implementation phases of the Dialysis Pilot that collectively  could limit the achievement of its goals. Specifically, weaknesses in pilot  location selection, cost estimation practices, and cost savings calculations  could hamper the Dialysis Pilot’s effectiveness.
subsections: section_title: VA’s Pilot Location Selection Process Was Not Transparent or Appropriately Documented
paragraphs: While the Dialysis Workgroup reported using several criteria to select the  Dialysis Pilot locations and documented some of these criteria in the  approval documents for the Dialysis Pilot, it did not document how these  criteria were applied or whether it assessed all 153 VAMCs for potential  inclusion in the Dialysis Pilot. According to GAO internal control  standards, clearly documenting key information is necessary to ensure  that appropriate internal controls for communicating and recording   According to Dialysis Workgroup  decision-making activities are in place.officials, the Dialysis Workgroup began its pilot location selection process  by identifying 13 potential pilot locations using several criteria, including  (1) the number of veterans receiving outpatient dialysis treatments living  within a 30-mile radius or a 30-minute drive of a VAMC, (2) a VAMC’s  potential to achieve cost savings by operating a pilot location, and (3) the  perceived level of dialysis-related clinical expertise available at each  VAMC. Dialysis Workgroup officials told us that the final four pilot  locations in Durham, Fayetteville, Philadelphia, and Cleveland were all  ultimately selected because they had a high number of veterans receiving  dialysis treatments and, in some cases, had access to high-quality clinical  expertise. However, no documentation was provided by VA discussing  how these criteria were applied to all 153 medical centers and why  Durham, one of the final four pilot locations, was omitted from the list of  13 potential pilot locations when it clearly met these selection criteria.addition, VA officials from the Dialysis Workgroup whom we spoke with  could not recall complete details regarding the pilot location selection  process that occurred in 2009, including whether additional VAMCs were  assessed against the various criteria. As a result, it is not possible for VA  or an external party to definitively determine if there were any other  VAMCs that could have been viable pilot locations beyond the   13 considered by the Dialysis Workgroup.
The transparency of the pilot site selection process was further  compromised by the manner in which the Durham VAMC was selected as  a pilot location. Dialysis Workgroup officials did not document their  rationale for selecting this VAMC—a site not included in the original   13 potential pilot locations—as one of the four final pilot locations.  According to Dialysis Workgroup officials, the VAMC in Salisbury was  originally selected as one of the four final pilot locations; however, this  VAMC was undergoing managerial changes at that time and the VAMC in  Durham, located in the same VISN, was selected as a replacement pilot  location. In April 2012, Dialysis Workgroup officials reported that the  Durham VAMC’s high number of veterans receiving dialysis and  nephrology expertise also contributed to its selection as one of the four  final pilot locations. However, VA did not document either the initial  selection of the VAMC in Salisbury as a pilot location or the rationale for  why the VAMC in Durham was a better final selection for the Dialysis Pilot  than one of the other 13 potential pilot locations that was not selected.  The lack of documentation on this particular selection further reduced the  transparency of the decision-making process.
VA Central Office officials do not have complete information about how or  why pilot locations were selected because key decisions and the rationale  behind them were not documented. Such documentation of decision- making processes is necessary to ensure that VA decision makers have  access to relevant, reliable, and timely information and could follow a  rigorous and fair decision-making process for this critical aspect of the  Dialysis Pilot. The lack of documentation related to a key planning  decision—such as the complete process used to select pilot locations— limits VA’s ability to access this information in the future, evaluate the  success of the Dialysis Pilot, and make decisions about how best to  expand the pilot to additional locations.
section_title: VA Did Not Produce Consistent and Comparable Cost Estimates for Pilot Locations
paragraphs: It is not possible to determine whether pilot locations completed reliable  cost estimations because these estimates are not consistent and  comparable. This will limit VA’s ability to determine if the Dialysis Pilot has  met its mission to reduce the cost of dialysis treatments paid for by VA.  Reliable cost estimates are necessary to ensure that pilot location costs   Generating reliable and  are comparable across the four pilot locations.comparable cost estimates prior to opening the pilot locations was critical  to the early implementation of the Dialysis Pilot in order to ensure that  appropriate site-specific baseline cost estimates were generated that  would allow VA to evaluate the cost of the Dialysis Pilot and ensure that  any cost savings generated by the pilot locations could be accurately  calculated. The importance of thorough and reliable cost estimation  processes was included in VA’s own business analysis of the Dialysis  Pilot, which stated that pilot locations were intended to use the same cost  estimation methodology to facilitate uniformity and ensure that all pilot  locations produced reliable information.
To its credit, the Dialysis Workgroup worked with VA systems redesign  engineers to develop a sophisticated cost estimation model to help VISN  and VAMC officials estimate costs for their pilot locations. VA systems  redesign engineers built the cost estimation model using validated  research as the foundation for its general baseline cost estimates. The  cost estimation model included information on several aspects of  establishing and operating an outpatient dialysis clinic—including  equipment costs, leased-space costs, staff costs, and veteran travel  costs. (See fig. 1.)
Despite this effort to build a robust model for estimating Dialysis Pilot  costs, VA did not maintain proper control over the cost estimation model  following its release for use by VISNs and VAMCs. While there were  designated areas in the cost estimation model for each pilot location to  enter its specific cost inputs, a VA systems redesign engineer we spoke  with explained that the formulas in the cost estimation model should not  be customized by pilot locations. These formulas were validated by VA  systems redesign engineers and were meant to remain constant across  all pilot locations to ensure that comparable and consistent cost estimates  were produced. However, VA Central Office requested that the cost  estimation model be fully customizable—including all cost inputs and  formulas—in order to encourage pilot locations to use the model.  Because the model was fully customizable, some pilot locations both  appropriately altered their pilot location-specific inputs and inappropriately  altered the formulas that were intended to remain constant. As a result of  the inappropriately altered formulas, final cost estimates for the four pilot  locations are inconsistent and do not include comparable information.
We found several inconsistencies in pilot locations’ use of the cost  estimation model, including the following:
Not all pilot locations used validated formulas for developing  cost estimates. We found that not all pilot locations used the  validated formulas developed by VA systems redesign engineers to  calculate their cost estimates. For example, Philadelphia’s cost  estimation model did not use the validated formula for calculating the  pilot location’s equipment costs. While the validated equipment cost  formula included a patient transport cardiac monitor for each dialysis  pilot location, the Philadelphia pilot location’s model omitted this  equipment from its calculation. Also, instead of using the validated  formulas, the Cleveland pilot location deleted some formulas related  to annual patient demand and leased-space costs and replaced them  with specific numeric values. As a result of these changes, some of  Cleveland’s cost estimations cannot be confidently compared with  those from the other three pilot locations because it is unclear what  the location used to calculate these numeric values. Consistent use of  the cost estimation model and its validated formulas is necessary to  ensure that the cost estimations of each pilot location can be  compared and evaluated.
Pilot location capacity changes. Two pilot locations increased the  capacity of their outpatient dialysis clinics, despite the fact that the  Dialysis Workgroup specifically established clear capacity limits and  the cost estimation model was developed for these specific capacity  limits. According to the Dialysis Workgroup and the Dialysis Pilot  approval document signed by the Secretary of Veterans Affairs, each  pilot location’s capacity was limited to 12 dialysis stations that could  provide up to 48 veterans with dialysis treatments each week.  However, the Fayetteville pilot location increased its capacity from 12  to 16 dialysis stations and the Cleveland pilot location increased its  capacity from 12 to 20 dialysis stations. These capacity increases  were not validated by VA systems redesign engineers, and as a  result, it is unclear how these changes may affect the efficiency of  these pilot locations. In addition, it is unclear whether these capacity  increases were approved by VA Central Office since the size of these  two pilot locations is larger than what was originally approved by the  Secretary of Veterans Affairs.
According to Dialysis Workgroup officials, pilot location-specific baseline  cost estimates were included in VA’s own business analysis of the  Dialysis Pilot. However, the baseline cost estimates included in this  document are unreliable for the following reasons:
The baseline cost estimates included in VA’s business analysis of the  Dialysis Pilot are based on the assumption that all pilot locations will  be limited to 12 dialysis stations. However, the Cleveland and  Fayetteville pilot locations currently have considerably more dialysis  stations, with 20 and 16 dialysis stations, respectively.
The baseline cost estimates included in VA’s business analysis of the  Dialysis Pilot were generated prior to the cost estimation model’s  distribution to pilot locations for customization. Therefore, these  estimates do not account for the pilot location-specific customization  of several model inputs, such as actual leased-space costs. During  the site-specific customization process, several of the costs  associated with these customizable inputs increased significantly due  to either changes in pilot location size or other factors. For example,  the Cleveland pilot location’s customized model includes about  $400,000 in annual lease expenses, while the baseline cost estimate  for Cleveland’s lease expenses from VA’s business analysis is only  about $220,000.
section_title: VA Did Not Provide Clear and Timely Guidance to Pilot Locations on Start-up Fund Repayment and Cost Savings Calculations
paragraphs: VA Central Office officials did not provide VISN and VAMC officials with  clear and timely written guidance or instructions on how to pay back   start-up funds or how to calculate cost savings from the Dialysis Pilot. VA  Central Office provided a total of approximately $10 million in start-up  funding for the Dialysis Pilot to the three VISNs associated with the pilot  locations. Each pilot location received about $2.5 million in start-up  funding to establish its outpatient dialysis clinic. Pilot locations are  expected to achieve cost savings through the outpatient dialysis clinics  and to repay their start-up funding. Specifically, the memorandum  approving the Dialysis Pilot signed by the Secretary of Veterans Affairs  and the Under Secretary for Health states that pilot location start-up funds  are to be repaid in two equal payments in fiscal year 2012 and fiscal year  2014.
A lack of communication—including ongoing discussions, reporting, and  guidance—regarding the repayment of pilot location start-up funds could  make it difficult for VA and external parties to determine if the pilot  locations are making reasonable progress toward repaying these funds  and realizing cost savings from the Dialysis Pilot.officials reported a lack of clarity about how Dialysis Pilot start-up funds  must be repaid. Specifically, officials from two of the three VISNs  associated with pilot locations—VISNs 4 and 10—told us that they have  not discussed the repayment of their pilot locations’ start-up funding with  VA Central Office. Similarly, officials from the VAMCs associated with the  two operational pilot locations acknowledged their understanding that  start-up fund repayment would likely be included as part of their cost   VISN and VAMC  savings calculations, but told us that they were not aware of any specific  agreements or plans for repayment.
In addition, while the Dialysis Workgroup provided pilot locations with   5-year cost savings projections and articulated plans for calculating actual  cost savings in a document that was published prior to the approval of the  Dialysis Pilot, VISN officials we spoke with were uncertain about how cost  savings would be calculated. According to VA’s own business analysis of  the Dialysis Pilot, actual cost savings will be calculated by comparing the  cost per treatment at each pilot location to actual fee basis per-treatment  rates for each pilot location’s corresponding VAMC.states that cost saving calculations will be a collaborative effort between  pilot location leadership, VA researchers, and the VHA Chief Business  Office. However, VISN officials stated that while they expect that the cost  savings from their pilot locations will be examined, they did not receive  written guidance about how cost savings will be calculated. Given the lack  of specific and timely guidance to VISN and VAMC officials on the  calculation of cost savings, officials at the pilot locations may not use the  same methodology to track these savings.
section_title: VA’s Dialysis Pilot Has a Stated Mission but Lacks Clearly Defined Performance Criteria and an Evaluation Plan
paragraphs: VA Central Office has not yet determined how it will define success for  the Dialysis Pilot or created clear performance measures linked to the  four Dialysis Pilot goals. When other leading public sector organizations  are engaged in efforts to improve their performance and help their  organizations become more effective—similar to VA’s goals for the  Dialysis Pilot—we found that these organizations commonly take three  steps: (1) define a clear mission and goals, (2) measure performance to  gauge progress toward achieving goals, and (3) use performance  information as a basis for decision making.
Step 1—Defining a clear mission and goals. VA has completed the first  of these steps by defining a clear mission and goals for the Dialysis Pilot.  Specifically, the Dialysis Workgroup noted that the Dialysis Pilot would  allow VA to develop a cost-effective business model that could be used to  optimize VA’s resources and increase its capacity to provide dialysis  treatment to veterans. This workgroup also outlined four goals of the  Dialysis Pilot: (1) improved quality of care, (2) increased access for  veterans, (3) additional dialysis research opportunities, and (4) cost  savings for VA-funded dialysis treatments. Through the participation of its  membership in developing this mission and these goals, the Dialysis  Workgroup was able to incorporate the input of several VA internal  stakeholders—including VA Central Office representatives, VISN  leadership, clinical experts with experience treating veterans with ESRD,  and VA systems redesign engineers. This process included a thorough  assessment of VA’s options for providing dialysis treatments to  veterans—including the resources, equipment, and staffing needed to  operate a cost-effective outpatient dialysis clinic.
Step 2—Measuring performance to gauge progress. Despite its  success in defining a clear mission and goals for the Dialysis Pilot, VA  has not developed a clear plan for evaluating the pilot. Specifically, while  two pilot locations (Durham and Fayetteville) began treating veterans in  June 2011, VA has not yet begun an evaluation of the establishment and  management of the pilot locations—including causes for opening delays,  operating challenges, or the sufficiency of start-up funding. Previously, we  found that developing sound evaluation plans before a pilot program is  implemented can increase confidence in results and facilitate decision  making about broader applications of the pilot. In March 2012, the VHA  Chief Business Office reported that VA is in the early stages of  establishing an agreement with a leading university research center to  conduct an evaluation of the Dialysis Pilot; however, no target dates were  provided for when this evaluation would begin or what aspects of the  Dialysis Pilot beyond cost-effectiveness it would evaluate.
In addition, VA Central Office has not developed a cohesive strategy for  evaluating the Dialysis Pilot and has not yet formally defined its criteria for  measuring the performance of the pilot locations or the success of the  Dialysis Pilot in general. Several potential performance measures could  be used to measure the pilot locations’ progress toward the achievement  of each Dialysis Pilot goal:  Improved quality of care. Officials from the Dialysis Workgroup told  us that quality assurance outcomes, specifically those used by CMS  to certify outpatient dialysis clinics, could be used to assess pilot  locations. These metrics would likely help VA assess the quality of  dialysis care provided by the pilot locations.
Increased access for veterans. Dialysis Workgroup officials told us  that patient satisfaction information could be used to assess pilot  locations. This potential metric could help determine if the pilot  locations increased veterans’ access to dialysis care.
Additional dialysis research opportunities. In its business analysis  of the Dialysis Pilot, the Dialysis Workgroup recommended that VA  fund a 4-year research study to evaluate the quality of care at all pilot  locations and identify best practices in veteran dialysis care.  According to this business analysis, the findings of this study would  enable VA to develop an evidence-based strategy for veteran dialysis  care that ensures veterans receive the highest quality of care.
Cost savings for VA-funded dialysis treatments. In its business  analysis of the Dialysis Pilot, the Dialysis Workgroup suggested that  pilot locations could use the cost estimation model to calculate cost  savings generated by the pilot locations by comparing the cost of  providing dialysis at each pilot location to the cost of providing this  treatment through fee basis providers. However, this potential  performance metric may be limited by VA’s failure to maintain control  over the cost estimation model or provide sufficient guidance to pilot  locations about how to properly use it.
Because VA has not yet developed an evaluation plan or formally defined  performance measures for pilot locations, it does not have access to  consistent and reliable information on the performance of the pilot  locations and may not have this information accessible when it is time to  either make midcourse corrections for the Dialysis Pilot or decide whether  and how to open additional VA-operated outpatient dialysis clinics.
Step 3—Using performance as a basis for decision making. Despite  not having fully developed performance measures for assessing the pilot  locations, VA has already begun planning for the expansion of the  Dialysis Pilot, which should not occur until after VA has defined clear  performance measures for the existing pilot locations and evaluated their  success. Specifically, a member of the VHA Dialysis Steering Committee  told us that the committee has already developed a limited plan for  expansion of the Dialysis Pilot. However, this plan does not incorporate  the results of a performance assessment for the existing four pilot  locations. In addition, VA systems redesign engineers have begun  developing three additional cost estimation models despite not having  fully evaluated the effectiveness of the cost estimation model used in the  Dialysis Pilot. Taken together, these two actions indicate that VA is  beginning to make decisions about the future of the Dialysis Pilot and the  cost estimation model, even though VA decision makers currently lack  critical performance information on the existing four pilot locations.
section_title: Conclusions
paragraphs: To its credit, VA developed the Dialysis Pilot as a potential way of  addressing the rising cost and utilization of fee basis dialysis treatments  among veterans. Through the Dialysis Pilot, VA intends to test the viability  of increasing its capacity to provide dialysis treatments in VA-operated  outpatient dialysis clinics. VA set four goals for its Dialysis Pilot:   (1) improve quality of care, (2) increase access for veterans, (3) provide  additional dialysis research opportunities, and (4) achieve cost savings for  VA-funded dialysis treatments. While these are commendable goals,  there were weaknesses in VA’s planning and early implementation of the  Dialysis Pilot that if not corrected, will make it difficult to determine  whether the Dialysis Pilot has met its goals and will provide cost-effective  care if expanded. Specifically, VA did not conduct a transparent and well- documented pilot location selection process, provide clear and timely  guidance to participating VISNs and VAMCs on key financial aspects of  the Dialysis Pilot, or articulate clear performance measures for pilot  locations. We believe that VA can rectify these weaknesses, but must act  prior to full implementation of the four pilot locations to ensure that the  Dialysis Pilot is not compromised and can serve as an effective  demonstration effort. Moreover, until VA has reliable data, we believe it  would be unwise for VA to expand the Dialysis Pilot beyond the current  four pilot locations, as doing so may risk investing resources  inappropriately.
Moving forward, we believe four critical areas should be addressed. First,  VA must clearly document the selection process it used to identify the  existing four pilot locations and may use to identify any future pilot  locations. VA relied on a decentralized and ad hoc selection process to  choose the existing four pilot locations and failed to properly document  the results of this key decision-making effort. Such inattention to  documenting critical decisions results in a lack of transparency and  weakens the credibility of the Dialysis Pilot.
Second, VA needs to ensure that changes to its cost estimation model  are reviewed by knowledgeable staff. This step is necessary to ensure  that this model produces comparable data for all pilot locations that can  serve as an accurate basis for evaluating the financial success of the  Dialysis Pilot. To date, pilot locations altered existing formulas and model  assumptions, which resulted in cost estimation data with questionable  reliability that may limit VA’s ability to compare results consistently across  the pilot locations.
Third, VISN and VAMC officials need specific guidance for the repayment  of Dialysis Pilot start-up funds and the calculation of cost savings realized  from the pilot locations. To date, VA Central Office has not articulated its  expectations regarding these two critical aspects of the Dialysis Pilot.  Without a clear understanding of the terms for start-up fund repayment, it  is difficult for VA or external entities to determine if the four pilot locations  are making significant progress toward repaying these funds and  generating cost savings that can be used to offset the cost of treating the  projected increased number of veterans who will need dialysis treatment  in the coming years.
Finally, VA must develop clear and measurable performance criteria that  can be consistently applied to evaluate the Dialysis Pilot. Despite defining  the mission and goals for the Dialysis Pilot, VA has not developed a plan  for evaluating its success or developed performance measures to track  pilot locations’ progress toward meeting its stated mission and goals. An  effective evaluation plan and clear performance measures are needed to  help ensure that the Dialysis Pilot operates in an environment of  accountability.
section_title: Recommendations for Executive Action
paragraphs: In order to increase VA’s attention to planning, implementation and  performance measurement of the Dialysis Pilot we are making five  recommendations.
To improve VA’s communication related to the Dialysis Pilot, we  recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Health to ensure that key decisions made regarding pilot  location selection and efforts to continue or expand the Dialysis Pilot are  clearly documented.
To ensure that reasonable cost estimates are created for the Dialysis  Pilot and other similar programs, we recommend that the Secretary of  Veterans Affairs direct the Under Secretary for Health to restrict or  evaluate changes made to cost estimation models at the VISN and VAMC  levels that affect pilot development and analysis.
To ensure that start-up funds are repaid and cost savings are accurately  calculated for the Dialysis Pilot, we recommend that the Secretary of  Veterans Affairs direct the Under Secretary for Health to develop written  guidance about expectations for the repayment of start-up funds and how  the cost savings generated by the four pilot locations should be  calculated.
To ensure that VA Central Office effectively evaluates the Dialysis Pilot,  we recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Health to take the following two actions:
Develop an evaluation plan that outlines how the Dialysis Pilot will be  assessed and provides target dates for the completion of this  assessment.
Develop clear measures for assessing the performance of the four  Dialysis Pilot locations in key areas—including quality, access, and  cost.
section_title: Agency Comments and Our Evaluation
paragraphs: VA provided written comments on a draft of this report, which we have  reprinted in appendix II. In its comments, VA generally agreed with our  conclusions, concurred with our recommendations, and described the  department’s plans to implement each of our five recommendations. VA  did not provide any technical comments.
In its general comments, VA noted that it has established a  comprehensive strategic plan for chronic kidney disease and dialysis  services; however, a copy of this strategic plan was not provided as part  of VA’s response to the draft report. According to VA, this plan  incorporates aspects of several of our recommendations. In addition, VA  stated that it is in the process of developing longer-range plans for the  expansion of dialysis services, including establishing additional  freestanding outpatient dialysis clinics similar to the current four pilot  locations. We support VA’s efforts to carefully analyze its delivery of  dialysis services to veterans, including the most cost-effective method of  providing these life-saving medical treatments, and make reasoned  decisions based on a thorough evaluation of its current pilot locations. In  this regard, we continue to believe that it is unwise to establish additional  freestanding outpatient dialysis clinics until the current four pilot locations  are fully evaluated and VA rectifies the weaknesses we identified in this  report.
In its plan for addressing our recommendations, VA stated that it is  developing a plan for the Dialysis Pilot that will address three of our  recommendations related to (1) the documentation of Dialysis Pilot key  decisions, including the selection of future pilot locations; (2) the creation  of reasonable cost estimates for pilot locations; and (3) guidance for the  repayment of start-up funding and cost savings calculations. According to  VA, this plan will ensure better communication and documentation of  decisions for future pilot site selections; more rigorous oversight of cost  estimation tools and analysis of financial and clinical outcomes; and a  thorough analysis of start-up fund repayment, including whether VA will  reverse its decision to require repayment of these funds. VA’s anticipated  completion date for these actions is July 1, 2012.
Finally, to address our remaining two recommendations, VA intends to  develop a detailed evaluation plan for the Dialysis Pilot by July 1, 2012.  According to VA, this plan will include specific criteria, target dates, and  activities that must occur throughout the remainder of the pilot. VA  intends to use this plan to periodically review and evaluate the Dialysis  Pilot. In addition, VA described its efforts to significantly enhance the  decision-making tools used for the Dialysis Pilot, including the cost  estimation model. VA reported that these enhancements will include more  rigorous accounting for facility costs, such as those for staffing and  equipment. In addition, VA plans to task its systems redesign engineers  with assessing pilot locations’ performance using metrics for cost, access,  and quality.
We are sending copies of this report to the Secretary of Veterans Affairs,  appropriate congressional committees, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.
section_title: Appendix I: Status of VA Dialysis Pilot Locations
paragraphs: Table 1 provides additional information on the current operating status of  the four Dialysis Pilot locations at the Department of Veterans Affairs (VA)  medical centers (VAMC) in Durham, North Carolina; Fayetteville, North  Carolina; Philadelphia, Pennsylvania; and Cleveland, Ohio.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Marcia A. Mann, Assistant  Director; Kathleen Diamond; Katherine Nicole Laubacher; Rebecca Rust;  and Malissa G. Winograd made key contributions to this report. Lisa  Motley provided legal support.