VA Health Care Quality: VA Should Improve the Information It Publicly Reports on the Quality of Care at Its Medical Facilities
Published: 2017-09-29T00:00:00

Fast Facts:
Health care quality measures provide standard data that help inform consumers about which providers are most likely to deliver high quality care. They also help providers improve the care they give patients.
To help veterans make informed choices, Veterans Affairs reports on 110 of these measures for VA medical centers on its website (as of June 2017). We found that information on these measures is in two separate parts of the site. The easily-accessible page contains 15 of the 110 measures. The other, older page contains 100 measures, but is hard to find and to understand.
We recommended two actions to improve reporting.

Highlights:
section_title: Why GAO Did This Study
paragraphs: To help veterans make informed choices about their care, the Veterans Access, Choice, and Accountability Act of 2014 (Choice Act) directs VA to publicly report applicable health care quality measures for its medical facilities on HHS's Hospital Compare website and on VA's own website.
The Choice Act also contains provisions for GAO to review the health care quality measures VA publicly reports. In this report, GAO 1) describes the quality measures VA reports on Hospital Compare and its own website; 2) evaluates VA's reporting of quality measures on its website; and 3) examines the extent to which VA has assessed the accuracy of the quality measures it publicly reports. GAO reviewed the quality measures VA publicly reports, reviewed studies and interviewed VA officials about the accuracy and completeness of the clinical information used to calculate the measures, and assessed the presentation and relevance of VA's information on quality of care using criteria identified in previous GAO work to evaluate health care websites.
section_title: What GAO Found
paragraphs: As of June 2017, the Department of Veterans Affairs (VA) publicly reported 35 health care quality measures on the Hospital Compare website, which is maintained by the Department of Health and Human Services. Veterans can use information on this website to compare the performance of VA medical centers (VAMC) and non-VA hospitals on a common set of quality measures. Those measures include patient reports of their experience of care, such as how well doctors and nurses communicated with them, and actual outcomes of care, such as readmissions to the hospital. On its own website, VA reported 110 quality measures, including some of the same measures reported on Hospital Compare . VA also reports quality measures not found on Hospital Compare , such as measures of how long veterans must wait to access care at VAMCs.
VA reports health care quality measures on two separate webpages of its website. VA launched the Access and Quality webpage in April 2017, which according to VA officials is the primary source of information for veterans on the quality of care at VAMCs. GAO found that this information is generally presented in a way that is accessible and easy to understand. However, GAO also found that the primary webpage provides information from a small subset—15—of the 110 measures VA reports on its website as of June 2017. Most of the other measures are available on a second, older webpage that resides elsewhere on VA's website and is generally not easily accessible and understandable. Until VA can provide information on a broader range of health care measures and services and present this information in a way that is easily accessible and understandable, VA cannot ensure that its website is functioning as intended in helping veterans make informed choices about their care.
Within VA, VA Central Office is responsible for calculating the health care quality measures that VA publicly reports for each of its VAMCs and ensuring that these measures provide accurate information on the VAMCs' quality of care. However, GAO found that VA Central Office has not systematically assessed the completeness and accuracy of the underlying clinical information that is used to calculate these measures. This clinical information is recorded in veterans' medical records and includes diagnoses given and treatments provided. Several studies have found potential problems with the accuracy and completeness of this clinical information at some VAMCs. For example, a 2015 independent assessment conducted by McKinsey & Company found that VA's clinical documentation procedures are below industry standards and that many VAMCs do not have programs in place to improve clinical documentation practices. VA Central Office officials told GAO that they have not systematically assessed the completeness and accuracy of the clinical information across VAMCs and the extent to which this affects the accuracy of its quality measures because they have focused on other priorities. However, the lack of such an assessment is inconsistent with federal standards for internal controls related to information and monitoring. As a result, VA does not have assurance that the quality measures it publicly reports on Hospital Compare and its own website accurately reflect the performance of its VAMCs and provide veterans with the information they need to make informed choices about their care.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA 1) report a broader range of health care quality measures in an accessible and understandable way on its website and 2) conduct a systematic assessment of the patient clinical information across VAMCs to ensure its accuracy and completeness. VA concurred with GAO's first recommendation and concurred in principle with the second recommendation, and described steps to implement the recommendations.

section_title: Letter
paragraphs: The Department of Veterans Affairs (VA) provided care to about 6.9  million veterans and obligated about $65 billion for their care in fiscal year  2016. The majority of veterans utilizing VA health care services receive  care in VA-operated medical facilities, including 170 VA medical centers  (VAMCs) as of June 2017. Comprehensive assessments of VA health  care have found that VAMCs vary significantly in the quality of their care,  just as non-VA operated hospitals do. As a result, the ability of veterans  to make informed choices among VA and non-VA medical facilities  depends in part on having access to valid, accurate, and understandable  information on the relative quality of the care at these facilities.
Congress passed the Veterans Access, Choice, and Accountability Act of  2014 (Choice Act) to help ensure, among other goals, greater  accountability and transparency within the VA health care system.  Section 206 of the Choice Act established requirements for VA to publicly  report information on the performance of VAMCs on various health care  quality measures, both on the Department of Health and Human Services’  (HHS) Hospital Compare website and on VA’s own website. Hospital  Compare publicly posts health care quality measures for over 4,000 non- VA hospitals that participate in Medicare, which enables veterans and  others the opportunity to compare the performance of non-VA hospitals  and VAMCs on a common set of quality measures. VA also separately  reports on its own website information on the performance of VAMCs on  a range of health care quality measures. Such information can assist  veterans in making their health care decisions, particularly if that  information is presented in a way that veterans can locate and  understand and if the information is relevant for making informed health  care decisions.
The Choice Act included a provision for GAO to assess VA’s  implementation of requirements for publicly reporting health care quality  measures on Hospital Compare and VA’s own website. In this report, we describe the health care quality measures VA reports on Hospital  Compare and its own website; evaluate VA’s reporting of health care quality measures on its  website, including the information’s presentation and relevance to  veterans; and examine the extent to which VA has assessed the accuracy of the  health care quality measures it publicly reports.
To describe the health care quality measures VA reports on Hospital  Compare and its own website, we reviewed the quality measures that VA  Central Office—which has responsibility for VA’s public reporting of  information on quality of care—has reported on Hospital Compare and  VA’s own website as of June 2017. We also reviewed VA documents  describing the quality measures and interviewed VA Central Office  officials to understand the rationale behind VA’s decisions to report  certain quality measures, and we examined VA’s plans to expand or  adjust its public reporting of quality measures in the future. Specifically,  we interviewed officials from entities within VA’s Central Office that are  responsible for calculating and reporting the quality measures, including  the Office of Reporting, Analytics, Performance, Improvement &  Deployment. We also spoke with HHS officials responsible for  maintaining the Hospital Compare website about the measures on the  website and the processes used to receive information from VA.
To evaluate VA’s reporting of health care quality measures on its website,  including the information’s presentation and relevance, we used 13  criteria to assess the webpages on VA’s website that contain information  on quality measures. We previously identified these criteria for assessing  the effectiveness of websites that provide consumers with comparative  information on the quality of care delivered by different health care  providers. Six of the 13 criteria examine whether the information is  presented in a way that enables the consumer to locate and interpret it  and the remaining 7 criteria address the extent to which a website  provides relevant, substantive information to consumers in making  informed health care decisions. Additionally, we spoke with  representatives of three veterans service organizations (VSO)—the  American Legion, the Veterans of Foreign Wars, and Disabled American  Veterans—to obtain their perspectives about the presentation and  relevance of the quality measures reported on VA’s website. Finally, we  asked two external experts to review and comment on our assessments  of the presentation and relevance of VA’s publicly reported quality  measures on its website, and we incorporated their input as appropriate.
To examine the extent to which VA has assessed the accuracy of the  health care quality measures it publicly reports, we reviewed VA  documents related to health care clinical information management and  interviewed VA Central Office officials about the clinical information used  to calculate VAMCs’ performance on these measures. Specifically, we  interviewed VA Central Office officials responsible for reporting of quality  measures and for managing the process of recording clinical information.  We also examined an evaluation by outside experts who conducted an  independent assessment of the processes that VA has implemented to  ensure the completeness and accuracy of the clinical information  recorded in patients’ medical records at VAMCs. Further, we reviewed  investigations of VA’s clinical documentation practices by the VA Office of  Inspector General and the report and recommendations the Commission  on Care issued in 2016. We assessed VA Central Office’s efforts in the  context of federal standards for internal control for information and  monitoring.
We conducted this performance audit from August 2016 to September  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Health Care Quality Measures
paragraphs: Health care quality measures are standard, evidence-based metrics  designed to assess the performance of health care providers, such as  hospitals, in providing care. These measures are intended to (1) inform  providers about opportunities for potential improvements in their delivery  of care, (2) incentivize providers to consistently provide high quality care,  and (3) inform consumers about which providers are most likely to deliver  high quality care. There are broad categories of clinical quality measures  that address various aspects of quality of care. See table 1 for a  description of these broad categories of quality measures.
These broad measure categories can be further broken down into more  specific groups of related measures. For example, outcome measures  can include measures of patient safety, such as the incidence of  healthcare-associated infections (HAI) or complications, as well as  measures of hospital readmissions and mortality, and results obtained  from ambulatory care, such as the proportion of patients with  hypertension whose blood pressure is reduced to the normal range.
The data used to calculate the results of health care quality measures can  come from a number of different sources. Some measures often require  detailed clinical information obtained from patient medical records, such  as process measures that indicate whether timely and effective care was  provided in a specific situation, for example, or whether stroke patients  received clot-dissolving medication appropriately. Other measures are  designed to use information on patient demographics and diagnoses that  can be obtained from more readily accessible sources, such as claims  data or other administrative data that have already been collected for  other purposes such as billing. In addition, patients can be asked directly,  usually through surveys, to report on their experiences receiving care.
section_title: Communicating Information on Health Care Quality Measures to Consumers
paragraphs: One key method for disseminating information on health care quality  measures to consumers—including which providers are delivering high or  low quality care and the costs of care—is through websites that can  convey this information to anyone with internet access. HHS is one of the  organizations that provide such information to consumers and others.  Specifically, HHS’s Centers for Medicare & Medicaid Services (CMS)  maintains a series of websites that provide information on health care  quality, including a series of websites for hospitals, nursing homes, and  certain other providers that participate in the Medicare program.
Since 2005, CMS has increased the number of health care quality  measures it posts on one of its websites, known as Hospital Compare,  that covers more than 4,000 hospitals that participate in the Medicare  program. These hospitals supply data to CMS for quality measures of  inpatient and outpatient care in return for higher payments on their  Medicare claims. Each year HHS goes through a formal process,  including receiving input from experts and stakeholders, to review and  revise the mix of quality measures that these hospitals report. The  Hospital Compare website allows anyone with internet access to select  up to three hospitals to compare their performance on each of these  measures side-by-side. CMS uses contractors to collect and process the  data submitted by individual hospitals, and it posts the results on each of  the quality measures on the Hospital Compare website.
section_title: VA Reports Some Health Care Quality Measures on Hospital Compare and a Broader Range of Measures on Its Website
subsections: section_title: On Hospital Compare, VA Reports 35 of the 79 Health Care Quality Measures That Non-VA Hospitals Report
paragraphs: In implementing a requirement under the Choice Act, VA reports some of  the 79 possible health care quality measures reported by non-VA  hospitals on HHS’s Hospital Compare website. As of June 2017, VA  was reporting 35 measures that it had determined were applicable to its  individual medical centers. Of the remaining 44 measures that non-VA  hospitals report on Hospital Compare, VA plans on reporting on 12  measures in future years and does not plan to report 32 measures. See  table 2 for a summary of VA’s reporting of quality measures on Hospital  Compare.
The 35 measures VA reported on Hospital Compare as of June 2017  include all of the available patient experience measures (e.g., patient  perspectives on how well physicians communicated with them), most of  the process measures (e.g., whether stroke patients received appropriate  clot-dissolving drugs), and some of the outcome measures (e.g., whether  pneumonia patients were readmitted to the hospital within 30 days). (See  app. I for more detailed information on the specific measures VA reports  on Hospital Compare.) VA officials told us that when they began to report  quality measures to Hospital Compare in 2010, the measures they first  decided to report included measures that VA was already reporting for  other purposes. For example, these included process measures of timely  and effective inpatient care that VA reported for Joint Commission  hospital accreditation. According to VA officials, choosing quality  measures it already reported for other purposes minimized the additional  resources needed to report the measures on Hospital Compare.
From 2017 through 2019, VA plans on reporting an additional 11 outcome  measures on Hospital Compare. These include 6 measures of various  HAIs at VAMCs, 4 measures of mortality and readmission rates  associated with additional medical conditions and procedures, such as  stroke and hip/knee replacement surgery, and 1 measure related to  patient safety. Specifically:
Regarding the 6 additional HAI measures, VA officials told us that  they will report these measures when they can develop a new data  collection process that will allow them to meet the requirements for  reporting HAI measures in a way that minimizes demands on VA  resources. According to VA officials, complying with the existing  process for collecting data for and reporting on HAIs would require  infection control staff at each VAMC to fill out forms with information  on individual patients, whereas VA currently collects its own  information on HAIs based on aggregated data assembled by VA  Central Office staff. VA officials told us that their new data collection  process for reporting HAI measures is intended to meet those  Hospital Compare requirements through automating much of the  required data entry at the VAMCs. However, VA officials noted that  this new reporting process is still in an early stage of development,  and they expressed uncertainty about how long it would take to  implement the process.
Regarding the 4 additional mortality and readmission measures, CMS  faces challenges in integrating VA’s clinical information into the  Hospital Compare database. According to both VA and CMS officials,  VA has had to implement new readmissions and mortality rate  measures incrementally due to the limited capacity of one of CMS’s  Hospital Compare contractors to develop the programming needed to  calculate these measures using VA patient data. Because the  readmissions and mortality rate measures require complex  programming to implement risk adjustments, based on a number of  different diagnoses recorded in patient medical records over time, VA  and CMS officials agreed to add only two new measures each year  until VA can report all 4 measures.
There are another 32 measures non-VA hospitals report on the Hospital  Compare website that VA has no plans to report. VA officials determined  that these measures are not relevant for VA’s health care system, given  its distinctive funding sources, patient population, and health care delivery  structure. For example, VA officials stated that they did not plan to report  any of the Hospital Compare measures related to costs of care, such as  Medicare spending per beneficiary, because those measures are based  on Medicare payments and VA does not receive any payments from  Medicare. In other cases, VA officials do not plan on reporting measures  that relate to health care services that VAMCs rarely, if ever, provide,  such as obstetrics and hip and knee replacements.
In implementing a requirement under the Choice Act, VA has posted on  the Hospital Compare website a link to the notification on the quality  measures VA is not reporting. The most recent notice, dated February  2015, broadly explains that VA faces three main challenges that affect the  availability of certain measures. According to the notice, these challenges  relate to data quality, standard data collection processes, and funding to  support the collecting and reporting of quality measures. VA officials told  us that they plan on adding to the notification information about the  specific measures VA expects to report in the future and when it expects  to report them.
section_title: VA Reports 110 Health Care Quality Measures on Its Website, Including Some Categories of Measures Not Available on Hospital Compare
paragraphs: As of June 2017, VA reported on its own website 110 health care quality  measures for its VAMCs. These include measures in many of the same  categories that VA reports on Hospital Compare, but they also include  several additional categories of quality measures that are not available on  Hospital Compare. These additional categories address quality issues VA  officials deem relevant for veterans, such as various measures of access  to care. For example, the additional measures include measures of how  long veterans must wait to obtain care at VAMCs and measures of the  quality of care related to ambulatory care, such as colorectal cancer  screening rates. See Table 3 for the categories of measures VA reports  on its website.
Some of the specific measures reported on VA’s website are the same as  those reported on Hospital Compare and some are entirely different. For  example, most of the patient experience measures reported on VA’s  website are the same as those reported on Hospital Compare, while the  access to care measures are only reported on VA’s website. In addition,  some measures reported on VA’s website are similar to, but not exactly  the same as, Hospital Compare measures. For example, among similar  measures, the target population may be defined somewhat differently or  the result may be calculated differently. (See app. I for more information  on the specific measures VA reports on its website.)
In implementing a requirement under the Choice Act, VA has also posted  a notification of unavailable measures on its website. This is the same  February 2015 notice VA provided on the Hospital Compare website.
section_title: VA Reports Health Care Quality Measures on Two Separate Webpages, and Its Primary Webpage Is Accessible and Understandable but Lacks Certain Relevant Information
subsections: section_title: VA Reports Health Care Quality Measures on Two Separate Webpages of Its Website, Considering One of Them as VA’s Primary Source of Information on Care Quality for Veterans
paragraphs: VA publicly reports 110 health care quality measures on two separate  webpages on its website—the “Access and Quality” webpage, launched  in April 2017, and the “Quality of Care” webpage, which VA has used  since 2008. The Access and Quality webpage is intended to be the  primary source of information on quality of care at VA for veterans,  according to VA officials. VA officials also told us that this webpage was  developed to present quality of care information in ways that are easy for  veterans and other stakeholders to understand. A link to the primary  Access and Quality webpage can be found on the homepage of VA’s  website, making it relatively easy to find. VA officials also told us that they  retained the older Quality of Care webpage in an effort to be transparent  as well as provide historical data on the many quality measures it has  tracked. The older Quality of Care webpage is not linked to the homepage  of VA’s website. Additionally, neither the primary Access and Quality  webpage nor the older Quality of Care webpage provides any link or  makes any mention of the other. See fig. 1 and fig. 2, which show the  primary Access and Quality and older Quality of Care webpages,  respectively.
On the two webpages, VA reports its 110 health care quality measures  across various subpages, with some measures reported on multiple  subpages. Specifically, as of June, 2017:
VA reported 15 of its 110 quality measures on the primary Access and  Quality webpage, which comprises three subpages. Two subpages  focus on measures related to access (which we refer to as the “Wait  Times” and “Experience With Access” subpages) and the third  subpage compares how VAMCs perform relative to non-VA hospitals  in their geographic area on a few selected Hospital Compare  measures (which we refer to as the “Non-VA Hospital Comparison”  subpage). According to VA officials, the 15 quality of care measures  on the primary Access and Quality webpage were selected because  they provide information that is useful to veterans in making health  care choices.
On the older Quality of Care webpage, which comprises four  subpages, VA reported 100 quality measures. According to VA  officials, each of the four subpages of this webpage was created for a  specific purpose and as a result, some measures are reported on  multiple subpages.
We also found that VA reported five of the same quality measures on  both the primary and older webpages, which include measures  reflecting patient ratings of their experience in the hospital and  healthcare-associated infection rates. VA officials told us that they do  not plan on consolidating the information currently reported on the two  webpages since they serve different purposes, as explained earlier.
See Table 4 for a summary of the health care quality measures reported  on VA’s Access and Quality and Quality of Care webpages.
section_title: VA’s Primary Webpage Provides Health Care Quality Measures That Are Generally Accessible and Understandable to Veterans
paragraphs: We found that VA’s primary Access and Quality webpage generally  reports health care quality of care information in ways that are more  accessible and understandable than VA’s older Quality of Care webpage.  We assessed both webpages using criteria we identified in prior work for  evaluating how well websites present information on health care quality to  the public. Specifically, we found that VA’s primary webpage meets four  of six presentation criteria compared with VA’s older webpage, which met  none of the criteria. (See table 5.)
We found that VA’s primary webpage does especially well at presenting  information on quality of care on the two subpages that focus on access  to care. In particular, these subpages: provide information written in plain language with accompanying enable consumers to customize the information presented, so that  they can, for example, select the types of medical appointments of  interest to them; and allow users to rank order VAMCs by level of performance on a given  quality measure.
In contrast with VA’s primary webpage, we found that VA’s older Quality  of Care webpage displays quality measures in ways that generally do not  meet any of the presentation criteria. In particular, we noted that the  information presented across the four subpages on the Quality of Care  webpage shares the following key limitations: none are written in plain language or use graphics to convey key none summarize related quality information or organize the data to  show patterns, such as rank ordering VAMCs on a given performance  measure; none enables comparison of multiple VAMCs in one view, but instead  requires users to look up information on VAMCs one at a time; none enables customization of how the information is displayed so  that users can focus on the quality measures most relevant to them;  and none are advertised to potential users, with no indication provided on  the VA website homepage that the subpages exist, what information  they provide, and where to find them.
Representatives from two of the three VSOs we spoke with said that the  information on the Quality of Care webpage is not utilized by veterans  because links to the webpage are not prominently displayed on the VA  website. According to one VSO official we spoke with, when VA sought  feedback from the veterans service organizations about its overall plans  for its primary Access and Quality webpage, the official found that the  main advantage of this webpage is that it presents information in a more  visually compelling way that allows veterans to directly compare quality  information for multiple VAMCs. According to officials from the three  VSOs with whom we spoke, this capability is important because veterans  can be overwhelmed by information that is not conveyed in an easily  digestible format.
section_title: VA’s Primary Webpage Lacks Certain Health Care Quality Measures That Are Relevant to Veterans
paragraphs: We also assessed VA’s webpages using criteria we identified in prior  work for evaluating the extent to which websites provide consumers with  information relevant for making health care decisions. Using these  criteria, we found that VA’s primary Access and Quality webpage does  not provide veterans as much relevant information as VA’s older Quality  of Care webpage. Our analysis shows that VA’s primary webpage meets  only two of the seven relevance criteria. In contrast, we found that VA’s  older webpage performs relatively well on the relevance criteria—meeting  six of the seven criteria. (See table 6.)
As Table 6 shows, VA’s primary webpage does not provide information  on a broad range of health care services, nor does it highlight key  differences in clinical quality of care. For example, the primary webpage  reports only on the incidence of two types of HAIs and not on any other  types of outcomes, such as readmissions and mortality rates associated  with different medical conditions. Additionally, VA’s primary webpage only  provides limited information about key differences in patient experiences  and the key strengths and limitations of the data reported. Although VA  officials told us that they expect to expand the number of reported  measures in future and intend to make the webpage more  comprehensive, they did not identify specific measures to be  implemented.
In contrast, VA’s Quality of Care webpage performs better on the  relevance criteria primarily because its four subpages together provide  information on 100 different measures spread across a broader range of  measure categories (see tables 4 and 6). As a result, it provides  information on a broad range of services and on key differences in clinical  quality of care. Furthermore, representatives of two veterans service  organizations told us that much of the information reported on VA’s older  webpage would be relevant to veterans, including rates of hospital  readmissions, mortality, and complications—none of which are included  on VA’s primary webpage.
While VA has improved how it presents information on quality to veterans  through its primary Access and Quality webpage, there are still gaps in  the relevance of information reported on the webpage. Specifically, while  it may not be necessary for VA to report on all 110 measures on its  primary webpage to meet the relevance criteria, the 15 measures  reported on the Access and Quality webpage (of which 10 focus on  access) clearly do not provide the same breadth of relevant information  relevant for veterans. VA officials told us that they began with these 15  quality measures that they deemed most useful to veterans for making  health care choices, and in the short term, they are focused on improving  the user experience of the webpage. However, until VA’s website  provides information on a broader range of health care services,  highlights key differences in clinical quality of care, and reports this  information in a manner that is easily accessible and understandable, VA  is missing an opportunity to provide veterans with relevant quality  information that it has already collected to help veterans make informed  decisions about their care.
section_title: VA Central Office Has Not Systematically Assessed the Completeness and Accuracy of the Clinical Information Used to Calculate Health Care Quality Measures
paragraphs: Studies and other evidence we reviewed indicate potential problems with  the completeness and accuracy of the clinical information in patient  records that is used to calculate VA’s publicly reported health care quality  measures on Hospital Compare and its own website. Moreover, VA  Central Office—which has responsibility for calculating and reporting  health care quality measures for each VAMC—has not systematically  assessed the completeness and accuracy of this clinical information  across its VAMCs and the effects, if any, on the accuracy of its health  care quality measures.
subsections: section_title: Evidence Indicates That Clinical Information in VA Patient Medical Records Used to Calculate Health Care Quality Measures May Not Be Complete and Accurate
paragraphs: Studies and other evidence we reviewed indicate potential problems with  the completeness and accuracy of the clinical information recorded in  patient medical records at some VAMCs (e.g., diagnoses given and  treatments received). This is significant because the accuracy of VAMCs’  performance on quality measures that are calculated and reported by VA  on Hospital Compare and its own website depends on the completeness  and accuracy of this clinical information. For example, VA reports  readmission measures on Hospital Compare and on its website that  compare VAMCs in terms of the proportion of their patients with a medical  condition, such as heart failure, who are readmitted to a hospital within 30  days of an initial inpatient stay. Because VAMCs as well as non-VA  hospitals that treat healthier patients are likely to have lower readmission  rates independent of the quality of care that they provide, these  readmission measures incorporate information about whether a hospital’s  patients have certain other diagnoses that indicate their overall level of  health. However, the calculation of hospital readmission rates will not  work as intended if VAMCs do not record in their medical records  complete and accurate information on patient diagnoses. In particular, if  some VAMCs record that information more completely and accurately  than others, that may distort any comparison of the readmission rates  among the VAMCs.
However, according to an independent assessment of VA’s clinical  documentation procedures conducted by McKinsey & Company in 2015,  VA as a whole is below industry standards. The independent  assessment’s report looked specifically at VA’s implementation of clinical  documentation improvement (CDI) programs, which typically provide a  combination of provider education and assessments of provider  performance to promote more complete and accurate clinical  documentation. The independent assessment noted that such CDI  programs have been widely adopted by hospitals across the U.S. health  care system and play a critical role in producing more complete and  accurate clinical information. However, the assessment found that only 62  of 134 VAMCs examined had a CDI program as of 2014. The number of  VAMCs with CDI programs has increased since the study was completed;  VA officials reported to us that as of July 2017, 99 VAMCs had a CDI  program in place, and an additional 11 VAMCs are planning to implement  a CDI program.
Moreover, the independent assessment found evidence suggesting that  deficiencies in the completeness and accuracy of the clinical information  recorded in VA patient medical records may have affected the assessed  quality performance of at least some VAMCs. Specifically, the  independent assessment cited multiple instances where VAMC officials  observed that their facility’s assessed performance on VA’s quality  measures markedly improved after the facility took steps to improve the  completeness and accuracy of the clinical information recorded in the  VAMC’s patient records. We could not quantify the potential effect of  incomplete and inaccurate clinical information on VAMC assessed quality  performance without conducting an audit of the clinical information  recorded at each VAMC.
Other components of the independent assessment also highlighted  longstanding issues with VA’s electronic health records (EHR) system,  which stores the clinical information used to calculate VAMC performance  on the quality measures. The summary report for all 12 independent  assessments noted that VA’s EHR system is outdated. The EHRs used at  each VAMC do not consistently use standard data elements and  algorithms for recording clinical information, and this variability in  underlying clinical information from one VAMC to the next has led to an  inability to convey consistent and complete information across the VA  health care system.  In 2016, the Commission on Care determined that  the deficiencies in VA’s clinical documentation were so pervasive that it  recommended that VA procure an entirely new EHR system that would  replace its increasingly obsolete EHR system.
These findings are similar to other studies conducted by the VA Office of  Inspector General (OIG) that found deficiencies in VAMCs’ operations  that can lead to the recording of incomplete and inaccurate patient  information. For example, in July 2017, a VA OIG investigation of  colonoscopy practices concluded that more accurate and stringent clinical  data collection would enable VA to improve its monitoring of the quality of  providers’ colonoscopies. Additionally, in May 2012, the OIG found that  some VAMCs do not consistently conduct reviews intended to ensure that  their clinicians properly enter information into EHRs, including ensuring  the appropriate use of cut and paste functions. In April 2010, the OIG  also found that VAMC emergency departments do not consistently  document all of the information that they are required to report when  transferring patients to other facilities.
Furthermore, as one VA official observed to us, because of the way in  which they are funded, VAMCs do not have the same financial incentives  that non-VA hospitals do to ensure that the clinical information stored in  patient medical records is complete and accurate. In addition, because  most of VA health care is funded through direct appropriations, rather  than by payments of claims filed with an insurance company or with  government programs such as Medicare, VA does not routinely produce  the claims-based data on health care services provided to patients that  non-VA hospitals and clinics typically generate in the course of doing  business. Thus, to calculate VAMC performance on quality measures, VA  officials told us that VA has to extract data from its patient medical  records specifically for this purpose. In contrast, non-VA hospitals  routinely collect this information—such as data on readmissions and  mortality—as part of their claims processing and reimbursement from  Medicare and other health care payers.
section_title: VA Central Office Has Not Systematically Assessed the Completeness and Accuracy of Clinical Information in Patient Records and How this Affects Accuracy of Health Care Quality Measures
paragraphs: Within VA, VA Central Office is responsible for calculating and reporting  the health care quality measures that VA publicly reports for each of its  VAMCs and ensuring that these measures provide accurate information  on the quality of care at these facilities. However, while studies and other  evidence we reviewed indicate potential problems with the clinical  information recorded at some VAMCs, VA Central Office has not  determined the extent to which these problems exist across VAMCs and  adversely affect the accuracy of the quality measures VA publicly reports  on Hospital Compare and its own website. Specifically, VA Central Office  has not conducted a systematic assessment of the completeness and  accuracy of the clinical data recorded in VA patient medical records  across VAMCs. (There are models for this type of assessment; see app.  III for examples of methodologies that have been applied to different  types of medical record systems.) When asked why they have not  conducted such an assessment, VA Central Office officials told us that  they have focused instead on improving the accuracy of medical coding  and on specific clinical quality issues identified at individual VAMCs.   However, accurate medical coding depends on the completeness and  accuracy of the underlying clinical information in patient records.
Moreover, VA Central Office policy assigns responsibility for monitoring  the completeness and accuracy of patients’ clinical information to each  individual VAMC. Each VAMC has a Health Record Review Committee  that is charged with monitoring the clinical documentation practices at that  facility. This committee determines the frequency of medical record  reviews at its facility, decides what focused reviews should occur, and  identifies clinicians that record clinical information poorly until they  improve to an acceptable rate of completeness and accuracy. According  to officials in VA’s Central Office who are responsible for health  information management, the results of individual VAMC Health Record  Review Committee reviews are not reported to VA Central Office. As a  result, VA Central Office lacks information that could help it systematically  determine whether or to what extent VAMCs’ clinical information is  incomplete and inaccurate, and if so, the extent to which these  deficiencies affect VAMCs’ reported performance on the quality measures  VA publicly reports. The results of such a systematic analysis could also  help identify the deficiencies, if any, in the recording of patient clinical  information and what steps, if any, VA Central Office may need to take to  address them.
VA Central Office’s lack of a systematic assessment of the completeness  and accuracy of clinical information recorded in patient medical records  and the extent to which this affects the accuracy of its quality measures is  inconsistent with federal standards for internal controls related to  information and monitoring. These standards call for agencies to use  accurate information to achieve objectives, and to monitor agency  activities and evaluate results. Without a systematic assessment of the  completeness and accuracy of the clinical information recorded in VAMC  patient medical records and their potential effects on the health care  quality measures VA reports, VA Central Office does not have reasonable  assurance that differences in VAMCs’ performance on VA’s quality  measures reflect true differences in the quality of care and not differences  in the accuracy and completeness of the underlying clinical information.  This may hinder VA Central Office’s ability to appropriately assess VAMC  performance and offer accurate publicly available information on its  website to veterans so that they can make informed choices about their  care.
section_title: Conclusions
paragraphs: VA uses Hospital Compare and its website to provide veterans with  information on how VAMCs perform on a range of health care quality  measures. By providing information on the quality of care at VA facilities,  the quality measures are intended to help veterans make informed  decisions about their care. However, two key limitations in VA’s efforts  may hinder veterans’ ability to use the information VA provides to make  informed decisions about their health care.
First, while VA’s primary Access and Quality webpage provides generally  accessible and understandable information on the quality of VA health  care, the breadth of information it provides is too limited for veterans to  make informed choice about their care. VA intends for this webpage to be  veterans’ primary source of information on the quality of care at VAMCs,  but the 15 measures reported on the webpage represent only a small  subset of the 110 measures and only a few of the measure categories VA  makes available elsewhere on its website. Until VA can provide  information on a broader range of health care measures and services,  highlight key differences in the quality of clinical care at VAMCs, and  present this information in a way that is easily accessible and  understandable, VA cannot ensure that its website is functioning as  intended in helping veterans make informed choices about their care.
Second, VA does not have reasonable assurance that the health care  quality measures it reports on Hospital Compare and its own website  accurately assess the relative performance of VAMCs. This is because  the quality measures are calculated using clinical information recorded in  patient medical records, but VA Central Office does not know to what  extent this information has been accurately and completely recorded  across VAMCs. This is because VA has not conducted a systematic  assessment and does not have reasonable assurance about the accuracy  of the information that its quality measures are based on. Therefore, VA  lacks reasonable assurance that the quality measures reported on  Hospital Compare and its own website provide accurate information to  veterans so they can make informed choices about their care.
section_title: Recommendations for Executive Action
paragraphs: We are making two recommendations to the Department of Veterans  Affairs.
We recommend that the Undersecretary for Health take additional steps  to ensure that VA’s website reports health care quality measures that  cover a broad range of health care services, highlights key differences in  the clinical quality of care, and presents this information in an easily  accessible and understandable way (Recommendation 1).
We recommend that the Undersecretary for Health direct VA Central  Office to conduct a systematic assessment of the completeness and  accuracy of patient clinical information across VAMCs that is used to  calculate the health care quality measures VA reports and address any  deficiencies that affect the accuracy of these measures  (Recommendation 2).
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to VA and HHS for review and  comment. HHS had no comments on this report. VA provided written  comments, which we have reprinted in appendix IV, and provided  technical comments, which we have incorporated as appropriate. In its  comments, VA concurred with our first recommendation to take additional  steps to ensure that VA’s website reports health care quality measures  that cover a broad range of health care services, highlights key  differences in the clinical quality of care, and presents this information in  an easily accessible and understandable way. VA provided additional  information about its plans to expand the quality measures reported on its  Access and Quality webpage, including information on which additional  measures it was planning to add to the webpage and the steps it planned  to take to continue to enhance the presentation of this information for  veterans. VA indicated that its target completion date for these activities  was December 2017.
In addition, VA concurred in principle with our second recommendation  for VA Central Office to conduct a systematic assessment of the  completeness and accuracy of patient clinical information across VAMCs  that is used to calculate the health care quality measures VA reports and  address any deficiencies that affect the accuracy of these measures. In  its comments, VA acknowledged the importance of improving the  reliability of data used to calculate its health care quality measures and  described its plans to establish a workgroup under the Deputy Under  Secretary for Health for Organizational Excellence to determine the best  approach for conducting a systematic assessment of the completeness  and accuracy of the patient clinical information across VAMCs. In its  response, VA described the workgroup’s focus on determining whether  current validation processes ensure completeness and accuracy and  stated that this determination would be completed by December 2017. In  conjunction with reviewing these validation processes, VA should also  assess the completeness and accuracy of clinical information recorded in  patient records that is used to calculate the quality measures. This will  allow VA to determine the extent to which the quality measures it reports  accurately reflect VAMCs’ performance in delivering care to veterans.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, the Secretary of Health  and Human Services, and other interested parties. In addition, the report  is available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Health Care Quality Measures that the Department of Veterans Affairs Publicly Reports
paragraphs: This appendix presents the specific health care quality measures that the  Department of Veterans Affairs (VA) publicly reports, plans to report, or  does not report on the Department of Human and Health Services’ (HHS)  Hospital Compare website. HHS’s Hospital Compare website provides  information on the quality of care at non-VA hospitals and VA Medical  Centers (VAMCs). This information is captured across various quality  measures covering topics such as patient experience. Additionally, this  appendix presents the specific health care quality measures that VA  reports on its own website about the quality of care at VAMCs. Some of  the specific measures reported on the Hospital Compare website are the  same as those reported on VA’s website and some are entirely different.  In the table below, measures listed on the same row (e.g., patient  experience measures) are measures that we determined were the same,  based on information we obtained from VA and HHS. Measures that are  grouped together but listed on different rows may address similar quality  issues (e.g., hospital readmissions) but are calculated differently.  Additionally, a dash in a table cell indicates that the measure listed on  that row is not reported on the webpage in question. See Table 7.
The VA website locations refer to subpages of VA’s two webpages with  quality measures. They are coded as:
Access and Quality webpage: (1) Wait Times; (2) Experience with  Access; and (3) Non-VA Hospital Comparison.
Quality of Care webpage: (1) VA Quality Scores (Strategic Analytics  for Improvement and Learning (SAIL)); (2) MCP--Medical Center  Performance; (3) WNTB--Why Not the Best; and (4) PT EXP--Patient  Experience on the Quality of Care.
section_title: Appendix II: Criteria for Assessing the Presentation and Relevance of Information on Health Care Quality Measures
paragraphs: The 13 criteria we used to assess the presentation and relevance of  information on quality of care reported on VA’s primary Access and  Quality and older Quality of Care webpages were identified as part of  GAO’s prior work on health care transparency.  These criteria identify the  key characteristics that make websites effective in communicating  information on health care quality of care to consumers. We used a four- scale rating system—yes, no, limited or very limited—to determine if each  criterion had been met. A “limited” rating indicates that the webpage has  discrete areas where the webpage has implemented the characteristic to  some degree, but those areas are not representative of the webpage as a  whole. A rating of “very limited” indicates that the webpage has largely not  implemented the characteristic (with a few exceptions). Two analysts  applied the criteria separately and reconciled any differences. Two  external experts we also consulted generally agreed with our  assessments of the presentation and relevance of VA’s publicly reported  quality measures on its website.
subsections: section_title: Presentation criteria
paragraphs: Six of the 13 characteristics of effective consumer websites focus on the  extent to which a website presents its information in a way that enables  the consumer to grasp and interpret it. Specifically, the research we  reviewed shows that more effective websites:  1.  Use plain language with clear graphics. Effective consumer  websites use labels and descriptions that make sense to consumers  who typically are unfamiliar with clinical terminology and who often  have difficulty interpreting numerical information. Graphics, including  symbols, can help to readily convey information on relative provider  performance, especially when they are designed to display a  summary assessment of that performance as part of the symbol itself,  for example one that incorporates the words “superior” or “poor.”  2.  Explain purpose and value of quality performance ratings to  consumers. Effective consumer websites address prevalent  misleading preconceptions by providing consumers coherent  explanations of how different quality measures relate to the aspects of  quality that consumers find relevant. These explanations work best  when they link individual measures to overarching categories  indicating what is being achieved, such as effectiveness of care,  safety, or patient-focused care.  3.  Summarize related information and organize data to highlight  patterns and facilitate consumer interpretation. Two techniques  that consumer websites can use to help consumers make sense of  large amounts of information are (a) combining information from  multiple related measures into summary or composite scores, and (b)  structuring presentation of the data in ways that make patterns  evident. For example, listing providers in rank order on selected cost  and quality dimensions greatly simplifies identification of high and low  performers.  4.  Enable consumers to customize information selected for  presentation to focus on what is most relevant to them.  Consumers differ in the priority they assign to different aspects of  quality. Websites that enable consumers to customize which quality  information is presented help consumers filter out information of  lesser consequence to them, and hone in on the information that they  find most compelling. For example, one consumer may choose to  focus on providers’ capacity to communicate well with patients, while  another may focus on providers’ rates of complications and infections.  5.  Enable consumers to compare quality performance of multiple  providers in one view. Websites are most effective when they  present side-by-side assessments of providers’ performance on a  given aspect of cost or quality, so consumers can most easily  compare providers.  6.  Enable easy use and navigation of the tool. Unless consumers can  quickly find information of interest to them, they are likely to quickly  dismiss the potential utility of a consumer website and move on.  Extensive testing with consumers can help public and private entities  providing websites to develop intuitive, user-friendly approaches to  website navigation and for manipulating how the data are presented.
section_title: Relevance criteria
paragraphs: Seven of the 13 characteristics of effective websites we identified address  the extent to which a website provides substantive quality and cost  information of relevance to consumers. Specifically, the research we  reviewed shows that more effective websites:   1.  Review a broad range of services so that more consumers’  particular needs are included. The more services that are covered  by the website, the more likely it is that the website will have  information relevant to the particular services of interest to any given  consumer. It is especially important to include services that are  predictable and non-urgent, because these services are most likely to  afford consumers the opportunity to evaluate cost and quality  information before receiving the service.   2.  Cover a broad range of providers. Websites that provide  information for all or most of the available providers in a given  geographic area, regardless of network status or practice setting, give  consumers more information about their full range of options. For  example, for procedures that can be conducted in either a hospital  outpatient department or ambulatory surgical center, it helps  consumers to provide comparable information for both settings, so  that consumers can choose from a larger number of providers that  offer those procedures.   3.  Describe key differences in clinical quality of care, particularly  patient-reported outcomes. Assessments of the clinical quality of  care that have been shown to have particular relevance to consumers  are those that relate to long-term outcomes of the care experienced  by other patients. Often this is best addressed by patient-reported  outcomes, which tell consumers the eventual outcome of treatments,  as reported by previous patients of a particular provider. For example,  patients receiving hip replacements can be asked, through such  patient-reported outcomes, to rate their ability to climb stairs both  before and after their procedures, which enables assessments of the  procedures’ effects on patients’ mobility.   4.  Describe key differences in patient experiences with providers.
Another outcome that matters is patients’ assessment of their  interactions with providers. Effective websites include information on  how past patients have evaluated providers in terms of dimensions  such as how well nurses communicate with patients, or the  responsiveness of clinicians to patients’ needs.  5.  Describe other information related to quality, where appropriate.
There may be other quality indicators that could have major  significance to consumers for certain types of services. For example,  facility inspection results and staffing levels are of particular relevance  to nursing home care.  6.  Provide timely Information. More recent data are intrinsically more  relevant than data that are several years old. Because consumer  websites necessarily rely on past data to assess likely cost and quality  performance in the future, some lag in collecting, analyzing, and  providing data is inevitable. Data that are no more than two years old  are generally considered timely.  7.  Describe key strengths and limitations of the data. Although the  research we reviewed shows that few consumers are inclined to delve  into the many methodological issues that concern appropriate  techniques for collecting, checking, and analyzing cost and quality  data, effective websites can provide both summary assessments of  strengths and limitations for most consumers, and links to more  complete explanations for those wanting to pursue these issues in  greater detail. Such information, along with identification of the  organization responsible for the website, provides consumers a basis  to judge the credibility of the cost and quality information provided.
section_title: Appendix III: Examples of Research Assessing the Completeness and Accuracy of Clinical Information in Patient Records
paragraphs: In recent years, a number of researchers have examined methods for  assessing the quality of clinical information recorded in electronic medical  records, including data completeness and accuracy. A wide variety of  approaches have been proposed and used, depending on the clinical  focus of the research and the institutional context in which the clinical  information is recorded. The following provides some examples drawn  from this body of research.   1.  Nicole Gray Weiskopf and Chunhua Weng, “Methods and dimensions  of electronic health record data quality assessment: enabling reuse for  clinical research,” Journal of the American Medical Informatics  Association, vol. 20, no. 1 (2013), 144–151.
This article identifies a range of different tests that had been applied  by various researchers to determine both data completeness and data  accuracy in electronic patient records. Most of these tests involved  comparison with another data source, such as paper records, patient  interviews, or other alternative data sources, that served as a “gold  standard” for purposes of assessing the information recorded in the  electronic medical record. Other tests involved comparison between  different pieces of information recorded in the electronic medical  record.  2.  Philip J.B. Brown and Victoria Warmington, “Data quality probes— exploiting and improving the quality of electronic patient record data  and patient care,” International Journal of Medical Informatics, vol. 68  (2002), 91-98.
The article describes the use of “data quality probes” to assess data  quality by matching the result of database queries relative to  established clinical knowledge for a given condition.  3.  Edwin R Faulconer and Simon de Lusignan, “An eight-step method for  assessing diagnostic data quality in practice: chronic obstructive  pulmonary disease as an exemplar,” Informatics in Primary Care, vol.  12, no. 4, (2004), 243–53.
This article describes an eight-step methodology for assessing the  completeness and accuracy of diagnoses for a specific condition.   4.  Michael G. Kahn, Marsha A. Raebel, Jason M. Glanz, Karen  Riedlinger, and John F. Steiner, “A Pragmatic Framework for Single- site and Multisite Data Quality Assessment in Electronic Health  Record-based Clinical Research,” Medical Care, vol. 50, no. 7 (2012),  S21-S29.
This article proposes an approach that prioritizes assessment of  selected variables and data quality dimensions; iterative cycles of  assessment within and between sites; targeting assessment toward  data domains known to be vulnerable to quality problems; and  detailed documentation of the rationale and outcomes of data quality  assessments to inform data users. It presents a comprehensive set of  data quality rules that look for anomalies in data values and  distributions as well as inconsistencies across related variables.  5.  William R. Hogan and Michael M. Wagner, “Accuracy of Data in  Computer-based Patient Records,” Journal of the American Medical  Informatics Association, vol. 4, no. 5 (1997), 342–355.
This article emphasizes the importance of assessing data  completeness and accuracy in conjunction with each other. It  recommends adopting a continuous improvement approach to  improving data quality based on regular cycles of monitoring, analysis  of errors, and interventions to address the factors found to cause  errors.
section_title: Appendix IV: Comments from the Department of Veterans Affairs
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Rashmi Agarwal, Assistant  Director; Eric Peterson, Analyst in Charge; Dee Abasute; Krister Friday;  Jacquelyn Hamilton; Wati Kadzai, and Vikki Porter made key  contributions to this report.
section_title: Related GAO Products
paragraphs: VA Health Care: Improvements Needed in Data and Monitoring of Clinical  Productivity and Efficiency. GAO-17-480 (Washington, D.C.: May 24,  2017).
Federal Health Care Center: VA and DOD Need to Develop Better  Information to Monitor Operations and Improve Efficiency. GAO-17-197  (Washington, D.C.: Jan. 23, 2017).
Veterans Health Care: Improvements Needed in Operationalizing  Strategic Goals and Objectives.GAO-17-50 (Washington, D.C.: Oct. 21,  2016).
VA Primary Care: Improved Oversight Needed to Better Ensure Timely  Access and Efficient Delivery of Care. GAO-16-83 (Washington, D.C.:  Oct. 8, 2015).
Health Care Transparency: Actions Needed to Improve Cost and Quality  Information for Consumers. GAO-15-11 (Washington, D.C.: Oct. 20,  2014).
VA Health Care Management and Oversight of Consult Process Need  Improvement to Help Ensure Veterans Receive Timely Outpatient  Specialty Care. GAO-14-808 (Washington, D.C.: Sep. 30, 2014).