Department of Health and Human Services 
OFFICE OF 
 
INSPECTOR GENERAL 
 
 
 
   
 
 
 
CMS V H
    ALIDATED  OSPITAL 
 
I Q
  NPATIENT  UALITY 
 
R P D ,
EP ORTING  ROGRAM  ATA  
 
B S U
  UT  HOULD  SE 
 
A T T
DDITIONAL  OOLS  O 
 
I G  
DENTIFY  AMING
 
Daniel R. Levinson 
Inspector General 
 
April 2017 
OEI-01-15-00320   
Report i n Brief   
April 2017 
 
OEI-01-15-00320 
 
 
 
Why OIG Did This Review   CMS Validated Hospital Inpatient Quality 
Accurate data are fundamental to the  Reporting Program Data, But Should Use 
integrity of the Centers for Medicare 
Additional Tools to Identify 
& Medicaid Services (CMS) quality- Key Takeaway 
based payment programs, several of  Gaming  
For payment year 2016, CMS 
which rely on data from Hospital 
validated sufficient IQR data to 
Inpatient Quality Reporting (IQR).   What OIG Found 
meet its regulatory 
This evaluation focuses on CMSs 
For payment year 2016, CMS met its  requirement, and it took action 
efforts to ensure the integrity of IQR 
regulatory requirement by validating  against the few hospitals that 
data. These data are used to adjust 
sufficient IQR data, which are used to adjust  failed validation.  However, 
payments on the basis of quality  payments on the basis of quality.  Almost   CMS made limited use of 
measures; thus, inaccurate data pose  analytics that can help identify 
99 percent of hospitals that CMS reviewed 
risks to payment accuracy.  CMS and  gaming of quality data. We 
passed validation, and CMS took action 
the Centers for Disease Control and  recommend that CMS make 
against the six that failed, including reducing 
Prevention (CDC) issued a Joint  better use of analytics to ensure 
their Medicare payments.  In addition, CMS 
Reminder regarding their concerns  the accuracy of Medicare 
and CDC offer training to hospitals to help 
that data was being manipulated, or  payment adjustments based on 
gamed, by hospital staff who did not  improve the accuracy of the quality data that  these data. 
hospitals report.  However, CMSs approach 
follow CDC definitions for reportable 
infections.  This report assesses  to selecting hospitals for validation for 
CMSs validation efforts and  payment year 2016 made it less likely to identify gaming of quality 
recommends ways to strengthen  reporting (i.e., hospitals manipulating data to improve their scores).  CMS 
program integrity safeguards.  did not include any hospitals in its targeted sample on the basis of their 
  having aberrant data patterns.  Targeting hospitals with aberrant patterns 
How OIG Did This Review   for further review could help identify inaccurate reporting and protect the 
We analyzed CMS validation data for  integrity of programs that make quality-based payment adjustments. 
payment year 2016 to determine the 
What OIG Recommends 
number of hospitals that CMS 
To identify potential gaming or other inaccurate reporting of quality data, 
selected for validation, why CMS 
we recommend that CMS make better use of analytics to ensure the 
selected them, and the outcome of 
integrity of hospital-reported quality data and the resulting payment 
the validation.  We conducted 
adjustments.  CMS could use analytics to select an increased number of 
structured interviews with five 
hospitals in its targeted validation sample.  It could analyze the data to 
stakeholder experts about any 
identify outliers (i.e., hospitals with data patterns that are substantially 
concerns they had about hospital 
different from other hospitals), determine which of those outliers warrant 
quality data or CMSs validation.  We 
further review, and then add them to the sample.  For example, CMS 
also conducted interviews with CMS 
could use analytics to identify hospitals with abnormal percentages of 
and CDC staff regarding any quality 
patients who had infections present on admission; this might help identify 
assurance activities or analyses they 
hospitals that engage in some of the data manipulation highlighted in 
conduct on the quality data.  Finally, 
CMS and CDCs Joint Reminder.  CMS concurred with our 
we reviewed training materials that 
recommendation.  
CMS and CDC offered to hospitals on 
how to report their quality data.   
Full report can be found at http://oig.hhs.gov/oei/reports/oei-01-15-00320.asp 
   
TABLE OF CONTENTS 
 
Objectives ....................................................................................................1 
Background ..................................................................................................1 
Methodology ................................................................................................7 
Findings........................................................................................................9 
For payment year 2016, CMS met its regulatory requirement by 
validating sufficient IQR data, which are used to adjust payments 
on the basis of quality ......................................................................9 
Almost 99 percent of hospitals that CMS reviewed passed 
validation, and CMS took action against the six that failed, 
including reducing their Medicare payments ...................................9 
CMS and CDC offer training to hospitals to help improve the 
accuracy of quality data that hospitals report ................................10 
However, CMSs approach to selecting hospitals for validation  
for payment year 2016 made it less likely to identify gaming of 
quality reporting ............................................................................. 11 
Conclusion and Recommendation .............................................................14 
Agency Comments and Office of Inspector General Response .....16 
Appendixes ................................................................................................17 
A:  Agency Comments ...................................................................17 
Acknowledgments......................................................................................19 
 
 
 
 
   
   
OBJECTIVES 
1.  To determine the extent to which the Centers for Medicare & Medicaid 
Services (CMS) validated hospital-reported inpatient quality data in 
accordance with regulatory requirements,  
2.  To determine the extent to which CMSs approach to validation is 
likely to identify gaming, and  
3.  To assess the outcomes of CMSs validation. 
BACKGROUND  
Accurate data are fundamental to the integrity of CMSs quality-based 
payment programs, several of which rely on data from Hospital Inpatient 
Quality Reporting (IQR).  These data are used to adjust payments on the 
basis of quality measures, so inaccurate data pose risks to payment 
accuracy.  This evaluation focuses on CMSs efforts to ensure the integrity 
of hospital-submitted data regarding healthcare-associated infections 
(HAIs) and clinical process of care. 
Hospital Inpatient Quality Reporting Program 
Beginning in fiscal year (FY) 2005, CMS required Medicare inpatient 
acute-care hospitals to report quality data or be subject to a payment 
reduction.  For FYs 2005 through 2006, Medicare reduced payment by 
0.4 percentage points for hospitals that did not submit the required data for 
those years.  For FYs 2007 through 2014, Medicare reduced payment by 
2 percentage points for failure to submit data.  Beginning in FY 2015 and 
all subsequent fiscal years, the reduction is set at one-quarter of the 
hospital market basket update.1  (The market basket update is 2.7 percent 
for FY 2017.2)  IQR measures include clinical process of care, such as 
whether aspirin was provided to patients with acute myocardial infarction; 
HAIs, such as bloodstream infections associated with central lines; claims-
based clinical outcomes measures, including readmissions; patient 
experience of care, which is measured by the Hospital Consumer 
Assessment of Healthcare Providers and Systems survey; and Medicare 
spending per beneficiary and payment measures.3  
                        ____________________________________________________________ 
1 Social Security Act  1886(b)(3)(B)(vii)-(viii); 42 CFR  412.64(d)(2); 81 Fed. Reg. 
56939 (Aug. 22, 2016). 
2 The CMS market baskets are used to update payments and cost limits in the various 
CMS payment systems.  The CMS market baskets reflect input price inflation facing 
providers in the provision of medical services. 
3 Social Security Act  1886(b)(3)(B)(viii). 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  1 
Gaming (OEI-01-15-00320) 
   
   
Hospitals submit HAI data, including infection data and information used 
to find the predicted number of infections for each hospital, to the 
infection database of the Centers for Disease Control and Prevention 
(CDC).  This database is called the National Healthcare Safety Network 
(NHSN).4, 5  CDC provides the numbers of reported and predicted 
infections for each hospital to CMS.  Hospitals submit data for the other 
measures directly to CMS via claims or through CMSs QualityNet 
website.  CMS posts the quality data on its Hospital Compare website.  
See Exhibit 1 for an overview of how hospitals report the various IQR 
measures.   
 Exhibit 1:  How Hospitals Report Certain IQR Data 
Category of Measures  Examples of Measures  How Hospitals Report the Measures 
Clinical Process of Care  Heart attack patients are given  Data are captured in claims sent to 
fibrinolytic medication within   CMS or are uploaded via QualityNet. 
30 minutes of arrival; pneumonia 
patients are given the most 
appropriate initial antibiotics 
HAIs  Central-line-associated bloodstream  Hospitals provide facility data and 
infections; catheter-associated  report infection cases to the NHSN.  
urinary tract infections  CDC then provides the number of 
reported and predicted infections to 
CMS. 
Patient Experience of Care  Hospital Consumer Assessment of  Hospitals can self-administer the 
Healthcare Providers and Systems  survey or use an approved survey 
survey  vendor.  Survey data are uploaded to 
the CMS data warehouse. 
Spending per beneficiary  Price-standardized payments for all  Data are captured in claims sent to 
Part A and Part B services provided  CMS. 
from 3 days prior to a hospital 
admission through 30 days after the 
hospital discharge 
Source:  CMS, Hospital IQR Program FY 2017 Reference Checklist and interviews with CMS and 
CDC staff. 
Medicare Programs Informed by IQR Data 
CMS uses the IQR data in certain Medicare programs, such as the 
Hospital Value-Based Purchasing Program, Hospital-Acquired Condition 
Reduction Program, and Hospital Readmissions Reduction Program, 
among others. 
The Hospital Value-Based Purchasing Program can trigger payment 
increases or payment reductions for participating hospitals.  Since fiscal 
year 2013, CMS has used a changing subset of IQR measures for this 
                        ____________________________________________________________ 
4 76 Fed. Reg. 51631 (Aug. 18, 2011). 
5 75 Fed. Reg. 50190, 50202 (Aug. 16, 2010). 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  2 
Gaming (OEI-01-15-00320) 
   
   
program.  In fiscal year 2017, hospitals may receive a payment increase or 
reduction in their Medicare payments of up to 2 percent.6 
The Hospital-Acquired Condition Reduction Program, which began in 
fiscal year 2015, reduces payments to hospitals with the highest rates of 
hospital-acquired conditions.  This program also uses a subset of 
IQR measures.  The Hospital-Acquired Condition Reduction Program 
includes HAI measures and other measures, including rates of pressure 
ulcers and hip fractures.  CMS reduces payments by 1 percent to the 
hospitals that have overall scores in the quartile with the worst 
performance.7   
The Hospital Readmissions Reduction Program began in fiscal year 2013.  
This program reduces payments to hospitals that have readmission rates 
that are higher than expected.  (A readmission is the admission of a patient 
to any hospital within 30 days of being discharged from a hospital.)  CMS 
uses a subset of IQR data to calculate expected readmission rates for 
hospitals, and it penalizes hospitals with rates that exceed these expected 
rates.8 
How CMS Validates IQR Data 
CMS has regulatory authority to validate IQR data by reviewing medical 
records from participating hospitals.9  In accordance with regulatory 
requirements, CMS validates the clinical-process-of-care measures and the 
HAI measures by annually selecting a random sample of 400 participating 
hospitals, and then requesting a sample of medical records from those 
hospitals.10  It validates the data for the two domains separately, then 
combines the outcomes into one overall score.  The year of reporting that 
CMS validates lags 1 to 2 years behind the payment year that the 
validation will affect.  For example, CMS validated data for payment year 
2016 using data on hospital discharges from the second half of 2013 
through the first half of 2014.   
In addition to what it is required to validate, CMS can also select 
a targeted sample of up to 200 additional hospitals on the basis of certain 
criteria.  Most of these criteria place hospitals into the targeted sample if 
they reach a certain thresholdfor example, if they failed validation the 
                        ____________________________________________________________ 
6 Social Security Act  1886(o). 
7 P.L. No. 111-148,  3008 (March 23, 2010), as amended by the Health Care 
Reconciliation Act of 2010, P.L. No. 111-152 (March 30, 2010), collectively known as 
the Affordable Care Act. 
8 Social Security Act  1886(q) and 42 CFR  412.150412.154. 
9 42 CFR  412.140(d). 
10 The Patient Experience of Care domain is not currently validated. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  3 
Gaming (OEI-01-15-00320) 
   
   
previous year, or if they submitted data to the NHSN after CMSs 
deadlines.  CMS also sets analysis-based criteria for the targeted sample, 
such as a hospitals having abnormal or conflicting data patterns.11  These 
criteria could include the use of analyticsfor example, identifying 
outliers on particular measures.  See Table 2 for a list of the criteria that 
CMS uses to select the targeted sample. 
 
Exhibit 2:  CMS Selection Criteria for the IQR Targeted Sample 
Selection Criterion   Description 
Threshold-Based Criteria 
Hospital failed to report to the NHSN at least 
Failed To Report Half of HAIs  half of actual HAIs as determined during the 
previous years validation 
In the previous years validation process, the 
Low Passing Score  hospitals score was in the range of 75% plus or 
minus the statistical margin of error 
Hospital reported HAI data within CDC deadline 
Late Reporting of HAIs 
but after CMS deadline 
Any hospital that has not been randomly 
Supplemental 
selected in the past 3 years  
New hospital that has not been randomly 
selected in the past 3 years (e.g., hospital has 
New Hospital  a participation date after January 1, 2011, but 
has not been selected for validation for  
fiscal years 20132015) 
Multiple Reasons  Multiple criteria 
Analysis-Based Criteria 
Inconsistencies, such as infection date coded  
Abnormal/Conflicting Data Patterns 
3 days after culture date  
Rapid Change in Data Patterns 
Hospital with rapidly changing data patterns 
Other Criteria 
CMS can select hospitals for other reasons* 
* For details, see the note following Exhibit 3, page 12. 
 
Source:  77 Fed. Reg. 5355253 (Aug. 31, 2012) and interviews with CMS staff.
To validate HAI measures, CMS first requests all lab culture results for the 
quarter being validated from certain units of sampled hospitals (e.g., 
intensive care units).  CMS uses the lab results to identify cases likely to 
have involved infections that are reportable to the NHSN.  It then selects 
                        ____________________________________________________________ 
11 77 Fed. Reg. 5355053553 (Aug. 31, 2012). 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  4 
Gaming (OEI-01-15-00320) 
   
   
a random sample of up to 12 of these cases for medical record review.  
Hospitals send the medical records for those cases to CMS, which then 
matches the medical records to the NHSN to determine whether the 
hospitals reported the infections documented in the medical records.12  For 
a given year of validation, CMS may review a total of up to 48 cases to 
validate a hospitals HAI measures.  For a given measure, such as 
bloodstream infections associated with central lines, CMS validates the 
number of infections (i.e., the numerator), but not the total number of 
patient-days that included central lines (i.e., the denominator). 
To validate the clinical process-of-care measures, CMS requests a sample 
of 15 medical records per quarter from sampled hospitals, including 
measures for acute myocardial infarction, pneumonia, and heart failure.  
CMS matches the medical records with the quality data that hospitals 
reported, or it identifies a mismatch.13  For a given year of validation, 
CMS may review a total of up to 60 cases to validate a hospitals 
process-of-care measures. 
CMS then calculates an overall score that combines the HAI and 
process-of-care validation results.  Because CMS reviews only a sample of 
records, it projects each hospitals score.  A hospital passes validation if its 
match ratethe rate at which CMS identifies a match between the 
medical records and the data that the hospital reportedis at least 
75 percent plus the statistical margin of error set by CMS.14  A hospital 
fails validation if the rate is below 75 percent minus the margin of error.  
A hospital neither passes nor fails if its projected score is within the range 
of 75 percent plus or minus the margin of error.  CMS may select 
a hospital that neither passes nor fails for targeted validation the following 
year.15  A hospital that fails validation can request a reconsideration, in 
which case the hospital must explain which data it believes were 
improperly validated by CMS and why it believes that they are correct.16 
CDC and CMS Concerns About HAI Data 
In October 2015, CDC and CMS issued a Joint Reminder on NHSN 
Reporting.17  This document highlighted concerns raised by staff at some 
                        ____________________________________________________________ 
12 77 Fed. Reg. 5354253548 (Aug. 31, 2012). 
13 76 Fed. Reg. 5164551648 (Aug. 18, 2011). 
14 42 CFR  412.140(d)(2); 77 Fed. Reg. 5355053551 (Aug. 31, 2012). 
15 77 Fed. Reg. 5355053551 (Aug. 31, 2012). 
16 42 CFR  412.140(e)(1)-(2). 
17 CDC and CMS, Adherence to the CDCs Infection Definitions and Criteria is Needed 
to Ensure Accuracy, Completeness, and Comparability of Infection Information (joint 
notice), no date.  Accessed at http://www.cdc.gov/nhsn/pdfs/cms/nhsn-reporting-
signed.pdf on March 1, 2017. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  5 
Gaming (OEI-01-15-00320) 
   
   
hospitals regarding three ways that hospitals may be deviating from 
CDCs definitions for reportable HAIs:  overculturing, underculturing, and 
adjudication.  These deviations could be ways in which hospitals game 
their quality data to obtain payment increases or avoid payment decreases. 
Overculturing:  Overculturing departs from standard clinical practice by 
ordering diagnostic tests in the absence of clinical symptoms.  Hospitals 
may order these tests when patients are admitted.  For example, hospitals 
may obtain a urine specimen taken from a patient who shows no 
symptoms of a urinary tract infection.  Many results are negative, 
subjecting patients to unnecessary tests.  Hospitals might use positive 
results to game their data by claiming that infections that appeared many 
days later during hospitalization were present on admission, and thus not 
reportable to the NHSN.18  
Underculturing:  Underculturing departs from standard clinical practice by 
discouraging the ordering of diagnostic tests in the presence of clinical 
symptoms.  Hospitals may engage in underculturing in cases in which 
a patient has likely developed an infection during the hospital stay.  By not 
ordering the test, the hospital does not learn whether the patient truly has 
an infection and therefore the hospital does not have to report an infection 
to the NHSN.  Furthermore, to deal with infections that are possible but 
unconfirmed, hospitals mightin the absence of diagnosestreat these 
patients with broad-spectrum antibiotics, contributing to poor antibiotic 
stewardship. 
Adjudication:  In the practice known as adjudication, administrative or 
clinical superiors inappropriately overrule the hospital staff who are 
responsible for reporting HAIs to the NHSN, with the result that the 
hospital does not report infections that should be reported per CDCs 
definitions for reportable HAIs. 
Finally, although the Joint Reminder on NHSN Reporting does not 
assert that these are widespread practices, it does remind hospitals that 
intentionally reporting incorrect data or deliberately failing to report 
required data may violate statute and regulations.  It recommends that 
hospital staff contact the Office of Inspector General (OIG) if they become 
aware of intentional deviations from CDCs reporting protocols. 
 
                        ____________________________________________________________ 
18 Hospitals can use a Present on Admission code on claims to indicate that a patient 
already had a particular condition upon admission. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  6 
Gaming (OEI-01-15-00320) 
   
   
METHODOLOGY 
Scope 
This inspection is national in scope.  It included data on CMSs validation 
of acute-care hospital inpatient quality reporting data used for payment 
year 2016 (October 1, 2015September 30, 2016), the most recent year for 
which complete data were available.  The data that CMS validated for 
payment year 2016 came from hospital discharges from the second half of 
2013 through the first half of 2014.  Specifically, this inspection examined 
CMSs validation of the HAI and process-of-care measures.  We did not 
examine the patient-experience-of-care measure, which is not subject to 
a validation process.   
Data Sources and Collection 
Validation Data.  We obtained summary data from CMS (maintained by 
its Clinical Data Abstraction Center contractor) on the hospitals that CMS 
selected for validation for payment year 2016; the reasons it selected 
hospitals for the targeted sample; and the number of hospitals that failed 
validation.  We analyzed the data to determine the number of hospitals 
selected for validation, the outcomes of the validation, and the most 
common reasons hospitals were selected for the targeted validation 
sample. 
Structured Interviews.  We conducted structured interviews with CMS 
staff regarding the actions taken with hospitals that failed validation; 
quality assurance activities that CMS conducts on the quality measure data 
it receives; hospital training and outreach; and concerns that CMS staff 
have regarding the data.  We conducted structured interviews with CDC 
staff regarding quality assurance activities they conduct on the NHSN 
data, and any concerns they have about the data and validation process.   
We conducted structured interviews with five stakeholder experts 
regarding their concerns about the validation process and data reporting.  
These experts included representatives from trade associations, members 
of the research community, and officials from a State department of health.  
We identified these experts through research, by reviewing published 
articles, and from individual recommendations from other stakeholders.   
Document Review.  We reviewed CMS and CDC training materials related 
to hospital outreach on data validation and submission (e.g., conference 
call agendas and related materials).  We analyzed CMS and CDC training 
materials to verify hospital training topics. 
Limitations 
We did not independently verify the results of CMSs validation process. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  7 
Gaming (OEI-01-15-00320) 
   
   
Standards 
We conducted this study in accordance with the Quality Standards for 
Inspection and Evaluation issued by the Council of the Inspectors General 
on Integrity and Efficiency. 
   
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  8 
Gaming (OEI-01-15-00320) 
   
   
FINDINGS 
For payment year 2016, CMS met its regulatory requirement by 
validating sufficient IQR data, which are used to adjust 
payments on the basis of quality 
CMSs validation for payment year 2016 comprised medical record 
reviews of hospital discharge data from the third quarter of 2013 through 
the second quarter of 2014. 
CMS validated data from a random sample of 400 hospitals, meeting its 
regulatory requirement, and also from a targeted sample of 49 hospitals.  
For this total of 449 hospitals, CMS reviewed 60 medical records per 
hospitala total of 26,940 recordsto validate process-of-care measures.  
It reviewed 11,097 cases to validate HAI measures. 
Of the 49 hospitals CMS selected for its target sample, it selected 
25 hospitals for failing to report half their HAIs and 14 hospitals for 
having low passing scores in the previous years validation process.  
Seven hospitals were selected on the basis of CMSs category Other.19 
Almost 99 percent of hospitals that CMS reviewed passed 
validation, and CMS took action against the six that failed, 
including reducing their Medicare payments 
Of the 449 hospitals in CMSs sample, 443 (98.7 percent) passed 
validation.  That means CMS found those hospitals data to have 
75 percent or more reliability.  Five hospitals from the random sample and 
one hospital from the targeted sample failed validation.20  The hospital 
from the targeted sample that failed also failed validation in the previous 
year.  CMS reduced the Medicare payment for these six hospitals by  
Six hospitals failed 
validation and, as  0.6 percent, which is one-quarter of the FY 2016 market basket update.21  
a result, received a  This is the same penalty for hospitals that do not submit IQR data.  These 
0.6 percent cut in their  hospitals will also be excluded from the Hospital Value-Based Purchasing 
Medicare payments.  Program and thus will be ineligible for any positive or negative payment 
adjustments made under that program the following year.22  The hospitals 
cannot appeal this action.  Excluding hospitals that submitted unreliable 
data maintains the integrity of the data that underlie this program. 
                        ____________________________________________________________ 
19 CMS did not provide a reason for three of the hospitals. 
20 A sixth hospital from the random sample initially failed validation but successfully 
appealed that determination. 
21 The FY 2016 market basket update was 2.4 percent.  See 80 Fed. Reg. 49508-49509 
(Aug. 17, 2015). 
22 We observe that this creates a situation in which a hospital that is penalized under the 
Hospital IQR Program may avoid a penalty under the Hospital Value-Based Purchasing 
Program. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  9 
Gaming (OEI-01-15-00320) 
   
   
CMS and CDC offer training to hospitals to help improve the 
accuracy of quality data that hospitals report 
Providing ongoing training for hospitals is important to reach new users, 
meet needs for refresher training, and to teach hospitals about changes to 
measures and definitions.  CMS revises the IQR measures annually; it 
revises definitions and, in some cases, adds entirely new measures.  CDC 
adjusts the HAI reporting definitions, as needed, in response to feedback 
from users and to incorporate advancements in infection detection.   
CMS and CDC offer training that attempts to reduce subjectivity in 
hospitals reporting by focusing on the reporting definitions, among other 
topics.  For example, for a patient with a central line and other clinical 
conditions, staff in one hospital may attribute an infection to the central 
line, whereas staff in another hospital may attribute the infection to one of 
the other clinical conditions.  This would affect how each hospital would 
report the rate of central line infections.  As a result, different hospitals 
would not be reporting rates that are based on consistent measurements. 
The experts with whom we spoke identified subjectivity in the HAI 
measures as a concern that could affect validation, though they also 
acknowledged that CDC and CMS are working to address this concern 
through training.  
CMS develops educational and resource materials and conducts outreach 
about IQR program requirements via a contractor.  The contractor hosts 
webinars and posts educational materials on its website.  It updates the 
materials about every 6 to 9 months to reflect CMSs revisions to the 
quality measures.  Past training sessions covered reporting to the NHSN; 
new IQR measures, such as sepsis rates; and the Hospital Consumer 
Assessment of Healthcare Providers and Systems.   
Additionally, hospitals that are being validated in a given year can opt to 
receive feedback.  Specifically, a hospital can ask CMSs validation 
contractor to conduct an educational review, in which the contractor 
reviews medical records that the validation process identified as 
mismatches and provides feedback to the hospital on how it can improve.  
In FY 2016, 35 hospitals took advantage of this review; in FY 2017,  
146 hospitals did so. 
CDC offers in-person training for staff from any hospital that submits data 
to the NHSN.  Training designed to improve data quality is also available 
to hospital staff as webinars, and includes training on definitions for 
reportable HAIs, entering data in the NHSN, and evaluating entered data 
for accuracy.  The stakeholders we interviewed who were concerned about 
subjectivity told us that this training helps address their concerns. 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  10 
Gaming (OEI-01-15-00320) 
   
   
However, CMSs approach to selecting hospitals for validation 
for payment year 2016 made it less likely to identify gaming of 
quality reporting 
CMS can use analytics to identify hospitals with data patterns that warrant 
additional review, such as patterns suggesting that a hospital is trying to 
game its reporting.  In fact, CMS can select up to 200 such hospitals for 
targeted validation.  However, despite recognizing the possibility of 
gaming in its 2015 Joint Reminder on NHSN Reporting (which 
highlighted three types of gaming related to HAIs:  overculturing, 
underculturing, and adjudication), CMS made limited use of these 
analytical tools in its validation for payment year 2016.  Gaming can 
involve hospitals manipulating their data to show better performance.   
CMS made limited use of analytics that can help identify 
suspected gaming 
A variety of analytic tools could help CMS identify hospitals that might be 
gaming the quality data that they report.  For example, CMS could identify 
hospitals that are outliers on particular measures (i.e., hospitals that have 
very high or low values on those measures compared to other hospitals) 
and include them in its targeted sample for validation.  If CMS confirms 
instances of gaming by certain hospitals, it could further analyze those 
hospitals data to refine its analytic tools to identify similar patterns of data 
from other hospitals that might also be gaming.     
These tools could be particularly useful with HAI data.  CMS relies on 
hospitals to submit complete and accurate data on lab cultures.  These data 
in turn become the basis for the individual medical records that CMS 
requests.  (CMS requests all lab cultures from hospitals, and then draws 
a sample of records on the basis of the results of those cultures.)  Hospitals 
could game their data by inappropriately excluding positive cultures that 
indicate infections.  This makes analytics an important tool to tease out 
hidden patterns in the data, which could help determine when the cause of 
variation might be the result of manipulation, rather than subjectivity in 
the measures or variation in medical practice.  CMS can analyze the HAI 
data to determine whether a hospitals reported rates are so low as to make 
that hospital an outlier that warrants further scrutiny. 
CMS did not include any hospitals in its targeted sample on 
the basis of aberrant data patterns 
Although CMS can include up to 200 hospitals in its targeted sample, it 
selected only 49 hospitals for payment year 2016.  Furthermore, CMS 
selected none of these hospitals using analysis-based criteria, such as 
aberrant data patterns or rapid changes in reporting.  CMS staff told us that 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  11 
Gaming (OEI-01-15-00320) 
   
   
they identified 96 hospitals with aberrant data patterns but did not select 
any of them for the targeted sample; instead, they prioritized the hospitals 
that had reported fewer than half of their HAIs.  In fact, CMS selected 
most of the hospitals for targeted review using threshold-based criteria.  
These criteria automatically place hospitals in the targeted sample because 
of an existing problem, such as failing to report HAI data.  See  
Exhibit 3 for the selection criterion of the hospitals that CMS selected. 
Exhibit 3:  Number of Hospitals Selected for Payment Year 2016 
Targeted Sample, by Selection Criterion 
Number of Hospitals 
Selection Criterion  
Selected 
Threshold-Based Criteria 
Failed To Report Half of HAIs  25 
Low Passing Score  14 
Late Reporting of HAIs  0 
Supplemental (hospital not 
0 
validated in prior 3 years) 
New Hospital  0 
Multiple Reasons  0 
Analysis-Based Criteria 
Abnormal/Conflicting Data 
0 
Patterns 
Rapid Change in Data 
0 
Patterns 
Other Criteria 
Other  7 
 
Notes:  Three targeted hospitals did not have targeting reasons available. 
The category of Other is not a criterion in its own respect, but rather a system designation.  
Hospitals that are in the Other category are randomly chosen by the system from any combination 
of meeting two of the following criteria: 
  Abnormal/conflicted data patterns (likely data entry errors) 
  Rapidly changing data patterns that improve quality 
  Submitted more than one NHSN case to CDC after CMS posted results 
  Any hospital that is new and/or has not been selected for validation in any of the previous 
3 years 
Source:  OIG analysis of CMS validation data and interviews with CMS staff. 
 
CMS and CDC analyze quality reporting data, but their analysis has not 
resulted in CMSs selecting hospitals for targeted review.  For example, 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  12 
Gaming (OEI-01-15-00320) 
   
   
CMS uses a contractor to analyze process-of-care data and claims data to 
evaluate two measures of HAI reporting:  surgical site infections and 
Clostridium difficile infections.  CDC analyzes NHSN data to identify 
outliers and reviews specific data elements related to changes in measure 
definitions or previously identified data inaccuracies.23   
Both CMS and CDC use edit checks (automated system processes) to help 
ensure the quality of data that hospitals enter into CMSs reporting portal 
and CDCs NHSN.  These edit checks test for acceptable value ranges for 
individual data elements and compare two or more data elements against 
one another for conflicting data.  For example, if a hospital enters a 
patients discharge date as having preceded the patients arrival date, the 
record would be rejected and the hospital would be required to fix the date 
discrepancy.  CMS and CDCs edit checks may be informed by experience 
over time with quality reporting data as a whole, but their purpose is to 
improve the quality and accuracy of individual records rather than to 
identify patterns within the data. 
CMS reported that it does not find claims data useful for identifying 
outliers in quality reporting.  For example, CMS staff told us that claims 
typically include a small number of HAIs and present-on-admission codes, 
making the claims a poor data source to identify outliers. 
CMS plans to obtain patient-level data to allow it to conduct 
analytics on hospitals HAI data 
CMS staff told us that to improve CMSs analysis of HAI reporting, they 
need access to the patient-level data that hospitals submit directly to the 
NHSN.  CMS and CDC staff told us that they are working together to 
provide CMS with patient-level HAI data.  CMS could use these data to 
identify gaming, outliers, and HAI underreporting, as well as to detect 
Currently, hospitals 
submit HAI to data  specific types of infection.  CMS could also use these data to better 
to CDC, which then  understand the reasons for mismatches identified during validation and 
provides aggregated  improve feedback to hospitals.  CDC has offered to pair its NHSN data 
data to CMS.  The  analysts with CMS analysts to assist CMS.  However, before CMS can 
two agencies are  obtain the data, it needs to address some technical and administrative 
working to give CMS 
matters.  CMS staff told us that the agency needs to establish the 
access to patient-level 
computing capacity for storing and safeguarding the data.  In addition, 
HAI data. 
CDC has asked CMS to ensure that CMS analysts have signed 
confidentiality agreements before using the NHSN data, and that CMS 
track and manage these agreements through a single point of contact.   
                        ____________________________________________________________ 
23 In one instance, CDCs analysis identified hospitals that incorrectly reported their 
Medicare provider numbers to the NHSN.  CDC informed CMS of this problem; in 
response, CMS initiated training for hospitals on how to report their provider numbers.   
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  13 
Gaming (OEI-01-15-00320) 
   
   
CONCLUSION AND RECOMMENDATION 
Through its validation of hospital IQR data, CMS has made important 
strides to ensure that Medicares payment adjustments linked to measures 
of quality are based on accurate data.  Hospitals have much at stake with 
their reporting of quality data, as do Medicare and its beneficiaries.  
Collecting and analyzing quality data is increasingly central to Medicare 
programs that link payments to quality and value.  Therefore, it is 
important for CMS to ensure that hospitals are not gaming their reporting 
of quality data.  
We found that CMS is validating data according to the process it 
established in regulation, and that most hospitals pass the validation.   
However, CMS has yet to take full advantage of data analytics in its 
process, leaving it less likely to identify patterns that might suggest 
gaming, including the practices that CMS and CDC listed in their 2015 
Joint Reminder on NHSN Reporting.  CMS is taking stepssuch as 
acquiring patient-level HAI datato enable it to use additional analytics.  
It is important that CMS follow through on that effort.  We recommend 
that CMS: 
Make better use of analytics to ensure the integrity of 
hospital-reported quality data and the resulting payment 
adjustments 
CMS should analyze IQR datacombining the patient-level HAI data 
with administrative data such as claimsto do the following:  identify 
hospitals with questionable patterns of data, prioritize hospitals that most 
warrant further review, and include those hospitals in the targeted sample 
for data validation.  For example, CMS could use analytics to identify 
hospitals with abnormal percentages of patients who had infections 
present on admission.  This might help identify hospitals that are engaging 
in overculturing patients, one of the concerns highlighted in the Joint 
Reminder on NHSN Reporting.  Obtaining patient-level HAI data from 
CDC will be helpful in these efforts.  Going further, CMS could use risk 
scoring to identify hospitals with a high possibility of manipulating their 
reporting.  CMS could use its experience with these hospitals to create and 
continuously improve models that identify hospitals most likely to be 
gaming their reporting. 
The benefits of using data analytics to target hospitals for validation 
extend beyond identifying and addressing gaming.  Analytics and 
patient-level HAI data may help CMS to improve its feedback to and 
training for hospitals, thereby improving the accuracy of quality reporting.  
Furthermore, using analytics may help CMS more accurately identify 
hospitals that are providing better care and those that are poor performers.  
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  14 
Gaming (OEI-01-15-00320) 
   
   
CMS could use such insight in its quality improvement effortsfor 
example, to select high-performing hospitals to serve as mentor hospitals, 
to target poor performers for outreach, and to assess the performance of its 
quality improvement contractors. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  15 
Gaming (OEI-01-15-00320) 
   
   
AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 
RESPONSE 
CMS concurred with our recommendation.  The agency stated that it will 
continue to evaluate the use of better analytics, such as including 
additional criteria for selecting hospitals in its targeted validation sample, 
and will use such criteria, as feasible. 
We support CMSs actions to continue to evaluate and implement the use 
of analytics.  OIG requests details on CMSs efforts and the results of 
those efforts in its final management decision. 
For the full text of CMSs comments, see Appendix A. 
 
   
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  16 
Gaming (OEI-01-15-00320) 
   
   
APPENDIX A 
Agency Comments 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
   
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  17 
Gaming (OEI-01-15-00320) 
   
   
 
 
 
 
 
 
 
 
 
 
 
 
 
ACKNOWLEDGMENTS 
This report was prepared under the direction of Joyce M. Greenleaf, 
Regional Inspector General for Evaluation and Inspections in the Boston 
regional office, and Kenneth R. Price, Deputy Regional Inspector General.  
Ivan Troy served as the team leader for this study.  Other Office of 
Evaluation and Inspections staff from the Boston regional office who 
conducted the study include Kimberly Ruppert.  Central office staff who 
provided support include Clarence Arnold, Evan Godfrey, and 
Christine Moritz. 
 
   
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  18 
Gaming (OEI-01-15-00320) 
   
   
ACKNOWLEDGMENTS 
This report was prepared under the direction of Joyce M. Greenleaf, 
Regional Inspector General for Evaluation and Inspections in the Boston 
regional office, and Kenneth R. Price, Deputy Regional Inspector General.  
Ivan Troy served as team leader for this study.  Other Office of Evaluation 
and Inspections staff from the Boston regional office who conducted the 
study include Kimberly Ruppert.  Central office staff who provided 
support include Clarence Arnold, Evan Godfrey, and Christine Moritz. 
 
CMS Validated Hospital Inpatient Quality Reporting Program Data, But Should Use Additional Tools To Identify  19 
Gaming (OEI-01-15-00320) 
   
Office of Inspector General
http://oig.hhs.gov  
 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95452, as  
amended, is  to protect the integrity of the Department of  Health and Human Services  
(HHS) programs, as  well  as the health  and welfare of individuals served by those programs.  
This statutory mission is carried  out through a nationwide network of audits, investigations,  
and inspections conducted by the following operating components: 
Office of Audit Services 
The Office  of  Audit Services  (OAS) provides auditing services  for HHS, either by  conducting  
audits  with its own audit resources or by  overseeing  audit work done by others.  Audits  
examine the performance of HHS programs and/or its grantees and contractors in carrying  
out their respective responsibilities and are intended  to provide independent assessments of 
HHS programs and operations.  These assessments help reduce waste, abuse, and  
mismanagement and promote economy and efficiency  throughout  HHS. 
Office of Evaluation and Inspections 
The Office  of  Evaluation and Inspections (OEI)  conducts national evaluations to  provide 
HHS, Congress, and the public with timely, useful, and reliable information on significant 
issues.  These evaluations focus on preventing fraud,  waste, or abuse  and promoting  
economy, efficiency, and effectiveness of departmental programs.  To promote impact, OEI 
reports also present practical recommendations for improving program operations.  
Office of Investigations 
The Office of Investigations (OI) conducts criminal, civil, and administrative investigations  
of fraud and misconduct  related to HHS programs, operations, and individuals.  With  
investigators working in all 50 States and the District of Columbia, OI  utilizes its resources 
by actively  coordinating with the Department  of Justice  and other Federal, State, and local 
law enforcement authorities.  The investigative efforts of OI often lead to  criminal 
convictions, administrative sanctions,  and/or  civil monetary  penalties.  
Office of Counsel to the Inspector General 
The Office of Counsel to the  Inspector  General (OCIG) provides general legal services to 
OIG, rendering  advice and opinions on HHS programs and operations and  providing all  
legal support for OIGs  internal operations.  OCIG represents  OIG in all civil and 
administrative fraud and  abuse cases involving HHS programs, including False Claims Act, 
program exclusion, and civil monetary penalty cases.  In  connection with these cases, OCIG 
also negotiates and monitors corporate integrity agreements.  OCIG renders advisory 
opinions, issues compliance program  guidance, publishes fraud alerts, and provides other  
guidance  to  the health care industry concerning the anti-kickback statute and other  OIG  
enforcement authorities.