Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
USING SOFTWARE TO DETECT 
UPCODING OF HOSPITAL BILLS 
JUNE GIBBS BROWN 
Inspector General 
August 1998 
OEI-01-97-00010  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 
protect the integrity of the Department of Health and Human Services programs as well as the 
health and welfare of beneficiaries served by them.  This statutory mission is carried out through a 
nationwide program of audits, investigations, inspections, sanctions, and fraud alerts.  The 
Inspector General informs the Secretary of program and management problems and recommends 
legislative, regulatory, and operational approaches to correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Boston office prepared this report under the direction of Mark R. Yessian, Ph.D., Regional 
Inspector General.  Principal OEI staff included: 
BOSTON REGION  HEADQUARTERS 
Russell Hereford, Project Leader  Mark Krushat 
Kenneth Price  Tricia Davis 
Nicola Pinson  Linda Moscoe 
Brian Ritchie 
To obtain a copy of this report, please contact the Boston Regional Office by telephone at 
(617) 565-1050 or by fax at (617) 565-3751. 
Reports are also available on the World Wide Web at our home page address: 
http://www.dhhs.gov/progorg/oei  E X E C U T I V E  S U M M A R Y

PURPOSE 
The purpose of this study is to test the ability of commercial software products to identify 
Diagnosis Related Group upcoding in Medicare hospital bills. 
BACKGROUND 
Since 1983, Medicare has paid acute care hospitals for the care of its beneficiaries under a 
prospective payment system using Diagnosis Related Groups (DRGs).  In fiscal year 1996, 
expenditures for inpatient hospital care under this system totaled $77.6 billion. 
Improper Payments 
Improper hospital payments are a continuing concern in the Medicare program.  In its 
Chief Financial Officers audit of Medicare, the Office of Inspector General (OIG) 
estimates that in fiscal year 1997, $4.1 billion of DRG payments were inappropriate due to 
lack of medical necessity, insufficient or no documentation, or incorrect coding. 
One particular concern is upcoding of hospital bills, the practice of billing for a hospital 
stay more expensive than the one actually incurred.  In previous studies, we found 
upcoding in DRGs ranging from 7 to 13 percent. 
Commercial Upcoding Detection Software 
Dozens of vendors now offer upcoding detection software that locates potentially 
upcoded DRGs by analyzing electronic files of hospital bills.  These products are likely to 
become increasingly sophisticated as the state of the art of computing races ahead. 
In this inquiry we evaluated the ability of two promising products to identify DRG 
upcoding.  First, we used these products to identify hospital bills with suspected upcoded 
DRGs.  Then we used professional record reviewers to perform a blinded medical review 
on a sample of cases to assess how well the products predicted DRG upcoding at the 
hospital, DRG, and case levels. 
FINDINGS 
Hospital Level 
Hospitals identified by the software had an average upcoding rate of 11.5 percent, more 
than double the 5.3 percent average upcoding rate of the control hospitals. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
i 
Bills  However, the software also identified as high upcoders a substantial number of hospitals in 
which our medical record review identified few or no upcoded cases. 
DRG Level 
The software performed best at identifying upcoded cases in three DRGs that show the 
highest rates of actual upcoding: DRG 87, pulmonary edema and respiratory failure; DRG 
79, respiratory infections and inflammations; and DRG 144, other circulatory system 
diagnoses.  These three DRGs comprise 3.5 percent of all Medicare discharges, or about 
350,000 discharges per year. 
However, among the most commonly occurring DRGs, we found that the software was no 
more effective in identifying upcoded cases than among other DRGs. 
Case Level 
The software successfully identified between 50 and 60 percent of cases that were actually 
upcoded.  Over 40 percent of upcoded cases went undetected. 
However, only 10 to 20 percent of cases that the software identified as upcoded were, in 
fact, upcoded. 
CONCLUSIONS 
Our analysis of the software products provides some basis for optimism about the role that 
such products can play in detecting DRG upcoding.  Yet we temper that optimism with 
strong caution as to the current state of the art of this software and the need to couple its 
use with other measures in the detection and prevention portfolio. 
The software we examined showed modest success in identifying hospitals with a high rate 
of upcoding and upcoded cases within a narrowly defined group of DRGs that exhibited 
the most frequent upcoding.  Thus, software could be used to identify hospitals that may 
need close scrutiny either before or after Medicare pays them.  However, because these 
products were distinctly less successful for most other DRGs, we see only a limited role 
for these products at the current time. 
It is likely, however, that the software market will continue to develop over time, and that 
products such as these will advance in sophistication and become more useful as part of a 
fraud detection strategy.  No doubt HCFA will want to stay abreast of opportunities that 
this technology may present. 
VENDOR COMMENTS 
We provided copies of our draft report and our contractors report to the three vendors 
whose software products we tested.  We wish to express our appreciation to these 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
ii 
Bills  companies for their willingness to let us use their products in this inspection, and for their 
comments and analysis of our draft report. 
These companies raised two general points in their responses.  First, the companies 
indicated that their products could be modified in ways that address the Medicare 
population more directly, and that they are continuously updating and enhancing their 
products.  We note that our purpose was not to develop new software, but to test 
commercially available off-the-shelf software.  We did not modify the vendors software, 
nor did we ask them to modify the software or to develop a specific software product for 
this purpose. 
Second, they questioned the methods we used to test the software.  We stand by our 
methodology.  We tested the software in a way that we considered would be useful to an 
agency such as HCFA.  We took the softwares underlying individual claims based 
approach and aggregated the results of individual claims analysis to the provider level.  We 
then verified the software products performance by reviewing cases among a sample of 
the providers that the software identified as having a high rate of upcoding.  In our 
judgement, this was a practical extension of the software.  We used these products in a 
manner that might identify and focus on providers that bear additional scrutiny in a fraud 
prevention and detection effort. 
We also address the methodological issues that one of the vendors raised in its response to 
the report. 
We include the full text of each vendors comments in Appendix D. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
iii 
Bills  T A B L E  O F  C O N T E N T S

PAGE

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  i

INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

FINDINGS

Hospital Level  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

DRG Level  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Case Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  8

VENDOR COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

APPENDICES

A: Software Vendor Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

B: Testing Methodology  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1

C: Statistical Tables  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1

D: Text of Software Vendors Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .D-1

E: Endnotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-1

OEI-01-97-00010  Software to Detect Upcoding of Hospital 
Bills  I N T R O D U C T I O N

PURPOSE 
The purpose of this study is to test the ability of commercial software products to identify 
Diagnosis Related Group upcoding in Medicare hospital bills. 
BACKGROUND 
Since 1983, Medicare has paid hospitals for the care of its beneficiaries under a 
prospective payment system (PPS) using Diagnosis Related Groups (DRGs).  In fiscal 
year 1996, expenditures for hospital care under this system totaled $77.6 billion.1  Under 
PPS, payment to hospitals for each Medicare case is based on a hospital-specific payment 
rate, multiplied by the weight of the DRG to which the case is assigned.  Each DRG 
weight represents the average resources required to care for cases in that particular DRG 
relative to the average resources used to treat cases in all DRGs. 
Cases are classified into DRGs based on the principal diagnosis, up to eight additional 
diagnoses, and up to six procedures performed during the stay, as well as the age, sex, and 
discharge status of the patient.  Upon discharge, the physician summarizes information on 
a discharge face sheet.  A hospital coder then reviews the entire medical record and uses 
that information to assign the most appropriate codes from the International Classification 
of Diseases, 9th Revision, Clinical Modification (ICD-9-CM).  The hospital uses this 
information to prepare a claim for payment, which it forwards to the Medicare fiscal 
intermediary.  The intermediary applies a series of edits to the claim, then groups the ICD-
9-CM codes in the claim  into the appropriate DRG for payment to the hospital. 
Improper Payments 
Improper hospital payments are a continuing concern for Medicares Part A trust fund.  In 
its Chief Financial Officers audit of Medicare, the Office of Inspector General (OIG) 
estimates that in Fiscal year 1997, $4.1 billion of hospital payments were inappropriate 
due to lack of medical necessity, insufficient or no documentation, or incorrect coding.2 
One particular concern is upcoding of hospital bills, the practice of billing for a hospital 
stay more expensive than the one actually incurred.  In previous OIG studies, we found 
upcoding in DRGs ranging from 7 to 13 percent.3,4 
Commercial Upcoding Detection Software 
As the pressure on public and private insurers to eliminate improper payments has risen, 
the market for software to detect upcoding has experienced rapid growth.  Dozens of 
vendors now offer such software.  These products analyze the diagnostic and 
administrative data from each hospital bill in an electronic claims file to predict whether 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
1 
Bills  the DRG contained in the bill is upcoded.  Many vendors sell their products off the 
shelf ready to be installed and utilized with minimal investment and setup time.  These 
products are likely to become increasingly sophisticated as the state of the art of 
computing races ahead. 
This Inquiry 
Within its fiscal year 1996 Chief Financial Officers audit of Medicare, OIG recommended 
that HCFA enhance prepayment and postpayment controls by updating computer systems 
to better detect improper claims.  In this inquiry we evaluated the ability of two promising 
products to identify DRG upcoding through electronic analysis of hospital bills.  We chose 
these two products from a field of 21 vendors who offer similar products.  We intended 
this test to be illustrative of how software might complement HCFAs existing program 
integrity initiatives by functioning as one part of a broad strategy for DRG payment 
safeguarding.  We based our evaluation on a blinded medical record review of a national 
sample of 2,622 Medicare cases from 1996.  The review was performed by an independent 
contractor using accredited medical records professionals. 
METHODOLOGY 
We executed the test in two phases, using a contractor with expertise in statistical 
sampling and medical record review for highly specialized tasks. 
In phase one we used our contractor to search for vendors of upcoding detection 
software.  The search initially identified 57 vendors whose product description indicated 
some type of claim auditing software or services.  Further research of these 57 vendors 
reduced the list to 21 vendors who had software that appeared relevant to our study.  Out 
of these 21 software vendors, 3 agreed to participate in our study.5 
Using software from these three vendors, we analyzed 100 percent of Medicare inpatient 
claims from January through June of 1996 to identify claims that appeared to be upcoded.6 
Next, we collapsed the output from each software product to generate three lists of 
hospitals with high predicted rates of upcoding.  Through correlation analysis, we 
discovered a strong relationship between the lists of hospitals generated by two of the 
software products, while the list from the third product differed significantly.  Due to 
limits on the number of medical records we could review for this study, we decided to 
focus our inquiry by testing only the output from the two software products whose lists 
were closely correlated.7  Therefore, as the first step of our sample, we selected 50 
hospitals that both products predicted had high rates of upcoding.  We refer to this group 
of hospitals as our test sample. 
As a control, we also selected a sample of 20 hospitals that fell into similar size strata as 
our test sample but did not have high predicted rates of upcoding.  This brought the total 
number of hospitals in our study to 70  50 hospitals with high predicted rates of 
upcoding and 20 hospitals without high predicted rates of upcoding.  From each hospital, 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
2 
Bills  we sampled 40 Medicare inpatient admissions billed under any of the 50 most prevalent 
DRGs in 1996.  This brought the total number of cases in our study to 2,800.  We were 
able to obtain and complete analysis on 2,622 (94 percent) of these cases.  Tables C-1 
through C-4 in Appendix C present data on the characteristics of the hospitals and cases 
examined in this review. 
In phase two our contractor performed a blinded medical record review of each case.  For 
this task, the contractor used Registered Records Administrators, Accredited Records 
Technicians, Certified Coding Specialists, and physicians.  Based on the contents of the 
medical record, the contractor derived a new set of ICD-9-CM diagnostic and procedure 
codes and used them to generate a new DRG for each case.  If there was discrepancy 
between the new DRG and the DRG for which the hospital had billed Medicare, the case 
was referred for a second blind review to determine the final DRG.  If the contractor 
calculated a final DRG that was less expensive than the DRG the hospital billed Medicare 
for, we defined a case as upcoded.8  Thus, for each of the 2,622 cases in our review, we 
knew which had been properly coded and which had been upcoded. 
We then used this information to determine if the two software products successfully 
predicted whether each case in our sample had an upcoded DRG.  We analyzed these data 
on three levels: by hospital, by  DRG, and by case.  To perform hospital-level and DRG-
level analysis, we aggregated our data by hospital and DRG to compare actual and 
predicted rates of upcoding.  Case-level analysis examined the success of the software in 
predicting DRG upcoding on a case-by-case basis.  Our analyses used t-tests and logistical 
regression to determine statistical differences. 
A detailed description of the software vendor search and testing methodology appears in 
Appendices A and B. 
We conducted this study in accordance with the Quality Standards for Inspections issued 
by the Presidents Council on Integrity and Efficiency. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
3 
Bills  F I N D I N G S

Hospital Level 
Hospitals identified by the software had an average upcoding rate of 11.5 percent, 
more than double the 5.3 percent average upcoding rate of the control hospitals. 
The 50 hospitals in the test group  those hospitals that the software identified as having 
high rates of upcoding  did in fact exhibit higher upcoding rates than the hospitals in our 
control sample.  The 12 hospitals in which our medical records reviewers found the 
highest upcoding rates were in the test sample.  Seven of these test hospitals had upcoding 
rates of 25 percent or higher; more than half (26 out of 50) had an upcoding rate of 10 
percent or higher. (See Table I.) 
The 20 hospitals in the control sample tended to have low upcoding rates.  In the sample 
of cases from control hospitals, our reviewers found upcoding rates below 5 percent in 
half the hospitals.  They found no upcoded cases at all in 5 of the 20 control hospitals. 
We performed logistic regression analysis to control for the effects of additional variables 
related to the hospitals (e.g., teaching status, ownership) and individual cases (e.g., patient 
gender and age).  Even taking these variables into account, we found that the likelihood of 
a case being upcoded in hospitals identified by the software was almost twice as high as it 
was for hospitals in the control sample.  Table C-5 in Appendix C presents the full results 
of this analysis. 
Table I:  Upcoding Rates for Test and Control Hospitals 
Entire Sample  Test Hospitals  Control Hospitals 
(n=70)  (n=50)  (n=20) 
Average upcoding rate 
9.8%  11.5%  5.3% 
(t=3.57 p<.001) 
Number (percent) with 
7 (10%)  7  (14%)  0  (0%) 
upcoding $$25% 
Number (percent) with 
22  (31%)  19  (38%)  3 (15%) 
upcoding $$10% and <25% 
Number (percent) with 
41  (59%)  24 (48%)  17 (85%) 
upcoding <10% 
However, the software also identified as high upcoders a substantial number of 
hospitals in which our medical record review identified few or no upcoded cases. 
In 6 of the 50 hospitals in the test sample, our medical records reviewers found no 
upcoded cases, even though the software predicted that these hospitals would have high 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
4 
Bills  rates of upcoding.  In 15 of these 50 hospitals we found upcoding rates of 5 percent or 
less. 
At the same time, it is worth noting that our reviewers found upcoding rates of 10 percent 
or higher in 3 of the 20 control hospitals. 
DRG Level 
Measuring the Effectiveness of Software in Identifying Upcoded Cases 
The effectiveness of a software product can be measured along two dimensions, referred 
to as sensitivity and specificity.  Each dimension can be expressed as a percentage. 
Sensitivity measures the extent to which the software identifies all cases that have been 
upcoded. In our sample of 2,622 cases, our independent medical records reviewers 
determined that 254 cases (9.7 percent) had been upcoded.  A software product that was 
perfectly sensitive would identify all 254 of these cases. 
Specificity assesses the softwares efficiency. Specificity measures the softwares 
ability to discriminate between those cases that were upcoded and those cases that were 
not upcoded, i.e., the extent to which the software identifies only those cases that really 
were upcoded. If the software were perfectly specific, every case that it identified would 
be upcoded. 
An ideal product would be perfectly sensitive and perfectly specific  in our review, for 
example, a perfect product would have selected all 254 upcoded cases and omitted the 
other 2,368 cases. 
In reality, there often is a trade-off between sensitivity and specificity:  To achieve 
greater sensitivity, the software must cast a wide net; this means that it might identify 
some cases that were not really upcoded, referred to as false positives. Conversely, to 
achieve greater specificity, the software risks missing some cases that actually were 
upcoded; those cases that it misses are referred to as false negatives. 
We examined the performance of the software among two sets of DRGs which we 
consider potentially high risk to the Medicare program in terms of potential dollars lost: 
those DRGs in which we found a high level of upcoding and those DRGs which occur 
most frequently.  We examined these DRGs to determine whether the software might be 
most efficiently utilized by identifying a subset of DRGs that represent potentially high 
cost to the Medicare program, either because they exhibited high rates of upcoding or 
because of the sheer volume of cases. 
The software performed best at identifying upcoded cases in DRGs that show the 
highest rates of actual upcoding. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
5 
Bills  The software was most accurate in identifying cases in the three DRGs with the highest 
actual rates of upcoding.  These 3 DRGs comprise 3.5 percent of all Medicare discharges, 
or about 350,000 discharges per year: 
DRG 87 (Pulmonary edema and respiratory failure).  Our medical records 
reviewers found an actual upcoding rate of 41 percent.  The software products had 
sensitivity rates of 69 percent and 61 percent, and specificity rates of 60 percent 
and 47 percent. 
DRG 79 (Respiratory infections and inflammations, age > 17 with complications 
or comorbidities).  Our medical records reviewers found an actual upcoding rate of 
35 percent.  The software products had sensitivity rates of 95 percent and  55 
percent, and specificity rates of 36 percent each. 
DRG 144 (Other circulatory system diagnoses with complications or 
comorbidities).  Our medical records reviewers found an actual upcoding rate of 
30 percent.  The software products had sensitivity rates of 86 percent and 
71 percent, and specificity rates of 86 percent and 42 percent. 
For DRGs with lesserbut still highrates of upcoding, however, the software was less 
accurate.  For example, one software product flagged no cases in the DRGs with the 
fourth highest upcoding rate (DRG 239, with 24 percent actual upcoding) or the fifth 
highest upcoding rate (DRG 429, with 23 percent upcoding); the other product was only 
slightly more successful.  For informational purposes, we present data on sensitivity and 
specificity of the software for the 10 DRGs with highest rates of upcoding in Appendix C, 
Table C-6. 
One implication arising from this analysis is that once DRGs that exhibit high levels of 
upcoding have been found  for example, through ongoing case review and analysis of 
discharges  the software products may have a role to play in helping to identify specific 
cases within those DRGs that merit further scrutiny. 
We also examined how well the software performed in detecting case-specific upcoding 
among the group of 10 DRGs with the highest upcoding rates, versus the other DRGs we 
reviewed in this inspection. (See Table C-7 in Appendix C.)  We found no statistical 
difference between these two groups in the softwares sensitivity (i.e., its ability to identify 
upcoded cases).  We did, however, find that the software was more specific among those 
frequently upcoded DRGs.  In other words, those cases that the software did identify 
tended to be actually upcoded. 
However, among the most commonly occurring DRGs, we found that the software 
was no more effective in identifying upcoded cases than among other DRGs. 
The 10 most commonly occurring DRGs comprise 10 percent of all Medicare discharges, 
or about 1 million discharges per year.  Within our sample, they comprised 13 percent of 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
6 
Bills  cases reviewed, yet they accounted for 53 percent of upcoded cases.9  Table C-8 in 
Appendix C provides data on these 10 most commonly occurring DRGs. 
We examined how well the software performed in detecting case-specific upcoding among 
this group of 10 DRGs, versus the other DRGs we reviewed in this inspection. (See Table 
C-9 in Appendix C.)  We found no statistical difference in sensitivity or specificity for 
either product in its ability to detect upcoding among these 10 most commonly occurring 
DRGs, compared with the performance of the products in correctly identifying upcoded 
cases in other DRGs. 
Case Level 
Our sensitivity analysis showed that the software products successfully identified 
between 50 and 60 percent of cases that were actually upcoded. 
Our medical records reviewers determined that our sample contained 254 cases that had 
been upcoded.  Of these 254 upcoded cases, one product identified 133 (52 percent) of 
these cases, and the other product identified 147 cases (58 percent).  This sensitivity rate 
has an important implication: over 40 percent of cases that actually were upcoded went 
undetected by these products. 
However, our specificity analysis showed that only 10 to 20 percent of cases that the 
software products identified as upcoded were, in fact, upcoded. 
One product identified 685 cases as upcoded, but only 133 (19 percent) of these cases 
were determined by our reviewers to be upcoded.  For the other product, out of 1,284 
cases it identified as upcoded, 147 (11 percent) were determined by our reviewers to 
actually be upcoded.  Such a low specificity rate reduces the efficiency of the software as a 
detection tool by requiring that multiple cases be reviewed in order to locate each upcoded 
case.  In essence, for the product with 19 percent specificity, reviewers would need to 
examine 4 false leads to find 1 case that truly was upcoded.  For the product with the 11 
percent specificity rate, that review level rises to 9 false leads for each truly upcoded case. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
7 
Bills  C O N C L U S I O N S

The software we examined showed modest success in identifying hospitals with a high rate 
of upcoding.  Those hospitals that the software identified were twice as likely to have 
upcoded cases as a control group of hospitals.  This finding leads us to believe that the 
software could be used in an ongoing way to identify hospitals that are likely to upcode 
their Medicare bills. 
There are various approaches as to how the software might be applied at the hospital 
level.  For example, HCFA might wish to use this software as a tool in its post-payment 
recovery efforts.  Based on the results we found, HCFA could use such software to 
retrospectively identify hospitals in which it would be likely to find a high level of upcoded 
cases and commensurate overpayment.  Alternatively, the agency could use this software 
to identify hospitals that have previously demonstrated a tendency to upcode, and then 
perform focused review on cases from these hospitals prior to making payments. 
The software also showed some success within the narrowly defined group of DRGs that 
exhibited the most frequent upcoding.  Because the software was relatively successful in 
identifying particular cases that were upcoded among these DRGs, its use here could be 
expected to yield significant returns.  For post-payment recovery efforts, HCFA could opt 
to focus on cases in the upcoded DRGs; analogously, the software could be used 
prospectively to identify cases in particular DRGs for review prior to payment. 
At the same time, our review leads us to raise caution about these products, particularly at 
the individual case level.  While they worked well for the most frequently upcoded DRGs, 
our review determined that these products were distinctly less successful for other DRGs. 
It is for this reason that we see only a limited role, as described above, for these products 
at the current time. 
The two software products that we reviewed were illustrative of what was available on the 
market in the Spring of 1997.  We believe, however, that it is likely that the software 
market will continue to develop, and that products such as these will advance in 
sophistication and expand in their usefulness as part of a fraud detection strategy.  No 
doubt HCFA will want to stay abreast of opportunities that this technology may present. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
8 
Bills  V E N D O R  C O M M E N T S

We provided copies of our draft report and our contractors (FMAS) report to the three vendors 
whose software products we tested.  We wish to express our appreciation to these companies for 
their willingness to let us use their products in this inspection, and for their review of and 
comments on our draft report.  Their overall comments reflect support for analytical work of this 
nature; however, the vendors express some concerns about our application of the software and 
raise some questions about our methodology.  We address their comments here, and we include 
the complete text in Appendix D. 
We wish to address two points that the vendors raised regarding the manner in which we applied 
the software.  First, each vendor indicates that it is continuously updating and enhancing its 
products.  One company even notes specifically that the software system we tested could be 
modified in ways that address the Medicare population more directly.  We are confident that 
software enhancements undoubtedly will continue to expand the potential for products such as 
these to play an important role in fraud prevention and detection. 
The purpose of this inspection, however, was not to develop new software, but to test 
commercially available off-the-shelf software.  Our interest was in determining if products that 
were on the market at the time we conducted our review (Spring 1997) could prove useful in 
identifying hospitals that showed a high rate of DRG upcoding.  Consequently, we did not modify 
the vendors software, nor did we ask them to modify the software or to develop a specific 
software product for this purpose.  Rather, we utilized the vendors software packages on an as-
is basis. 
Second, the vendors raised concerns about our use of the software to go beyond identification of 
individual claims that may have been coded incorrectly.  We recognize that, to some extent, our 
test was a modification of the original intent of these software products, which is to detect 
specific clinical claims that are questionable.  In essence, we took this underlying approach and 
extended it.  We aggregated the results of the individual claims analysis in order to identify 
hospitals that the software showed have a tendency toward upcoding.  We then verified the 
software products performance by reviewing cases among a sample of the providers that the 
software identified as having a high rate of upcoding.  In our judgement, this was a logical 
extension of the software to the practical realities of how it could be used in the Medicare 
program.  We used these products in a manner that might identify and focus on providers that 
may bear additional scrutiny in a fraud prevention and detection effort. 
We also wish to address the methodological issues that Dhrystone Systems raised in its response 
to the report.  First, this vendor questions the methods that we used to select our sample, in that 
the sample contains outliers.  We stand by our methodology; indeed we designed the 
methodology specifically to concentrate on the outlying providers that are most problematic.  The 
experimental group comprised hospitals that the software identified as lying at least two standard 
deviations above the mean in the proportion of upcoding identified by the software.  The control 
group comprised all remaining hospitals. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
9 
Bills  Second, Dhrystone also questions the appropriateness of our limiting the universe of cases 
reviewed to the 50 most common DRGs.  In response, we note that we selected records from 
these 50 DRGs to focus our review effort.  These 50 DRGs comprise nearly 70 percent of all 
discharges, and over 60 percent of all Medicare PPS reimbursement.  Consequently, targeting 
these 50 DRG strikes us as a prudent means of focusing on where the greatest concentration of 
Medicare dollars lies.  We do not generalize these results to broader populations of DRGs or of 
hospitals. 
Dhrystone also states that the study purports to have been conducted in a double blind manner, 
and questions whether we did, in fact, do so.  In response, we note that our review was conducted 
in a blinded manner; but we do not claim it was a double blind study.  The initial review was 
conducted by a registered records coder in a fully blinded manner.  If the coder found a 
discrepancy, the record was then unblinded; the coder then compared the hospitals reasoning 
with her reasoning, and arrived at a determination of the appropriate coding.  If disagreement 
persisted between the coder and the hospital, a second blind review was conducted, and the 
results of both reviews compared.  In essence, this is a conservative way of conducting a review 
such as this.  It clearly gives the benefit of any initial doubt to the hospital.  We consider such a 
conservative approach to be prudent and likely reflective of any practical application of such 
software by HCFA and the Office of Inspector General. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
10 
Bills  A P P E N D I X  A

Software Vendor Search 
Our contractor, FMAS Corporation, consulted with the World Development Group 
(WDG) to search for commercial software vendors that had products designed to locate 
DRG upcoding using claims data.10,11 
To begin the search for software vendors, WDG identified and interviewed relevant 
experts and companies to obtain names of probable software vendors, additional experts, 
and any other relevant information to assist in the search.  Initially, WDG identified 33 
experts in health informatics, medical expert systems, electronic medical records, and 
Medicare Part A claims payment. 
WDG sent each expert a fax describing the purpose of the project, the software of interest, 
and the questions that it would ask in a telephone interview.  WDG contacted and 
interviewed 25 of the experts.  This effort resulted in the identification of 11 additional 
experts, 5 of which WDG interviewed.  In total, WDG interviewed 30 experts. 
Concurrent with interviewing experts, WDG conducted a literature and Internet search to 
identify relevant software vendors.  WDG searched the following print sources: 
<  1996 Annual Market Directory Issue.  Health Management Technology. 1996. 
<  Medical Hardware and Software Buyers Guide.  M.D. Computing 1995; 12 (6). 
<  Ankrapp, Betty (ed). Health Care Software Sourcebook.  Gaithersburg, MD: 
Aspen Publishers, Inc., 1996. 
<  Frisch, Bruce (ed). The HCP Directory of Medical Software. Brooklyn, NY: 
Healthcare Computing Publications Inc., 1996. 
The literature and Internet searches and interviews with industry experts identified 57 
vendors whose product description indicated some type of claim auditing software or 
services.  WDG faxed a letter to each of these vendors to determine if they sold a product 
that met the projects criteria of relevance.  This search identified three more vendors who 
claimed to have a relevant product. 
In total, 21 of the 57 vendors appeared to meet the initial criteria of relevance.  In 
preparation for the telephone interview, WDG sent a fax to these vendors. 
WDG interviewed 20 of the 21 vendors.  One vendor did not respond to repeated calls. 
Of the 20 vendors interviewed, 6 confirmed having a relevant product.  Interviews with 
the 6 confirmed vendors lasted an average of an hour.  During these interviews, WDG 
requested brochures and any other available product literature, as well as a contact name 
for the software testing phase of the study. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
A - 1 
Bills  Of the six vendors, five agreed to a test of their software with certain conditions.  WDG 
conducted follow-up interviews to obtain client references and discuss the test that would 
be conducted.  In preparation of these follow-up interviews, WDG developed questions to 
query vendors about their software, clients, and willingness to test their software.  WDG 
sent each of the 5 vendors a fax describing the purpose of the test and the topics to be 
discussed during the follow-up interview. 
WDG contacted and interviewed all 5 vendors.  Each interview lasted an average of 15 
minutes.  During these interviews, WDG requested as references the names of two payers 
or fiscal intermediaries.  If the vendor did not have payer or fiscal intermediary references, 
WDG accepted any client references.  Subsequently, WDG interviewed two client 
references for an average of 10 minutes each. 
Vendors concerns about the test fell into three categories:  1) the size of our test (5-10 
million claims records) was too large; 2) vendors were uncertain about how OIG would 
utilize the results of the test; and 3) OIGs desired layout of the output was not clear 
enough. 
Because of these concerns, only three vendors chose to remain in the study and participate 
in a test of their software. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
A - 2 
Bills  A P P E N D I X  B

Testing Methodology 
We executed the study in two phases, utilizing a contractor with expertise in medical 
record review and statistical sampling for highly specialized tasks.  In phase one, we 
located software products that might detect upcoding, used these products to generate a 
sample of hospitals, and drew a sample of medical records from these hospitals.  In phase 
two, we performed a DRG validation on each case in our sample and used the results of 
this validation to determine if the software products used in stage one accurately predicted 
DRG upcoding. 
We began phase one by issuing a Request for Proposals to locate a contractor with 
expertise in medical record review and statistical sampling to assist in the study.  We 
contracted with FMAS Corporation, a company with extensive experience performing 
case review and analysis for the health care programs of the U.S. Department of Health 
and Human Services and the Department of Defense.12 
FMAS worked with World Development Group (WDG) to locate vendors of software 
that detects DRG upcoding.  From a field of 57 probable vendors, WDG identified 3 
vendors that had relevant software and were willing to participate in our test. (See 
Appendix A). 
Sample Selection 
We used the software from these three vendors to process 100 percent of Medicare 
Prospective Payment System (PPS) cases from January through June 1996.13  As output, 
each software flagged cases that it deemed likely to have an upcoded DRG.  Next, we 
made 3 lists of hospitals with high predicted rates of upcoding by collapsing each 
softwares output by hospital.  Through correlation analysis, we discovered a strong 
relationship between the lists of hospitals from two of the software, while the list from the 
third software differed significantly.  This meant that we would have to draw two separate 
samples to have a sample of hospitals that was representative of hospitals identified by all 
three software.  Thus, due to limits on the number of medical records we could review for 
this study, we decided to focus our inquiry by testing only the output from the two 
software whose lists were closely correlated.14 
To build our experimental (test) sample, we first selected hospitals that either of the two 
software indicated had a predicted upcoding rate of the mean rate plus two standard 
deviations.  This process led to a group of 299 hospitals, which we stratified into three 
groups according to number of Medicare discharges in the 6-month file we analyzed: 300 
or fewer discharges, 301 to 1,000 discharges, and over 1,000 discharges.  Next, in 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
B - 1 
Bills  proportion to the total number of hospitals in each stratum, we randomly selected a total 
of 50 hospitals from across the 3 strata. 
From each hospital, we then randomly selected 40 Medicare cases billed under any of the 
50 DRGs that were most commonly used across the country during fiscal year 1996.  As a 
control sample, we executed the same sampling strategy to select 800 cases from 20 
hospitals that had did not have high predicted rates of upcoding.  This brought our total 
sample to 2,800 cases: 2,000 of which were from hospitals that had high predicted rates of 
upcoding, 800 of which were from hospitals that did not have high predicted rates of 
upcoding.  We then merged claims data from each case against Medicares Enrollment 
Data Base (EDB) to obtain beneficiary name and the Online Survey Certification and 
Reports (OSCAR) system to obtain hospital name and address.  We used this information 
to mail medical record request letters and case listings to the administrator of each hospital 
in our sample.  Hospitals sent medical records to the OIG, where we logged them, gave 
them a quality check, and assigned each a tracking number.  We then sent the records to 
FMAS for DRG coding validation. 
DRG Coding Validation 
During phase two of the study the contractor, FMAS, performed a DRG coding validation 
on 2,622 (94 percent) of the 2,800 records in our sample.  FMAS, using Registered 
Records Analysts and Accredited Records Technicians, performed a blinded record 
review, in which the original ICD-9-CM and DRG codes were hidden.  This review 
generated new ICD-9-CM codes and a new DRG code for each case in the sample.  When 
FMAS completed reviewing a record, it compared the new codes to the previously hidden 
codes used by the hospital.  Below is the DRG reconciliation process: 
If FMAS codes and the hospitals codes matched, FMAS noted the DRG as correctly 
coded by the hospital.  Depending on the specific ICD-9-CM codes assigned by FMAS, it 
assigned one of the following two reconciliation reason codes to the case: 
1.  Confirm: Face Sheet, UB-92, FMAS codes and DRGs match. 
2.  DRGs match, but there is some variance in codes. 
If FMAS codes initially disagreed with those of the hospital, FMAS still noted the 
hospitals DRG as correctly coded by the hospital if its reviewer agreed with the hospitals 
coding after performing an unblinded reconciliation review.  FMAS reviewer then 
assigned one of the following reconciliation codes to the case: 
3.	 DRGs differed because more than one diagnosis could have been the principal 
diagnosis according to guidelines and hospital selected principal diagnosis leading 
to lower-weighted DRG.  FMAS did not recode or regroup these cases either in 
software or on its hardcopy worksheet. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
B - 2 
Bills  4.	 DRGs differed because of a judgement-call situation not covered by guidelines or 
Coding Clinic.  FMAS reviewer gave the hospital the benefit of the doubt.  FMAS 
did not recode or regroup these cases either in software or on its hardcopy 
worksheet. 
5.	 DRGs differed but FMAS reviewer, upon reviewing hospitals codes/DRG, noted 
that the hospitals DRG was correct.  This was the only reconciliation reason 
category that FMAS recoded and regrouped in software and on the hardcopy 
worksheet so that its final DRG matched the initial hospital DRG. 
6.	 UB-92 DRG differed but hospital face sheet matched FMAS DRG.  FMAS did 
not recode or regroup these cases either in software or on its hardcopy worksheet. 
This category was selected whenever the codes on the face sheet would have led 
to the same DRG as the FMAS DRG, but the UB-92 DRG and related codes were 
different. 
7.	 FMAS reserved this reconciliation code for potential additional reconciliation 
reasons, but did not use it during the study. 
Whenever the DRGs differed after reconciliation, FMAS assigned the following 
reconciliation reason code to the case: 
8.	 DRGs differ.  Upon review of hospitals DRG codes, FMAS reviewer confirmed 
that FMAS DRG was correct based upon coding guidelines and Coding Clinic. 
FMAS did not recode or regroup these cases either in software or on its hardcopy 
worksheet.  FMAS recorded all applicable DRG variance reasons and one DRG 
variance type (described below) on its DRG variance worksheet.  FMAS then 
completed a second blinded review of the case using a different reviewer. 
Variance types for reconciliation reason 8: 
Misspecification: The narrative principal diagnosis, a secondary diagnosis, or a 
procedure is not supported by the medical record. 
Miscoding: The medical records department selected an incorrect ICD-9-CM 
numeric code for a correct narrative diagnosis or procedure. 
Resequencing: The hospital substituted a secondary diagnosis for the correctly 
attested and coded principal diagnosis. 
Other: The hospital made another type of error (such as incorrect discharge 
status) that led to DRG variance but cannot be categorized as numbers 1-3 above. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
B - 3 
Bills  OIG Analysis 
FMAS sent data for the completed medical record reviews to OIG in electronic format, 
keyed by our tracking number.  We merged these data with the original inpatient claims 
data and additional administrative data to create our analytical files for the study. 
We analyzed these files on three levels:  by hospital, by DRG, and by case.  To perform 
hospital-level and DRG-level analyses, we aggregated our data by hospital and DRG to 
compare actual and predicted rates of upcoding.  Case-level analysis examined the success 
of the software in predicting DRG upcoding on a case-by-case basis.  We used  t-tests 
and logistical regression to determine statistical differences.  We performed data analysis 
using SAS software.15 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
B - 4 
Bills  A P P E N D I X  C

Statistical Tables

TABLE C-1 
CHARACTERISTICS OF HOSPITALS REVIEWED 
Control Sample  Test Sample  Total 
(n=20)  (n=50)  Sample (n=70) 
n  (%)  n (%)  n (%) 
Number of Beds 
1-99  14  (70.0)  42  (84.0)  56  (80.0) 
100-299  5  (25.0)  4  (8.0)  9  (12.9) 
300+  1  (5.0)  4  (8.0)  5  (7.1) 
Teaching Status 
Teaching  3  (15.0)  10  (20.0)  13  (18.6) 
Nonteaching  17  (85.0)  40  (80.0)  57  (81.4) 
Location 
Metropolitan  11  (55.0)  10  (20.0)  21  (30.0) 
Nonmetropolitan  9  (45.0)  40  (80.0)  49  (70.0) 
Control 
For profit  3  (15.0)  3  (6.0)  6  (8.6) 
Nonprofit  14  (70.0)  20  (40.0)  34  (48.6) 
Government  3  (15.0)  27  (54.0)  30  (42.9) 
Number of 
Discharges, 1/96-6/96 
9  (45.0)  32  (64.0)  41  (58.6) 
1-300 
8  (40.0)  13  (26.0)  21  (30.0) 
301-1000 
3  (15.0)  5  (10.0)  8  (11.4) 
1001+ 
Source: OIG analysis of the FY 1996 Medicare Provider Analysis and Review (MEDPAR) file and data 
from the Online Survey Certification Reports (OSCAR) system. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
C - 1 
Bills  TABLE C-2 
COMPARISON OF TEST SAMPLE WITH ALL HOSPITALS WITH 
HIGH PREDICTED RATES OF UPCODING 
Test Sample  Total High Predicted 
(n=50)  Group (n=299) 
n  (%)  n (%) 
Number of Beds 
1-99  42  (84.0)  219  (73.2) 
100-299  4  (8.0)  49  (16.4) 
300+  4  (8.0)  31  (10.4) 
Teaching Status 
Teaching  10  (20.0)  62  (20.7) 
Nonteaching  40  (80.0)  237  (79.3) 
Location 
Metropolitan  10  (20.0)  89  (29.8) 
Nonmetropolitan  40  (80.0)  210  (70.2) 
Control 
For profit  3  (6.0)  30  (10.0) 
Nonprofit  20  (40.0)  124  (41.5) 
Government  27  (54.0)  145  (48.5) 
Number of 
Discharges, 1/96-6/96 
32  (64.0)  155  (51.8) 
1-300 
13  (26.0)  109  (36.5) 
301-1,000 
5  (10.0)  35  (11.7) 
1,001+ 
Source: OIG analysis of the FY 1996 Medicare Provider Analysis and Review 
(MEDPAR) file and data from the Online Survey Certification Reports (OSCAR) 
system. 
OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
C - 2 
Bills  TABLE C-3 
HOSPITAL CHARACTERISTICS BY CASE CHARACTERISTICS 
FOR CASES REVIEWED 
Control Sample  Test Sample  Total 
(n=744)  (n=1,878)  (n=2,622) 
n  (%)  n  (%)  n  (%) 
Number of Beds 
1-99  531  (71.4)  1,583  (84.3)  2,114  (80.6) 
100-299  175  (23.5)  156  (8.3)  331  (12.6) 
300 +  38  (5.1)  139  (7.4)  177  (6.8) 
Teaching Status 
Teaching  108  (14.5)  372  (19.8)  480  (18.3) 
Nonteaching  636  (85.5)  1,506  (80.2)  2,142  (81.7) 
Location 
Metropolitan  407  (54.7)  383  (20.4)  790  (30.1) 
Nonmetropolitan  337  (45.3)  1,495  (79.6)  1,832  (69.9) 
Control 
For profit  119  (16.0)  120  (6.4)  239  (9.1) 
Nonprofit  511  (68.7)  770  (41.0)  1,281  (48.9) 
Government  114  (15.3)  988  (52.6)  1,102  (42.0) 
Source: OIG analysis of the FY 1996 Medicare Provider Analysis and Review (MEDPAR) file and 
data from the Online Survey Certification Reports (OSCAR) system. 
OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
C - 3 
Bills  TABLE C-4 
BENEFICIARY CHARACTERISTICS FOR CASES REVIEWED 
Control Sample  Test Sample  Total 
(n=744)  (n=1,878)  (n=2,622) 
n  (%)  n (%)  n  (%) 
Age (years) 
<65  106  (14.3)  268  (14.3)  374  (14.3) 
65-74  196  (26.3)  489  (26.0)  685  (26.1) 
75-84  269  (36.2)  695  (37.0)  964  (36.8) 
85+  173  (23.3)  426  (22.7)  599  (22.9) 
Sex 
Male  313  (42.1)  827  (44.0)  1,140  (43.5) 
Female  431  (57.9)  1,051  (56.0)  1,482  (56.5) 
Race 
White  648  (87.1)  1,573  (83.8)  2,221  (84.7) 
Black  59  (7.9)  201  (10.7)  260  (9.9) 
Other  28  (3.8)  87  (4.6)  115  (4.4) 
Unknown  9  (1.2)  17  (0.9)  26  (1.0) 
Source: OIG analysis of the FY 1996 Medicare Provider Analysis and Review (MEDPAR) file. 
OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
C - 4 
Bills  TABLES C-5

RESULTS OF LOGISTIC REGRESSION MODEL

TABLE C-5A 
ODDS RATIOS ESTIMATES FOR STATISTICALLY 
SIGNIFICANT VARIABLES 
90% Confidence Interval 
Variable  Estimate  Lower  Upper 
Facility in Selected Group  1.94  1.41  2.66 
Teaching Hospital  0.52  0.37  0.74 
Publicly Owned  1.79  1.39  2.32 
High Case Mix Index  2.59  1.96  3.43 
Male  0.68  0.54  0.87 
Age 75 to 84  1.40  1.07  1.85 
Age 85+  1.60  1.18  2.17 
Flagged by Product A  3.49  2.73  4.46 
Flagged by Product B*  1.25  0.99  1.58 
*This variable was not significant. 
TABLE C-5B 
DEPENDENT VARIABLE IN LOGISTIC REGRESSION MODEL 
Case Actually Upcoded as Determined by Our Medical Records Reviewers 
Change  1 = DRG Upcoded (N=254)  0 = DRG not Upcoded (N=2,368) 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
C - 5 
Bills  TABLE C-5C 
INDEPENDENT VARIABLES IN LOGISTIC REGRESSION MODEL 
Facility Characteristics 
Profit  1 = For profit  0 = Non profit 
Public  1 = Public  0 = Non profit 
Location  1 = Nonmetropolitan  0 = Metropolitan 
Teaching  1 = Teaching  0 = Nonteaching 
Smallbed  1 = 1-99 beds  0 = 100-299 beds 
Bigbed  1 = 300 + beds  0 = 100-299 beds 
FewDC  1= 300 or fewer discharges  0 = 301 - 1,000 discharges 
ManyDC  1 = Over 1,000 discharges  0 = 301 - 1,000 discharges 
LowCMI  1 = CMI less than 0.9  0 = CMI between .9 and 1.1 
High CMI  1 = CMI over 1.1  0 = CMI between .9 and 1.1 
ExpSamp  1= Facility in test group of hospitals  0=Facility in control group of hospitals 
Patient Characteristics 
Gender  1 = Male  0 = Female 
Black  1 = Black  0 = White 
Other  1 = Other  0 = White 
Unknown  1 = Unknown  0 = White 
Young  1 = Under 65  0=65-74 
Seven5  1 = 75-84  0=65-74 
Eight5  1 = 85 and Older  0=65-74 
Case Characteristics 
Surgical  1 = Surgical Claim  0 = Nonsurgical Claims 
Software Characteristics 
A_Hit  1 = Flagged by Software A  0=Not flagged by Software A 
B_Hit  1= Flagged by Software B  0= Not flagged by Software B 
OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
C - 6 
Bills  TABLE C-6 
SOFTWARE PERFORMANCE ON THE 10 DRGS WITH HIGHEST RATES OF UPCODING 
Cases  Percent 
DRG  Product A  Product B 
Reviewed  Upcoded 
(% of Medicare discharges)  (% of reviewed cases)  Sensitivity  Specificity  Sensitivity  Specificity 
87  Pulmonary edema & 
32 
respiratory failure (0.6%)  41%  69%  60%  62%  47% 
(1.2%) 
79  Respiratory infections & 
186 
inflammations age >17  35%  95%  36%  55%  36% 
(7.1%) 
w/cc (2.2%) 
144 Other circulatory system 
23 
diagnoses w/cc (0.7%)  30%  86%  86%  71%  42% 
(0.9%) 
239 Pathological fractures & 
25 
musculoskeletal & conn  24%  0%  0%  33%  18% 
(1.0%) 
tiss malignancy (0.5%) 
429 Organic disturbances & 
13 
mental retardation (0.4%)  23%  0%  N/A*  33%  50% 
(0.5%) 
416 Septicemia age >17 
(2.0%)  84 
20%  94%  23%  82%  22% 
(3.2%) 
475 Respiratory system 
26 
diagnosis with ventilator  19%  100%  20%  100%  26% 
(1.0%) 
support (0.9%) 
188 Other digestive system 
17 
diagnoses age >17 w/cc  18%  0%  0%  67%  20% 
(0.6%) 
(0.6%) 
121 Circulatory disorders 
54 
w/AMI & C.V. comp  15%  25%  15%  88%  15% 
(2.1%) 
disch alive (1.5%) 
316 Renal failure (0.8%) 
34 
15%  20%  33%  20%  6% 
(1.3%) 
*Note:

Sensitivity = N/A when we found no upcoded cases within a DRG, i.e., the denominator in our sensitivity

calculation is zero.

Specificity = N/A when the software did not flag any cases within a DRG, i.e., the denominator of our specificity

calculation is zero.

OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
C - 7 
Bills  TABLE C-7 
SOFTWARE PERFORMANCE ON TEN DRGS WITH HIGHEST RATES OF 
UPCODING VERSUS ALL OTHERS 
MEAN  STD DEV  t  P< 
PRODUCT A SENSITIVITY 
Top 10 Upcoded DRGs  0.4894  0.4367 
All Others (25 with a score*)  0.2763  0.3351  1.558  n/s 
PRODUCT A SPECIFICITY 
Top 10 Upcoded DRGs (9 with a score*)  0.3045  0.2787 
All Others (25 with a score)  0.0891  0.1038  2.265  .10 
PRODUCT B SENSITIVITY 
Top 10 Upcoded DRGs  0.6115  0.2596 
All Others (25 with a score)  0.4808  0.3560  1.203  n/s 
PRODUCT B SPECIFICITY 
Top 10 Upcoded DRGs  0.2819  0.1479 
All Others (36 with a score)  0.0594  0.0826  4.562  .05 
*Note:

When a DRG may not have a sensitivity score: DRGs with no upcoding will not have a sensitivity score, as the

sensitivity denominator, the number of upcoded cases, is zero. Fifteen of the 50 DRGs in our sample had no

upcoding.

When a DRG may not have a specificity score: DRGs that had no cases flagged by the software will not have a 
specificity score, as the specificity denominator, the number of flagged cases, is zero. Sixteen DRGs had no cases 
flagged by Product A. Four DRGs had no cases flagged by Product B. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
C - 8 
Bills  TABLE C-8 
SOFTWARE PERFORMANCE ON THE 10 MOST COMMONLY OCCURRING DRGS 
DRG  Cases Reviewed  Product A  Product B 
(% of Medicare discharges)  (% of reviewed cases)  Sensitivity Specificity  Sensitivity  Specificity 
127  Heart failure & shock (6.3%) 
245 
0%  0%  55%  5% 
(9.3%) 
89  Simple pneumonia & pleurisy age 
299 
>17 w/cc (4.0%)  11%  4%  67%  8% 
(11.4%) 
14  Specific cerebrovascular disorders 
129 
except TIA (3.4%)  58%  14%  58%  8% 
(4.9%) 
88  Chronic obstructive pulmonary 
163 
disease (3.3%)  0%  0%  0%  0% 
(6.2%) 
209 Major joint & limb reattachment 
47 
procedures- lower extremity (3.2%)  N/A*  N/A*  N/A  0% 
(1.8%) 
79  Respiratory infections & 
186 
inflammations age >17 w/cc (2.2%)  95%  36%  55%  36% 
(7.1%) 
174 G.I. hemorrhage w/cc (2.2%) 
82 
25%  7%  38%  8% 
(3.1%) 
182 Esophagitis, gastroent & misc digest 
105 
disorders age >17 w/cc (2.1%)  10%  5%  30%  7% 
(4.0%) 
296 Nutritional & misc metabolic 
108 
disorders age >17 w/cc (2.1%)  36%  26%  50%  10% 
(4.1%) 
112 Percutaneous cardiovascular 
9 
procedures (2.0%)  N/A  N/A  N/A  0% 
(0.3%) 
*Note:

Sensitivity = N/A when we found no upcoded cases within a DRG, i.e., the denominator in our sensitivity

calculation is zero.

Specificity = N/A when the software did not flag any cases within a DRG, i.e., the denominator of our specificity

calculation is zero.

OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
C - 9 
Bills  TABLE C-9 
SOFTWARE PERFORMANCE ON TEN MOST COMMON DRGS VERSUS 
ALL OTHERS 
MEAN  STD DEV  t  P< 
PRODUCT A SENSITIVITY 
10 Most Common DRGs (8 with a score*)  0.2944  0.3314 
All Others (27 with a score)  0.3499  0.3896  0.364  n/s 
PRODUCT A SPECIFICITY 
10 Most Common DRGs (9 with a score*)  0.1159  0.1327 
All Others (27 with a score)  0.1529  0.2028  0.484  n/s 
PRODUCT B SENSITIVITY 
10 Most Common DRGs (8 with a score)  0.4405  0.2128 
All Others (27 with a score)  0.5412  0.3611  0.745  n/s 
PRODUCT B SPECIFICITY 
10 Most Common DRGs  0.0817  0.1050 
All Others (36 with a score)  0.1150  0.1430  0.684  n/s 
*Note:

When a DRG may not have a sensitivity score: DRGs with no upcoding will not have a sensitivity score, as the

sensitivity denominator, the number of upcoded cases, is zero. Fifteen of the 50 DRGs in our sample had no

upcoding.

When a DRG may not have a specificity score: DRGs that had no cases flagged by the software will not have a 
specificity score, as the specificity denominator, the number of flagged cases, is zero. Sixteen DRGs had no cases 
flagged by Product A. Four DRGs had no cases flagged by Product B. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
C - 10 
Bills  APPENDIX D

Software Vendors Comments

OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
D - 1 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 2 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 3 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 4 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 5 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 6 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 7 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 8 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 9 
Bills  OEI-01-97-00010  )))))))))))  Software to Detect Upcoding of Hospital
D - 10 
Bills  A P P E N D I X  E

Notes 
1.  Department of Health and Human Services, Health Care Financing Administration, Office of 
the Actuary.  Personal communication April, 1998. 
2.  Department of Health and Human Services, Office of Inspector General, Report on the 
Financial Statement Audit of the Health Care Financing Administration for Fiscal Year 1997, 
A-01-97-00520, May 1998. 
3.  Department of Health and Human Services, Office of Inspector General, National DRG 
Validation Study Special Report on Coding Accuracy, OAI-12-88-01010, February 1988. 
4.  Department of Health and Human Services, Office of Inspector General, National DRG 
Validation Study Update: Summary Report, OEI-12-89-00190, August 1992. 
5.  The main issues of concern for those declining to participate fell into three categories:  1) the 
size of our test (5-10 million claims records) was too large; 2) vendors were uncertain about how 
OIG would utilize the results of the test; and 3) OIGs desired layout of the output was not clear 
enough. 
6.  We used the Medicare Provider Analysis and Review (MEDPAR) file as input for the software 
products.  This file contains diagnostic, billing, and beneficiary demographic data for each stay in 
an inpatient hospital by a Medicare beneficiary.  Our test ran approximately 6 million MEDPAR 
records through each software product. 
7.  The fact that output from one vendors software differed significantly and that we decided not 
to test it is in no way a reflection on the potential merit of that software. 
8.  Although not the purpose of this evaluation, we also kept track of cases that were undercoded 
(i.e., cases in which the hospital billed for a less expensive DRG than it should have).  Our review 
found that out of 2,622 cases, 124 cases (4.73 percent) were undercoded while 254 cases (9.69 
percent) were upcoded. 
9.  The 10 most frequent DRGs in Medicare comprise a higher percentage of the discharges in our 
sample compared to the all Medicare discharges (13 percent versus 10 percent) due to our 
sampling strategy.  We sampled only among the top 50 most common DRGs. 
10.  FMAS Corporation.  11300 Rockville Pike.  Rockville, MD 20852. 
11.  World Development Group, Incorporated.  5101 River Road, Suite 1913.  Bethesda, MD 
20816-1574. 
12.  FMAS Corporation.  11300 Rockville Pike, Rockville, MD 20852. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
E - 1 
Bills  13.  This represents about 6 million admissions.  We used the Medicare Provider Analysis and 
Review (MEDPAR) file as input. 
14.  The fact that output from one product differed significantly and that we decided not to test it 
is in no way a reflection on the potential merit of that product. 
15.  SAS Institute, Inc. SAS Campus Drive, Cary, NC 27513. 
OEI-01-97-00010  ))))))))))) Software to Detect Upcoding of Hospital 
E - 2 
Bills