Department of Health and Human Services
OFFICE OF 
  
INSPECTOR GENERAL 
 
 
 
 
 
 
 
 
ADVERS E EVENTS IN HOSPITALS:  
 
M  
ETHODS FOR 
 
 
I E  
DENTIFYING  VENTS 
 
 
 
 
 
 
Daniel R. Levinson  
Inspector General 
 
March 2010 
OEI-06-08-00221   
   
 E X E C U T I V E   S U M M A R Y  
OBJECTIVE
 
To evaluate the usefulness of selected methods for identifying events 
that harm hospitalized Medicare beneficiaries.    
BACKGROUND 
The term adverse event describes harm to a patient as a result of 
medical care or harm that occurs in a health care setting.  The term 
never events refers to a specific list of serious events, such as surgery 
on the wrong patient, that the National Quality Forum deemed should 
never occur in a healthcare setting.  The Tax Relief and Health Care 
Act of 2006 (the Act) mandated that the Office of Inspector General 
(OIG) report to Congress about such events, including making 
recommendations about processes for identifying events.  To meet the 
requirements of the Act, OIG published a series of reports in 2008 and 
will publish additional reports based on ongoing work.   
In 2008, we conducted a case study to determine the incidence of 
adverse events (hereinafter referred to as events) by reviewing a 
random sample of 278 Medicare beneficiary hospitalizations selected 
from all Medicare discharges from acute care hospitals in two selected 
counties during a 1-week period in August 2008.  Using a two-stage 
review process, the case study identified 120 events.  The first stage 
consisted of using five selected methods to screen for events, including 
nurse reviews of medical records, interviews of Medicare beneficiaries, 
two types of billing data analysis, and reviews of internal hospital 
incident reports.  Each time a screening method indicated the possibility 
that an event occurred during the hospitalization, we designated the 
possible event as a flag.  The second stage consisted of physician 
reviews of medical records for 183 of the 278 beneficiary 
hospitalizationsthose with at least 1 flag.  This report provides an 
indepth examination of the usefulness of the five screening methods 
used for identifying events.  We considered the most useful methods to 
be those that identified the greatest number of events. 
FINDINGS 
The five screening methods were useful in identifying events that 
harmed patients; however, most flags were not associated with 
events.   Physician reviews determined that 256 of the 662 flags  
(39 percent) generated by the screening methods were associated with  
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  i   
   
E X E C U T I V E   S U M M A R Y  
1 or more of the 120 events found by the case study.  Nurse reviews and 
one type of billing data analysis identified the greatest number of 
events.  Although the five screening methods helped to identify events, 
they also generated many flags (61 percent) that were not associated 
with events.     
Shortcomings in two screening methods have implications for 
Medicare payments and Federal initiatives to identify, track, and 
monitor events.  Our analysis revealed vulnerabilities regarding both 
accuracy and completeness of two critical sources of information about 
events.  Through analysis of the billing data, we found that diagnosis 
codes were inaccurate or absent for 7 of the 11 Medicare hospital-
acquired conditions identified by the case study.  These problems would 
prevent Medicares automated payment software from identifying the 
hospital-acquired conditions, which could result in Medicare 
overpayments and inhibit use of billing data to monitor quality of care 
in hospitals.  We also found that hospitals participating in the case 
study apparently did not have any internal incident reports for 112 of 
the 120 events (93 percent), including some of the most serious events 
involving death or permanent disability to the patients.  The lack of 
such reports could prevent hospitals from tracking events as required by 
regulation or reporting events to outside entities.  It also suggests that 
hospital incident-reporting systems may be an unreliable source of 
information for Patient Safety Organizations (PSO), which are entities 
that aggregate and analyze information about events voluntarily 
reported by hospitals. 
RECOMMENDATIONS 
Overall, the case study findings suggest that an effective way to identify 
events is through review of medical records by nurses and/or physicians, 
whereas other screening methods identified far fewer events.  
Additionally, it demonstrated opportunities to address shortcomings that 
limited the usefulness of some screening methods.   
Therefore, we recommend to the Centers for Medicare & Medicaid 
Services (CMS) and the Agency for Healthcare Research and Quality 
(AHRQ) that: 
CMS and AHRQ should explore opportunities to identify events 
when conducting medical record reviews for other purposes.  
Examples of such efforts include, but are not limited to:  CMSs Medicare 
Comprehensive Error Rate Testing and the work of Quality Improvement 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  ii   
   
E X E C U T I V E   S U M M A R Y  
Organizations; AHRQ grant awards for patient safety research; and State 
survey agency work in certifying hospital compliance with the Medicare 
conditions of participation. 
CMS should ensure that hospitals code claims accurately and 
completely to allow for identification of hospital-acquired conditions 
affected by Medicares payment policy.  To identify Medicare hospital-
acquired conditions, CMS relies on hospitals to code diagnoses and 
present on admission (POA) indicators (which indicate whether 
diagnoses were present upon admission).  CMS should determine 
whether additional guidance is needed to ensure that hospitals code 
diagnoses and POA indicators accurately and completely.   
CMS should provide interpretive guidelines for State survey 
agencies to assess hospital compliance with requirements to track 
and monitor adverse events.  The CMS State Operations Manual 
contains no guidance to State survey agencies regarding assessing 
hospital compliance with Federal requirements to track and monitor 
events; therefore, it is unclear how surveyors are to assess hospitals 
compliance. 
AHRQ should inform PSOs that internal hospital incident reporting 
may be insufficient to provide needed information about events to 
PSOs.  As the key Federal agency involved with PSOs, AHRQ indicated 
that PSOs will rely primarily upon hospitals for identifying, tracking, 
and reporting information about events.  However, we found that 
hospital incident reports existed for only 8 of the 120 events identified 
in the case study.  In providing technical assistance to support PSOs, 
AHRQ should convey the importance of hospitals having strong 
internal incident-reporting procedures. 
AGENCY COMMENTS 
We received comments on our draft of this report from AHRQ and CMS.  
AHRQ concurred with the report as written. 
CMS stated that it agrees with recommendations relevant to its 
programs. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  iii   
 T A B L E   O F   C O N T E N T S  
EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
 
I N T RO DU C T IO N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 
 
F I N D ING S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 
The five screening methods were useful in identifying events           
that caused harm; however, most flags were not associated with 
events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 
Shortcomings in two screening methods have implications for       
Medicare payments and Federal initiatives to identify, track,          
and monitor events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 
 
R E C OMME N DA TIO N S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 
Agency Comments and Office of Inspector General Response  . . . 19 
 
SCREENING METHODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 
 
APPENDIXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 
  A:  Tax Relief and Health Care Act of 2006 . . . . . . . . . . . . . . . . . . 29 
  B:  Medicare List of Hospital-Acquired Conditions . . . . . . . . . . . . 31 
  C:  Glossary of Selected Clinical Terms . . . . . . . . . . . . . . . . . . . . . 32 
  D:  Case Study Protocol for Nurse Reviews . . . . . . . . . . . . . . . . . . 34 
  E:  Medicare Beneficiary Interview Questions. . . . . . . . . . . . . . . . 36 
  F:  Patient Safety Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 
  G:  Case Study Protocol for Physician Reviews . . . . . . . . . . . . . . . 41 
  H:  Results of Screening Methods. . . . . . . . . . . . . . . . . . . . . . . . . . 48 
  I:   Agency Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 
 
A CKN OW LEDGMEN TS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54   
   
 I N T R O D U C T I O N  
OBJECTIVE 
To evaluate the usefulness of selected methods for identifying events 
that harm hospitalized Medicare beneficiaries.  
BACKGROUND 
Statutory Mandate and Office of Inspector General Response 
The Tax Relief and Health Care Act of 2006 (the Act) requires that the 
Office of Inspector General (OIG) report to Congress regarding the 
incidence of never events among Medicare beneficiaries, payment by 
Medicare or beneficiaries for services furnished in connection with such 
events, and the processes that the Centers for Medicare & Medicaid 
Services (CMS) uses to identify such events and deny or recoup 
payment.1, 2  (For relevant text of the Act, see Appendix A.)  The Act 
also requires that OIG make recommendations, as appropriate, 
regarding processes for identifying such events.  To meet the 
requirements of the Act, OIG published a series of reports and will 
publish additional reports based on ongoing work.3  
Expanding beyond the term never events, OIG studies use the term 
adverse event to allow for a more comprehensive examination of the 
topic.  As used in these studies, the term adverse event describes harm 
to a patient as a result of medical care or harm that occurs in a health 
care setting.  Although an adverse event often indicates that the care 
resulted in an undesirable clinical outcome and may involve medical 
errors, adverse events do not always involve errors, negligence, or poor 
quality of care and may not always be preventable.4   
 
1 The Act, P.L. 109-432  203. 
2 For purposes of the Act, the term never events means an event that is listed and 
endorsed as a serious reportable event by the National Quality Forum (NQF) as of 
November 16, 2006.  The Act,  203(d).  The NQF used the term Serious Reportable 
Events to describe a specific list of events associated primarily with patient death or 
serious disability that should never occur in a healthcare setting.  These colloquially 
became known as never events.  Available online at 
http://www.qualityforum.org/Topics/Safety.aspx.  Accessed on August 12, 2009. 
3 OIG, Adverse Events in Hospitals:  Overview of Key Issues, OEI-06-07-00470, 
December 2008; OIG, Adverse Events in Hospitals:  State Reporting Systems;                 
OEI-06-07-00471, December 2008; and OIG, Adverse Events in Hospitals:  Case Study of 
Incidence Among Medicare Beneficiaries in Two Counties, OEI-06-08-00220,          
December 2008; OIG, Adverse Events in Hospitals:  Public Disclosure of Information About 
Events, OEI-06-09-00360. 
4 R.M. Wachter, Understanding Patient Safety, McGraw-Hill, 2008. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  1   
   
I N T R O D U C T I O N  
Following a review of Medicare policies and expenditures, as well as 
consultation with officials from CMS and the Agency for Healthcare 
Research and Quality (AHRQ), we chose to focus much of our work on 
inpatient acute care hospitals.  In 2006, 12.5 million Medicare 
beneficiaries were hospitalized,5 with inpatient hospital costs 
constituting the largest portion of Medicare expenditures (32 percent in 
2006).6  Federal regulations require that hospitals, as a condition of 
participation in the Medicare and Medicaid programs, develop and 
maintain a quality assessment and performance improvement (QAPI) 
program.7  As a part of the QAPI program, hospitals must measure, 
analyze, and track quality indicators, including adverse patient 
events.8  To accomplish this, hospitals must track medical errors and 
adverse patient events, analyze their causes, and implement preventive 
actions and mechanisms that include feedback and learning throughout 
the hospital.9  State health agencies, otherwise referred to as State 
survey agencies, perform survey and review functions for Medicare and 
certify that hospitals comply with these Federal requirements.10 
Hospitals may also report information about adverse events to various 
entities, such as Patient Safety Organizations (PSO), which seek to 
improve quality of patient care by identifying and reducing the risks 
and hazards associated with care. 11, 12  PSOs must certify that they 
 
5 CMS, Statistics Book, p. 43, Table IV.1:  Medicare/short-stay hospital utilization,    
2008. 
6 Based on data from the Congressional Budget Office (CBO), Fact Sheet for CBOs 
March 2007 Baseline:  Medicare, March 7, 2007.  Available online at   
http://www.cbo.gov/budget/factsheets/2007b/medicare.pdf.  Accessed on September 8, 2008. 
7 42 CFR  482.21.  In the preamble to the final rule adding QAPI to the conditions of 
participation, CMS said QAPI focuses providers on the care delivered to patients, the 
performance of the hospital as an organization, and the effect of treatment.                         
68 Fed. Reg. 3435 (Jan. 24, 2003).  
8 42 CFR  482.21(a)(2). 
9 42 CFR  482.21(c)(2). 
10 42 CFR  488.10.  Providers accredited by an accreditation organization are deemed to 
be in compliance with Medicare conditions of participation.  42 CFR  488.5.    
11 Sections 923 and 924 of the Public Health Service Act, which were added by the 
Patient Safety and Quality Improvement Act of 2005, required the Department of Health 
and Human Services (HHS) to determine that PSOs meet certain criteria to perform 
patient safety activities and establish a Network of Patient Safety Databases to receive, 
analyze, and report on patient safety information submitted by the PSOs.  Patient Safety 
and Quality Improvement Act of 2005, P.L. 109-41  2, Public Health Service Act,  923 
and 924, 42 U.S.C.  299b-23 and 24.   
12 Other entities, such as States operating adverse event reporting systems and the Joint 
Commission, also accept reports from hospitals regarding adverse events. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  2   
   
I N T R O D U C T I O N  
have policies and procedures in place to perform patient safety 
activities, such as aggregation and analysis of reported events received 
from hospitals that voluntarily report patient safety information.13  The 
Secretary of Health & Human Services has delegated to AHRQ the 
responsibility for determining whether certifications submitted by 
entities seeking to be PSOs meet Federal requirements.14  The 
Secretary may also provide technical assistance to PSOs on matters 
such as methodology, communication, data collection, or privacy 
concerns.15  An AHRQ-funded study found that 98 percent of hospitals 
reported having adverse event reporting systems and nearly all 
hospitals have safety and quality functions that would facilitate 
participating in a PSO.16  HHS indicated that PSOs would rely 
primarily on existing hospital activities, such as internal procedures 
for identifying and reporting adverse events.17  
Present on Admission Indicators and Medicares Hospital-Acquired 
Conditions Policy 
Since October 1, 2007, hospitals have been required to assign a present 
on admission (POA) indicator to each principal and secondary diagnosis 
for acute Inpatient Prospective Payment System (IPPS) claims for all 
discharges.18  This was an initial step in complying with the Deficit 
Reduction Act of 2005 (DRA), which required CMS to select at least two 
hospital-acquired conditions for which hospitals would not be paid 
higher Medicare reimbursement than if the conditions had not occurred 
for discharges occurring on or after October 1, 2008.19  In the fiscal year 
(FY) 2009 IPPS Final Rule, CMS established the Medicare policy to 
deny hospitals higher payment for hospital admissions complicated by 
any of 10 categories of hospital-acquired conditions, that is, conditions 
that were not present upon admission.20  Appendix B contains a list of 
these 10 hospital-acquired conditions. 
 
13 Public Health Service Act,  924(a), 42 U.S.C.  299b-24(a).  
14 73 Fed. Reg. 70732 (Nov. 21, 2008).  
15Public Health Service Act,  925, 42 U.S.C.  922b-25.  
16 73 Fed. Reg. 70732, 70793 (Nov. 21, 2008). 
17 Ibid. 
18 CMS, CMS Manual System, Change Request 5679 (July 20, 2007).  
19 DRA, P.L. 109-171  5001(c)(1), Social Security Act,  1886(d)(4)(D),                             
42 U.S.C.  1395ww(d)(4)(D). 
20  FY 2009 Final IPPS Rule, 73 Fed. Reg. 48434, 48471-48491 (Aug. 19, 2008).     
Effective October 1, 2008, CMS began denying hospitals higher payment for Medicare 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  3   
   
I N T R O D U C T I O N  
Efforts by CMS to identify and deny payment for certain adverse events 
are part of a broader quality improvement initiative.  In 2005, CMS 
released its Quality Improvement Roadmap, which expressed a vision 
for health care qualityThe right care for every person every time.21  
To achieve this vision, CMS stated its commitment to safe, effective, 
timely, patient-centered, efficient, and equitable medical care.  A central 
part of quality improvement is the value-based purchasing (VBP) 
initiative, which ties payment to quality medical care.  Medicares 
hospital-acquired conditions policy is part of the VBP initiative.22   
In the preamble to the FY 2009 IPPS Proposed Rule, CMS indicated 
that evaluating POA indicators in Medicare billing data could be used to 
better understand and prevent the occurrences of hospital-acquired 
conditions.  CMS posed that this information could be used to measure 
hospital performance as part of the VBP program.  Such information 
could also be publicly reported, enabling consumers to make more 
informed choices about their health care.  Additionally, CMS indicated 
that researchers could use POA data in a variety of ways to increase 
understanding and identify best practices for prevention of hospital-
acquired conditions.23 
Case Study of Events That Harmed Hospitalized Medicare Beneficiaries 
To best respond to the Act, we conducted a pilot study in 2008 to 
determine the incidence of adverse events among the Medicare 
population in two selected counties so that we could learn about various 
methods for identifying adverse events.  The results of that study, 
Adverse Events in Hospitals:  Case Study of Incidence Among Medicare 
Beneficiaries in Two Counties (hereinafter referred to as case study), 
are the basis of this report.24   
In the case study, we found that an estimated 15 percent of hospitalized 
Medicare beneficiaries in two selected counties experienced adverse 
 
admissions complicated by these conditions.  CMS, CMS Manual System, Change      
Request 6189 (October 3, 2008). 
21 CMS Quality Improvement Roadmap.  Available online at 
http://www.cms.hhs.gov/CouncilonTechInnov/downloads/qualityroadmap.pdf.  Accessed on 
June 30, 2009. 
22 FY 2009 Final IPPS Rule, 73 Fed. Reg. 48434, 48471 (Aug. 19, 2008); CMS, Roadmap 
for Implementing Value Driven Healthcare in the Traditional Fee-for-Service Program,  
pp. 10-12.  
23 FY 2009 Proposed IPPS Rule, 73 Fed. Reg.  23528, 23561 (Apr. 30, 2008). 
24 OIG, Adverse Events in Hospitals:  Case Study of Incidence Among Medicare 
Beneficiaries in Two Counties, OEI-06-08-00220, December 2008. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  4   
   
II NN TT RR OO DD UU CC TT II OO NN   
events that resulted in harm during their hospital stays.25  The case 
study was based on a random sample of 278 Medicare beneficiary 
hospitalizations selected from all Medicare discharges from acute care 
hospitals in two selected counties during a 1-week period in August 
2008.  We calculated the estimated adverse event incidence rate as the 
percentage of Medicare beneficiaries with at least one qualifying 
adverse event.   
Qualifying adverse events met one or more of the following criteria:   
1.  included in NQFs list of Serious Reportable Events; 
2.  included in Medicares list of hospital-acquired conditions; and 
3.  resulted in a level of patient harm determined by physician 
reviewers to be associated with a prolonged hospital stay, 
permanent harm, life-sustaining intervention, or death.   
Further, we found that another 15 percent of beneficiaries experienced 
less serious occurrences that resulted in harm that was temporary but 
required intervention, referred to as temporary harm events in the 
case study.26  For the purposes of this report, we refer to all occurrences 
that harmed patients (both adverse events and temporary harm events) 
as events.  The case study identified a total of 120 events that affected 
sample beneficiaries.     
Methods for Identifying Events in the Case Study 
We used a two-stage review process to identify events in the case study.  
The first stage consisted of screening for possible events using the 
following five screening methods:  nurse reviews of medical records, 
interviews of Medicare beneficiaries, two types of billing data analysis, 
and reviews of internal hospital incident reports.  If any screening 
method identified a possible event, it was labeled a flag and the 
medical record proceeded to the second stage of review.  The second 
stage consisted of physician reviews of those medical records for which 
at least one of the screening methods indicated that an event had 
possibly occurred.  
First-Stage Review:  Screening Methods 
 Method 1:  Nurse Reviews.  Contracted registered nurses reviewed 
medical records for each sampled Medicare beneficiarys 
 
25 Case study, p. 10.  
26 Case study, p. 16.  
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  5   
   
I N T R O D U C T I O N  
hospitalization.  Nurses used a standardized review process developed 
by the Institute for Healthcare Improvement (IHI) as part of its 
Global Trigger Tool (GTT) protocol, which we adapted for use in the 
case study.27  The nurse review used IHIs GTT worksheet that listed 
54 triggers that could be found within a medical record to indicate 
the possibility of an event.  Examples of triggers include transfusions 
or a return to surgery.28  (For a glossary of clinical terms, see 
Appendix C.)  When a trigger was found, the nurse reviewer explored 
the medical record further to identify possible events and associated 
level of harm.  (See Appendix D for a copy of the nurse review protocol 
and the IHI GTT worksheet.) 
 Method 2:  Analysis of POA Indicators.  For the case study, we 
obtained administrative billing data directly from hospitals for each of 
the 278 sample Medicare beneficiary hospitalizations.29  We used 
POA indicators in the billing data to identify hospitalizations that 
may have had events.  When the POA indicator showed that a 
diagnosis was not present upon admission, we concluded that the 
condition developed during the hospital stay and might have been the 
result of an event.30  For example, a diagnosis code for acute renal 
failure that was not present upon admission may be a sign that an 
event occurred during the hospital stay. 
 Method 3:  Beneficiary Interviews.  We conducted telephone 
interviews with 220 of the 278 Medicare beneficiaries or their family 
members to learn about the medical care experienced during sampled 
hospitalizations.31  The interview protocol was designed to determine 
 
27 For more information about the IHI GTT protocol, see F.A. Griffin and R.K. Resar, IHI 
Global Trigger Tool for Measuring Adverse Events, IHI Innovation Series white paper.  
Cambridge, MA:  Institute for Healthcare Improvement Innovation Series 2007.  
28 The case study used the 2007 version of the IHI GTT worksheet.  A revised version 
with 53 triggers was published in 2009.  Available online at 
http://www.ihi.org/IHI/Results/WhitePapers/IHIGlobalTriggerToolWhitePaper.htm.  
Accessed June 15, 2009. 
29 Hospital administrative billing data may include up to 30 diagnosis codes with POA 
indicators.  In contrast, Medicare claims data submitted by hospitals to CMS for payment 
are limited to nine diagnosis codes with POA indicators. 
30 The ICD-9-CM system assigns diagnoses and procedure codes associated with 
hospitalizations and is maintained jointly by the National Center for Health Statistics 
(NCHS) and CMS.  NCHS, The International Classification of Diseases, 9th Rev., Clinical 
Modification (ICD-9-CM), 6th Edition, was issued for use beginning October 1, 2007. 
31 The remaining 58 beneficiaries or family members either could not be reached or 
declined to be interviewed. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  6   
   
I N T R O D U C T I O N  
whether beneficiaries experienced any episodes while in the hospital 
that might have involved events.  It also included questions about 
such topics as medications, procedures, infections, and falls.  
Interviews typically lasted 57 minutes, but often took longer when 
interviewees reported occurrences and the interviewer probed for 
details.  (See Appendix E for a list of interview questions.) 
 Method 4:  Hospital Incident Reports.  We requested that hospitals 
provide any internal incident reports, such as submissions to any 
hospital incident-reporting systems, adverse drug reaction reports, 
complaints, peer reviews, and mortality and morbidity reviews 
associated with the 278 sample Medicare beneficiary hospitalizations.  
Reports provided by hospitals included issues related to risk 
management, hospital infections, surgical management, and others. 
 Method 5:  Analysis of Patient Safety Indicators.  We applied AHRQs 
Patient Safety Indicator (PSI) software program to hospital 
administrative billing data for the 278 sample Medicare beneficiary 
hospitalizations.  AHRQ developed the PSI software to monitor health 
care quality using administrative data, such as patient demographics 
(e.g., age, gender), and diagnoses and procedure codes.32  The PSI 
software is based upon a series of algorithms that detect 20 provider-
level complications that indicate possible events (e.g., death of a low-
risk patient).33  (See Appendix F for a list of provider-level PSIs.) 
Flags.  As mentioned, each time a screening method indicated the 
possibility that an event occurred during the hospitalization, we 
designated the possible event as a flag.  For example, if a nurse review 
indicated that a patient contracted an infection during the sampled 
hospitalization, we considered that a flag.  If the nurse review also 
indicated that the same patient fell during the hospital stay, the fall 
was considered another flag for the same patient.  Similarly, possible 
events identified through any of the other screening methods were all 
considered flags.  Thus, each sampled hospitalization could have 
multiple flags from one or more screening methods.  Medical records for 
 
32 AHRQ, Guide to Patient Safety Indicators, AHRQ Quality Indicators.  March 2003.  
Available online at http://www.qualityindicators.ahrq.gov/archives/psi/psi_guide_v30.pdf.  
Accessed on June, 4, 2008. 
33 AHRQ also has seven area-level PSIs designed to indicate possible events on a 
regional level, which were not used for the case study. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  7   
   
II NN TT RR OO DD UU CC TT II OO NN   
hospitalizations with one or more flags were forwarded to the second 
stage of review (i.e., physician review of medical records). 
Second-Stage Review:  Physician Review of Medical Records 
During the second stage of the case study review, 3 contracted 
physicians reviewed medical records for 183 of the 278 beneficiary 
hospitalizationsthose with at least 1 flag.  The physicians reviewed 
both the full medical records and the information concerning each flag.  
Physicians completed a structured medical review protocol through 
which they described each incident that had been flagged in the first-
stage review and noted what documentation led to flagging the incident. 
The physicians then determined whether the incidents flagged in the 
first-stage review qualified as events for purposes of our study.  (See 
Appendix G for a copy of the Protocol for Physician Reviews of Medical 
Records.)   
To ensure consistency among the three physician reviewers, we 
conducted weekly conference calls to discuss issues and reach consensus 
about complex cases.  Additionally, after physicians initially identified 
122 events, we gave the associated hospitals an opportunity to provide 
additional information or documentation.  We received 18 hospital 
submissions; the additional information led the physicians to change 
their determinations for 2 cases and reduce the number of events to 120.  
Only events that were identified and confirmed by physicians were 
included in the case study results.   
METHODOLOGY 
Scope 
This study provides an in-depth examination of the usefulness of the 
five screening methods for identifying events in the case study.  The 
case study was based on a random sample of 278 Medicare beneficiary 
hospitalizations selected from all Medicare discharges from acute care 
hospitals in 2 selected counties during a 1-week period in August 2008. 
Data Collection and Analysis 
To evaluate the usefulness of the methods for identifying events,  
we compared all flags generated by each method to the 120 events 
identified and/or confirmed through physician reviews.  We classified 
each flag as either associated with an event (i.e., it provided 
information that led the physician to identify an event) or not 
associated with an event (i.e., the possible event did not meet our 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  8   
   
I N T R O D U C T I O N  
criteria to be classified as an event).  We then calculated the number 
and percentage of flags associated and not associated with events.  We 
considered the most useful methods to be those that identified the 
greatest number of events.  We also examined the types of events 
identified through each method and determined which events had no 
associated flags (i.e., events identified only through physician reviews of 
the medical records).   
Limitations  
Several aspects of the case study warrant mention because of possible 
implications and limitations for this report.  First, results of the case 
study can be projected only to the population of Medicare beneficiaries 
hospitalized in the two selected counties during a 1-week period in 
August 2008.  Second, it is possible that the case study underestimated 
incidence of adverse events in the sample.  Specifically, the two-stage 
methodology meant that physicians conducted medical record reviews 
for only 183 of the 278 Medicare beneficiary hospitalizations.  Therefore, 
it is unknown whether the 95 hospitalizations with no flags had any 
events.  Third, the case study included beneficiary interviews for only 
220 of the 278 sampled hospitalizations.  It is unknown whether 
interviews with the other 58 beneficiaries would have identified 
additional flags or events.  Fourth, it is possible that hospitals did not 
provide all incident reports associated with the Medicare beneficiaries 
hospitalizations.  Finally, because data collection occurred soon after the 
sample beneficiary hospitalizations, staff from a few hospitals indicated 
that the hospital billing data had not undergone complete data cleaning 
and validation.  We do not know whether this additional processing by 
hospitals would have affected any screening results.  
Standards 
This study was conducted in accordance with the Quality Standards for 
Inspections approved by the Council of the Inspectors General on 
Integrity and Efficiency.
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  9   
   
 F I N D I N G S  
The five screening methods were useful in  The 5 screening methods used in 
identifying adverse events; however, most flags  the case study generated a total of 
662 flags (i.e., possible events).  
were not associated with events 
(See Figure 1.)  Physician reviews 
determined that 256 of the 662 flags (39 percent) were associated with 
events and 406 of the 662 flags (61 percent) were not associated with 
events.  Because more than 1 flag identified many events, the 256 flags 
revealed a total of 114 events.  Physicians identified an additional  
6 (of the 120) events in their medical record reviews, finding events 
within cases that were flagged for other reasons.  (See Appendix H for 
an expanded graphic of the results of the screening methods.) 
Figure 1:  Flags and Events Identified Through Case Study 
Screening Methods
All Flags Identified 
in Case Study
(n = 662)
Flags Not Associated 
Flags Associated 
With Events 
With Events (n = 256)
(n = 406)
Events Identified  Unique Events Identified 
Only Through Physician  Through Screening 
Reviews Methods
(n = 6) (n = 114)
Events Identified in 
Case Study
(n = 120)
Source:  OIG analysis of 278 Medicare beneficiaries hospitalizations in 2 selected counties, 2008.
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  10   
   
F I N D I N G S  
Nurse reviews and POA analysis identified the greatest number of events  
Nurse reviews identified 93 of the 120 events in the case study and  
POA analysis identified 61 events.  (See Table 1.)  The third most  
useful screening method, beneficiary interviews, identified 22 events, 
and the remaining 2 screening methods identified 8 events each. 
Table 1:  Number and Percentage of Events Identified 
Through Each Screening Method 
Percentage of     
Method  Number of Events* 
120 Events 
Nurse Reviews  93  78% 
POA Analysis  61  51% 
Beneficiary Interviews  22  18% 
Hospital Incident Reports  8  7% 
PSI Analysis  8  7% 
Source:  OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008. 
*Column does not sum to 120 because many events were identified through more than 1 method. 
The usefulness of nurse reviews and POA analysis appears attributable 
to both the breadth and precision of each of these methods.  Both were 
broad, in that nurse reviews examined all documents in the medical 
records and POA analysis considered up to 30 diagnosis codes 
representing patient conditions.  This allowed for a review of a range of 
possible events unlike, for example, a hospital incident report related 
specifically to a medication error.  Both methods were also precise.  The 
nurse review protocol provided a clinical description of the health care 
condition; the ICD-9-CM codes used for POA analysis specified each 
patient diagnosis.  This specific clinical information later enabled the 
physicians to more easily pinpoint related information in the medical 
records. 
The usefulness of nurse reviews and POA analysis is further 
demonstrated by their performance in identifying events not flagged by 
any other screening methods.  Of the 120 events, 55 (46 percent) were 
identified by only 1 screening method.  Nurse reviews identified  
35 events not flagged by any other screening method, (29 percent of the 
120 events) and POA analysis alone flagged 14 events (12 percent).   
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  11   
   
F I N D I N G S  
Although the five screening methods were useful in identifying events,  
most flags were not associated with events 
As reported on page 11, 406 of the 662 flags generated by the screening 
methods were not associated with any of the 120 events identified in the 
case study.  The POA analysis generated the most flags that were not 
associated with events (183 flags) and PSI analysis generated the fewest 
(4 flags).  (See Figure 2.)   
Figure 2:  Number of Flags Not Associated With Events and Associated With Events by 
Screening Method
200
183
150
Flags Not Associated With Events
123
113
104 Flags Associated With Events
100
50
46
50
21
9 9
4
0
POA  Nurse  Beneficiary  Hospital  PSI 
Analysis Reviews Interviews Incident  Analysis
(n = 296) (n = 227) (n = 71) Reports (n = 13)
(n = 55)
Source:  OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008.
Note:  Some flags were associated with more than one event, and some events were identified through more than one flag.
 
We noted several possible explanations for the large number of flags 
that were not associated with events: 
 First, physician reviews determined that many flags represented 
conditions or occurrences that did not cause harm to the patients.  For 
example, six hospital incident reports described beneficiary falls that, 
according to the medical record, did not result in injury or require 
additional treatment.   
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  12   
   
FFF III NNN DDD III NNN GGG SSS    
 Second, physician reviews determined that some flags represented 
occurrences that were part of the natural course of the patients 
disease or medical condition.  In one such case, a flag from the nurse 
review indicated that a beneficiary experienced postoperative atrial 
fibrillation.  However, the physician review determined that this was 
not an event because the beneficiarys underlying medical condition 
likely led to the atrial fibrillation, rather than care provided during 
the hospitalization.   
 Third, in some cases, physician reviews did not find supporting 
documentation in the medical records to determine that flags were 
associated with events.  For example, a beneficiary interview included 
a report of pressure ulcers, yet the physician reviewer did not find 
pressure ulcers documented in the medical record and therefore could 
not determine that an event occurred.   
 Finally, to ensure that physicians reviewed medical records with any 
chance of an event, we defaulted to including all flags, even when the 
screening method suggested only a slight possibility of harm.  For 
example, physicians reviewed all cases for which the hospitals 
provided associated incident reports, even if the reports did not 
initially appear to be related to patient harm.  
Although some events were flagged by one screening method, other 
events were flagged by multiple methods, especially in cases in which 
the patients had several complicating health care conditions.  For 
example, one of the most complex hospitalizations involved a patient 
who died during the hospitalization.  Even though the family declined to 
be interviewed, the 4 remaining screening methods generated a total of 
29 flags.  Physician reviews determined that the patient experienced     
4 events.  One of the four events was a cascade with 19 associated 
flags.34  This cascade included several events, such as deep vein 
thrombosis, respiratory failure, renal failure, and sepsis.  Two other 
events were hospital-acquired infections, which were both flagged by 
nurse reviews and POA analysis.  The fourth event involved kidney 
damage, which was flagged by nurse review and POA analysis.  Overall, 
 
34 IHI defines cascade event as one in which an initial event causes a series of additional, 
related events for the same patient, and advocates collapsing these series into single events 
in determining event counts.  F.A. Griffin and R.K. Resar, IHI Global Trigger Tool for 
Measuring Adverse Events, IHI Innovation Series white paper.  Cambridge, MA:  Institute 
for Healthcare Improvement Innovation Series 2007, p. 11.   
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  13   
   
F I N D I N G S  
physician reviews determined that 26 of the 29 flags were associated 
with events, and the remaining 3 flags were not associated with events.  
For more details on each screening method, see page 21.   
 
Shortcomings in two screening methods have  For the Medicare program and 
implications for Medicare payments and Federal  other entities to learn of events 
that harm patients, they must 
initiatives to identify, track, and monitor events 
receive accurate and complete 
information from hospitals.  Two critical sources of such information are 
hospital billing data (diagnosis codes and associated POA indicators) 
and hospital incident reports.  Our analysis of data from these sources 
revealed vulnerabilities regarding both accuracy and completeness, 
which could result in Medicare overpayments and inhibit initiatives to 
improve patient safety.  
POA analysis revealed problems that could inhibit CMSs ability to identify  
hospital-acquired conditions and appropriately deny Medicare payments  
In the case study, we found that only 4 of the 11 events involving 
Medicare hospital-acquired conditions could be identified through the 
POA indicators contained in billing data.35  As previously stated, 
Federal regulations require that hospitals submit with each diagnosis 
code on a Medicare claim a POA indicator designating whether the 
condition was present upon admission.  The ICD-9-CM codes and POA 
indicators submitted by hospitals enable Medicares automated payment 
processing software to identify claims with hospital-acquired conditions, 
enabling denial of payment for associated care and also providing 
valuable information for use in monitoring quality of care in hospitals.   
Among the seven hospital-acquired conditions found in the case study 
that POA analysis did not flag, five had no related ICD-9-CM code in 
the hospital billing data.36  Thus, Medicare automated payment 
processing software could not determine that a hospital-acquired 
condition had occurred.  Although the hospital would not receive 
additional payment for care associated with the conditions (the desired 
 
35 Medicares payment policy for hospital-acquired conditions was not yet in effect at the 
time of the case study, and therefore these findings do not indicate that Medicare made any 
improper or incorrect reimbursements for claims associated with these hospital-acquired 
conditions. 
36 It was beyond the scope of the case study to determine the underlying causes for these 
absent or inaccurate codes. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  14   
   
FF II NN DD II NN GG SS   
outcome of the Medicare payment policy), the absence of these codes 
inhibits the usefulness of billing data for gauging how often such events 
occur and for monitoring quality of care in hospitals.    
In the two remaining cases, hospital billing data contained diagnosis 
codes related to the hospital-acquired condition, but the codes were not 
specific enough to invoke the Medicare payment policy.  In each of these 
two cases, physician reviews determined that the beneficiary 
experienced a catheter-associated urinary tract infection, yet the 
billing data included a more general diagnosis code for urinary tract 
infections, not otherwise specified.  Although these less descriptive 
diagnosis codes led physicians to identify the events in these cases, they 
would not have allowed Medicares automated payment-processing 
software to identify the hospital-acquired conditions and, therefore, 
could have resulted in Medicare overpayments to the hospitals if the 
payment policy had been in effect at the time of the case study.  Such 
discrepancies would also affect use of Medicare claims and POA 
indicator data to monitor events and quality of care in hospitals. 
The lack of hospital incident reports could prevent hospitals from tracking 
events as required or reporting events to outside entities  
Identifying events within hospitals and capturing event information in 
hospital incident reports are critical first steps toward addressing 
problems that lead to patient harm.  We requested that hospitals 
provide any internal incident reports involved with the hospitalization, 
including submissions to the hospital incidence-reporting systems, 
adverse drug reaction reports, complaints, peer reviews, and morbidity 
and mortality reviews, for the 278 sample Medicare beneficiary 
hospitalizations.  However, hospitals did not provide, and apparently 
did not have, any reports for 112 of the 120 events (93 percent) found in 
the case study.  Further, hospitals had no incident reports for two of the 
three events that resulted in death to the patients or two of the four 
events that resulted in serious disability.   
The implications of hospitals failure to identify and capture event 
information can be significant.  First, although we did not assess 
hospital compliance with Federal requirements to track medical errors 
and adverse patient events, it raises concerns that only four of the 
seven most serious events had no associated hospital incident reports.  
Further, the lack of incident reports for 93 percent of events suggests 
that hospital incident-reporting systems may be an unreliable source of 
information for PSOs, States operating adverse event reporting 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  15   
   
F I N D I N G S  
systems, and other entities.  These entities often seek to learn from the 
combined experiences of many hospitals to generate lessons to improve 
patient safety.  Unless events are reported within the hospital first, the 
event information is unlikely to be available to outside entities for 
learning.
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  16   
   
F I NR DEI CN OG SM  M E N D A T I O N S  
 
Identifying events that harm hospital patients is a topic of interest to 
Congress, HHS, and the health care community and is important to 
reduce their occurrence.  This report found that, combined, the selected 
screening methods used for the case study were useful in identifying 
events.  However, we also identified weaknesses in their use; physicians 
determined that most flags generated by the five screening methods 
were not associated with events.  Further, we found that shortcomings 
revealed within two screening methods have implications for Medicare 
payments and Federal initiatives to identify, track, and monitor events.   
Overall, the case study findings suggest that an effective way to identify 
events that result in harm to patients is through review of medical 
records by nurses and/or physicians, methods that are both costly and 
time consuming.  Other less costly and time-consuming methods, such 
as analyzing billing data or examining hospital incident reports, would 
identify far fewer events.  Nonetheless, there is no current mandate for 
HHS to conduct medical reviews to identify events. 
Therefore, we recommend that: 
CMS and AHRQ should explore opportunities to identify events when 
conducting medical record reviews for other purposes 
CMS and AHRQ could reduce the cost of efforts to identify adverse 
events by adding such efforts to ongoing medical record reviews.  
Examples of such efforts include, but are not limited to:  CMSs 
Medicare Comprehensive Error Rate Testing (CERT) and the work of 
Quality Improvement Organizations (QIO); AHRQ grant awards for 
patient safety research; and State survey agency work in certifying 
hospital compliance with the Medicare conditions of participation. 
CMS should ensure that hospitals code claims accurately and completely to 
allow for identification of hospital-acquired conditions affected by 
Medicares payment policy 
CMS relies on hospitals to code diagnoses and POA indicators to identify 
Medicare hospital-acquired conditions subject to the payment policy.  
However, we found that ICD-9-CM codes were inaccurate or absent for  
7 of the 11 events that involved Medicare hospital-acquired conditions 
found in the case study.  Because the payment component of the Medicare 
hospital-acquired conditions policy was not in effect at the time of the case 
study, this problem did not result in improper payments to any providers.  
However, given that the payment policy is now in effect, similar problems 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  17   
   
R E C O M M E N D A T I O N S  
could result in improper payments.  Further, incorrect or absence of 
diagnosis codes would prevent CMS from determining that the events 
occurred, which would inhibit its ability to measure rates of hospital-
acquired conditions and limit public reporting of the measured rates.   
CMS should determine whether additional guidance or other action is 
needed to ensure that hospitals code diagnoses and POA indicators 
accurately and completely.  We are conducting further research examining 
the accuracy of claims related to hospital-acquired conditions for a 
nationally representative sample of Medicare beneficiaries 
hospitalizations. 
CMS should provide interpretive guidelines for State survey agencies to 
assess hospital compliance with requirements to track and monitor adverse 
events  
Although hospitals must track medical errors and adverse patient 
events under Medicares conditions of participation, few events found 
in the case study had associated hospital incident reports.  The State 
Operations Manual, through which CMS provides guidance to State 
survey agencies that assess hospital compliance with Federal 
regulations, contains no interpretive guidelines regarding these 
requirements.37  Therefore, when State agency staff perform surveys of 
hospitals (e.g., standard compliance surveys and surveys based on 
complaints), it is unclear how surveyors are to assess hospital 
operations for tracking medical errors and other events that result in 
harm to patients.  CMS should provide guidance to surveyors for 
assessing hospitals compliance with these requirements.  Such 
guidance could include what types of medical errors and adverse patient 
events hospitals should track and monitor, as well as what information 
should be captured in hospital incident reports.   
AHRQ should inform PSOs that internal hospital incident reporting may be 
insufficient to provide needed information about events to PSOs 
PSOs were established to aggregate and analyze events that pose risks 
to patient safety.  AHRQ indicated that it expects PSOs will rely 
primarily upon existing hospital infrastructures for identifying, 
tracking, and reporting information about events to PSOs.  However, we 
found that hospital incident reports existed for only 8 of the 120 events 
 
37 CMS, State Operations Manual, 2009, Appendix A, A-0267; A-0286 through A-0288.  
Available online at http://www.cms.hhs.gov/manuals/downloads/som107ap_a_hospitals.pdf.  
Accessed on September 1, 2009. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  18   
   
R E C O M M E N D A T I O N S  
identified in the case study.  In providing technical assistance to 
support PSOs, AHRQ should convey the importance of strong hospital 
internal incident-reporting procedures that capture relevant 
information about events that cause harm to patients. 
AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 
RESPONSE 
We received comments on our draft of this report from AHRQ and CMS.  
AHRQ concurred with the report as written.  CMS stated that it 
appreciates the original contribution that the report makes in 
advancing the patient safety agenda in hospitals and agrees with the 
recommendations made to CMS.  We made no changes to the report 
based on agency comments. 
In its comments, CMS outlined several opportunities to identify events 
when conducting medical record reviews for other purposes.  First, CMS 
noted that it expects State Survey Agencies to investigate patient 
adverse events while conducting a survey of a hospitals compliance 
with the Medicare conditions of participation.  Second, CMS stated that 
it is increasingly investigating the extent to which adverse events are 
tracked and analyzed by hospital internal quality assurance and 
performance systems.  Third, CMS noted that the QIO program 
addresses many quality-of-care concerns through the beneficiary 
complaint and medical review processes or through referral by other 
entities.  Regarding QIOs, CMS indicated that it has proposed policy 
revisions to the QIO beneficiary complaint and quality-of-care review 
process to include adverse events as indicators of quality and will 
explore a possible quality-referral mechanism between the CERT 
contractor and the QIO program.  Lastly, CMS stated that it is 
collaborating with AHRQ on the Medicare Patient Safety Monitoring 
System to estimate prevalence of certain conditions and adverse events.   
CMS indicated that to ensure that hospitals accurately code claims to 
enable identification of hospital-acquired conditions, it has established a 
process for providing coding advice to hospitals.  Further, CMS 
indicated that it will study the accuracy of coding hospital-acquired 
conditions and reporting of POA indictors to use in developing 
refinements in coding instructions or to focus review on areas of 
inaccurate reporting. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  19   
   
R E C O M M E N D A T I O N S  
CMS indicated that it will ensure that the State Operations Manual 
includes full guidance for surveyors to assess hospital QAPI systems 
and strengthen CMS surveyor-training programs to enhance surveyor 
abilities to evaluate compliance with the QAPI requirements.  
For the full text of AHRQ and CMS comments, see Appendix I.
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  20   
   
 S C R E E N I N G   M E T H O D S  
The following section provides specific results for each of the screening 
methods used for identifying events in the case study, as well as factors 
that appeared to affect their usefulness. 
 
Method 1:  Nurse Reviews of Medical Records 
Nurse reviews identified the most events among the screening methods and 
also generated many flags that were not associated with events  
Nurse reviews generated 227 flags across 127 Medicare beneficiaries:   
1 flag each for 70 beneficiaries and more than 1 flag for the remaining 
57 beneficiaries.  In an unusually complex case, the nurse reviewer 
identified 11 flags.  Because the nurse review protocol required a broad-
based examination of the medical record, nurses identified flags 
associated with a variety of conditions and occurrences.  The most 
prominent types of flags related to surgery and other procedures, 
medication, and infections.  (See Table 2.)   
Table 2:  Number of Flags and Events Identified by Nurse 
Reviews of Medical Records 
Number of Flags 
Type of Conditions or Occurrences 
Identified by  Associated 
Method  With Events 
Surgery and other procedures  58  34 
Medication  51  29 
Infections  45  20 
Skin care  19  11 
Patient care  9  5 
Glycemic control  8  3 
Patient fall  7  1 
Readmission  7  0 
Intravenous fluid  4  2 
Use of restraints  4  0 
Other  15  0 
     Total  227  104* 
Source:  OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008. 
*Column will not sum to 104 because some flags were associated with more than 1 event and some events 
were identified through more than 1 flag. 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  21   
   
SS CC RR EE EE NN II NN GG    MM EE TT HH OO DD SS   
Physician reviews determined that 104 of the 227 flags (46 percent) 
identified by nurse reviews were associated with events.  Because more 
than 1 flag was associated with some of the events, this method 
identified a total of 93 events (78 percent of the 120 events in the case 
study).  Physician reviews determined that 123 of the 227 flags           
(54 percent) generated by nurse reviews were not associated with 
events.  These flags were typically not associated with events because 
the condition or occurrence flagged did not involve harm to the patient 
or was related to the patients underlying disease or medical condition 
or because the medical record did not contain sufficient documentation 
to determine that an event occurred.  
 
Method 2:  Analysis of Present on Admission Indicators 
POA analysis generated the most flags among the five methods and 
identified about half of the events 
To identify diagnoses that were not coded as present on admission and 
therefore presumed to be hospital acquired, POA analysis of hospital 
billing data considered up to 30 ICD-9-CM codes and POA indicators for 
each case.  This analysis generated 296 flags across 98 Medicare 
beneficiaries:  1 flag for 40 beneficiaries and more than 1 flag for the 
remaining 58 beneficiaries.  The method generated 10 or more flags for 
each of 4 beneficiaries who had very complex hospitalizations.   
The 296 flags included 155 different ICD-9-CM codes across the  
98 beneficiaries.  Of these 155 codes, 95 were flagged once (i.e., for only 
1 beneficiary).  The most frequently represented code, anemia because 
of blood loss, was flagged for 16 different beneficiaries.  (See Table 3.) 
Physician reviews determined that 113 of the 296 flags (38 percent) 
identified through POA analysis were associated with events.  Because 
more than 1 flag was associated with some of the events, the method 
identified a total of 61 events (51 percent of the 120 events in the case 
study).38  However, POA analysis generated the largest number flags 
 
38  POA analysis would have identified 34 percent fewer events (21 of 61 events) if we 
had used Medicare claims data rather than billing data obtained directly from hospitals.    
For these 21 events, either there were no Medicare claims for the hospitalizations or the 
relevant diagnosis codes were not in the Medicare claims.  (Medicare claims data include 
only the first nine diagnosis codes and POA indicators for each hospitalization.)  This 
suggests that researchers who cannot obtain billing data directly from hospitals may 
identify a smaller percentage of adverse events than suggested by the case study results. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  22   
   
S C R E E N I N G   M E T H O D S  
that were not associated with events (183) among the screening 
methods (62 percent of the 296 POA analysis flags).  For POA flags that 
were not associated with events, physician reviews determined that 
they typically did not involve harm to the patient or were related to the 
patients diseases or medical conditions or that the medical records did 
not contain documentation to determine that events occurred.  For 
example, physician reviews often did not find accompanying patient 
signs or symptoms associated with laboratory tests that caused actual 
harm or required an intervention or prolonging of the hospital stay.  
Physician reviewers reported that the three most frequently occurring 
POA flagsanemia because of acute blood loss, potassium deficiency, 
and acute renal failureeach represented temporary laboratory 
findings common during care for complex patients.    
Table 3:  Seven Most Frequently Occurring Flags With Diagnoses 
Not Coded as Present on Admission 
Number of Flags 
ICD-9-CM 
Description 
Code  Identified by  Associated 
Method  With Events 
285.1  Anemia because of acute blood loss  16  2 
276.8  Potassium (K) deficiency  12  0 
584.9  Acute renal failure  10  4 
Other enterovirus diseases of central 
48.6  6  4 
nervous system 
511.9  Unspecified pleural effusion  6  1 
707.03  Decubitus ulcer  5  5 
997.1  Cardiac complications  5  1 
Source:  OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008.  
 
 
Method 3:  Beneficiary Interviews 
Beneficiary interview was the only method to rely on information from 
outside the hospitals and identified 21 percent of events  
Beneficiary interviews identified 70 flags across 42 Medicare 
beneficiaries:  1 flag each for 25 beneficiaries and more than 1 flag for 
each of the remaining 17 beneficiaries.  Two beneficiaries each reported 
five different flags, the most reported during interviews.  Flags from 
interviews covered a smaller range of conditions and occurrences than 
nurse reviews or POA analysis.  (See Table 4.)   
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  23   
   
S C R E E N I N G   M E T H O D S  
Table 4:  Number of Types of Flags and Events Identified 
Through Beneficiary Interviews  
   Number of Flags 
Identified by  Associated 
Type of Condition or Care 
Method  With Events 
Medication  18  8 
Skin care  14  3 
Hospital-acquired infections  13  3 
Surgery and other procedures  12  6 
Patient falls  4  1 
Intravenous volume overload  1  1 
Other general care complaints  8  0 
     Total  70  21* 
Source:  OIG analysis of 220 Medicare beneficiaries interviews. 
*Column does not sum to 21 because 1 flag was associated with 2 events. 
During interviews, beneficiaries and their representatives revealed both 
specific and general concerns.  Among the 70 flags, 62 involved specific 
health care circumstances (e.g., medication problems, infections).  For 
the other eight flags, interviews gleaned general complaints about the 
quality or outcome of care.  For example, one beneficiary indicated that 
his mobility changed shortly after the hospital stay.  To ensure that we 
did not miss events, we considered each of these 8 as flags and 
forwarded them to physicians for medical record reviews, along with the 
62 specific flags.   
Several reasons may explain why beneficiary interviews identified 
relatively few flags as compared to some of the other methods.  
Although we conducted interviews within about 1 month of their 
hospitalizations, a number of beneficiaries had difficulty recalling their 
experiences, although some remembered more as the interview 
proceeded.  For example, during the initial series of general questions, 
several respondents indicated that they had no problems, yet later 
identified problems when asked about specific topic areas such as 
medication or falls.  Respondents who had more than one recent 
hospital stay often had difficulty distinguishing during which stay 
problems occurred.  Although assured that information would be kept 
confidential, a few beneficiaries expressed concern that hospitals would 
disapprove of their participating in the interviews.  Finally, this method 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  24   
   
S C R E E N I N G   M E T H O D S  
generated no flags for the 58 sample beneficiaries that we could not 
contact or declined to be interviewed.   
Despite these barriers, physician reviews determined that 21 of the  
70 flags (30 percent) generated from beneficiary interviews were 
associated with events.  One event that was flagged through an 
interview but missed by the other screening methods demonstrates the 
usefulness of beneficiary interviews in learning about events.  In this 
case, the beneficiary experienced an allergic reaction determined by the 
physicians to be an event.  The beneficiary indicated in the interview 
that hospital staff had been informed in advance of the allergy.   
 
Method 4:  Hospital Incident Reports 
Although few hospital incident reports were associated with events, these 
were useful in understanding what transpired 
Counting each incident report as a flag, this method identified 55 flags 
across 40 Medicare beneficiaries.  Hospitals submitted a single incident 
report each for 29 of these beneficiaries and multiple reports for each of 
the other 11 beneficiaries.  In a few of the cases with multiple reports, 
the reports were related to the same condition or occurrence, with initial 
reports and then subsequent reports detailing further analysis and 
outcomes.  In the other cases, the multiple reports indicate that the 
beneficiaries experienced multiple unrelated occurrences.  
We grouped the 55 incident reports into 3 distinct categories:  patient 
incidents (29 reports), systems incidents (21 reports), and nonmedical 
incidents (5 reports).  (See Table 5.)  Each patient incident report 
involved a medical condition or occurrence specific to one of the sample 
Medicare beneficiaries.  The most common of these reports involved 
adverse medication reactions.  The systems incident reports also related 
to particular patients, but the reports described incidents related to 
hospital systems rather than the patients conditions.  The most 
common hospital systems incident report addressed medication 
administration errors that, despite involving errors, did not cause harm 
to the patients.  Finally, nonmedical incident reports were unrelated to 
the health care provided to the patients, such as two reports about 
stolen property.  
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  25   
   
S C R E E N I N G   M E T H O D S  
Table 5:  Hospital Incident Reports by Category 
Number of Flags 
Category of Hospital Incident Reports 
Identified  Associated 
by Method   With Events  
PATIENT INCIDENTS  29  9 
Adverse medication reactions  11  3 
Skin care-related incidents  7  3 
Falls  6  0 
Other patient incidents  5  3 
SYSTEMS INCIDENTS  21  0 
Medication administration errors  7   0 
Hospital policy-driven reports*   6  0 
Other systems incidents  5  0 
Staff care incidents  3   0 
NONMEDICAL INCIDENTS  5  0 
     Total  55  9 
Source:  OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008. 
*These reports were generated because of hospital policies that called for such reports whenever specific 
circumstances occurred, such as an unplanned readmission of a patient. 
Physician reviews determined that 9 of the 55 flags (16 percent) were 
associated with events.  (The method identified eight unique events 
because one event had two associated incident reports.)  The events 
involved three adverse medication reactions, two skin care events, two 
surgical complications, and one blood-clotting event.  The eight events 
involved only reports that we categorized as patient incident reports, 
indicating that reports related to hospital systems and nonmedical 
incidents were not useful for identifying events.   
The detail included in the hospital incident reports was useful in 
understanding what transpired.  These reports typically described the 
nature and types of incidents and their effect on patients.  Some reports 
also indicated whether the events were reportable to other entities, such 
as States operating adverse event reporting systems.  More serious 
events tended to require reporting to other entities and sometimes 
resulted in highly detailed and/or multiple reports.  For example, for 
one beneficiary, who experienced multiple complications following 
surgery, the hospital generated three separate incident reports:  a 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  26   
   
SS CC RR EE EE NN II NN GG    MM EE TT HH OO DD SS   
report describing a hospital-acquired infection that led to sepsis, a 
report describing the patients return to surgery for a bowel obstruction, 
and a more general report about hypotension and confusion.   
The low number of events identified through hospital incident reports 
(8), and the relatively high number of incident reports not related to 
events (46), suggest a disconnect between the hospitals purpose for 
internal incident-reporting systems and the goal of identifying events 
that result in patient harm.  Reports often referred to incidents with 
hospital systems and nonmedical issues that could be useful for hospital 
administration purposes, but the incidents did not involve harm to 
patients.   
 
Method 5:  Analysis of Patient Safety Indicators 
PSI analysis identified the fewest flags among the screening methods and 
did not identify some events related to conditions covered by PSI software 
PSI analysis identified 13 flags across 11 Medicare beneficiaries, the 
lowest number of flags among the screening methods used in the case 
study.39  (See Table 6.)  This low number of flags is, in part, because  
PSI software is limited to only 20 health care conditions.40  The most 
frequently occurring condition with a PSI was pressure ulcer (five flags).  
Physician reviews determined that 9 of the 13 flags (69 percent) 
generated by PSI analysis were associated with events.  (The method 
identified eight events because one of the events had two associated 
PSIs.)  Although this method was one of the methods that identified the 
fewest events among the screening methods, it also had the fewest flags 
that were not associated with events (four).  
 
 
39 The PSI analysis identified 1 flag for 10 of these 11 beneficiaries and 3 flags for the 
other beneficiary. 
40 Additionally, 4 of these 20 PSIs involved obstetrics and were therefore unlikely to 
affect the Medicare beneficiary population included in the case study.  
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  27   
   
S C R E E N I N G   M E T H O D S  
 
Table 6:  Number of Flags and Events Identified Through  
PSI Analysis 
Number of Flags 
Patient Safety Indicator 
Identified by  Associated 
Method  With Events 
Pressure ulcer  5  5 
Postoperative pulmonary embolism or deep vein 
3  2 
thrombosis 
Accidental puncture or laceration  2  1 
Selected infections because of medical care  1  1 
Iatrogenic pneumothorax  1  0 
Complications of anesthesia  1  0 
     Total  13  9 
Source: OIG analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008.  
Additionally, PSI analysis did not identify as flags some events 
involving health care conditions targeted by the PSI software typically 
because related diagnosis codes were not included in the billing data 
provided to OIG.  For example, although PSI software is designed to 
identify pressure ulcers, our PSI analysis identified only 5 of the          
13 pressure ulcers found in the case study.  Regarding the other eight 
hospitalizations with pressure ulcers, billing data for seven had no  
ICD-9-CM diagnosis codes indicating that the Medicare beneficiaries 
had pressure ulcers.  The remaining case had an ICD-9-CM diagnosis 
code for the pressure ulcer, but the PSI software intentionally invoked 
an exemption because the hospitalization had another complication, 
specifically anoxic brain damage.  In the absence of the brain damage, 
PSI analysis would likely have flagged the pressure ulcer.   
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  28   
   
 A P P E N D I X   ~   A  
Tax Relief and Health Care Act of 2006 
P.L.  109-432  
 
DIVISION B  MEDICARE AND OTHER HEALTH PROVISIONS 
TITLE IIMEDICARE BENEFICIARY PROTECTIONS 
SEC 203 OIG STUDY OF NEVER EVENTS 
 
(a) Study. 
(1) In general.The Inspector General in the Department of Health and 
Human Services shall conduct a study on 
(A) incidences of never events for Medicare beneficiaries, including 
types of such events and payments by any party for such events; 
(B) the extent to which the Medicare program paid, denied payment, or 
recouped payment for services furnished in connection with such events 
and the extent to which beneficiaries paid for such services; and 
(C) the administrative processes of the Centers for Medicare & Medicaid 
Services to detect such events and to deny or recoup payments for 
services furnished in connection with such an event. 
(2) Conduct of study.In conducting the study under paragraph (1), the 
Inspector General 
(A) shall audit a representative sample of claims and medical records of 
Medicare beneficiaries to identify never events and any payment (or 
recouping of payment) for services furnished in connection with such 
events; 
(B) may request access to such claims and records from any Medicare 
contractor; and 
(C) shall not release individually identifiable information or facility-
specific information. 
(b) Report.Not later than 2 years after the date of the enactment of 
this Act, the Inspector General shall submit a report to Congress on the 
study conducted under this section.  Such report shall include 
recommendations for such legislation and administrative action, such as 
a noncoverage policy or denial of payments, as the Inspector General 
determines appropriate, including 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  29   
   
A P P E N D I X   ~   A 
(1) recommendations on processes to identify never events and to deny 
or recoup payments for services furnished in connection with such 
events; and 
(2) a recommendation on a potential process (or processes) for public 
disclosure of never events which 
(A) will ensure protection of patient privacy; and  
(B) will permit the use of the disclosed information for a root cause 
analysis to inform the public and the medical community about safety 
issues involved. 
(c) Funding. Out of any funds in the Treasury not otherwise 
appropriated, there are appropriated to the Inspector General of the 
Department of Health and Human Services $3,000,000 to carry out this 
section, to be available until January 1, 2010. 
(d) Never Events Defined. For purposes of this section, the term 
never event means an event that is listed and endorsed as a serious 
reportable event by the National Quality Forum as of               
November 16, 2006. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  30   
   
 A P P E N D I X   ~   B  
Medicare List of Hospital-Acquired Conditions 
The Centers for Medicare & Medicaid Services (CMS) list of          
hospital-acquired conditions is divided into 10 categories.  Effective 
October 1, 2008, CMS no longer pays a higher reimbursement for 
hospitalizations complicated by these categories of conditions that were 
not present upon admission. 
Table B-1:  Medicare List of Hospital-Acquired Conditions 
01.  Foreign object retained after surgery 
02.  Air embolism 
03.  Blood incompatibility 
04.  Pressure ulcers (stages III and IV) 
05.  Falls 
A.  Fracture 
B.  Dislocation 
C.  Intracranial injury 
D.  Crushing injury 
E.  Burn 
F.  Electric shock 
06.  Manifestations of poor glycemic control 
A.  Hypoglycemic coma 
B.  Diabetic ketoacidosis 
C.  Nonkeototic hyperosmolar coma 
D.  Secondary diabetes with ketoacidosis 
E.  Secondary diabetes with hyperosmolarity 
07.  Catheter-associated urinary tract infection 
08.  Vascular catheter-associated infection 
09.  Deep vein thrombosis/pulmonary embolism associated with 
A.  Total knee replacement 
B.  Hip replacement 
10.  Surgical site infection 
A.  Mediastinitis after coronary artery bypass graft 
B.  Associated with certain orthopedic procedures involving the 
a.  Spine 
b.  Neck 
c.  Shoulder 
d.  Elbow 
C.  Associated with certain bariatric surgical procedures for obesity 
a.  Laprascopic gastric bypass 
b.  Gastroenterostomy 
c.  Laparoscopic gastric restrictive surgery 
Source:  Fiscal Year 2009 Final Inpatient Prospective Payment System Rule, 73 Fed. Reg. 48434,           
4849048491 (Aug. 19, 2008). 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  31   
   
A P PA EPNPD EI NX  DB I  X   ~   C  
 
Glossary of Selected Clinical Terms41 
AcidosisAn abnormal condition of reduced alkalinity of the blood and 
tissues that is marked by sickly sweet breath, headache, nausea and 
vomiting, and visual disturbances and is usually a result of excessive 
acid production. 
Acute renal failureA sudden loss of the ability of the kidneys to remove 
waste and concentrate urine without losing electrolytes. 
AnemiaA condition in which the blood is deficient in red blood cells, in 
hemoglobin, or in total volume. 
Anoxic brain damageBrain damage that occurs when the brain does 
not receive sufficient oxygen. 
Atrial fibrillationVery rapid uncoordinated contractions of the atria of 
the heart resulting in a lack of synchronism between the heartbeat and 
pulse beat. 
Blood clotA coagulated mass produced by clotting of blood. 
Coronary artery bypass graftHeart bypass surgery performed to route 
blood flow around clogged arteries supplying the heart. 
Deep vein thrombosisA condition marked by the formation of a 
thrombus within a deep vein (as of the leg or pelvis) that may be 
asymptomatic or be accompanied by symptoms (such as swelling and 
pain) and that is potentially life threatening if dislodgment of the 
thrombus results in pulmonary embolism. 
Enterovirus DiseaseA kind of ribonucleic virus that multiplies 
especially in the gastrointestinal tract but may infect other tissues, such 
as nerve and muscle.   
HypotensionAbnormally low pressure of the blood; also called low 
blood pressure. 
HypoglycemiaAbnormal decrease of sugar in the blood. 
 
41 National Institutes of Health, U.S. National Library of Medicine, Medline Plus 
Medical Dictionary, updated February 4, 2003.  Available online at 
http://www.nlm.nih.gov/medlineplus/mplusdictionary.html.  Accessed on June 23, 2009. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  32   
   
A P P E N D I X   ~   C  
Pleural effusionAn oozing of fluid from the blood or lymph into a 
pleural cavity. 
Potassium deficiencyAlso known as hypokalemia, a lower than 
normal amount of potassium in the blood.  It may result from a number 
of conditions. 
Pressure ulcerAn ulceration of tissue deprived of adequate blood 
supply by prolonged pressure; called also decubitus, decubitus ulcer, 
and pressure sore. 
Pulmonary embolismA sudden blockage of a lung artery or one of its 
branches that is produced by foreign matter; most often a blood clot 
originating in a vein of the leg or pelvis. 
SepsisA systemic response typically to a serious, usually localized 
infection (as of the abdomen or lungs) especially of bacterial origin that 
is usually marked by abnormal body temperature and white blood cell 
count, tachycardia, and tachypnea; specifically, systemic inflammatory 
response syndrome induced by a documented infection. 
Urinary tract infectionAn infection of the tract through which urine 
passes.  It consists of the renal tubules and renal pelvis of the kidney, 
the ureters, the bladder, and the urethra. 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  33   
   
   A P P E N D I X   ~   D  
Case Study Protocol for Nurse Reviews 
Figure D-1 presents the form used by nurses to record the results of 
medical record reviews for the first stage for the case study.  Figure D-2 
contains the Institute for Healthcare Improvement (IHI) Global Trigger 
Tool (GTT) worksheet.. 
Figure D-1:  Nurse Review Results Form 
 
IHI GTT  TRIGGER TOOL RESULTS  OEI-06-08-00220           Start Time:__________AM / PM 
 
DATE:  __________  RN ID:  __________  OIG CASE ID:  __________  HOSPITAL: ______________ 
 
Beneficiary Information   
   
NAME:  ________________________________   MR#:  ___________________________ 
 
 
 
 
PHYSICIAN REVIEW INFORMATION 
 
Should this case be reviewed by a physician? 
  YES, Adverse Event Identified 
  YES, Other (please explain reason in notes below) 
  NO  
Notes:   
 
 
ADVERSE EVENT INFORMATION 
  Harm Category Key:
Assign AE Number:_______(#) of _______(#)    
(Ex:  2 of 3  this is 2nd of 3 adverse events   E  Temporary harm, intervention required 
identified for this patient)  F  Temporary harm, initial or prolonged hospitalization 
  G  Permanent patient harm 
H  Life-sustaining intervention required 
Harm Category:   _________ 
I  Contributing to death 
 
Adverse Event Description: 
 
   
 
 
 
Notes to Physician about Adverse Event (e.g., date, location in chart, questions) 
 
   
 
 
 
        End Time:__________AM / PM 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  34   
   
A P P E N D I X   ~   D  
Figure D-2:  IHI GTT Worksheet (2007)42 
 
42 F.A. Griffin and R.K. Resar, IHI Global Trigger Tool for Measuring Adverse Events, 
IHI Innovation Series white paper.  Cambridge, MA:  Institute for Healthcare Improvement 
Innovation Series 2007, Appendix B. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  35   
   
 AA PP PP EE NN DD II XX    ~~    DE   
 
Medicare Beneficiary Interview Questions 
Table E-1 provides the list of the questions asked of Medicare 
beneficiaries to screen for possible events that may have occurred during 
the target hospitalization in the case study. 
 
Table E-1.  Beneficiary Interview Questions 
 
Part A:  General Questions 
1.  Why did you go to the hospital? 
2.  In general, were there any problems that happened during this hospital stay?  (If NO, go to Question 4.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
3.  Were there any other problems?  (If NO, go to Question 4.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
4.  Was there anything [else] that seemed unusual about your medical care or anything else during your stay in this 
hospital?  ( If NO, go to Question 5) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
continued on next page 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  36   
   
A P P E N D I X   ~   E  
Table E-1.  Beneficiary Interview Questions  
Part B:  Medication Questions 
5.  Did you receive any medications or anesthesia during this hospital stay?  (If NO, go to Question 8.) 
6.  Were there any problems with the medication you were given?  (If NO, go to Question 8.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Were you given the wrong type of medication?  Yes or No. 
B.  Were you given the wrong dosage or amount of medication?  Yes or No. 
C.  Did you have an allergic reaction to the medication?  Yes or No. 
D.  Or did you have some other problem with the medication?  Yes or No. 
E.  Was the medicine administered correctly?  Yes or No. 
Please tell me more about that [these issues]: 
A.  What was the name of the medication?  If you cant remember the name, please tell me what the 
medication was for:  [Name/ Purpose] 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
7.  Were there any other problems related to the medication you received during your stay at this hospital?  (If NO, go to 
Question 8.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
Part C:  Procedure Questions 
8.  Did you have any surgeries or procedures in this hospital stay?  (If NO, go to Question 11.) 
9.  Were there any problems related to the surgery?  (If NO, go to Question 11.) 
  If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
10.  Were there any other problems related to the surgery?  (If NO, go to Question 11.) 
        If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
continued on next page
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  37   
   
A P P E N D I X   ~   E  
Table E-1.  Beneficiary Interview Questions 
 
Part C:  Procedure Questions (Continued) 
11.  Did you have any blood transfusions while you were in this hospital stay?  (If NO, go to Question 14.) 
12.  Were there any problems related to the transfusion?  (If NO, go to Question 14.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
13.  Were there any other problems related to the blood transfusion?  (If NO, go to Question 14.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
14.  Were there any problems with the equipment?  (If NO, go to Question 15.) 
If YES or UNSURE/ CANNOT DECIDE: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital? 
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
Part D:  Infections and Other Issues 
15.  Did you develop any infections during this hospital stay?  (If NO, go to Question 17.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
16.  Did you develop any additional infections during this hospital stay that we have not already discussed?  (If NO, go 
to Question 17.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
continued on next page
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  38   
   
A P P E N D I X   ~   E  
Table E-1.  Beneficiary Interview Questions  
Part D:  Infections and Other Issues (Continued) 
17.  Did you develop any bedsores during the hospital stay?  (If NO, go to Question 18.)  
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
D.  Describe how you became aware of this problem.  For example, did you notice it yourself or did a 
physician, nurse, or someone else from this hospital tell you about it? 
18.  Did you ever fall during this hospital stay?  (If NO, go to Question 19.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
19.  Did you ever feel unsafe during this hospital stay?  (If NO, go to question 20.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
20.  Did a physician or other health care official tell you that you received the wrong diagnosis while you were in this 
hospital?  (If NO, go to Question 21.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
21.  Did you discover any problems with the care you received at this hospital, once you were discharged?  (If NO, go to 
Question 22.) 
If YES: 
A.  Please tell me more about that. 
B.  Please describe how this affected you. 
C.  How, if at all, did this affect the rest of your medical care at this hospital?   
Part E:  Hospital Response 
22.  Did you file a formal complaint about this hospital stay for any reason?  (If NO, go to Question 23.) 
If YES: 
A.  Please describe the nature of the complaint. 
B.  How did this hospital stay respond to your complaint? 
C.  Were you fully satisfied by this hospital stays response to your complaint?   
23.  Did anyone from this hospital talk to you about the issues you have described today?  (Note:  Asked only if a 
possible event was discussed in prior questions.  (If NO, go to Question 24.) 
If YES: 
A.  If you can recall, who spoke to you about these issues? 
B.  What did they say? 
C.  How, if at all, did they address your concerns? 
D.  Were you fully satisfied by this hospital stays response?  
Part F:  Close 
24.  Is there anything else you would like to tell us about the care you received at this hospital? 
Source:  Office of Inspector General interview protocol used to collect information about Medicare beneficiaries. 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  39   
   
 AA PP PP EE NN DD II XX    ~~    FF   
Patient Safety Indicators 
Table F-1 lists the provider-level Patient Safety Indicators developed by 
the Agency for Healthcare Research and Quality (AHRQ), which, when 
flagged, may indicate possible adverse events. 
Table F-1:  AHRQ Patient Safety Indicators (Provider-Level) 
PSI 
 Patient Safety Indicators   
Number
Complications of anesthesia   1 
Death in low mortality  2 
Decubitus ulcer  3 
Failure to rescue  4 
Foreign body left in during procedure  5 
Iatrogenic pneumothorax  6 
Selected infections due to medical care  7 
Postoperative hip fracture  8 
Postoperative hemorrhage or hematoma  9 
Postoperative physiologic and metabolic derangements  10 
Postoperative respiratory failure  11 
Postoperative pulmonary embolism or deep vein thrombosis  12 
Postoperative sepsis  13 
Postoperative wound dehiscence in abdominopelvic surgical patients  14 
Accidental puncture and laceration  15 
Transfusion reaction  16 
Birth trauma - injury to neonate  17 
Obstetric trauma - vaginal delivery with instrument  18 
Obstetric trauma - vaginal delivery without instrument  19 
Obstetric trauma - cesarean delivery  20 
Source:  AHRQ, Guide to Patient Safety Indicators, AHRQ Quality Indicators.  March 2003.  Available online at 
http://www.qualityindicators.ahrq.gov/archives/psi/psi_guide_v30.pdf.  Accessed on June, 4, 2008 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  40   
   
 A P P E N D I X   ~   G  
Case Study Protocol for Physician Reviews 
Figure G-1 presents the form used by physicians to record the results of 
medical record reviews for the second stage of review in the case study.  
Physicians generated one form for each adverse event.  Figure G-2 is the 
case summary completed by physicians at the end of their reviews.  Table 
G-1 describes the categories used to classify the level of harm to the 
patient caused by the event. 
 
      Figure G-1:  Adverse Event Review Form  
CASE ID: _______ 
Medical Review Protocol - OIG Case Study of Adverse Events 
 
Adverse Event Review  
 
1.  Adverse Event #: ____ (Case # + Alpha) 
 
2.  Please briefly identify the adverse event:   _______________________________ 
 
____________________________________________________________________ 
 
3.  Please provide a full description of the adverse event, including the: 
  circumstances of the adverse event (note as AE),  
  possible contributing factors (note as CF), and  
  impact of the adverse event on the patient (note as PI).   
 
AE: 
 
 
 
 
CF: 
 
 
 
 
PI: 
 
 
 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  41   
   
A P P E N D I X   ~   G  
Figure G-1:  Adverse Event Review Form (Continued) 
CASE ID: ______  
 
4.  Using the NCC MERP categories of harm, what level of harm resulted from the event? 
 
  E  Temporary harm, intervention required 
  F  Temporary harm, initial or prolonged hospitalization 
  G  Permanent patient harm 
  H  Life-sustaining intervention required 
  I  Contributing to death 
 
5.  Does the harm incurred appear to be the result of a cascade? 
  
  Yes (If yes, please insure that Q3 includes the sequence of occurrences.) 
  No 
 
If yes, please lay out the sequence of occurrences below (with arrows):   
 
 
 
 
 
 
 
 
 
 
6.  On what date or range of dates did the adverse event occur?  
 
   
 
 
 
 
 
 
7.  During which time period did the adverse event occur?  
      During time in emergency department 
      During time in observation 
      During the targeted inpatient hospitalization 
   Other, please describe:  
 
   
 
 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  42   
   
A P P E N D I X   ~   G  
Figure G-1:  Adverse Event Review Form (Continued) 
CASE ID: ______  
 
8.  Please review the provided billing summary of diagnosis and procedure codes.  Which (if any) 
appear to result from this adverse event? 
 
ICD-9 Code  Brief Description 
     
     
     
     
     
     
     
     
     
     
     
 
 
9.  Which case flags reflected information about the specific adverse event? (Check all that apply.) 
 
  Patient interview 
  Global Trigger Tool summary 
  POA analysis 
  PSI analysis 
  Hospital incident report or other hospital report 
  No case flags reflected this adverse event 
 
10.  Is there anything additional you would like to note about the case flags for this adverse event?  
(case flag or element of a case flag that was particularly valuable in leading you to the event, thoughts 
about selecting case flags for our future work, etc.)  
 
   
 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  43   
   
A P P E N D I X   ~   G  
Figure G-1:  Adverse Event Review Form (Continued) 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  44   
   
A P P E N D I X   ~   G  
Figure G-2:  Case Summary  
CASE ID: _______ 
Medical Review Protocol - OIG Case Study of Adverse Events 
Case Cover Page  
 
Review Date:   _______________, 2008        Hospital: __________________ 
  Patient name: _______________ 
Physician Initials: _________________ 
  OIG ID#:  ______     MR#: ________    
 
Directions:  Review the provided case documents and medical record. 
Complete Adverse Event Review for any adverse events. 
Complete Summary Questions and Exit Check List. 
 
Beneficiary Information  
 
Name: _______________________________        D.O.B.:_________________ 
 
Dates of Stay: __________________________       L.O.S. (days):____________ 
 
Principal Diagnosis: _______________________ 
 
Flags:    ___ Beneficiary Interview      ___ IHI Global Trigger Tool Review 
    ___ POA Analysis      ___ Hospital Incident Report   
    ___ PSI Analysis      ___ Other: ________________ 
 
Summary Questions 
1. Is this review complete? 
  Yes 
  Partially complete, estimate _______ additional time needed 
 
2. Did you identify any adverse event(s)? 
  Yes, how many?  _______________ 
  No 
 
AE #s and brief descriptions: 
________    _____________________________________ 
________    _____________________________________ 
________    _____________________________________ 
________    _____________________________________ 
 
3. Should this case be targeted for further discussion? 
  Yes 
  No 
If yes, please briefly state reason: 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  45   
   
A P P E N D I X   ~   G  
Figure G-2:  Case Summary (Continued) 
CASE ID: _______ 
Case Exit Check List 
1.  Which of the following information sources did you review?  (Check all that apply.) 
  Medical record 
  Results of administrative data reviews 
  Patient interview 
  Hospital incident report or other hospital report 
  Other, please describe:_____________________________________________________ 
 
2.  Please circle on the scale below to assess the documentation in the medical record.  A scale of 
1=worst (unclear, difficult to determine event) and 5=best (well-described and definitive). 
 
1      2      3      4      5 
 
3.  Was the medical record recorded as (check both if applies): 
  Paper records? 
  Easy to navigate 
  Some difficulty navigating 
  Difficult to navigate 
  Electronic records? 
  Easy to navigate 
  Some difficulty navigating 
  Difficult to navigate 
 
4.  Did you consult with another physician reviewer? 
  Yes 
  No 
If yes, please describe: _________________________________________________________ 
 
5.  How much time did you spend on this review (including any consultation with others)? 
  Less than 30 minutes 
  Between 30 minutes  one hour 
  Between one hour  two hours 
  More than two hours 
  Difficult to determine because of stop and start and/or consultation 
 
6.  Did this case suggest modifications of this form to improve subsequent studies? 
  Yes 
  No 
If yes, please describe: ________________________________________________________ 
 
___________________________________________________________________________ 
 
7.  Is there anything else about this case that you would like to note? _________________________ 
 
______________________________________________________________________________ 
 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  46   
   
 A P P E N D I X   ~   G  
 
Patient Harm Index 
Table G-1 lists the categories of patient harm adapted by the Institute 
for Healthcare Improvement (IHI) and used by the Office of Inspector 
General in the case study.  The National Coordinating Council for 
Medication Errors Reporting and Prevention (NCC MERP) developed an 
index to categorize the level of patient harm resulting from medication 
errors.  Researchers have modified this index for use in measuring and 
distinguishing adverse events of all types, rather than only medication 
errors.  IHI adapted the index to reflect only events that cause harm to 
patients.   
Table G-1:  IHI Global Trigger Tool Adaptation of the NCC 
MERP Index to Categorize Harm 
Category  Description 
E  Temporary harm to the patient and required intervention 
F  Temporary harm to the patient and required an initial or prolonged hospital stay 
G  Permanent patient harm 
H  Intervention required to sustain life 
I  Patient death 
Source:  F.A. Griffin and R.K. Resar, IHI Global Trigger Tool for Measuring Adverse Events, IHI Innovation 
Series white paper.  Cambridge, MA: Institute for Healthcare Improvement Innovation Series 2007, pp. 45.    
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  47   
   
 A P P E N D I X   ~   H  
Results of Screening Methods 
 
Figure H-1:  Results of Screening Methods Used for Identifying Events in Case Study
Flags Associated 
Flags by  With Events by 
Screening Method Screening Method Events*  Events
(n = 662) (n = 254) (Unduplicated) (n = 120)
Present on 
Admission (POA)  POA Analysis
Analysis (n = 111)
(n = 296)
Unique Events 
Identified Through 
Screening Methods
(n = 114)
Nurse Review Nurse Review
(n = 227) (n = 104)
Physician Review of 
Medical Records 
With Flags
Beneficiary  Beneficiary 
Events
Interviews Interviews
(n = 120)
(n = 71) (n = 21)
Hospital Incident  Hospital Incident 
Reports Reports
(n = 55) (n = 9)
Patient Safety 
Indicators (PSI)  PSI Analysis
Analysis (n = 9)
(n = 13)
Events Identified 
Only Through 
Physician Reviews
(n = 6)
Source:  Office of Inspector General analysis of 278 Medicare beneficiary hospitalizations in 2 selected counties, 2008.
*Some flags were associated with more than one event, and some events were identified through more than one flag.
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  48  x 
A  P  PEN  D 
Agency Comments:  Agency for Healthcare Research and Quality 
DEPARTMEN1'OF HEALTH & HUMAN SERVICES  Agency for H~althcare. 
Research and Quality 
540 Gaither Road 
Rockville MD 20850 
www,ahrq.gQv 
DEC 162009 
TO:  Daniel Levinson, Inspector GeneraJ 
FROM:  Cal'olynM, Clancy,M,D Dii'ectol' 
SUBJECT:  Comments on draft Office of Inspec(orGenetal Report entitled, "Adverse 
BVeiltsin Hospitals: Metl1odsJ'or Identffying Events" 
Thmlk you [wr the oPpOI:lunity to review the draft Qffj.ceoflirspe.ctor General Report 
entftled: "Ad.\lerse EVents in Hospitals: Methbds fDr Identifying Events," The Agency 
for Hefllthcare Re:seal'ch.and Quality (AHRQ)cqllctJfswith thereport.as written. 
If you have any questions; please feel freeto call Wendy Perry of my staff at 301-427
l216.  .  ,  . 
oE1 -0 6-08-00 221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  49  APPEND  X 
Agency Comments:  Centers for Medicare &Medicaid Services 
DEPARTMENT OF HEALTH & HUMAN SERVICES  Centers lor Medicare & Medicaid Services 
-----------------
Administrator 
DEC  1 7 2009  Washington. DC 20201
TO: 	 Daniel R. Levinson 

Deputy Inspector General 

FROM: 	 Charlene Frizzera 

Acting Administrator 

SUBJECT: 	 Office OfInspector General (OlG) Draft Report: "Adverse Events in Hospitals: 

Methods for Identifying Events," (OEI-06-08-00221) 

Thank you for the opportunity 10 review and comment on this draft report. "Adverse Events in 
Hospitals: Methods for Identifying Events." This is a particularly useful report that offers timely 
recommendations. The OlG undertook original data collection to assess five different methods 
of identifying the occurrence of serious adverse events in short-term acute care hospitals. The 
OlG then identified strengths and weaknesses ofeach method.  Finally, the OlG explored 
implications of its findings with regard to the ability of the Centers for Medicare & Medicaid 
Services (CMS) to monitor adverse events, promote value-based purchasing, and enforce quality 
ofcare and patient safety requirements. 
We appreciate the original contribution that this study makes in advancing the patient safety 
agenda in hospitals and bolstering CMS efforts to prevent adverse events. As the OlG noted, 
CMS enforces quality ofcare expectations through its survey and certification onsite reviews. 
eMS has actively advanced value-based purchasing across a variety ofprovider types.  CMS has 
also instituted payment policies that prevent coverage of certain serious but clearly avoidable 
medical events, often referred to as "Never Events." Information about the most recent policies 
(relating to performance of the wrong surgery or surgery on the wrong body part or on the wrong 
patient) can be found at: 
http://W'oNw.cms.hhs.gov/MLNMattersArticlcs/dowl1load~nY!M6405.pdf. 
We endorse each recommendation addressed to eMS, and provide more detail below. 
OIG Recommendation 1: 
eMS and the Agency for Healthcare Research and Quality (AHRQ) should explore opportunities 
to identify events when conducting' medical record reviews for other purposes. 
CMS Response 
We agree with the recommendation and will consider creating or taking advantage ofexisting 
opportunities to identi fy adverse events, including consideration ofthe specific options identified 
by the OlG. 
oEI-06-08-002 21  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  50   
   
A P P E N D I X   ~  I  
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  51   
   
A P P E N D I X   ~   I  
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  52   
   
A P P E N D I X   ~   I  
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  53   
   
 A C K N O W L E D G M E N T S  
This report was prepared under the direction of Kevin K. Golladay, 
Regional Inspector General for Evaluation and Inspections in the Dallas 
regional office, and A. Blaine Collins, Deputy Regional Inspector 
General.   
Amy Ashcraft and Ruth Ann Dorrill served as the team leaders for this 
study.  Deborah Cosimo served as the lead analyst.  Other principal 
Office of Evaluation and Inspections staff from the Dallas regional office 
who contributed to the report include Anthony Guerrero-Soto, Deborah 
McGurk, and Lyndsay Patty; other central office staff who contributed 
include Rob Gibbons and Rita Wurm. 
 
  OEI-06-08-00221  ADVERSE EVENTS IN HOSPITALS:  METHODS FOR IDENTIFYING EVENTS  54   
Office of Inspector General 
http://oig.hhs.gov 
 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, 
as amended, is to protect the integrity of the Department of Health and Human Services 
(HHS) programs, as well as the health and welfare of beneficiaries served by those 
programs.  This statutory mission is carried out through a nationwide network of audits, 
investigations, and inspections conducted by the following operating components: 
Office of Audit Services 
The Office of Audit Services (OAS) provides all auditing services for HHS, either by 
conducting audits with its own audit resources or by overseeing audit work done by others.  
Audits examine the performance of HHS programs and/or its grantees and contractors 
in carrying out their respective responsibilities and are intended to provide independent 
assessments of HHS programs and operations.  These assessments help reduce waste, 
abuse, and mismanagement and promote economy and efficiency throughout HHS. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) conducts national evaluations to provide 
HHS, Congress, and the public with timely, useful, and reliable information on 
significant issues.  Specifically, these evaluations focus on preventing fraud, waste, or 
abuse and promoting economy, efficiency, and effectiveness in departmental programs.  
To promote impact, the reports also present practical recommendations for improving 
program operations.  
Office of Investigations 
The Office of Investigations (OI) conducts criminal, civil, and administrative 
investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries 
and of unjust enrichment by providers.  The investigative efforts of OI lead to criminal 
convictions, administrative sanctions, or civil monetary penalties. 
Office of Counsel to the Inspector General 
The Office of Counsel to the Inspector General (OCIG) provides general legal services to 
OIG, rendering advice and opinions on HHS programs and operations and providing all 
legal support in OIG's internal operations.  OCIG imposes program exclusions and civil 
monetary penalties on health care providers and litigates those actions within HHS.  
OCIG also represents OIG in the global settlement of cases arising under the Civil False 
Claims Act, develops and monitors corporate integrity agreements, develops compliance 
program guidances, renders advisory opinions on OIG sanctions to the health care 
community, and issues fraud alerts and other industry guidance.