Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
The External Review of Hospital Quality 
Holding the Reviewers Accountable 
JUNE GIBBS BROWN 
Inspector General 
JULY 1999 
OEI-01-97-00053  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 
protect the integrity of the Department of Health and Human Services programs as well as the 
health and welfare of beneficiaries served by them.  This statutory mission is carried out through a 
nationwide program of audits, investigations, inspections, sanctions, and fraud alerts.  The 
Inspector General informs the Secretary of program and management problems and recommends 
legislative, regulatory, and operational approaches to correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Boston regional office prepared this report under the direction of Mark R. Yessian, Ph.D., 
Regional Inspector General.  Principal OEI staff included: 
BOSTON  HEADQUARTERS 
Joyce M. Greenleaf, Project Leader  Jennifer Antico, Program Specialist 
Lynne G. Dugan, Senior Analyst  Alan Levine, Program Specialist 
Meredith A. Vey, Program Analyst 
To obtain copies of this report, please call the Boston Regional Office at 617/565-1050. 
Reports are also available on the World Wide Web at our home page address: 
http://www.dhhs.gov/progorg/oei  E X E C U T I V E   S U M M A R Y  
PURPOSE 
To assess how the Health Care Financing Administration holds the Joint Commission and 
State agencies accountable for the external review of hospital quality. 
BACKGROUND 
External Quality Review of Hospitals in the Medicare Program 
Hospitals routinely offer valuable services, but also are places where poor care can 
lead to unnecessary harm.  The external quality review of hospitals plays an important role 
not only in protecting patients from such harm, but also in complementing the hospitals 
own internal quality efforts.  The Federal Government relies on two types of external 
review to ensure hospitals meet the minimum requirements for participating in Medicare: 
accreditation, usually by the Joint Commission on Accreditation of Healthcare 
Organizations, and Medicare certification, by State agencies.  About 80 percent of the 
6,200 hospitals that participate in Medicare are accredited by the Joint Commission. 
This Inquiry 
This report, part of a series of four companion reports that resulted from our 
inquiry, focuses on the Health Care Financing Administrations (HCFAs) oversight of 
both the Joint Commission and State agencies.  Our inquiry draws on aggregate data, file 
reviews, surveys, and observations from a rich variety of sources, including HCFA, the 
Joint Commission, State agencies, and other stakeholders. 
We organized this report around a three-part framework that HCFA can use to 
hold accrediting bodies and State agencies accountable: (1) obtaining information on 
performance, (2) providing feedback on performance, and (3) disclosing information 
publicly. 
FINDINGS 
The HCFA obtains limited information on the performance of the Joint Commission 
or the States.  In both cases, HCFA asks for little in the way of routine performance 
reports.  To assess the Joint Commissions performance, HCFA relies mainly on validation 
1 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  surveys conducted, at HCFAs expense, by the State agencies.  But for a number of 
reasons the value of these surveys has been limited.  The methodology for selecting the 
hospitals to survey fails to consider hospital size or type and draws on hospitals surveyed 
only in certain months.  More fundamentally, the surveys have been  based on different 
standards ( the Medicare conditions of participation as opposed to the Joint Commission 
standards) and have been conducted subsequent to the Joint Commissions survey (when 
hospital conditions could have changed).  During 1996 and 97, HCFA piloted 20 
observation surveys--during which State and HCFA officials accompanied Joint 
Commission surveyors.  This approach appears to have much promise, but HCFA has not 
yet issued any evaluation of the pilots. 
The HCFA rarely observes State agencies survey hospitals, and conducts no validation 
surveys of them. 
The HCFA provides limited feedback to the Joint Commission and the State 
agencies on their overall performance.  Its feedback to the Joint Commission is more 
deferential than directive.  Its major vehicle for feedback to the Joint Commission is its 
annual Report to Congress, which is based on the flawed validation surveys and has 
typically been submitted years late.  The HCFA is more directive to the State agencies, 
which carry out their survey work in accord with HCFA protocols, but gives them little 
feedback on how well they perform their hospital oversight work. 
Public disclosure plays only a minimal role in holding Joint Commission and State 
agencies accountable.  The HCFA makes little information available to the public on the 
performance of either hospitals or of the external reviewers.  By contrast, HCFA posts 
nursing home survey findings on the Internet and requires nursing homes to post them 
within the facility as well.  The Joint Commission has been more proactive than HCFA in 
making hospital survey results widely available on the Internet and through other means. 
CONCLUSION 
The clear and disturbing conclusion of this report is that both the Joint 
Commission and State agencies are only minimally accountable to HCFA for their 
performance in reviewing hospitals.  While we recognize that these entities are also 
accountable to others and that they must have considerable flexibility to function 
effectively, we maintain that it is vitally important for HCFA to ensure that they 
adequately fulfill their responsibilities to protect Medicare beneficiaries.  How, then, can 
HCFA hold these entities accountable while minimizing burdensome oversight?  How can 
it recognize their inherent strengths and limitations, and tailor performance measurement 
accordingly?  We address these and other related questions in our summary report, A Call 
for Greater Accountability.  That report also contains our recommendations, which we 
direct to HCFA. 
2 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  COMMENTS 
Within the Department of Health and Human Services, we received comments 
from HCFA.  We also solicited and received comments from the following external 
parties:  Joint Commission on Accreditation of Healthcare Organizations, Association of 
Health Facility Survey Agencies, American Osteopathic Association, American 
Association of Retired Persons, Service Employees International Union, National Health 
Law Program, and Public Citizens Health Research Group.  We include the detailed text 
of all of these comments and our responses to them in our summary report, The External 
Review of Hospital Quality: A Call for Greater Accountability (OEI-01-97-00050). 
3 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  T A B L E  O F  C O N T E N T S  
PAGE 
EXECUTIVE SUMMARY 
INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

FRAMEWORK FOR ACCOUNTABILITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

FINDINGS: 
Obtaining Information on the Joint Commissions Performance  . . . . . . . . . . . . . . . . . . .  9

Providing Feedback to the Joint Commission  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Disclosing Information on Joint Commission Performance  . . . . . . . . . . . . . . . . . . . . . .  14

Obtaining Information on State Agencies Performance  . . . . . . . . . . . . . . . . . . . . . . . . 15

Providing Feedback to State Agencies  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Disclosing Information on State Agencies Performance . . . . . . . . . . . . . . . . . . . . . . . . 18

CONCLUSION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  20

APPENDICES 
A: Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21

B: HCFAs Reinvention Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

C: SAQIP  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

D: Endnotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  I N T R O D U C T I O N  
PURPOSE 
To assess how the Health Care Financing Administration holds the Joint

Commission and State agencies accountable for the external review of hospital quality.

BACKGROUND 
Hospital Safety 
Hospitals are an integral part of our healthcare system, offering services that 
improve, extend, and even save lives.  But they are also places where inappropriate care 
can lead to unnecessary harm.  This reality was clearly underscored in 1991, when a 
Harvard medical practice revealed the results of its review of about 30,000 randomly 
selected records of patients hospitalized in New York during 1984.  The study found that 
1 percent of the hospitalizations involved adverse events caused by negligence.1  On the 
basis of its sample, the study team estimated that during that year, negligent care provided 
in New York State hospitals was responsible for 27,179 injuries, including 6,895 deaths 
and 877 instances of permanent and total disability.  Many other more recent studies 
have reinforced the concerns raised by the Harvard study.  Of particular note was one that 
focused on the care received by 1,047 hospitalized  patients in a large teaching hospital 
affiliated with a medical school.  It found that 17.7 percent of these patients received 
inappropriate care resulting in a serious adverse event--ranging from temporary disability 
to death.2  In the public eye, these scholarly inquiries have been overshadowed by media 
reports that describe, often in graphic detail, the harm done to patients because of poor 
hospital care.3 
Hospitals rely upon a variety of internal mechanisms, from physician credentialing, 
to peer review and benchmarking, in order to try to avoid such incidents and to improve 
the quality of care provided in their facilities.  External quality review serves as a vital 
additional safeguard.  It provides a more detached, independent mechanism for assessing 
the adequacy of hospital practices.  Such oversight is of fundamental importance to 
patients and to the public and private entities that purchase health care services on their 
behalf.  Protecting patient safety and improving the quality of patient care must be a top 
priority of external review. 
Medicares Interest in External Hospital Quality Review 
The Medicare program covers about 38 million elderly and disabled individuals, 
many of whom are high users of hospital care.4  In 1997, Medicare spent about $136 
5 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  billion on Part A, the hospital insurance benefit.5  This figure is just over half the total 
amount the Federal government spent on all Medicare benefits.6  In the same year, 
Medicare spent over $80 billion for inpatient acute hospital care alone.7 
Since Medicares inception, external quality review has been a part of the 
Medicare program.  When Congress enacted the Medicare Act in 1965, it required 
hospitals to meet certain minimum health and safety requirements to participate in the 
program.8  Those minimum requirements, called the Medicare conditions of participation, 
were published in 1966, revised in 1986, and are now being revised again (see appendix 
A).9 
The Joint Commission and State Agencies as External Reviewers 
The Federal Government relies on two processes of external review to determine 
hospital compliance with these conditions of participation: accreditation, usually by the 
Joint Commission on Accreditation of Healthcare Organizations (Joint Commission), and 
Medicare certification, by State survey and certification agencies.10  Both processes 
involve a team of trained surveyors visiting a hospital, interviewing staff, reviewing 
documents, and inspecting the facility.11  However, the nature of these processes is very 
different. 
Accreditation is a voluntary conformity-assessment process whereby industry 
experts define what standards organizations must conform to in order to be accredited and 
then surveyors systematically assess the organizations performance against those 
standards.12  It is a form of self-regulation for which hospitals pay a fee.13  The Joint 
Commission enjoys a special status because, by Federal statute, hospitals accredited by the 
Joint Commission are deemed to meet the Medicare conditions of participation.14  As the 
largest accreditor of hospitals, accrediting about 80 percent of the nations 6,200 
hospitals, the Joint Commission is responsible for the majority of the nations external 
quality review of hospitals.  In 1972, Congress enacted amendments that gave the Health 
Care Financing Administration (HCFA) responsibility for overseeing the Joint 
Commission.15 
The Medicare certification process, on the other hand, is a public regulatory 
process that aims to ensure hospitals wishing to serve Medicare beneficiaries, but not 
wishing to be accredited, meet the conditions of participation.  The HCFA relies on the 51 
State survey and certification agencies (hereafter called State agencies) to determine 
compliance with the Medicare conditions of participation.  Hospitals pay no fee for this 
process.  States agencies currently certify 1,442 nonaccredited hospitals nationwide.16 
These State agencies are paid and trained by HCFA and use HCFAs survey instrument to 
survey nonaccredited hospitals. 
6 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  This Inquiry and Report 
This report describes how HCFA holds the Joint Commission and the State 
agencies accountable for the external review of hospital quality.  Toward that end, on the 
following page we offer a framework for considering how external reviewers can be held 
accountable.  We offer our findings in accord with this framework. 
This report is one of four companion reports that resulted from our inquiry. 
Another report, The Role of Accreditation (OEI-01-97-00051), examines the 
contributions of accreditation in external hospital review. A third, The Role of Medicare 
Certification (OEI-01-97-00052), describes the extent and nature of HCFA and State 
agencies external review of nonaccredited hospitals. The fourth report, A Call for 
Greater Accountability (OEI-01-97-00050), provides a summary of external review of 
hospital quality and presents the recommendations emerging from our inquiry. 
Our inquiry draws on a variety of sources.  These include:  data from HCFAs 
Online Survey Certification and Reporting System; aggregate data from the Joint 
Commission concerning hospital survey activity; a mail survey to State agencies in the 50 
States and District of Columbia (hereafter referred to as a State); observations of 7 
hospital surveys conducted by the Joint Commission and one conducted by a State agency; 
observation of two complaint investigations by State agencies; reviews of accreditation 
manuals, policies, and hospital survey files from the Joint Commission; the systematic 
gathering of information from representatives of HCFA central and regional offices, State 
agencies, the Joint Commission, American Hospital Association, consumer groups, 
professional associations, and representatives of other organizations we considered to be 
stakeholders in hospital oversight; and reviews of laws and regulations and articles from 
newspapers, journals, newsletters, and magazines.  We also interviewed officials from the 
American Osteopathic Association and reviewed their accreditation materials.  The 
American Osteopathic Association accredits about 100 to 150 hospitals, some of which 
are also accredited by the Joint Commission.  For purposes of this inquiry, however, we 
focused on the Joint Commission.  See appendix B for more details on our methodology. 
We conducted this inspection in accordance with the Quality Standards for 
Inspections issued by the Presidents Council on Integrity and Efficiency. 
7 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  A C C O U N T A B I L I T Y   F R A M E W O R K  
The following is a generic framework for considering the mechanisms that HCFA can use 
to hold accrediting bodies and State agencies accountable. We base the framework on existing 
mechanisms. The framework serves as a starting place for discussions, to which mechanisms can 
be added or deleted. We present it in the context of hospitals, but it can also be considered in the 
context of other provider types.  We apply it to the Joint Commission and State agencies on the 
following pages. 
Obtaining Information on Performance 
1. Validation Surveys. These are onsite reviews of hospitals that are conducted some time after 
the accreditors or State agencies own visits. Validation surveys aim to ensure that reviewed 
hospitals meet certain minimums. 
2.  Observation Surveys.  These are onsite reviews conducted at the same time as the accreditors 
or State agencies surveys.  Their intent is to directly observe the surveyors and to use those 
observations as the basis for constructive feedback. 
3.  Performance Reports. Such reports involve the regular sharing of data and analysis to enable 
HCFA to remain well-informed of the hospital review activities. 
4. Policy and Procedural Updates. These are regular notices to HCFA on the key policies and 
procedures governing the bodies reviews of hospitals. 
Providing Feedback on Performance 
1. Performance Assessments. These represent HCFAs feedback to the accreditors and state 
agencies on how well they are performing.  They can be regularly scheduled or on an as-needed 
basis. 
2. Policy and Procedural Guidance. Through such guidance, HCFA can convey to the bodies 
those approaches and priorities that it regards to be of national significance. 
Disclosing Information Publicly 
1. Performance of Hospitals. Publicly disclosing information on hospitals conveys not only 
something about the hospitals performance but also assurance that a process exists for overseeing 
hospitals that wish to serve Medicare beneficiaries. Such disclosure fosters a trust that the publics 
safety is paramount. 
2.  Performance of Accrediting Bodies/State Agencies.  By publicly disclosing information on 
these entities, HCFA conveys that it is monitoring the work they do on its behalf. It fosters trust 
that HCFA is promoting public safety. 
8 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  J o i n t  C o m m i s s i o n  A c c o u n t a b i l i t y  
The HCFA obtains little information on Joint Commission 
performance. 
Validation Surveys:  These surveys are fundamentally limited as tool of 
accountability because they are based on different standards that are applied at 
different points in time.  They are also costly to implement and based on a 
sample that not only is flawed, but also has been shrinking over time. 
The validation survey process is the centerpiece of HCFAs approach to holding 
the Joint Commission accountable.17  Yet its value is limited.  It seeks to measure the Joint 
Commissions performance through hospitals compliance with the Medicare conditions of 
participation, not the Joint Commissions own standards.18  This difference in reference 
points not only complicates the interpretation of validation surveys, but also inherently 
compromises their value.  Any reasonable tool of oversight needs a common frame of 
reference.19  The validation survey now must bridge the Joint Commissions several 
hundred standards, which are categorized into 45 groups, with the few dozen Medicare 
conditions.  Moreover, Joint Commission standards--considered by many even within 
HCFA to be state-of-the-art--are constantly evolving, whereas the Medicare conditions 
rarely change:  first published in 1966, revised substantially in 1986, and under revision 
again now.20 
Further undermining the validation process as a tool for accountability is its 
retrospective nature:  validation surveys occur within 60 days of the Joint Commissions 
routine hospital survey.21  During that elapsed time, hospitals have likely corrected some 
problems and experienced new ones, undergone staff changes, expanded or eliminated 
services--in other words, any number of changes could affect the validation survey results. 
Validation surveys are labor-intensive.  A team of at least two surveyors conducts 
a full hospital survey, lasting 2 to 5 days, depending on the size of the hospital.  Alone, the 
survey portion of the validation probably costs upwards of $2 million per year.22  Beyond 
that, though, are costs associated with the preparation and analysis.  For example,  HCFA 
staff choose the sample for validation, collect and compare the accreditation and validation 
survey results, and analyze them.  Considering the basic limitations identified above, 
devoting these resources to validation surveys is questionable. 
Other concerns about HCFAs approach to validation stem from how it selects 
hospitals for validation surveys.  For example, until recently, HCFA selected hospitals for 
9 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  validation surveys only from those having undergone Joint Commission accreditation 
surveys between January and July of each year.  The HCFA reports that in January 1998 it 
began drawing a random sample of accredited hospitals each month to correct this 
problem.  While an improvement, HCFAs approach to sampling still fails to consider 
hospital characteristics such as past performance, bedsize or rural or urban location. 
Other problems with the validation sample appear to be improving.  For example, 
although the size of the sample for validation fell over the past few years, it appears to be 
on the rise.  Between 1991 and 1997, the size of the validation sample fell 50 percent, 
from 227 surveys in 1991 to 113 surveys in 1997 (about 2 percent of all accredited 
hospitals).  In 1998, HCFA reports that it conducted 164 validation surveys, an increase of 
31 percent over the previous year.  Still, some hospitals selected for validation never get 
validated, although that also appears to be improving.  In 1996, for example, HCFA 
validated no hospitals in 16 States, including Missouri, Iowa, Nebraska, and Kansas--an 
area comprising an entire Federal region of the country.  Likewise, in 1997, HCFA 
validated none of the 30 selected hospitals in New Mexico, Texas, Oklahoma, Arkansas, 
and Louisiana--another Federal region.  In 1998, however, HCFA reports that each 
Federal region had at least one validation survey conducted in it and 76 percent of the 
hospitals selected for validation did, in fact, have a validation survey. 
Observation Surveys:  The HCFA has piloted an observation survey that has 
promise but remains largely undeveloped. 
The HCFA developed an observation survey process as part of a larger initiative to 
reinvent its oversight of accreditation (HCFA generally refers to these surveys as 
reinvention surveys, see appendix B for more information on the reinvention initiative). 
The process developed called for State and HCFA surveyors to silently follow and observe 
while Joint Commission surveyors conduct accreditation surveys--a major shift from 
traditional validation surveys.  In developing the observation surveys, HCFA aimed to 
address the fundamental problems with the traditional validation (such as different 
reference points and time frames).  A workgroup involving officials from HCFA, State 
agencies, and the Joint Commission devoted considerable effort to defining an observation 
process as a tool for accountability.  The observation survey represented an opportunity 
for State and HCFA surveyors to assess first-hand the process of an accreditation survey. 
It promised to provide what the traditional validation could not:  the same snapshot of a 
hospital and insights into accreditation surveyor skills and processes (defined as survey 
preparation, survey process, interpersonal, presentation, and teamwork).23 
In 1996 and 1997 HCFA and State agency surveyors tested their newly developed 
observation survey process by observing 20 accreditation surveys across 4 States.  But 
HCFA has yet to report on the results.  Indeed, its plans for revising or expanding the 
observation surveys are undeveloped.  Using skilled Medicare surveyors to be what 
essentially amounts to silent observers of an accreditation survey process is a questionable 
10 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  use of scarce resources.  Furthermore, we observed neither much interaction with, nor any 
feedback to, accreditation surveyors on the one survey observation in which we 
participated.  In fact, because HCFA has yet to produce an evaluation on the pilot, it has 
yet to result in feedback to the Joint Commission. 
Performance Reports:  The HCFA obtains few reports addressing Joint 
Commission performance. 
The HCFA asks for little regular reporting from the Joint Commission beyond the 
survey scheduling and results it needs to carry out its validation surveys.  For example, it 
asks for no regular reports from the Joint Commission on topics such as the nature or 
pattern of complaints received about accredited hospitals, appeals of accreditation 
decisions, or the extent and nature of actions taken as a follow-up to surveys.  It also asks 
for no aggregate data on the timeliness of surveys or training of surveyors.  Nor does it 
ask for any regular reports on how many special surveys (for example in response to 
unexpected deaths or preventable injuries) the Joint Commission conducts.  These types of 
data are already being collected by the Joint Commission and could provide HCFA with 
insights into how well accredited hospitals are performing, what new problem areas are 
emerging, and how well the accreditation results are being received.24 
Regular reporting from the Joint Commission to HCFA has been largely limited to 
information HCFA needs to carry out its validation surveys.  For example, the Joint 
Commission provides HCFAs central office with lists of upcoming surveys, accredited 
hospitals and their accreditation status, and individual hospital performance reports.  The 
HCFA has also received data on Joint Commission standards commonly found out of 
compliance and a report on hospital withdrawals from accreditation, mergers, and 
acquisitions. 
Policy Updates: The HCFA receives regular updates on Joint Commission 
policies and procedures. 
The Joint Commission communicates with HCFA regularly and through several 
avenues.  For example, officials from HCFA meet several times a year with 
representatives of the Joint Commission.  Their meetings cover an array of topics, 
including the reinvented survey process, electronic exchange of data, and changes in 
the accreditation survey process, among others.  In the course of developing and 
implementing the observation survey pilot, HCFA and Joint Commission officials met 
regularly and HCFA staff even attended Joint Commission surveyor training.  Staff 
from the Joint Commission communicate regularly to clarify policy and procedural 
issues, either in person, in writing, on the telephone, or, more recently, via electronic 
mail.  And at least two HCFA officials serve as members of Joint Commission 
committees, which provides additional opportunities for HCFA to remain well-
11 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  informed of existing and emerging policies.25  The communication involves largely the 
HCFA central rather than regional office staff. 
The HCFA provides minimal feedback to the Joint 
Commission. 
Performance Assessments:  The main vehicle for feedback on the Joint 
Commissions performance is the Report to Congress.  That report is of limited 
value. 
The Congressional mandate that requires HCFA to publish a summary of the 
validation reports as part of its Medicare Annual Report to Congress shapes both the 
method and content of HCFAs feedback to the Joint Commission.26  The HCFA uses this 
report to provide feedback about the validation surveys to the Joint Commission.  It uses 
no other formal mechanism for feedback.  Thus, outside of validation survey results, the 
Joint Commission receives little in the way of detailed, operational feedback or 
recommendations for improvement. 
We have already raised questions about the basic value of the validation surveys. 
Beyond that, by the time HCFA releases its Annual Reports, the validation data are dated, 
further limiting their value.  For example, in 1993 HCFA released the reports to Congress 
for the Fiscal Years 1989 and 1990.  It published the results of the 1991 and 1992 
validation surveys in the 1992 Report, released in September 1994.  The 1994 validation 
survey results never left the Department, and HCFA released the most recent reports, 
containing the 1995 and 1996 validation results, in late 1997.  Given these delays, the 
Joint Commissions standards and procedures could have changed between the time the 
validation surveys were conducted and the time the Joint Commission receives its copy of 
the Medicare Annual Report. 
The delays of HCFAs validation feedback impede meaningful discussion with the 
Joint Commission about the findings.  While HCFA does not provide the Joint 
Commission with an opportunity to formally respond to the validation findings, some such 
discussion has occurred.  For example, in response to the results of the 1991 and 1992 
validation findings, which raised concerns about the Joint Commissions attention to life 
safety codes, HCFA observed the Joint Commissions surveyor training and consulted 
with the it about interpreting life safety code standards.27 
12 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  Policy and Procedural Guidance:  The HCFAs guidance to the Joint Commission 
on policy and procedural matters is negligible. 
The HCFAs posture toward the Joint Commission is more deferential than 
directive.  The Joint Commission, which has dominated the hospital accreditation market 
for the past 30 years, has amassed expertise that HCFA relies upon.  Indeed, many 
officials in HCFA refer to the Joint Commission as maintaining the gold standard or 
higher bar toward which HCFA can aspire.  The HCFA even modeled the recently 
proposed hospital Conditions of Participation after Joint Commissions Agenda for 
Change.28 
On occasion, representatives of both HCFA and the Joint Commission will be on-
site at the same hospital at the same time.  This is especially apt to occur following an 
incident of patient harm in an accredited hospital, to which both parties are responding. 
According to HCFA regional officials with whom we spoke, the HCFA takes on no 
coordinating nor information-sharing roles during those responses.  In fact, during one 
high profile case of a preventable death in a hospital, the HCFA regional office staff 
reported that they played a significant role in coordinating the responses of HCFA and 
State officials, but not the Joint Commission.  A HCFA official from another region 
captured this hands-off approach when he said [We] wouldnt want to interfere with their 
process. 
While the HCFA lacks a seat on the Joint Commissions Board of Commissioners, 
it does have opportunities to provide policy and procedural guidance through its 
participation in two Joint Commission committees.29  In the Professional and Technical 
Advisory Committee for hospital standards, HCFA staff can weigh in as new standards are 
discussed and developed.  Likewise, HCFA can weigh in on discussions of performance 
measurement by participating in the Council on Performance Measurement.  However, the 
expectations of Committee membership, as defined by the Joint Commission, call for 
participants to not only provide advice, but also to serve as [an] external advocate[s] for 
the accreditation program.30  Such expectations could run counter to HCFAs role in 
holding the Joint Commission accountable. 
Other factors also play some role in limiting HCFAs ability to provide meaningful 
policy or procedural guidance to the Joint Commission.  For example, within HCFA itself, 
no center of responsibility exists for hospital-related issues.  Rather, responsibility for 
issues related to hospitals is divided among several divisions and groups in the Center for 
Medicaid and State Operations, the Center for Health Plans and Providers, and the Office 
of Clinical Standards and Quality.  The HCFAs organizational changes in the last few 
years have left many HCFA staff and managers with steep learning curves with respect to 
hospital quality review.  Indeed, many HCFA officials with whom we spoke characterized 
their interactions with the Joint Commission as more educational than evaluative. 
13 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  The Joint Commission and especially HCFA make limited 
use of public disclosure as a tool of accountability. 
Performance of Hospitals:  The Joint Commission makes basic information on 
accredited hospitals survey results public, but keeps most information about 
adverse events confidential. 
In December 1994, the Joint Commission began releasing accreditation 
information about hospitals, on request, in the form of performance reports.31  These 
reports include the hospitals overall survey score from its most recent triennial survey, the 
hospitals scores in the 45 areas surveyed, comparative data on the percentage of hospitals 
receiving each possible score in the 45 areas, and a list of the areas in having specific 
recommendations for improvement, including the status of the improvement.32  The 
reports exclude any details on the problems identified during the surveys.  But the reports 
do come with a guide for interpreting and understanding the scores and an explanation of 
the 45 performance areas. 
As of late 1998, the Joint Commission had 13,408 performance reports for 
hospitals, nursing homes, and other provider types available to the public, and that list 
keeps growing.  Between late 1996 and early 1997, requests for performance reports grew 
from 13 per quarter to over 2,000.  These reports are free and readily available by mail, 
phone, or through the Internet.33 
However, the Joint Commission keeps most information on adverse events in 
hospitals confidential.  In April 1998, it implemented a policy fostering hospitals to self-
report such events and conduct their own analysis of the cause of the events.34  By keeping 
the information confidential, the Joint Commission aims to provide hospitals with a safe 
venue for reporting adverse events, thereby allowing it to develop a database for 
researching and preventing such events. 
Performance of Accrediting Bodies:  The HCFA has little to disclose on the 
performance of the Joint Commission, but does make its Report to Congress 
available on request. 
The HCFA does not mandate that information on the performance of the Joint 
Commission be disclosed.  In fact, it has little to disclose.  It does have the Report to 
Congress, its main vehicle for providing feedback to the Joint Commission.  That report is 
available to anyone who requests it, but, as we have seen, it is of limited value. 
Independent of HCFA, the Joint Commission makes some information about its own 
performance public through its newsletter, website, and occasional articles in magazines.35 
14 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  S t a t e  A g e n c y  A c c o u n t a b i l i t y  
The HCFA obtains little information on State agencies 
performance. 
Validation Surveys:  The HCFA does not conduct such surveys of State agencies 
hospital activities. 
None of the 10 HCFA regional offices reported conducting validation surveys of 
nonaccredited hospitals over the past few years.  Based on our accountability framework 
presented on page 7, by validation survey we mean an onsite survey, conducted sometime 
after a State agency survey, that aims to verify that the hospital meets certain minimums. 
The HCFAs workload, like that of State agencies, is dominated by nursing homes and 
home health agencies rather than hospitals.  And while HCFA may validate those surveys, 
it has not validated hospital certification surveys in recent years. 
Observation Surveys:  The HCFA rarely conducts such surveys of State agencies 
hospital activities. 
As we have seen, the bulk of State agencies work is directed toward nursing 
homes and home health agencies rather than hospitals, and HCFAs oversight of the 
agencies reflects this.  State agencies conduct relatively few surveys of nonaccredited 
hospitals each year, thereby limiting the opportunities for HCFA to observe their survey 
process.36  Indeed, 4 of the 10 HCFA regional offices reported conducting no onsite 
monitoring at all over the past year or so.  The other 6 regional offices reported observing 
15-20 State agency surveys of hospitals over the past 3 years.  This represents about 2 
percent of all nonaccredited hospitals, or less than 1 percent per year.  The six regions that 
did observe State agency survey hospitals reporting using different approaches in their 
observation.  For example, some regional offices may provide feedback to the State 
surveyors while observing, others may wait until after the survey is complete. 
Performance Reports:  Nationally, HCFA obtains little information on State 
agencies performance concerning hospitals. 
In 1996, HCFA established the State Agency Quality Improvement Plan (SAQIP) 
to promote ongoing improvement in all aspects of survey and certification, including 
hospitals.37  Toward that end, HCFA developed a set of core performance standards for 
agencies (see appendix C for standards).  For the most part, the standards developed by 
HCFA apply generally to all provider types rather than being specific to hospitals.  For 
example, All surveys are conducted by qualified individuals, applies across provider 
15 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  types, including nursing homes and home health agencies.  In fact, when we asked States 
about their improvement plans, only six identified goals specific to hospitals, such as 
improving the review of complaints related to hospital dumping.38 
Under SAQIP, HCFA requires States to design a plan for quality improvement 
specific to the performance standard.  The HCFA regional offices assist the State agencies 
and receive copies of the quality improvement plans, thus they have a general picture of 
the State agencies progress.  Centrally, HCFA relies on its mid- and year-end SAQIP 
reports for a picture of the States performance.39 
This approach, however, provides HCFA with a superficial and somewhat blurred 
picture of States performance in overseeing hospitals.  For example, the 1997 year-end 
SAQIP report indicated State agencies had taken on general improvement projects ranging 
from training programs for documenting deficiencies to better tracking systems for survey 
data.40  But the report also highlights such vague examples of improvement as initial 
success was opening communication, review done, goal met, and State intends to use 
SAQIP in its licensure program.41  The HCFA grouped these and other examples into 
such categories as enthusiasm, positive feedback received, and improved 
communication.  Overall, such examples provide little in the way of meaningful insights 
into a States performance. 
While SAQIP provides a standard means for State agencies to approach quality 
improvement, it fails to provide either a means of comparing the performance across 
States or insights specific to hospitals.  To some extent, HCFA can rely on other means 
for these.  For example, it logs all hospital anti-dumping complaints received and 
investigated by the States.  It also maintains a database of State agencies survey activities 
and findings.  The States input data reflecting their survey work into the database, which 
HCFA then can access and aggregate.  The HCFA can glean the extent of survey activity 
and trends in hospitals choosing or dropping accreditation, complaints, and survey 
findings, among others.  Staff in the HCFA regional offices reported relying on such data 
occasionally, but the data have limits.  For example, HCFA maintains logs of patient 
dumping that list complaints chronologically, but lack summary data and analysis of 
trends.  Also, HCFA cannot determine trends in hospital termination notices.  In other 
words, HCFA does not know how many times it issued hospital termination notices based 
on findings during Medicare certification surveys of nonaccredited hospitals.42  It does 
know, however, how many hospitals are actually terminated. 
Policy and Procedural Updates:  The HCFA determines the policies and 
procedures used by the States. 
The HCFA provides State agencies with the survey protocols and interpretive 
guidelines they need to survey hospitals.  Its relationship with State agencies is akin to a 
contractual relationship, with HCFAs expectations spelled out in an agreement renewed 
16 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  annually.43  It ranks their activities by priority in the annual budgeting process.44  Some 
regional offices hold regular conference calls with their States to pass along updates, 
others hold regional conferences.  The HCFA regional offices provide technical assistance 
to State agencies as needed, and generally have close working relationships with them. 
The HCFA also regularly provides training for State agency staff.  The training agenda 
reflects statutory, regulatory, and procedural changes as well as the workload of the 
States, which means long-term care dominates the training agenda.  For example, among 
about 20 courses, HCFA offered 1 that was specific to hospitals in Fiscal Year 1997.  In 
Fiscal Year 1998, 3 of 25 courses HCFA offered were specific to hospitals. 
The HCFA provides limited feedback to State agencies. 
Performance Assessments:  The HCFA largely relies on the State agencies to 
conduct their own performance assessments.  It provides little feedback specific 
to their hospital work. 
The HCFAs SAQIP is the primary means for assessing State agencies 
performance.  By design, SAQIP calls for State agencies to work with HCFA regional 
offices to develop and implement States own quality improvement plans.  Since 
implementing SAQIP, HCFA has centrally produced two status reports:  one in mid-year 
1997 and one at the years end.  The HCFA sent copies of both reports to all State 
agencies.  These reports, though, provide no State-specific feedback.  They neither 
identify States nor offer assessments of performance specific to hospitals or other provider 
types.  They include no summaries of activities such as hospital surveys conducted or 
complaints received.  They also lack any feedback on specific areas for States to focus 
their improvements, or actions such as follow-up.  In fact, the status reports give almost as 
much attention to the SAQIP process and its challenges as they do to the challenges of 
survey and certification activities. 
As we have seen, the State agencies tend to work closely with the HCFA regional 
office staff in the SAQIP process, offering opportunities for feedback.  In fact 32 of 51 
State agencies reported receiving feedback from HCFA regarding their hospital oversight 
activities, and 20 reported making improvements based on that feedback.45  For example, 
several States reported implementing training programs, changing the emphasis of their 
approach to validation surveys, and establishing protocols for handling complaints based 
on HCFA feedback. 
Nevertheless, the feedback that State agencies receive from HCFA appears to be 
of limited value.  For example, when asked about the usefulness of HCFAs oversight, 29 
of 51 State agencies rated it not or somewhat useful, 15 as moderately useful, and 7 as 
very or extremely useful.  While it varies by region, just one HCFA regional office 
indicated that it provides narrative assessments of its State agencies performance.  Others 
17 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  indicated that feedback to the State agencies tends to be narrow and focused on a specific 
hospital rather than broad and relevant to the range of responsibilities States hold with 
hospitals.  For example, HCFAs feedback on a States performance is more akin to 
technical assistance in documenting a hospitals condition than assistance in prioritizing or 
managing their resources devoted to hospital activities. 
Policy and Procedural Guidance:  The HCFA gives State agencies considerable 
guidance, often on a case-by-case basis involving specific hospitals. 
The bulk of State agencies hospital work is related to investigating complaints, 
adverse events, and conducting validation surveys.46  The HCFA works closely with the 
agencies in carrying out each these activities.  When it comes to hospital complaints, for 
example, the agencies workload extends to accredited as well nonaccredited hospitals, 
and HCFA regional offices tend to be closely involved in determining not only which get 
investigated, but also to what extent.  This is particularly true when the complaint relates 
to an accredited hospital.  In those cases, the HCFA reviews the complaint in light of the 
Medicare Conditions of Participation and determines whether the State needs to 
investigate.  If an investigation ensues, the HCFA staff oversee the process step-by-step: 
they review the survey findings, summary of deficiencies, and any follow-up.  In some 
cases, especially high-profile complaints, the HCFA may send one if its own surveyors to 
accompany the State agency.  Indeed, in one State with a highly publicized series of 
hospital deaths, HCFA was so concerned about the State agencys ability to handle the 
case that it stepped in with its own surveyors to respond. 
The HCFA regional offices also provide considerable guidance on routine 
certification surveys.  As with complaint surveys, the HCFA staff will review the survey 
documentation.  Indeed, if the State agency finds a hospital out of compliance with a 
Medicare condition of participation, then HCFA is directly involved in the follow-up, by 
issuing termination notices, for example.  The HCFA also provided feedback for those 15-
20 monitoring surveys HCFA conducted over the past few years, usually by means of a 
formal debriefing with the State surveyors. 
The HCFA makes limited use of public disclosure as a means 
of holding hospitals or State agencies accountable. 
Performance of Hospitals:  The HCFA discloses results of its hospital 
certification and complaint surveys upon request. 
The HCFA will disclose most of the survey documents, except those related to 
peer review that are considered confidential.  Thus, upon request, HCFA will disclose the 
hospitals statement of deficiencies, which spells out, condition by condition, the problems 
18 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  identified during the survey and the hospitals plan for correction.  It only lists those areas 
where the surveyors identified problems, and excludes any comparative or trend 
information.  The HCFA will also disclose data from its database that records the findings 
and trends of hospital surveys.47  Neither the statement of deficiencies nor the data itself is 
available through HCFAs website.  Rather, interested parties must contact HCFA 
directly.  By contrast, Federal law mandates that nursing homes post their statements of 
deficiencies in a manner readily accessible to residents and their families.48  In addition, 
HCFAs website allows visitors to view the results of nursing home surveys, including the 
scope and severity of any deficiencies.49 
The survey and certification agencies in some States have gone further than HCFA 
in disclosing hospital-specific information, based on their own States authority rather than 
HCFAs.  For example, New Jersey maintains a website that displays resolved hospital 
violations and New York collects and reports hospital- and physician-specific performance 
data on cardiac surgery. 
Performance of State Agencies:  The HCFA discloses little on the performance of 
State agencies. 
Unlike the mandate for the Report to Congress on the Joint Commissions 
performance, no such mandate exists for HCFA to report on how well State agencies are 
performing.  Nevertheless, HCFA has some, albeit limited, information on States 
performance.  As we have seen, it is has mid-year and year-end SAQIP reports plus 
databases reflecting the State agencies survey work.  As noted above, HCFA will release 
the data, but it would take considerable knowledge and work to array the data to facilitate 
State-by-State comparisons or in some way measure State agency performance. 
According to HCFA officials with whom we spoke, although HCFA does not distribute 
the SAQIP reports beyond the States themselves, it will release them upon request.  Of 
course, the SAQIP reports lack any State-specific assessment of performance. 
19 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  C O N C L U S I O N  
The clear and disturbing conclusion of this report is that both the Joint 
Commission and State Agencies are only minimally accountable to HCFA for their 
performance in reviewing hospitals.  While we recognize that these entities are also 
accountable to others and that they must have considerable flexibility to function 
effectively, we maintain that it is vitally important for HCFA to ensure that they 
adequately fulfill their responsibilities to protect Medicare beneficiaries.  How, then, can 
HCFA hold these entities accountable while minimizing burdensome oversight?  How can 
it recognize their inherent strengths and limitations, and tailor performance measurement 
accordingly?  We address these and other related questions in our summary report, A Call 
for Greater Accountability.  That report also contains our recommendations, which we 
direct to HCFA. 
20 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX A 
Methodology

We collected information presented in this report from the following sources: 
The HCFA 
We obtained dates of certification surveys from HCFAs Online Survey 
Certification and Reporting System (OSCAR).  The HCFA authorizes States to update 
and maintain this database with survey information.  We extracted survey data pertaining 
to the frequency of certification surveys.  We subsequently verified the accuracy of our 
extraction by comparing it to on-line OSCAR system information to ensure the dates we 
used corresponded to routine certification surveys, rather than complaint investigations or 
other types of surveys.  We are satisfied that our information is as accurate as HCFAs 
OSCAR system. 
Additionally, we selected 4 States (California, Kansas, Minnesota, and Texas) that 
contain over 50 nonaccredited hospitals and represented different regions of the United 
States.  We then examined the OSCAR data for those States in greater detail.  We verified 
the operational status of  the nonaccredited hospitals in those States that had not been 
surveyed in over 5 years using the American Hospital Associations 1997 Hospital Guide. 
We also interviewed staff and managers at each HCFA regional office and the 
central office.  We reviewed a variety of HCFA documents, including budget call letters, 
reinvention materials, and reports to Congress, among others. 
The State Survey and Certification Agencies 
In August 1997, we mailed a pretested survey to the hospital certification agencies 
in the 50 States and the District of Columbia.  The response rate for this survey was 100 
percent.  The State survey addressed four areas of hospital quality oversight:  private 
accreditation of hospitals, Medicare certification of hospitals, HCFA oversight of State 
certification agencies, and State licensure of hospitals.  We interviewed State officials on 
the telephone or in person as well. 
21 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX A 
Accrediting Organizations 
We interviewed officials from both the Joint Commission on Accreditation of 
Healthcare Organizations and the American Osteopathic Association.  We also reviewed 
documents from both organizations, including mission statements, accreditation manuals, 
policies, and hospital survey files.  We requested and received aggregate data from the 
Joint Commission reflecting its recent hospital survey activity.  All Joint Commission data 
are presented as reported by the Joint Commission, unless otherwise noted.  For purposes 
of this inquiry, we focused our analysis on the Joint Commission. 
Survey Observations 
Based on schedules of upcoming triennial surveys, we identified nine hospitals in 
which to observe triennial Joint Commission surveys.  Of those, we were able to observe 
seven.  In two cases, the hospitals declined the Joint Commissions request that we be 
allowed to observe.  The 7 hospitals varied in size from 80 to 775 beds, represented both 
teaching and community hospitals, and were located in different areas of the country (both 
rural and urban).  We also observed one random unannounced Joint Commission survey. 
Although we observed different teams of surveyors, the survey agenda, lines of inquiry, 
and tone were consistent across the surveys, which were conducted in accordance with 
Joint Commission policy, based on review of survey manuals and interviews with 
representatives of the Joint Commission.  Finally, we observed a certification survey and 
parts of two complaint investigations performed by State surveyors under HCFAs 
auspices. 
Stakeholder Interviews 
We interviewed representatives of organizations we considered to be stakeholders 
in hospital oversight.  These stakeholder organizations included a union, professional 
organizations, hospital associations, and consumer groups. 
Other Documents 
In addition to the documents referenced above, we reviewed statutory and 
regulatory language and a variety of articles from newspapers, journals, magazines, and 
newsletters. 
22 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX B 
HCFAs Reinvented Oversight of Hospital Accreditation 
Dissatisfaction with Validation 
Interest in reinventing HCFAs approach to overseeing accreditation stemmed 
from growing dissatisfaction with the validation survey process.  In 1990, 16 years after 
HCFA began to validate the Joint Commission surveys, the General Accounting Office 
(GAO) published a report criticizing HCFAs validation program.  The GAO found the 
comparison of HCFA and Joint Commission survey findings yielded little information 
about the effectiveness of Joint Commission process.  Because HCFA measures hospital 
quality using the Medicare Conditions of Participation and the Joint Commission uses its 
own standards, the GAO saw little use in comparing the results of these surveys without a 
crosswalk linking the two sets of measures.  In response to this report, HCFA developed a 
crosswalk.  The process was labor-intensive and time-consuming, resulting in a document 
over 800 pages long.  However, HCFA found using this cumbersome crosswalk produced 
the same results as its old method of comparing survey results and discontinued use of the 
crosswalk. 
Other concerns about the validation process emerged and gained credence through 
the early 1990s.  For example, HCFA came to believe that using hospitals as a proxy to 
evaluate the Joint Commission accreditation process was not only indirect but also could 
be inaccurate if hospital performance changed since an accreditation survey. 
The Reinvention Workgroup 
In 1993, HCFA organized a workgroup comprising members from HCFA, States, 
the American Hospital Association, American Osteopathic Association, and Joint 
Commission to reinvent the validation.  This workgroup identified an alternative or 
reinvented approach to the validation through which HCFA would evaluate the Joint 
Commissions accreditation process by conducting concurrent, on-site observations of the 
Joint Commission surveys instead of comparing HCFA and Joint Commission survey 
results.  This reinvented validation survey would involve HCFA and State surveyors 
participating in Joint Commission surveys as silent observers.  The workgroup also 
proposed that HCFA and the Joint Commission should mutually develop both surveyor-
based and hospital-based performance indicators to be used by HCFA and State observers. 
This on-site assessment was planned to be one part of a larger comprehensive evaluation 
(see box). 
23 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX B 
Planned Components of the Reinvention 
Review of Accreditation Organizations 
<  Philosophy and Mission 
<  Survey Protocols 
<  Surveyor Support 
<  On-Site Survey 
<  Decision process 
<  Deficiency Resolution 
<  Timeliness 
<  Performance Improvement 
<  Intra-Cycle Activity 
<  Response to Complaints 
Source: HCFA documents, June 1997. 
The Two Completed Reinvention Phases 
The reinvention workgroup has planned and completed two phases of the 
reinvention, both pertaining to the on-site assessment piece of the validation.  As Phase I, 
the workgroup drafted a methodology and developed evaluation criteria for the 
reinvention surveys.  The workgroup identified five surveyor based performance 
indicators: survey preparation, survey process, interpersonal, presentation, and teamwork. 
It also agreed upon 11 core hospital functions:  assessment, patient preparation, treatment, 
nutrition, medication use, discharge coordination, environmental safety, patient safety, 
infection control, quality improvement, and information management.  The HCFA regional 
office and State teams of surveyors attended Joint Commission surveyor training and as 
well as training on how to use the newly developed surveyor and hospital criteria.  The 
HCFA  piloted a total of 20 reinvention surveys in 4 States: Pennsylvania, Georgia, 
Illinois, and Colorado. 
24 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX C 
State Agency Quality Improvement Plan 

Standards

The HCFA developed the following standards through a workgroup that included 
State agency representatives: 
Quality Improvement Standards 
Standard 1:	 The State agency documents deficiencies consistent with the Principles of 
Documentation. 
Standard 2: 	 The Plans of Correction accepted by the State agency reflect appropriate 
action and time frames to correct cited deficiencies. 
Standard 3:  All surveys are conducted by qualified individuals. 
Standard 4:  The State agency assures consistency in survey performance. 
Standard 5:	 The State agency has an effective program to measure accuracy and 
improve consistency int he application of long-term care enforcement 
remedies. 
Standard 6:	 The State agency effectively investigates and processes complaints 
(including those related to Section 1867) about participating providers and 
suppliers. 
Standard 7:	 The State agency monitors expenditures and supports charges to Federal 
programs in accordance with existing regulations, guidelines, policies, 
approved funding levels and allocation methodologies. 
Quality Control Standards 
Standard 8:	 The State agency ensures that each home health agency (HHA), skilled 
nursing facility (SNF) nursing facility (NF), and SNF/NF is subject to a 
standard survey not later than 15 months after the previous survey, and that 
the statewide average interval between standard surveys does not exceed 
12 months. 
25 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX C 
Standard 9:	 The State agency ensures that it meets all survey coverage percentages 
negotiated with the regional office for Title XVIII and/or Title XIX 
providers and suppliers, excluding HHAs, SNFs, NFs, and SNF/NFs. 
Standard 10:	 All surveys, including complaint investigations, are conducted 
unannounced, or are announced consistent with HCFA instructions. 
Standard 11:	 The State agency budget request, activity plan, and expenditure reports are 
prepared and submitted in accordance with Federal instructions and 
accurately reflect the allocation of costs between State and Federal 
programs. 
Standard 12:  The State agency effectively maintains the OSCAR system database. 
Standard 13:	 Annual surveys of intermediate care facilities for the mentally retarded and 
recertification actions completed before the expiration of the time-limited 
agreement. 
26 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
Endnotes

1.  Troyen A. Brennan et al, Incidence of Adverse Events and Negligence in Hospitalized 
Patients, The New England Journal of Medicine 324 (February 7, 1991) 6: 370-376. 
2.  The 17.7 percent refers to adverse events considered by the authors to be serious.  The authors 
defined adverse events as situations in which an inappropriate decision was made when, at the 
time, an appropriate alternative could have been chosen and serious as ranging from temporary 
disability to death.  See Lori B. Andrews et al, An Alternative Strategy for Studying Adverse 
Events in Medical Care, The Lancet 349 (February 1, 1997)309-313. 
See also Lucian L. Leape, Error in Medicine, Journal of the American Medical 
Association 272 (December 21, 1994) 23: 1851-1857; David C. Classes et al, Adverse Drug 
Events in Hospitalized Patients, Journal of the American Medical Association 277 (January 
22/29, 1997) 4: 301-306; Mark R. Chassin et al, The Urgent Need to Improve Health Care 
Quality:  Institute of Medicine National Roundtable on Health Care Quality, Journal of the 
American Medical Association 280 (September 16, 1998) 11: 1000-1005; David W. Bates et 
al,Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of 
Serious Medication Errors, Journal of the American Medical Association 280 (October 21, 
1998) 1311-1316; Robert A. Raschke et al, A Computer Alert System to prevent Injury from 
Adverse Drug Events:  Development and Evaluation in a Community Teaching Hospital, Journal 
of the American Medical Association 280 (October 21, 1998) 1317-1320; and David C. Classen, 
Clinical Decision Support Systems to Improve Clinical Practice and Quality of Care, Journal of 
the American Medical Association 280 (October 21, 1998) 1360-1361. 
3.  One example is a November 1998 New York Times article under the headline: Death in 
Surgery Reveals Troubled Practice and Lax Hospital.  The article described a botched 
operation on a young woman by a surgeon who was on probation by the State medical board and 
who used unauthorized medical equipment brought in to the operating room by a medical supply 
salesman.  Such incidents can happen even in the best of hospitals, but they underscore the point 
that hospitals can be dangerous places and that oversight systems can be lax.  See also 
Overdoses Still Weigh heavily at Dana Farber, The Boston Globe (26 December 1995); Florida 
Doctor Sanctioned in New Amputation, The Boston Globe (19 July 1995); Two Surgeons 
Surrender Licenses After Mistakenly Removing Kidney, The Boston Globe (6 June 1996); How 
Can We Save the Next Victim? New York Times Magazine (15 June 1997); Another Hospital 
Death Probed, The Boston Globe (26 July 1997); Patient Dies After Drinking Poison Left on 
Nightstand, San Diego Union-Tribune (6 March 1998); Man Arrested for Posing as Doctor for 
4 years, posted at the CNN interactive webpage (16 May 1998); Deadly Restraint:  Patients 
Suffer in a System Without Oversight, The Hartford Courant (13 October 1998). 
27 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
4.  http://WWW.hcfa.gov/Medicare/Medicare.htm 
5.  Ibid. 
6.  Ibid. 
7.  Ibid. 
8.  P.L. 89-97. 
9.  The HCFA published its proposed hospital conditions of participation on December 19, 1997 
(62 Fed. Reg. 66,726).  On July 2, 1999, it published an interim final rule concerning patients 
rights (64 Fed.  Reg. 36,070). 
10.  Congress also provided that hospitals accredited by the American Osteopathic Association 
could be considered in compliance, but only to the extent than the Secretary deemed appropriate. 
11. Two companion reports entitled The Role of Accreditation and The Role of Medicare 
Certification explore the role of accreditation and Medicare certification in greater detail. 
12. Michael S. Hamm, The Fundamentals of Accreditation (Washington, D.C.: American Society 
of Association Executives, 1997) 3. 
13. Nonaccredited hospitals can go through the Medicare certification process for free. 
14. Social Security Act, sec. 1865 (a), 42 U.S.C. 1395bb. 
15. Social Security Act, sec. 1864(c), 42 U.S.C. 1395aa. 
16. Health Care Financing Administration, Oscar 10 Report, as of May 30, 1998. 
17. Validation surveys are onsite reviews of hospitals conducted after a Joint Commission survey. 
State agency surveyors conduct the validation surveys based ont he Medicare conditions of 
participation. 
18. We acknowledge the dual purpose of validation surveys.  As originally envisioned, they were 
to serve as an assurance that accredited hospitals met the Medicare conditions of participation: 
Our validation program is concerned chiefly with the equivalency of accreditation standards to 
Federal requirements, and assuring that Medicare participating facilities deemed by virtue of 
accreditation would meet those requirements if surveyed against them.  (58 CFR 61831, see also 
Senate Report 92-1230).  Over time, the purpose and use of validation surveys evolved to a more 
28 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
encompassing role as a tool of oversight for the Joint Commission.  Indeed, these roles overlap 
and their distinctions may be blurred.  For purposes of this report, we focus on validation surveys 
and their potential as a tool of oversight. 
19. In 1990, the General Accounting Office criticized HCFAs approach to validation surveys, 
citing specifically the lack of a crosswalk linking the Medicare conditions with the Joint 
Commissions own standards.  In response, HCFA created such a crosswalk, but the undertaking 
was not only labor-intensive, but also cumbersome:  it resulted in an 800-page document.  And 
when HCFA relied upon that crosswalk for validation surveys in 1989 and 1990, the results were 
the same as without a crosswalk (See Health Care Financing Administration, The Reinvention of 
the Hospital Validation Survey, undated, and U.S. General Accounting Office, Health Care: 
Criteria Used to Evaluate Hospital Accreditation Process Needs Reevaluation, GAO/HRD-90-
89, 11 June 1990). 
20.  See our companion report entitled The Role of Accreditation for a discussion of the Joint 
Commissions standards. 
21.  The HCFAs instructions to State agencies also allow for validation surveys to occur within 
60 days of HCFAs notice to the State agency. 
22.  While we acknowledge our estimate is a rough one, we base it the conservative assumption 
that 2-3 surveyors would travel to a hospital for 2-3 days.  While highly variable, we estimated the 
costs of surveyor salary and travel would be $2,000 to $4,000 each.  And considering about 150 
surveys per year, we believe the estimate of about $2 million is reasonable. 
23. Health Care Financing Administration, The Reinvention of the Hospital Validation Survey, 
undated. 
24. In his book, The Fundamentals of Accreditation, Michael Hamm identifies some generic 
criteria for evaluating accreditation bodies, and among the criteria is the organizations 
performance record.  (Michael S. Hamm, The Fundamentals of Accreditation, Washington, DC: 
American Society of Association Executives, 1997, 10-11). 
25. The Joint Commission currently has HCFA officials serving on its Hospital Professional and 
Technical Advisory Committee and its Council on Performance Measurement. 
26. Social Security Act, sec. 1875(b), 42 U.S.C. 1395ll. 
27. Health Care Financing Administration, Medicare Annual Report to Congress: Fiscal Year 
1992. 
29 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
28.  62 Fed. Reg. 66726, December 19, 1997. 
29. The Joint Commissions board has 28 members, including 6 public members and an at-large 
nursing representative.  Most members are appointed by organizations such as the American 
College of Physicians, the American College of Surgeons, the American Dental Association, the 
American Hospital Association, and the American Medical Association. 
30. Joint Commission on Accreditation of Healthcare Organizations, Professional and Technical 
Advisory Committee (PTAC) Rules and Procedures, January 16, 1993. 
31. In November 1996, the Joint Commission waived the $30 report fee. 
32. The 45 performance areas, divided into 15 categories, are: 
Patient Rights and Organization Ethics:  Patient Rights, Organization Ethics

Assessment of Patients: Initial Assessment Procedures, Pathology and Clinical Laboratory

Services, Reassessment Procedures, Processes for patient Care Decision, Relevant Policies, Needs

Assessment for Specific Patient Populations

Care of Patients: Planning and Providing Care, Anesthesia Care, Medication Use, Nutrition

Care, Operative Procedures, Rehabilitation Care, Special Treatment Procedures

Education: Patient and Family Education

Continuum of Care: Continuity of Care

Improving Organizational Performance: Improvement Planning, Design of New Services,

Measurement of Processes and Outcomes, Assessment of Data, Improvement of Performance

Leadership: Strategic Planning, Departmental Leadership.  Integrating and Coordinating

Services, Leaders Role in Improving Performance

Management of the Environment of Care: Design of the Environment, Implementation of the

Safety Plans, Monitoring of the Safety Plans, Social Environment

Management of Human Resources: Human resources Planning; Orienting, Training, and

Education of Staff; Assessing Staff Competence; Managing Staff Requests

Management of Information: Information Management Planning, Availability of Patient-

Specific Information, Data Collection and Analysis, Literature to Support Decision-Making, Use

of Comparative Information

Infection Control: Infection Control

Governance: Governance

Medical Staff: Organization, Bylaws, Rules, and Regulations; Credentialing

Nursing: Nursing

Management: Management

30 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
33. The HCFA staff we contacted, particularly in the regional offices, were largely unaware of 
these reports and their availability. 
34.  For further discussion of this policy, see our companion report, The Role of Accreditation. 
35.  For examples, see Dennis S. OLeary, On Becoming Ten, Joint Commission Perspectives 
16 (May/June 1996)3:  2-3; John Morrissey, JCAHO Outlines Action to Improve Performance, 
Modern Healthcare, January 30, 1995. 
36. As we noted in our companion report, The Role of Medicare Certification, nursing homes, 
intermediate care facilities for the mentally retarded, and home health agencies all precede 
nonaccredited hospitals in survey priority. 
37. HCFA, The State Agency Quality Improvement Program, May 1996. 
38.  The Emergency Medical Treatment and Labor Act (42 U.S.C. 1395dd) prohibits hospitals 
from refusing emergency patients or women in labor medical screening and/or treatment.  If a 
hospital willfully or negligently fails to meet the requirements of the act, it can be fined or its 
participation in Medicare can be suspended or terminated. 
39.  The HCFA also has its On-Line Survey and Certification data system (referred to as OSCAR) 
for insights into the extent of hospital surveys conducted by the States. 
40.  Health Care Financing Administration, State Agency Quality Improvement Program End-of-
Year Implementation Status - Fiscal Year 1997, May 15, 1997. 
41. Ibid. 
42.  The HCFA tracks survey results through its OSCAR database, but it does not track 
termination notices, which can be 23-day or 90-day notices based on the severity of the survey 
findings. 
43.  Referred to as the 1864 Agreement, this document spells out HCFAs delegation of authority 
to the States and the functions States must carry out. 
44.  See the companion report, The Role of Medicare Certification, for more detail on the budget 
priorities. 
45.  Based on the Office of Inspector General mail survey in September 1997.  The survey defined 
hospital activities to include Medicare certification surveys for nonaccredited hospitals, responses 
to complaints, and validation surveys of accredited hospitals. 
31 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053  APPENDIX D 
46.  See the companion report, The Role of Medicare Certification, for more detail on State 
agencies workloads. 
47.  Referred to as the On-line Survey and Certification Reporting System.  The HCFA may 
charge a small fee for processing this data. 
48.  Social Security Act, sec. 1864 (42 U.S.C. 1395aa). 
49.  The HCFA calls this Nursing Home Compare and it is available at 
http://www.medicare.gov. 
32 
Hospital Quality: Holding the Reviewers Accountable  OEI-01-97-00053