Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

JUNE GIBBS BROWN 

Inspector General 

JULY 1999 

OEI-01-97-00053 

The External Review of Hospital Quality 

Holding the Reviewers Accountable 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through a 

nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Boston regional office prepared this report under the direction of Mark R. Yessian, Ph.D., 

Regional Inspector General. Principal OEI staff included: 

BOSTON HEADQUARTERS 

Joyce M. Greenleaf, Project Leader Jennifer Antico, Program Specialist 

Lynne G. Dugan, Senior Analyst Alan Levine, Program Specialist 

Meredith A. Vey, Program Analyst 

To obtain copies of this report, please call the Boston Regional Office at 617/565-1050. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SUMMARY 

PURPOSE 

To assess how the Health Care Financing Administration holds the Joint Commission and 

State agencies accountable for the external review of hospital quality. 

BACKGROUND 

External Quality Review of Hospitals in the Medicare Program 

Hospitals routinely offer valuable services, but also are places where poor care can 

lead to unnecessary harm. The external quality review of hospitals plays an important role 

not only in protecting patients from such harm, but also in complementing the hospitals’ 

own internal quality efforts. The Federal Government relies on two types of external 

review to ensure hospitals meet the minimum requirements for participating in Medicare: 

accreditation, usually by the Joint Commission on Accreditation of Healthcare 

Organizations, and Medicare certification, by State agencies. About 80 percent of the 

6,200 hospitals that participate in Medicare are accredited by the Joint Commission. 

This Inquiry 

This report, part of a series of four companion reports that resulted from our 

inquiry, focuses on the Health Care Financing Administration’s (HCFA’s) oversight of 

both the Joint Commission and State agencies. Our inquiry draws on aggregate data, file 

reviews, surveys, and observations from a rich variety of sources, including HCFA, the 

Joint Commission, State agencies, and other stakeholders. 

We organized this report around a three-part framework that HCFA can use to 

hold accrediting bodies and State agencies accountable: (1) obtaining information on 

performance, (2) providing feedback on performance, and (3) disclosing information 

publicly. 

FINDINGS 

The HCFA obtains limited information on the performance of the Joint Commission 

or the States. In both cases, HCFA asks for little in the way of routine performance 

reports. To assess the Joint Commission’s performance, HCFA relies mainly on validation 

Hospital Quality: Holding the Reviewers Accountable 1 OEI-01-97-00053 
surveys conducted, at HCFA’s expense, by the State agencies. But for a number of 

reasons the value of these surveys has been limited. The methodology for selecting the 

hospitals to survey fails to consider hospital size or type and draws on hospitals surveyed 

only in certain months. More fundamentally, the surveys have been based on different 

standards ( the Medicare conditions of participation as opposed to the Joint Commission 

standards) and have been conducted subsequent to the Joint Commission’s survey (when 

hospital conditions could have changed). During 1996 and 97, HCFA piloted 20 

observation surveys--during which State and HCFA officials accompanied Joint 

Commission surveyors. This approach appears to have much promise, but HCFA has not 

yet issued any evaluation of the pilots. 

The HCFA rarely observes State agencies survey hospitals, and conducts no validation 

surveys of them. 

The HCFA provides limited feedback to the Joint Commission and the State 

agencies on their overall performance. Its feedback to the Joint Commission is more 

deferential than directive. It’s major vehicle for feedback to the Joint Commission is its 

annual Report to Congress, which is based on the flawed validation surveys and has 

typically been submitted years late. The HCFA is more directive to the State agencies, 

which carry out their survey work in accord with HCFA protocols, but gives them little 

feedback on how well they perform their hospital oversight work. 

Public disclosure plays only a minimal role in holding Joint Commission and State 

agencies accountable. The HCFA makes little information available to the public on the 

performance of either hospitals or of the external reviewers. By contrast, HCFA posts 

nursing home survey findings on the Internet and requires nursing homes to post them 

within the facility as well. The Joint Commission has been more proactive than HCFA in 

making hospital survey results widely available on the Internet and through other means. 

CONCLUSION 

The clear and disturbing conclusion of this report is that both the Joint 

Commission and State agencies are only minimally accountable to HCFA for their 

performance in reviewing hospitals. While we recognize that these entities are also 

accountable to others and that they must have considerable flexibility to function 

effectively, we maintain that it is vitally important for HCFA to ensure that they 

adequately fulfill their responsibilities to protect Medicare beneficiaries. How, then, can 

HCFA hold these entities accountable while minimizing burdensome oversight? How can 

it recognize their inherent strengths and limitations, and tailor performance measurement 

accordingly? We address these and other related questions in our summary report, A Call 

for Greater Accountability. That report also contains our recommendations, which we 

direct to HCFA. 

Hospital Quality: Holding the Reviewers Accountable 2 OEI-01-97-00053 
COMMENTS 

Within the Department of Health and Human Services, we received comments 

from HCFA. We also solicited and received comments from the following external 

parties: Joint Commission on Accreditation of Healthcare Organizations, Association of 

Health Facility Survey Agencies, American Osteopathic Association, American 

Association of Retired Persons, Service Employees International Union, National Health 

Law Program, and Public Citizen’s Health Research Group. We include the detailed text 

of all of these comments and our responses to them in our summary report, The External 

Review of Hospital Quality: A Call for Greater Accountability (OEI-01-97-00050). 

Hospital Quality: Holding the Reviewers Accountable 3 OEI-01-97-00053 
TABLE OF CONTENTS 

PAGE 

EXECUTIVE SUMMARY 

INTRODUCTION .......................................................... 5

FRAMEWORK FOR ACCOUNTABILITY ...................................... 8

FINDINGS: 

Obtaining Information on the Joint Commission’s Performance ................... 9

Providing Feedback to the Joint Commission ................................ 12

Disclosing Information on Joint Commission Performance ...................... 14

Obtaining Information on State Agencies’ Performance ........................ 15

Providing Feedback to State Agencies ..................................... 17

Disclosing Information on State Agencies’ Performance ........................ 18

CONCLUSION ........................................................... 20

APPENDICES 

A: Methodology ...................................................... 21

B: HCFA’s Reinvention Project .......................................... 23

C: SAQIP .......................................................... 25

D: Endnotes ......................................................... 27

Hospital Quality: Holding the Reviewers Accountable 4 OEI-01-97-00053 
INTRODUCTION 

PURPOSE 

To assess how the Health Care Financing Administration holds the Joint

Commission and State agencies accountable for the external review of hospital quality.

BACKGROUND 

Hospital Safety 

Hospitals are an integral part of our healthcare system, offering services that 

improve, extend, and even save lives. But they are also places where inappropriate care 

can lead to unnecessary harm. This reality was clearly underscored in 1991, when a 

Harvard medical practice revealed the results of its review of about 30,000 randomly 

selected records of patients hospitalized in New York during 1984. The study found that 

1 percent of the hospitalizations involved adverse events caused by negligence.1 On the 

basis of its sample, the study team estimated that during that year, negligent care provided 

in New York State hospitals was responsible for 27,179 injuries, including 6,895 deaths 

and 877 instances of “permanent and total disability.” Many other more recent studies 

have reinforced the concerns raised by the Harvard study. Of particular note was one that 

focused on the care received by 1,047 hospitalized patients in a large teaching hospital 

affiliated with a medical school. It found that 17.7 percent of these patients received 

inappropriate care resulting in a serious adverse event--ranging from temporary disability 

to death.2 In the public eye, these scholarly inquiries have been overshadowed by media 

reports that describe, often in graphic detail, the harm done to patients because of poor 

hospital care.3 

Hospitals rely upon a variety of internal mechanisms, from physician credentialing, 

to peer review and benchmarking, in order to try to avoid such incidents and to improve 

the quality of care provided in their facilities. External quality review serves as a vital 

additional safeguard. It provides a more detached, independent mechanism for assessing 

the adequacy of hospital practices. Such oversight is of fundamental importance to 

patients and to the public and private entities that purchase health care services on their 

behalf. Protecting patient safety and improving the quality of patient care must be a top 

priority of external review. 

Medicare’s Interest in External Hospital Quality Review 

The Medicare program covers about 38 million elderly and disabled individuals, 

many of whom are high users of hospital care.4 In 1997, Medicare spent about $136 

Hospital Quality: Holding the Reviewers Accountable 5 OEI-01-97-00053 
billion on Part A, the hospital insurance benefit.5 This figure is just over half the total 

amount the Federal government spent on all Medicare benefits.6 In the same year, 

Medicare spent over $80 billion for inpatient acute hospital care alone.7 

Since Medicare’s inception, external quality review has been a part of the 

Medicare program. When Congress enacted the Medicare Act in 1965, it required 

hospitals to meet certain minimum health and safety requirements to participate in the 

program.8 Those minimum requirements, called the Medicare conditions of participation, 

were published in 1966, revised in 1986, and are now being revised again (see appendix 

A).9 

The Joint Commission and State Agencies as External Reviewers 

The Federal Government relies on two processes of external review to determine 

hospital compliance with these conditions of participation: accreditation, usually by the 

Joint Commission on Accreditation of Healthcare Organizations (Joint Commission), and 

Medicare certification, by State survey and certification agencies.10 Both processes 

involve a team of trained surveyors visiting a hospital, interviewing staff, reviewing 

documents, and inspecting the facility.11 However, the nature of these processes is very 

different. 

Accreditation is a voluntary “conformity-assessment process” whereby industry 

experts define what standards organizations must conform to in order to be accredited and 

then surveyors systematically assess the organization’s performance against those 

standards.12 It is a form of self-regulation for which hospitals pay a fee.13 The Joint 

Commission enjoys a special status because, by Federal statute, hospitals accredited by the 

Joint Commission are deemed to meet the Medicare conditions of participation.14 As the 

largest accreditor of hospitals, accrediting about 80 percent of the nation’s 6,200 

hospitals, the Joint Commission is responsible for the majority of the nation’s external 

quality review of hospitals. In 1972, Congress enacted amendments that gave the Health 

Care Financing Administration (HCFA) responsibility for overseeing the Joint 

Commission.15 

The Medicare certification process, on the other hand, is a public regulatory 

process that aims to ensure hospitals wishing to serve Medicare beneficiaries, but not 

wishing to be accredited, meet the conditions of participation. The HCFA relies on the 51 

State survey and certification agencies (hereafter called State agencies) to determine 

compliance with the Medicare conditions of participation. Hospitals pay no fee for this 

process. States agencies currently certify 1,442 nonaccredited hospitals nationwide.16 

These State agencies are paid and trained by HCFA and use HCFA’s survey instrument to 

survey nonaccredited hospitals. 

Hospital Quality: Holding the Reviewers Accountable 6 OEI-01-97-00053 
This Inquiry and Report 

This report describes how HCFA holds the Joint Commission and the State 

agencies accountable for the external review of hospital quality. Toward that end, on the 

following page we offer a framework for considering how external reviewers can be held 

accountable. We offer our findings in accord with this framework. 

This report is one of four companion reports that resulted from our inquiry. 

Another report, The Role of Accreditation (OEI-01-97-00051), examines the 

contributions of accreditation in external hospital review. A third, The Role of Medicare 

Certification (OEI-01-97-00052), describes the extent and nature of HCFA and State 

agencies’ external review of nonaccredited hospitals. The fourth report, A Call for 

Greater Accountability (OEI-01-97-00050), provides a summary of external review of 

hospital quality and presents the recommendations emerging from our inquiry. 

Our inquiry draws on a variety of sources. These include: data from HCFA’s 

Online Survey Certification and Reporting System; aggregate data from the Joint 

Commission concerning hospital survey activity; a mail survey to State agencies in the 50 

States and District of Columbia (hereafter referred to as a State); observations of 7 

hospital surveys conducted by the Joint Commission and one conducted by a State agency; 

observation of two complaint investigations by State agencies; reviews of accreditation 

manuals, policies, and hospital survey files from the Joint Commission; the systematic 

gathering of information from representatives of HCFA central and regional offices, State 

agencies, the Joint Commission, American Hospital Association, consumer groups, 

professional associations, and representatives of other organizations we considered to be 

stakeholders in hospital oversight; and reviews of laws and regulations and articles from 

newspapers, journals, newsletters, and magazines. We also interviewed officials from the 

American Osteopathic Association and reviewed their accreditation materials. The 

American Osteopathic Association accredits about 100 to 150 hospitals, some of which 

are also accredited by the Joint Commission. For purposes of this inquiry, however, we 

focused on the Joint Commission. See appendix B for more details on our methodology. 

We conducted this inspection in accordance with the Quality Standards for 

Inspections issued by the President’s Council on Integrity and Efficiency. 

Hospital Quality: Holding the Reviewers Accountable 7 OEI-01-97-00053 
ACCOUNTABILITY FRAMEWORK 

The following is a generic framework for considering the mechanisms that HCFA can use 

to hold accrediting bodies and State agencies accountable. We base the framework on existing 

mechanisms. The framework serves as a starting place for discussions, to which mechanisms can 

be added or deleted. We present it in the context of hospitals, but it can also be considered in the 

context of other provider types. We apply it to the Joint Commission and State agencies on the 

following pages. 

Obtaining Information on Performance 

1. Validation Surveys. These are onsite reviews of hospitals that are conducted some time after 

the accreditors’ or State agencies’ own visits. Validation surveys aim to ensure that reviewed 

hospitals meet certain minimums. 

2. Observation Surveys. These are onsite reviews conducted at the same time as the accreditors’ 

or State agencies’ surveys. Their intent is to directly observe the surveyors and to use those 

observations as the basis for constructive feedback. 

3. Performance Reports. Such reports involve the regular sharing of data and analysis to enable 

HCFA to remain well-informed of the hospital review activities. 

4. Policy and Procedural Updates. These are regular notices to HCFA on the key policies and 

procedures governing the bodies’ reviews of hospitals. 

Providing Feedback on Performance 

1. Performance Assessments. These represent HCFA’s feedback to the accreditors and state 

agencies on how well they are performing. They can be regularly scheduled or on an as-needed 

basis. 

2. Policy and Procedural Guidance. Through such guidance, HCFA can convey to the bodies 

those approaches and priorities that it regards to be of national significance. 

Disclosing Information Publicly 

1. Performance of Hospitals. Publicly disclosing information on hospitals conveys not only 

something about the hospitals’ performance but also assurance that a process exists for overseeing 

hospitals that wish to serve Medicare beneficiaries. Such disclosure fosters a trust that the public’s 

safety is paramount. 

2. Performance of Accrediting Bodies/State Agencies. By publicly disclosing information on 

these entities, HCFA conveys that it is monitoring the work they do on its behalf. It fosters trust 

that HCFA is promoting public safety. 

Hospital Quality: Holding the Reviewers Accountable 8 OEI-01-97-00053 
Joint Commission Accountability 

The HCFA obtains little information on Joint Commission 

performance. 

Validation Surveys: These surveys are fundamentally limited as tool of 

accountability because they are based on different standards that are applied at 

different points in time. They are also costly to implement and based on a 

sample that not only is flawed, but also has been shrinking over time. 

The validation survey process is the centerpiece of HCFA’s approach to holding 

the Joint Commission accountable.17 Yet its value is limited. It seeks to measure the Joint 

Commission’s performance through hospitals’ compliance with the Medicare conditions of 

participation, not the Joint Commission’s own standards.18 This difference in reference 

points not only complicates the interpretation of validation surveys, but also inherently 

compromises their value. Any reasonable tool of oversight needs a common frame of 

reference.19 The validation survey now must bridge the Joint Commission’s several 

hundred standards, which are categorized into 45 groups, with the few dozen Medicare 

conditions. Moreover, Joint Commission standards--considered by many even within 

HCFA to be state-of-the-art--are constantly evolving, whereas the Medicare conditions 

rarely change: first published in 1966, revised substantially in 1986, and under revision 

again now.20 

Further undermining the validation process as a tool for accountability is its 

retrospective nature: validation surveys occur within 60 days of the Joint Commission’s 

routine hospital survey.21 During that elapsed time, hospitals have likely corrected some 

problems and experienced new ones, undergone staff changes, expanded or eliminated 

services--in other words, any number of changes could affect the validation survey results. 

Validation surveys are labor-intensive. A team of at least two surveyors conducts 

a full hospital survey, lasting 2 to 5 days, depending on the size of the hospital. Alone, the 

survey portion of the validation probably costs upwards of $2 million per year.22 Beyond 

that, though, are costs associated with the preparation and analysis. For example, HCFA 

staff choose the sample for validation, collect and compare the accreditation and validation 

survey results, and analyze them. Considering the basic limitations identified above, 

devoting these resources to validation surveys is questionable. 

Other concerns about HCFA’s approach to validation stem from how it selects 

hospitals for validation surveys. For example, until recently, HCFA selected hospitals for 

Hospital Quality: Holding the Reviewers Accountable 9 OEI-01-97-00053 
validation surveys only from those having undergone Joint Commission accreditation 

surveys between January and July of each year. The HCFA reports that in January 1998 it 

began drawing a random sample of accredited hospitals each month to correct this 

problem. While an improvement, HCFA’s approach to sampling still fails to consider 

hospital characteristics such as past performance, bedsize or rural or urban location. 

Other problems with the validation sample appear to be improving. For example, 

although the size of the sample for validation fell over the past few years, it appears to be 

on the rise. Between 1991 and 1997, the size of the validation sample fell 50 percent, 

from 227 surveys in 1991 to 113 surveys in 1997 (about 2 percent of all accredited 

hospitals). In 1998, HCFA reports that it conducted 164 validation surveys, an increase of 

31 percent over the previous year. Still, some hospitals selected for validation never get 

validated, although that also appears to be improving. In 1996, for example, HCFA 

validated no hospitals in 16 States, including Missouri, Iowa, Nebraska, and Kansas--an 

area comprising an entire Federal region of the country. Likewise, in 1997, HCFA 

validated none of the 30 selected hospitals in New Mexico, Texas, Oklahoma, Arkansas, 

and Louisiana--another Federal region. In 1998, however, HCFA reports that each 

Federal region had at least one validation survey conducted in it and 76 percent of the 

hospitals selected for validation did, in fact, have a validation survey. 

Observation Surveys: The HCFA has piloted an observation survey that has 

promise but remains largely undeveloped. 

The HCFA developed an observation survey process as part of a larger initiative to 

reinvent its oversight of accreditation (HCFA generally refers to these surveys as 

reinvention surveys, see appendix B for more information on the reinvention initiative). 

The process developed called for State and HCFA surveyors to silently follow and observe 

while Joint Commission surveyors conduct accreditation surveys--a major shift from 

traditional validation surveys. In developing the observation surveys, HCFA aimed to 

address the fundamental problems with the traditional validation (such as different 

reference points and time frames). A workgroup involving officials from HCFA, State 

agencies, and the Joint Commission devoted considerable effort to defining an observation 

process as a tool for accountability. The observation survey represented an opportunity 

for State and HCFA surveyors to assess first-hand the process of an accreditation survey. 

It promised to provide what the traditional validation could not: the same snapshot of a 

hospital and insights into accreditation surveyor skills and processes (defined as survey 

preparation, survey process, interpersonal, presentation, and teamwork).23 

In 1996 and 1997 HCFA and State agency surveyors tested their newly developed 

observation survey process by observing 20 accreditation surveys across 4 States. But 

HCFA has yet to report on the results. Indeed, its plans for revising or expanding the 

observation surveys are undeveloped. Using skilled Medicare surveyors to be what 

essentially amounts to silent observers of an accreditation survey process is a questionable 

Hospital Quality: Holding the Reviewers Accountable 10 OEI-01-97-00053 
use of scarce resources. Furthermore, we observed neither much interaction with, nor any 

feedback to, accreditation surveyors on the one survey observation in which we 

participated. In fact, because HCFA has yet to produce an evaluation on the pilot, it has 

yet to result in feedback to the Joint Commission. 

Performance Reports: The HCFA obtains few reports addressing Joint 

Commission performance. 

The HCFA asks for little regular reporting from the Joint Commission beyond the 

survey scheduling and results it needs to carry out its validation surveys. For example, it 

asks for no regular reports from the Joint Commission on topics such as the nature or 

pattern of complaints received about accredited hospitals, appeals of accreditation 

decisions, or the extent and nature of actions taken as a follow-up to surveys. It also asks 

for no aggregate data on the timeliness of surveys or training of surveyors. Nor does it 

ask for any regular reports on how many special surveys (for example in response to 

unexpected deaths or preventable injuries) the Joint Commission conducts. These types of 

data are already being collected by the Joint Commission and could provide HCFA with 

insights into how well accredited hospitals are performing, what new problem areas are 

emerging, and how well the accreditation results are being received.24 

Regular reporting from the Joint Commission to HCFA has been largely limited to 

information HCFA needs to carry out its validation surveys. For example, the Joint 

Commission provides HCFA’s central office with lists of upcoming surveys, accredited 

hospitals and their accreditation status, and individual hospital performance reports. The 

HCFA has also received data on Joint Commission standards commonly found out of 

compliance and a report on hospital withdrawals from accreditation, mergers, and 

acquisitions. 

Policy Updates: The HCFA receives regular updates on Joint Commission 

policies and procedures. 

The Joint Commission communicates with HCFA regularly and through several 

avenues. For example, officials from HCFA meet several times a year with 

representatives of the Joint Commission. Their meetings cover an array of topics, 

including the reinvented survey process, electronic exchange of data, and changes in 

the accreditation survey process, among others. In the course of developing and 

implementing the observation survey pilot, HCFA and Joint Commission officials met 

regularly and HCFA staff even attended Joint Commission surveyor training. Staff 

from the Joint Commission communicate regularly to clarify policy and procedural 

issues, either in person, in writing, on the telephone, or, more recently, via electronic 

mail. And at least two HCFA officials serve as members of Joint Commission 

committees, which provides additional opportunities for HCFA to remain wellHospital Quality: Holding the Reviewers Accountable 11 OEI-01-97-00053 
informed of existing and emerging policies.25 The communication involves largely the 

HCFA central rather than regional office staff. 

The HCFA provides minimal feedback to the Joint 

Commission. 

Performance Assessments: The main vehicle for feedback on the Joint 

Commission’s performance is the Report to Congress. That report is of limited 

value. 

The Congressional mandate that requires HCFA to publish a summary of the 

validation reports as part of its Medicare Annual Report to Congress shapes both the 

method and content of HCFA’s feedback to the Joint Commission.26 The HCFA uses this 

report to provide feedback about the validation surveys to the Joint Commission. It uses 

no other formal mechanism for feedback. Thus, outside of validation survey results, the 

Joint Commission receives little in the way of detailed, operational feedback or 

recommendations for improvement. 

We have already raised questions about the basic value of the validation surveys. 

Beyond that, by the time HCFA releases its Annual Reports, the validation data are dated, 

further limiting their value. For example, in 1993 HCFA released the reports to Congress 

for the Fiscal Years 1989 and 1990. It published the results of the 1991 and 1992 

validation surveys in the 1992 Report, released in September 1994. The 1994 validation 

survey results never left the Department, and HCFA released the most recent reports, 

containing the 1995 and 1996 validation results, in late 1997. Given these delays, the 

Joint Commission’s standards and procedures could have changed between the time the 

validation surveys were conducted and the time the Joint Commission receives its copy of 

the Medicare Annual Report. 

The delays of HCFA’s validation feedback impede meaningful discussion with the 

Joint Commission about the findings. While HCFA does not provide the Joint 

Commission with an opportunity to formally respond to the validation findings, some such 

discussion has occurred. For example, in response to the results of the 1991 and 1992 

validation findings, which raised concerns about the Joint Commission’s attention to life 

safety codes, HCFA observed the Joint Commission’s surveyor training and consulted 

with the it about interpreting life safety code standards.27 

Hospital Quality: Holding the Reviewers Accountable 12 OEI-01-97-00053 
Policy and Procedural Guidance: The HCFA’s guidance to the Joint Commission 

on policy and procedural matters is negligible. 

The HCFA’s posture toward the Joint Commission is more deferential than 

directive. The Joint Commission, which has dominated the hospital accreditation market 

for the past 30 years, has amassed expertise that HCFA relies upon. Indeed, many 

officials in HCFA refer to the Joint Commission as maintaining the “gold standard” or 

higher bar toward which HCFA can aspire. The HCFA even modeled the recently 

proposed hospital Conditions of Participation after Joint Commission’s Agenda for 

Change.28 

On occasion, representatives of both HCFA and the Joint Commission will be onsite at the same hospital at the same time. This is especially apt to occur following an 

incident of patient harm in an accredited hospital, to which both parties are responding. 

According to HCFA regional officials with whom we spoke, the HCFA takes on no 

coordinating nor information-sharing roles during those responses. In fact, during one 

high profile case of a preventable death in a hospital, the HCFA regional office staff 

reported that they played a significant role in coordinating the responses of HCFA and 

State officials, but not the Joint Commission. A HCFA official from another region 

captured this hands-off approach when he said “[We] wouldn’t want to interfere with their 

process.” 

While the HCFA lacks a seat on the Joint Commission’s Board of Commissioners, 

it does have opportunities to provide policy and procedural guidance through its 

participation in two Joint Commission committees.29 In the Professional and Technical 

Advisory Committee for hospital standards, HCFA staff can weigh in as new standards are 

discussed and developed. Likewise, HCFA can weigh in on discussions of performance 

measurement by participating in the Council on Performance Measurement. However, the 

expectations of Committee membership, as defined by the Joint Commission, call for 

participants to not only provide advice, but also “to serve as [an] external advocate[s]” for 

the accreditation program.30 Such expectations could run counter to HCFA’s role in 

holding the Joint Commission accountable. 

Other factors also play some role in limiting HCFA’s ability to provide meaningful 

policy or procedural guidance to the Joint Commission. For example, within HCFA itself, 

no center of responsibility exists for hospital-related issues. Rather, responsibility for 

issues related to hospitals is divided among several divisions and groups in the Center for 

Medicaid and State Operations, the Center for Health Plans and Providers, and the Office 

of Clinical Standards and Quality. The HCFA’s organizational changes in the last few 

years have left many HCFA staff and managers with steep learning curves with respect to 

hospital quality review. Indeed, many HCFA officials with whom we spoke characterized 

their interactions with the Joint Commission as more educational than evaluative. 

Hospital Quality: Holding the Reviewers Accountable 13 OEI-01-97-00053 
The Joint Commission and especially HCFA make limited 

use of public disclosure as a tool of accountability. 

Performance of Hospitals: The Joint Commission makes basic information on 

accredited hospitals’ survey results public, but keeps most information about 

adverse events confidential. 

In December 1994, the Joint Commission began releasing accreditation 

information about hospitals, on request, in the form of performance reports.31 These 

reports include the hospital’s overall survey score from its most recent triennial survey, the 

hospital’s scores in the 45 areas surveyed, comparative data on the percentage of hospitals 

receiving each possible score in the 45 areas, and a list of the areas in having specific 

recommendations for improvement, including the status of the improvement.32 The 

reports exclude any details on the problems identified during the surveys. But the reports 

do come with a guide for interpreting and understanding the scores and an explanation of 

the 45 performance areas. 

As of late 1998, the Joint Commission had 13,408 performance reports for 

hospitals, nursing homes, and other provider types available to the public, and that list 

keeps growing. Between late 1996 and early 1997, requests for performance reports grew 

from 13 per quarter to over 2,000. These reports are free and readily available by mail, 

phone, or through the Internet.33 

However, the Joint Commission keeps most information on adverse events in 

hospitals confidential. In April 1998, it implemented a policy fostering hospitals to selfreport such events and conduct their own analysis of the cause of the events.34 By keeping 

the information confidential, the Joint Commission aims to provide hospitals with a safe 

venue for reporting adverse events, thereby allowing it to develop a database for 

researching and preventing such events. 

Performance of Accrediting Bodies: The HCFA has little to disclose on the 

performance of the Joint Commission, but does make its Report to Congress 

available on request. 

The HCFA does not mandate that information on the performance of the Joint 

Commission be disclosed. In fact, it has little to disclose. It does have the Report to 

Congress, its main vehicle for providing feedback to the Joint Commission. That report is 

available to anyone who requests it, but, as we have seen, it is of limited value. 

Independent of HCFA, the Joint Commission makes some information about its own 

performance public through its newsletter, website, and occasional articles in magazines.35 

Hospital Quality: Holding the Reviewers Accountable 14 OEI-01-97-00053 
State Agency Accountability 

The HCFA obtains little information on State agencies’ 

performance. 

Validation Surveys: The HCFA does not conduct such surveys of State agencies’ 

hospital activities. 

None of the 10 HCFA regional offices reported conducting validation surveys of 

nonaccredited hospitals over the past few years. Based on our accountability framework 

presented on page 7, by validation survey we mean an onsite survey, conducted sometime 

after a State agency survey, that aims to verify that the hospital meets certain minimums. 

The HCFA’s workload, like that of State agencies, is dominated by nursing homes and 

home health agencies rather than hospitals. And while HCFA may validate those surveys, 

it has not validated hospital certification surveys in recent years. 

Observation Surveys: The HCFA rarely conducts such surveys of State agencies’ 

hospital activities. 

As we have seen, the bulk of State agencies’ work is directed toward nursing 

homes and home health agencies rather than hospitals, and HCFA’s oversight of the 

agencies reflects this. State agencies conduct relatively few surveys of nonaccredited 

hospitals each year, thereby limiting the opportunities for HCFA to observe their survey 

process.36 Indeed, 4 of the 10 HCFA regional offices reported conducting no onsite 

monitoring at all over the past year or so. The other 6 regional offices reported observing 

15-20 State agency surveys of hospitals over the past 3 years. This represents about 2 

percent of all nonaccredited hospitals, or less than 1 percent per year. The six regions that 

did observe State agency survey hospitals reporting using different approaches in their 

observation. For example, some regional offices may provide feedback to the State 

surveyors while observing, others may wait until after the survey is complete. 

Performance Reports: Nationally, HCFA obtains little information on State 

agencies’ performance concerning hospitals. 

In 1996, HCFA established the State Agency Quality Improvement Plan (SAQIP) 

to promote ongoing improvement in all aspects of survey and certification, including 

hospitals.37 Toward that end, HCFA developed a set of core performance standards for 

agencies (see appendix C for standards). For the most part, the standards developed by 

HCFA apply generally to all provider types rather than being specific to hospitals. For 

example, “All surveys are conducted by qualified individuals,” applies across provider 

Hospital Quality: Holding the Reviewers Accountable 15 OEI-01-97-00053 
types, including nursing homes and home health agencies. In fact, when we asked States 

about their improvement plans, only six identified goals specific to hospitals, such as 

improving the review of complaints related to hospital dumping.38 

Under SAQIP, HCFA requires States to design a plan for quality improvement 

specific to the performance standard. The HCFA regional offices assist the State agencies 

and receive copies of the quality improvement plans, thus they have a general picture of 

the State agencies’ progress. Centrally, HCFA relies on its mid- and year-end SAQIP 

reports for a picture of the States’ performance.39 

This approach, however, provides HCFA with a superficial and somewhat blurred 

picture of States’ performance in overseeing hospitals. For example, the 1997 year-end 

SAQIP report indicated State agencies had taken on general improvement projects ranging 

from training programs for documenting deficiencies to better tracking systems for survey 

data.40 But the report also highlights such vague examples of improvement as “initial 

success was opening communication,” “review done, goal met,” and “State intends to use 

SAQIP in its licensure program.”41 The HCFA grouped these and other examples into 

such categories as “enthusiasm,” “positive feedback received,” and “improved 

communication.” Overall, such examples provide little in the way of meaningful insights 

into a States’ performance. 

While SAQIP provides a standard means for State agencies to approach quality 

improvement, it fails to provide either a means of comparing the performance across 

States or insights specific to hospitals. To some extent, HCFA can rely on other means 

for these. For example, it logs all hospital anti-dumping complaints received and 

investigated by the States. It also maintains a database of State agencies’ survey activities 

and findings. The States input data reflecting their survey work into the database, which 

HCFA then can access and aggregate. The HCFA can glean the extent of survey activity 

and trends in hospitals choosing or dropping accreditation, complaints, and survey 

findings, among others. Staff in the HCFA regional offices reported relying on such data 

occasionally, but the data have limits. For example, HCFA maintains logs of patient 

dumping that list complaints chronologically, but lack summary data and analysis of 

trends. Also, HCFA cannot determine trends in hospital termination notices. In other 

words, HCFA does not know how many times it issued hospital termination notices based 

on findings during Medicare certification surveys of nonaccredited hospitals.42 It does 

know, however, how many hospitals are actually terminated. 

Policy and Procedural Updates: The HCFA determines the policies and 

procedures used by the States. 

The HCFA provides State agencies with the survey protocols and interpretive 

guidelines they need to survey hospitals. Its relationship with State agencies is akin to a 

contractual relationship, with HCFA’s expectations spelled out in an agreement renewed 

Hospital Quality: Holding the Reviewers Accountable 16 OEI-01-97-00053 
annually.43 It ranks their activities by priority in the annual budgeting process.44 Some 

regional offices hold regular conference calls with their States to pass along updates, 

others hold regional conferences. The HCFA regional offices provide technical assistance 

to State agencies as needed, and generally have close working relationships with them. 

The HCFA also regularly provides training for State agency staff. The training agenda 

reflects statutory, regulatory, and procedural changes as well as the workload of the 

States, which means long-term care dominates the training agenda. For example, among 

about 20 courses, HCFA offered 1 that was specific to hospitals in Fiscal Year 1997. In 

Fiscal Year 1998, 3 of 25 courses HCFA offered were specific to hospitals. 

The HCFA provides limited feedback to State agencies. 

Performance Assessments: The HCFA largely relies on the State agencies to 

conduct their own performance assessments. It provides little feedback specific 

to their hospital work. 

The HCFA’s SAQIP is the primary means for assessing State agencies’ 

performance. By design, SAQIP calls for State agencies to work with HCFA regional 

offices to develop and implement States’ own quality improvement plans. Since 

implementing SAQIP, HCFA has centrally produced two status reports: one in mid-year 

1997 and one at the year’s end. The HCFA sent copies of both reports to all State 

agencies. These reports, though, provide no State-specific feedback. They neither 

identify States nor offer assessments of performance specific to hospitals or other provider 

types. They include no summaries of activities such as hospital surveys conducted or 

complaints received. They also lack any feedback on specific areas for States to focus 

their improvements, or actions such as follow-up. In fact, the status reports give almost as 

much attention to the SAQIP process and its challenges as they do to the challenges of 

survey and certification activities. 

As we have seen, the State agencies tend to work closely with the HCFA regional 

office staff in the SAQIP process, offering opportunities for feedback. In fact 32 of 51 

State agencies reported receiving feedback from HCFA regarding their hospital oversight 

activities, and 20 reported making improvements based on that feedback.45 For example, 

several States reported implementing training programs, changing the emphasis of their 

approach to validation surveys, and establishing protocols for handling complaints based 

on HCFA feedback. 

Nevertheless, the feedback that State agencies receive from HCFA appears to be 

of limited value. For example, when asked about the usefulness of HCFA’s oversight, 29 

of 51 State agencies rated it not or somewhat useful, 15 as moderately useful, and 7 as 

very or extremely useful. While it varies by region, just one HCFA regional office 

indicated that it provides narrative assessments of its State agencies’ performance. Others 

Hospital Quality: Holding the Reviewers Accountable 17 OEI-01-97-00053 
indicated that feedback to the State agencies tends to be narrow and focused on a specific 

hospital rather than broad and relevant to the range of responsibilities States hold with 

hospitals. For example, HCFA’s feedback on a States’ performance is more akin to 

technical assistance in documenting a hospital’s condition than assistance in prioritizing or 

managing their resources devoted to hospital activities. 

Policy and Procedural Guidance: The HCFA gives State agencies considerable 

guidance, often on a case-by-case basis involving specific hospitals. 

The bulk of State agencies’ hospital work is related to investigating complaints, 

adverse events, and conducting validation surveys.46 The HCFA works closely with the 

agencies in carrying out each these activities. When it comes to hospital complaints, for 

example, the agencies’ workload extends to accredited as well nonaccredited hospitals, 

and HCFA regional offices tend to be closely involved in determining not only which get 

investigated, but also to what extent. This is particularly true when the complaint relates 

to an accredited hospital. In those cases, the HCFA reviews the complaint in light of the 

Medicare Conditions of Participation and determines whether the State needs to 

investigate. If an investigation ensues, the HCFA staff oversee the process step-by-step: 

they review the survey findings, summary of deficiencies, and any follow-up. In some 

cases, especially high-profile complaints, the HCFA may send one if its own surveyors to 

accompany the State agency. Indeed, in one State with a highly publicized series of 

hospital deaths, HCFA was so concerned about the State agency’s ability to handle the 

case that it stepped in with its own surveyors to respond. 

The HCFA regional offices also provide considerable guidance on routine 

certification surveys. As with complaint surveys, the HCFA staff will review the survey 

documentation. Indeed, if the State agency finds a hospital out of compliance with a 

Medicare condition of participation, then HCFA is directly involved in the follow-up, by 

issuing termination notices, for example. The HCFA also provided feedback for those 15-

20 monitoring surveys HCFA conducted over the past few years, usually by means of a 

formal debriefing with the State surveyors. 

The HCFA makes limited use of public disclosure as a means 

of holding hospitals or State agencies accountable. 

Performance of Hospitals: The HCFA discloses results of its hospital 

certification and complaint surveys upon request. 

The HCFA will disclose most of the survey documents, except those related to 

peer review that are considered confidential. Thus, upon request, HCFA will disclose the 

hospital’s statement of deficiencies, which spells out, condition by condition, the problems 

Hospital Quality: Holding the Reviewers Accountable 18 OEI-01-97-00053 
identified during the survey and the hospital’s plan for correction. It only lists those areas 

where the surveyors identified problems, and excludes any comparative or trend 

information. The HCFA will also disclose data from its database that records the findings 

and trends of hospital surveys.47 Neither the statement of deficiencies nor the data itself is 

available through HCFA’s website. Rather, interested parties must contact HCFA 

directly. By contrast, Federal law mandates that nursing homes post their statements of 

deficiencies in a manner readily accessible to residents and their families.48 In addition, 

HCFA’s website allows visitors to view the results of nursing home surveys, including the 

scope and severity of any deficiencies.49 

The survey and certification agencies in some States have gone further than HCFA 

in disclosing hospital-specific information, based on their own States’ authority rather than 

HCFA’s. For example, New Jersey maintains a website that displays resolved hospital 

violations and New York collects and reports hospital- and physician-specific performance 

data on cardiac surgery. 

Performance of State Agencies: The HCFA discloses little on the performance of 

State agencies. 

Unlike the mandate for the Report to Congress on the Joint Commission’s 

performance, no such mandate exists for HCFA to report on how well State agencies are 

performing. Nevertheless, HCFA has some, albeit limited, information on States’ 

performance. As we have seen, it is has mid-year and year-end SAQIP reports plus 

databases reflecting the State agencies’ survey work. As noted above, HCFA will release 

the data, but it would take considerable knowledge and work to array the data to facilitate 

State-by-State comparisons or in some way measure State agency performance. 

According to HCFA officials with whom we spoke, although HCFA does not distribute 

the SAQIP reports beyond the States themselves, it will release them upon request. Of 

course, the SAQIP reports lack any State-specific assessment of performance. 

Hospital Quality: Holding the Reviewers Accountable 19 OEI-01-97-00053 
CONCLUSION 

The clear and disturbing conclusion of this report is that both the Joint 

Commission and State Agencies are only minimally accountable to HCFA for their 

performance in reviewing hospitals. While we recognize that these entities are also 

accountable to others and that they must have considerable flexibility to function 

effectively, we maintain that it is vitally important for HCFA to ensure that they 

adequately fulfill their responsibilities to protect Medicare beneficiaries. How, then, can 

HCFA hold these entities accountable while minimizing burdensome oversight? How can 

it recognize their inherent strengths and limitations, and tailor performance measurement 

accordingly? We address these and other related questions in our summary report, A Call 

for Greater Accountability. That report also contains our recommendations, which we 

direct to HCFA. 

Hospital Quality: Holding the Reviewers Accountable 20 OEI-01-97-00053 
APPENDIX A 

Methodology

We collected information presented in this report from the following sources: 

The HCFA 

We obtained dates of certification surveys from HCFA’s Online Survey 

Certification and Reporting System (OSCAR). The HCFA authorizes States to update 

and maintain this database with survey information. We extracted survey data pertaining 

to the frequency of certification surveys. We subsequently verified the accuracy of our 

extraction by comparing it to on-line OSCAR system information to ensure the dates we 

used corresponded to routine certification surveys, rather than complaint investigations or 

other types of surveys. We are satisfied that our information is as accurate as HCFA’s 

OSCAR system. 

Additionally, we selected 4 States (California, Kansas, Minnesota, and Texas) that 

contain over 50 nonaccredited hospitals and represented different regions of the United 

States. We then examined the OSCAR data for those States in greater detail. We verified 

the operational status of the nonaccredited hospitals in those States that had not been 

surveyed in over 5 years using the American Hospital Association’s 1997 Hospital Guide. 

We also interviewed staff and managers at each HCFA regional office and the 

central office. We reviewed a variety of HCFA documents, including budget call letters, 

reinvention materials, and reports to Congress, among others. 

The State Survey and Certification Agencies 

In August 1997, we mailed a pretested survey to the hospital certification agencies 

in the 50 States and the District of Columbia. The response rate for this survey was 100 

percent. The State survey addressed four areas of hospital quality oversight: private 

accreditation of hospitals, Medicare certification of hospitals, HCFA oversight of State 

certification agencies, and State licensure of hospitals. We interviewed State officials on 

the telephone or in person as well. 

Hospital Quality: Holding the Reviewers Accountable 21 OEI-01-97-00053 
APPENDIX A 

Accrediting Organizations 

We interviewed officials from both the Joint Commission on Accreditation of 

Healthcare Organizations and the American Osteopathic Association. We also reviewed 

documents from both organizations, including mission statements, accreditation manuals, 

policies, and hospital survey files. We requested and received aggregate data from the 

Joint Commission reflecting its recent hospital survey activity. All Joint Commission data 

are presented as reported by the Joint Commission, unless otherwise noted. For purposes 

of this inquiry, we focused our analysis on the Joint Commission. 

Survey Observations 

Based on schedules of upcoming triennial surveys, we identified nine hospitals in 

which to observe triennial Joint Commission surveys. Of those, we were able to observe 

seven. In two cases, the hospitals declined the Joint Commission’s request that we be 

allowed to observe. The 7 hospitals varied in size from 80 to 775 beds, represented both 

teaching and community hospitals, and were located in different areas of the country (both 

rural and urban). We also observed one random unannounced Joint Commission survey. 

Although we observed different teams of surveyors, the survey agenda, lines of inquiry, 

and tone were consistent across the surveys, which were conducted in accordance with 

Joint Commission policy, based on review of survey manuals and interviews with 

representatives of the Joint Commission. Finally, we observed a certification survey and 

parts of two complaint investigations performed by State surveyors under HCFA’s 

auspices. 

Stakeholder Interviews 

We interviewed representatives of organizations we considered to be stakeholders 

in hospital oversight. These stakeholder organizations included a union, professional 

organizations, hospital associations, and consumer groups. 

Other Documents 

In addition to the documents referenced above, we reviewed statutory and 

regulatory language and a variety of articles from newspapers, journals, magazines, and 

newsletters. 

Hospital Quality: Holding the Reviewers Accountable 22 OEI-01-97-00053 
APPENDIX B 

HCFA’s Reinvented Oversight of Hospital Accreditation 

Dissatisfaction with Validation 

Interest in reinventing HCFA’s approach to overseeing accreditation stemmed 

from growing dissatisfaction with the validation survey process. In 1990, 16 years after 

HCFA began to validate the Joint Commission surveys, the General Accounting Office 

(GAO) published a report criticizing HCFA’s validation program. The GAO found the 

comparison of HCFA and Joint Commission survey findings yielded little information 

about the effectiveness of Joint Commission process. Because HCFA measures hospital 

quality using the Medicare Conditions of Participation and the Joint Commission uses its 

own standards, the GAO saw little use in comparing the results of these surveys without a 

crosswalk linking the two sets of measures. In response to this report, HCFA developed a 

crosswalk. The process was labor-intensive and time-consuming, resulting in a document 

over 800 pages long. However, HCFA found using this cumbersome crosswalk produced 

the same results as its old method of comparing survey results and discontinued use of the 

crosswalk. 

Other concerns about the validation process emerged and gained credence through 

the early 1990s. For example, HCFA came to believe that using hospitals as a proxy to 

evaluate the Joint Commission accreditation process was not only indirect but also could 

be inaccurate if hospital performance changed since an accreditation survey. 

The Reinvention Workgroup 

In 1993, HCFA organized a workgroup comprising members from HCFA, States, 

the American Hospital Association, American Osteopathic Association, and Joint 

Commission to reinvent the validation. This workgroup identified an alternative or 

reinvented approach to the validation through which HCFA would evaluate the Joint 

Commission’s accreditation process by conducting concurrent, on-site observations of the 

Joint Commission surveys instead of comparing HCFA and Joint Commission survey 

results. This reinvented validation survey would involve HCFA and State surveyors 

participating in Joint Commission surveys as silent observers. The workgroup also 

proposed that HCFA and the Joint Commission should mutually develop both surveyorbased and hospital-based performance indicators to be used by HCFA and State observers. 

This on-site assessment was planned to be one part of a larger comprehensive evaluation 

(see box). 

Hospital Quality: Holding the Reviewers Accountable 23 OEI-01-97-00053 
APPENDIX B 

Planned Components of the Reinvention 

Review of Accreditation Organization’s 

< Philosophy and Mission 

< Survey Protocols 

< Surveyor Support 

< On-Site Survey 

< Decision process 

< Deficiency Resolution 

< Timeliness 

< Performance Improvement 

< Intra-Cycle Activity 

< Response to Complaints 

Source: HCFA documents, June 1997. 

The Two Completed Reinvention Phases 

The reinvention workgroup has planned and completed two phases of the 

reinvention, both pertaining to the on-site assessment piece of the validation. As Phase I, 

the workgroup drafted a methodology and developed evaluation criteria for the 

reinvention surveys. The workgroup identified five surveyor based performance 

indicators: survey preparation, survey process, interpersonal, presentation, and teamwork. 

It also agreed upon 11 core hospital functions: assessment, patient preparation, treatment, 

nutrition, medication use, discharge coordination, environmental safety, patient safety, 

infection control, quality improvement, and information management. The HCFA regional 

office and State teams of surveyors attended Joint Commission surveyor training and as 

well as training on how to use the newly developed surveyor and hospital criteria. The 

HCFA piloted a total of 20 reinvention surveys in 4 States: Pennsylvania, Georgia, 

Illinois, and Colorado. 

Hospital Quality: Holding the Reviewers Accountable 24 OEI-01-97-00053 
APPENDIX C 

State Agency Quality Improvement Plan 

Standards

The HCFA developed the following standards through a workgroup that included 

State agency representatives: 

Quality Improvement Standards 

Standard 1: The State agency documents deficiencies consistent with the Principles of 

Documentation. 

Standard 2: The Plans of Correction accepted by the State agency reflect appropriate 

action and time frames to correct cited deficiencies. 

Standard 3: All surveys are conducted by qualified individuals. 

Standard 4: The State agency assures consistency in survey performance. 

Standard 5: The State agency has an effective program to measure accuracy and 

improve consistency int he application of long-term care enforcement 

remedies. 

Standard 6: The State agency effectively investigates and processes complaints 

(including those related to Section 1867) about participating providers and 

suppliers. 

Standard 7: The State agency monitors expenditures and supports charges to Federal 

programs in accordance with existing regulations, guidelines, policies, 

approved funding levels and allocation methodologies. 

Quality Control Standards 

Standard 8: The State agency ensures that each home health agency (HHA), skilled 

nursing facility (SNF) nursing facility (NF), and SNF/NF is subject to a 

standard survey not later than 15 months after the previous survey, and that 

the statewide average interval between standard surveys does not exceed 

12 months. 

Hospital Quality: Holding the Reviewers Accountable 25 OEI-01-97-00053 
APPENDIX C 

Standard 9: The State agency ensures that it meets all survey coverage percentages 

negotiated with the regional office for Title XVIII and/or Title XIX 

providers and suppliers, excluding HHAs, SNFs, NFs, and SNF/NFs. 

Standard 10: All surveys, including complaint investigations, are conducted 

unannounced, or are announced consistent with HCFA instructions. 

Standard 11: The State agency budget request, activity plan, and expenditure reports are 

prepared and submitted in accordance with Federal instructions and 

accurately reflect the allocation of costs between State and Federal 

programs. 

Standard 12: The State agency effectively maintains the OSCAR system database. 

Standard 13: Annual surveys of intermediate care facilities for the mentally retarded and 

recertification actions completed before the expiration of the time-limited 

agreement. 

Hospital Quality: Holding the Reviewers Accountable 26 OEI-01-97-00053 
APPENDIX D 

Endnotes

1. Troyen A. Brennan et al, “Incidence of Adverse Events and Negligence in Hospitalized 

Patients,” The New England Journal of Medicine 324 (February 7, 1991) 6: 370-376. 

2. The 17.7 percent refers to adverse events considered by the authors to be serious. The authors 

defined adverse events as “situations in which an inappropriate decision was made when, at the 

time, an appropriate alternative could have been chosen” and serious as ranging from “temporary 

disability to death.” See Lori B. Andrews et al, “An Alternative Strategy for Studying Adverse 

Events in Medical Care,” The Lancet 349 (February 1, 1997)309-313. 

See also Lucian L. Leape, “Error in Medicine,” Journal of the American Medical 

Association 272 (December 21, 1994) 23: 1851-1857; David C. Classes et al, “Adverse Drug 

Events in Hospitalized Patients,” Journal of the American Medical Association 277 (January 

22/29, 1997) 4: 301-306; Mark R. Chassin et al, “The Urgent Need to Improve Health Care 

Quality: Institute of Medicine National Roundtable on Health Care Quality,” Journal of the 

American Medical Association 280 (September 16, 1998) 11: 1000-1005; David W. Bates et 

al,“Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of 

Serious Medication Errors,” Journal of the American Medical Association 280 (October 21, 

1998) 1311-1316; Robert A. Raschke et al, “A Computer Alert System to prevent Injury from 

Adverse Drug Events: Development and Evaluation in a Community Teaching Hospital,” Journal 

of the American Medical Association 280 (October 21, 1998) 1317-1320; and David C. Classen, 

“Clinical Decision Support Systems to Improve Clinical Practice and Quality of Care,” Journal of 

the American Medical Association 280 (October 21, 1998) 1360-1361. 

3. One example is a November 1998 New York Times article under the headline: “Death in 

Surgery Reveals Troubled Practice and Lax Hospital.” The article described a “botched” 

operation on a young woman by a surgeon who was on probation by the State medical board and 

who used unauthorized medical equipment brought in to the operating room by a medical supply 

salesman. Such incidents can happen even in the best of hospitals, but they underscore the point 

that hospitals can be dangerous places and that oversight systems can be lax. See also 

“Overdoses Still Weigh heavily at Dana Farber,” The Boston Globe (26 December 1995); “Florida 

Doctor Sanctioned in New Amputation,” The Boston Globe (19 July 1995); “Two Surgeons 

Surrender Licenses After Mistakenly Removing Kidney,” The Boston Globe (6 June 1996); “How 

Can We Save the Next Victim?” New York Times Magazine (15 June 1997); “Another Hospital 

Death Probed,” The Boston Globe (26 July 1997); “Patient Dies After Drinking Poison Left on 

Nightstand,” San Diego Union-Tribune (6 March 1998); ”Man Arrested for Posing as Doctor for 

4 years,” posted at the CNN interactive webpage (16 May 1998); “Deadly Restraint: Patients 

Suffer in a System Without Oversight,” The Hartford Courant (13 October 1998). 

Hospital Quality: Holding the Reviewers Accountable 27 OEI-01-97-00053 
APPENDIX D 

4. http://WWW.hcfa.gov/Medicare/Medicare.htm 

5. Ibid. 

6. Ibid. 

7. Ibid. 

8. P.L. 89-97. 

9. The HCFA published its proposed hospital conditions of participation on December 19, 1997 

(62 Fed. Reg. 66,726). On July 2, 1999, it published an interim final rule concerning patients’ 

rights (64 Fed. Reg. 36,070). 

10. Congress also provided that hospitals accredited by the American Osteopathic Association 

could be considered in compliance, but only to the extent than the Secretary deemed appropriate. 

11. Two companion reports entitled The Role of Accreditation and The Role of Medicare 

Certification explore the role of accreditation and Medicare certification in greater detail. 

12. Michael S. Hamm, The Fundamentals of Accreditation (Washington, D.C.: American Society 

of Association Executives, 1997) 3. 

13. Nonaccredited hospitals can go through the Medicare certification process for free. 

14. Social Security Act, sec. 1865 (a), 42 U.S.C. 1395bb. 

15. Social Security Act, sec. 1864(c), 42 U.S.C. 1395aa. 

16. Health Care Financing Administration, Oscar 10 Report, as of May 30, 1998. 

17. Validation surveys are onsite reviews of hospitals conducted after a Joint Commission survey. 

State agency surveyors conduct the validation surveys based ont he Medicare conditions of 

participation. 

18. We acknowledge the dual purpose of validation surveys. As originally envisioned, they were 

to serve as an assurance that accredited hospitals met the Medicare conditions of participation: 

“Our validation program is concerned chiefly with the equivalency of accreditation standards to 

Federal requirements, and assuring that Medicare participating facilities deemed by virtue of 

accreditation would meet those requirements if surveyed against them.” (58 CFR 61831, see also 

Senate Report 92-1230). Over time, the purpose and use of validation surveys evolved to a more 

Hospital Quality: Holding the Reviewers Accountable 28 OEI-01-97-00053 
APPENDIX D 

encompassing role as a tool of oversight for the Joint Commission. Indeed, these roles overlap 

and their distinctions may be blurred. For purposes of this report, we focus on validation surveys 

and their potential as a tool of oversight. 

19. In 1990, the General Accounting Office criticized HCFA’s approach to validation surveys, 

citing specifically the lack of a crosswalk linking the Medicare conditions with the Joint 

Commission’s own standards. In response, HCFA created such a crosswalk, but the undertaking 

was not only labor-intensive, but also cumbersome: it resulted in an 800-page document. And 

when HCFA relied upon that crosswalk for validation surveys in 1989 and 1990, the results were 

the same as without a crosswalk (See Health Care Financing Administration, “The Reinvention of 

the Hospital Validation Survey,” undated, and U.S. General Accounting Office, Health Care: 

Criteria Used to Evaluate Hospital Accreditation Process Needs Reevaluation, GAO/HRD-90-

89, 11 June 1990). 

20. See our companion report entitled The Role of Accreditation for a discussion of the Joint 

Commission’s standards. 

21. The HCFA’s instructions to State agencies also allow for validation surveys to occur within 

60 days of HCFA’s notice to the State agency. 

22. While we acknowledge our estimate is a rough one, we base it the conservative assumption 

that 2-3 surveyors would travel to a hospital for 2-3 days. While highly variable, we estimated the 

costs of surveyor salary and travel would be $2,000 to $4,000 each. And considering about 150 

surveys per year, we believe the estimate of about $2 million is reasonable. 

23. Health Care Financing Administration, “The Reinvention of the Hospital Validation Survey,” 

undated. 

24. In his book, The Fundamentals of Accreditation, Michael Hamm identifies some generic 

criteria for evaluating accreditation bodies, and among the criteria is the organizations’ 

performance record. (Michael S. Hamm, The Fundamentals of Accreditation, Washington, DC: 

American Society of Association Executives, 1997, 10-11). 

25. The Joint Commission currently has HCFA officials serving on its Hospital Professional and 

Technical Advisory Committee and its Council on Performance Measurement. 

26. Social Security Act, sec. 1875(b), 42 U.S.C. 1395ll. 

27. Health Care Financing Administration, Medicare Annual Report to Congress: Fiscal Year 

1992. 

Hospital Quality: Holding the Reviewers Accountable 29 OEI-01-97-00053 
APPENDIX D 

28. 62 Fed. Reg. 66726, December 19, 1997. 

29. The Joint Commission’s board has 28 members, including 6 public members and an at-large 

nursing representative. Most members are appointed by organizations such as the American 

College of Physicians, the American College of Surgeons, the American Dental Association, the 

American Hospital Association, and the American Medical Association. 

30. Joint Commission on Accreditation of Healthcare Organizations, “Professional and Technical 

Advisory Committee (PTAC) Rules and Procedures,” January 16, 1993. 

31. In November 1996, the Joint Commission waived the $30 report fee. 

32. The 45 performance areas, divided into 15 categories, are: 

Patient Rights and Organization Ethics: Patient Rights, Organization Ethics

Assessment of Patients: Initial Assessment Procedures, Pathology and Clinical Laboratory

Services, Reassessment Procedures, Processes for patient Care Decision, Relevant Policies, Needs

Assessment for Specific Patient Populations

Care of Patients: Planning and Providing Care, Anesthesia Care, Medication Use, Nutrition

Care, Operative Procedures, Rehabilitation Care, Special Treatment Procedures

Education: Patient and Family Education

Continuum of Care: Continuity of Care

Improving Organizational Performance: Improvement Planning, Design of New Services,

Measurement of Processes and Outcomes, Assessment of Data, Improvement of Performance

Leadership: Strategic Planning, Departmental Leadership. Integrating and Coordinating

Services, Leader’s Role in Improving Performance

Management of the Environment of Care: Design of the Environment, Implementation of the

Safety Plans, Monitoring of the Safety Plans, Social Environment

Management of Human Resources: Human resources Planning; Orienting, Training, and

Education of Staff; Assessing Staff Competence; Managing Staff Requests

Management of Information: Information Management Planning, Availability of PatientSpecific Information, Data Collection and Analysis, Literature to Support Decision-Making, Use

of Comparative Information

Infection Control: Infection Control

Governance: Governance

Medical Staff: Organization, Bylaws, Rules, and Regulations; Credentialing

Nursing: Nursing

Management: Management

Hospital Quality: Holding the Reviewers Accountable 30 OEI-01-97-00053 
APPENDIX D 

33. The HCFA staff we contacted, particularly in the regional offices, were largely unaware of 

these reports and their availability. 

34. For further discussion of this policy, see our companion report, The Role of Accreditation. 

35. For examples, see Dennis S. O’Leary, “On Becoming Ten,” Joint Commission Perspectives 

16 (May/June 1996)3: 2-3; John Morrissey, “JCAHO Outlines Action to Improve Performance,” 

Modern Healthcare, January 30, 1995. 

36. As we noted in our companion report, The Role of Medicare Certification, nursing homes, 

intermediate care facilities for the mentally retarded, and home health agencies all precede 

nonaccredited hospitals in survey priority. 

37. HCFA, “The State Agency Quality Improvement Program,” May 1996. 

38. The Emergency Medical Treatment and Labor Act (42 U.S.C. 1395dd) prohibits hospitals 

from refusing emergency patients or women in labor medical screening and/or treatment. If a 

hospital willfully or negligently fails to meet the requirements of the act, it can be fined or its 

participation in Medicare can be suspended or terminated. 

39. The HCFA also has its On-Line Survey and Certification data system (referred to as OSCAR) 

for insights into the extent of hospital surveys conducted by the States. 

40. Health Care Financing Administration, “State Agency Quality Improvement Program End-ofYear Implementation Status - Fiscal Year 1997,” May 15, 1997. 

41. Ibid. 

42. The HCFA tracks survey results through its OSCAR database, but it does not track 

termination notices, which can be 23-day or 90-day notices based on the severity of the survey 

findings. 

43. Referred to as the 1864 Agreement, this document spells out HCFA’s delegation of authority 

to the States and the functions States must carry out. 

44. See the companion report, The Role of Medicare Certification, for more detail on the budget 

priorities. 

45. Based on the Office of Inspector General mail survey in September 1997. The survey defined 

hospital activities to include Medicare certification surveys for nonaccredited hospitals, responses 

to complaints, and validation surveys of accredited hospitals. 

Hospital Quality: Holding the Reviewers Accountable 31 OEI-01-97-00053 
APPENDIX D 

46. See the companion report, The Role of Medicare Certification, for more detail on State 

agencies’ workloads. 

47. Referred to as the On-line Survey and Certification Reporting System. The HCFA may 

charge a small fee for processing this data. 

48. Social Security Act, sec. 1864 (42 U.S.C. 1395aa). 

49. The HCFA calls this “Nursing Home Compare” and it is available at 

http://www.medicare.gov. 

Hospital Quality: Holding the Reviewers Accountable 32 OEI-01-97-00053