Report to Congressional Requesters

United States Government Accountability Office

GAO 

July 2004 MEDICARE

CMS Needs Additional 

Authority to 

Adequately Oversee 

Patient Safety in 

Hospitals 

GAO-04-850 
www.gao.gov/cgi-bin/getrpt?GAO-04-850.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Janet Heinrich 

at (202) 512-7119. 

Highlights of GAO-04-850, a report to 

congressional requesters 

July 2004

MEDICARE

CMS Needs Additional Authority to 

Adequately Oversee Patient Safety in 

Hospitals 

JCAHO’s pre-2004 hospital accreditation process did not identify most of the 

hospitals found by state survey agencies in CMS’s annual validation survey 

sample to have deficiencies in Medicare requirements. In comparing the 

results of the two surveys, CMS considered whether it was reasonable to 

conclude that the deficiencies found by state survey agencies existed at the 

time JCAHO surveyed the hospital. In a sample of 500 JCAHO-accredited 

hospitals, state agency validation surveys conducted in fiscal years 2000 

through 2002 identified 31 percent (157 hospitals) with deficiencies in 

Medicare requirements. Of these 157 hospitals, JCAHO did not identify 78 

percent (123 hospitals) as having deficiencies in Medicare requirements. For 

the same validation survey sample, JCAHO also did not identify the 

majorityabout 69 percentof deficiencies in Medicare requirements found 

by state agencies. Importantly, the number of deficiencies found by 

validation surveys represents 2 percent of the 11,000 Medicare requirements 

surveyed by state agencies in the sample during this time period. At the 

same time, a single deficiency in a Medicare requirement can limit the 

hospital’s capability to provide adequate care and ensure patient safety and 

health. Inadequacies in nursing practices or deficiencies in a hospital’s 

physical environment, which includes fire safety, are examples of 

deficiencies in Medicare requirements that could endanger multiple patients. 

The potential of JCAHO’s new hospital accreditation process to improve the 

detection of deficiencies in Medicare requirements is unknown because the 

process was just implemented in January 2004. JCAHO plans to move from 

using announced to unannounced surveys in 2006, which would afford 

JCAHO the opportunity to observe hospitals’ operations when the hospitals 

have not prepared in advance to be surveyed. In addition, the pilot test of the 

new accreditation process was of limited value in predicting whether it will 

be an improvement over the pre-2004 process in detecting deficiencies. 

Limitations in the pilot test included that hospitals were not randomly 

selected to participate; that observers from JCAHO accompanied each 

surveyor, thus possibly affecting surveyors’ actions; and that JCAHO 

evaluated the results instead of an independent entity. 

CMS has limited oversight authority over JCAHO’s hospital accreditation 

program because the program’s unique legal status effectively prevents CMS 

from taking actions that it has the authority to take with other health care 

accreditation programs to ensure satisfactory performance. For example, 

requiring JCAHO’s hospital accreditation program to submit to a direct 

review process or placing the program on probation while monitoring its 

performance. Further, CMS relies on a measure to evaluate how well 

JCAHO’s hospital accreditation program detects deficiencies in Medicare 

requirements that provides limited information and can mask problems with 

program performance, uses statistical methods that are insufficient to assess 

JCAHO’s performance, and has reduced the number of validation surveys it 

conducts.

Hospitals accredited by the Joint 

Commission on Accreditation of 

Healthcare Organizations (JCAHO) 

are considered in compliance with 

Medicare participation 

requirements. GAO examined the 

extent to which JCAHO’s pre-2004 

hospital accreditation process 

identified hospitals not complying 

with Medicare requirements, the 

potential of JCAHO’s new process 

for improving the detection of 

deficiencies in Medicare 

requirements, and the effectiveness 

of CMS’s oversight of JCAHO’s 

hospital accreditation program. 

GAO analyzed CMS data on 

hospitals state surveyors found to 

have deficiencies in Medicare 

requirements that JCAHO 

surveyors did not detect, analyzed 

CMS’s measure of JCAHO’s ability 

to detect noncompliance with 

Medicare requirements, and 

interviewed JCAHO officials. 

GAO believes that Congress should 

consider giving CMS the authority 

over JCAHO’s hospital 

accreditation program that it has 

over other accreditation programs 

and recommends that CMS modify 

its methods for assessing JCAHO’s 

performance. CMS agreed with 

GAO’s recommendations. JCAHO 

stated that GAO’s methodology was

incomplete and did not 

comprehensively assess its overall 

performance. GAO emphasized that

its engagement was limited to one 

aspect of deficiency detection and 

was not intended to reflect 

JCAHO’s overall performance. 
Page i GAO-04-850 Medicare Patient Safety in Hospitals 

Letter 1

Results in Brief 4

Background 6

JCAHO’s Pre-2004 Hospital Accreditation Process Often Did Not 

Detect Serious Deficiencies Found by State Survey Agencies 10

Potential of JCAHO’s New Hospital Accreditation Process Is 

Unknown, and Testing Was Limited 16

CMS Oversight Authority of JCAHO’s Hospital Accreditation 

Program Is Limited and Needs Improvement 20

Conclusions 28

Matter for Congressional Consideration 28

Recommendations for Executive Action 29

Agency and Other External Comments and Our Evaluation 29

Appendix I Scope and Methodology 32 

Appendix II Medicare Conditions of Participation 36 

Appendix III Features of JCAHO’s New Accreditation Process 38 

Appendix IV Comments from the Centers for Medicare & Medicaid 

Services 40 

Appendix V Comments from the Joint Commission on 

Accreditation of Healthcare Organizations 46 

Appendix VI GAO Contact and Staff Acknowledgments 50 

GAO Contact 50

Acknowledgments 50

Related GAO Products 51 

Contents 
Page ii GAO-04-850 Medicare Patient Safety in Hospitals 

Tables 

Table 1: Hospitals in CMS’s Validation Survey Sample with Serious 

Deficiencies that State Survey Agencies Identified but 

JCAHO Surveyors Did Not, Fiscal Years 2000-2002 11

Table 2: Percentage of Serious Deficiencies Identified by State 

Survey Agencies but Not by JCAHO Surveyors in CMS’s 

Validation Survey Sample, Fiscal Years 2000-2002 12

Table 3: Number of Serious Deficiencies, by COP, Identified by 

State Survey Agencies but Not by JCAHO Surveyors in 

CMS’s Validation Survey Sample, Fiscal Years 2000-2002 14

Table 4: Accreditation Decisions for Hospitals Surveyed Under 

JCAHO’s New Survey Process Pilot Test as Compared to 

Results from JCAHO’s Pre-2004 Survey Process 19

Table 5: Hypothetical Examples of the Effect on the Rate of 

Disparity of a Decrease in the Number of Hospitals with 

Serious Deficiencies in a Sample of 200 Hospitals 25

Table 6: Number of Hospitals Targeted for Validation Surveys 

Compared with Usable Traditional Validation Surveys 

Completed 26

Table 7: Medicare Conditions of Participation 36

Table 8: JCAHO’s Description of Features of Its New Hospital 

Accreditation Process 38
Page iii GAO-04-850 Medicare Patient Safety in Hospitals 

Abbreviations 

AOA American Osteopathic Association 

CMS Centers for Medicare & Medicaid Services 

COP condition of participation 

HHS Department of Health and Human Services 

JCAHO Joint Commission on Accreditation of Healthcare 

 Organizations 

OIG Office of Inspector General 

PFP priority focus process 

PPR periodic performance review 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
Page 1 GAO-04-850 Medicare Patient Safety in Hospitals 

July 20, 2004 

The Honorable Charles E. Grassley 

Chairman 

Committee on Finance 

United States Senate 

The Honorable Pete Stark 

Ranking Minority Member 

Subcommittee on Health 

Committee on Ways and Means 

House of Representatives 

In fiscal year 2002, nearly 7.4 million Medicare beneficiaries received 

inpatient health care at hospitals that participated in Medicare. Federal 

law establishes criteria for hospitals for purposes of Medicare. The 

Centers for Medicare and Medicaid Services (CMS), the agency 

responsible for administering Medicare, has established quality and patient 

safety requirements called conditions of participation (COP) that hospitals 

must meet in order to be eligible for Medicare payment. Hospitals that are 

accredited by the Joint Commission on Accreditation of Healthcare 

Organizations (JCAHO) are generally deemed under federal law to be 

compliant with Medicare requirements for patient safety and health and 

become eligible for payments from Medicare.1,2

 No other health care 

accreditation program has this same statutory authority. 

JCAHO is a private, not-for-profit organization that accredits most of the 

hospitals that participate in Medicare. JCAHO sets standards that 

accredited hospitals must meet and reports that these standards are more 

comprehensive than the Medicare COPs.3

 In January 2004, JCAHO 

implemented a new hospital accreditation process with goals that included 

further enhancing health care quality and safety. 

 1

See 42 U.S.C. § 1395bb(a) (2000). 

2

JCAHO is referred to in statute under its former name, the Joint Commission on 

Accreditation of Hospitals. 

3

JCAHO develops its standards with a committee of experts and stakeholders, such as the 

government, hospitals, and consumers. 

United States Government Accountability Office

Washington, DC 20548 
Page 2 GAO-04-850 Medicare Patient Safety in Hospitals 

CMS oversight of JCAHO’s hospital accreditation program is limited 

because it cannot restrict or remove JCAHO’s accreditation authority if the 

agency detects problems. To oversee the program, CMS conducts on-site 

validation surveys of a sample of JCAHO-accredited hospitals and reports 

annually to Congress on the results of these surveys. The validation 

surveys, which are performed by agencies that CMS has agreements with 

in each state, help CMS determine whether Medicare quality and safety 

requirements are being met. CMS compares the results of these state 

surveys against survey results obtained through JCAHO’s hospital 

accreditation program. CMS uses a measure called the rate of disparity 

that summarizes the extent to which an accreditation program has failed 

to cite deficiencies identified by state agency validation surveys. We are 

using the term serious deficiency in this report to indicate a deficiency in 

one or more Medicare COPs. Examples of serious deficiencies include a 

hospital’s inability to provide adequate nursing services or failure to 

implement and enforce infection control policies. According to CMS, 

serious deficiencies substantially limit a hospital’s capability to render 

adequate care and adversely affect the safety and health of patients. 

Questions have been raised by the Department of Health and Human 

Services’ (HHS) Office of Inspector General (OIG) and others regarding 

whether accreditation by JCAHO ensures that hospitals provide adequate 

care. Specifically, experts have questioned how well JCAHO’s hospital 

accreditation process identifies deficiencies in hospitals that could 

jeopardize patient safety and health. A comprehensive study by the HHS 

OIG found that JCAHO’s surveys were not likely to identify patterns of 

deficient care.4

You asked that we examine the effectiveness of JCAHO’s hospital 

accreditation process in ensuring that hospitals comply with Medicare 

COPs to ensure the safety and health of Medicare beneficiaries. 

Specifically, we (1) examined the extent to which JCAHO’s pre-2004 

hospital accreditation process identified deficiencies in Medicare COPs 

that were identified by state survey agencies, (2) determined whether 

JCAHO’s new hospital accreditation process has potential for improving 

the detection of deficiencies in Medicare COPs and whether the process 

was adequately tested, and (3) examined the effectiveness of CMS’s 

oversight of JCAHO’s hospital accreditation program. 

 4

HHS OIG, The External Review of Hospital Quality: A Call for Greater Accountability, 

OEI-01-97-00050 (Washington, D.C.: July 1999). 
Page 3 GAO-04-850 Medicare Patient Safety in Hospitals 

To determine the extent to which JCAHO’s pre-2004 hospital accreditation 

process identified deficiencies in Medicare COPs that were identified by 

state survey agencies, we used data from a CMS comparison of state 

validation survey findings with findings of JCAHO’s hospital accreditation 

surveys, which indicated whether JCAHO found deficiencies in its 

standards. Of the four possible outcomes to this comparison of survey 

findings—(1) JCAHO and state agencies both identify no deficiencies, (2) 

JCAHO identifies deficiencies not found by state agencies, (3) JCAHO and 

state agencies both identify the same deficiencies, and (4) state agencies 

identify deficiencies that JCAHO does not—we focused on the fourth 

because it highlights the need for CMS oversight of the hospital 

accreditation program. For the second outcome, there could be two 

reasons for the disparity between JCAHO’s and state survey agencies’ 

findings: hospitals corrected deficiencies identified by JCAHO prior to the 

state agency survey or the state survey agency did not identify a deficiency 

that existed. In addition, not all JCAHO findings are equivalent to 

noncompliance with a Medicare COP. To determine whether JCAHO’s 

findings on deficiencies in its standards were comparable to the state 

agencies’ findings, CMS staff compared the two surveys and considered 

whether it was reasonable to conclude that the deficiencies found by state 

survey agencies existed at the time JCAHO surveyed the hospital. For 

deficiencies that CMS determined that JCAHO failed to identify, CMS met 

with JCAHO to address disputed findings and consider additional evidence 

on comparability offered by JCAHO. CMS provided results for a sample of 

500 JCAHO-accredited hospitals from fiscal years 2000 through 2002. We 

determined that the data CMS provided on serious deficiencies were 

adequate for addressing the issues in this report. On the basis of this 

sample of 500 JCAHO-accredited hospitals, we determined, using CMS’s 

data, both the percentage of serious deficiencies and the percentage of 

hospitals with serious deficiencies identified by the state survey agencies 

where JCAHO surveyors did not find comparable deficiencies. The 

analysis we performed on the results of the validation surveys was limited 

to the hospitals included in the validation survey sample and cannot be 

generalized to all JCAHO-accredited hospitals. 

To determine whether JCAHO’s new hospital accreditation process has 

potential for improving the detection of serious deficiencies, we identified 

changes in the accreditation process and analyzed significant new 

features. To determine whether JCAHO’s new hospital accreditation 

process was adequately tested, we reviewed the testing procedures and 

results that JCAHO used to determine the effectiveness of its new survey 

process in identifying quality and safety deficiencies. Because the new 

accreditation process was implemented recently, we did not have 
Page 4 GAO-04-850 Medicare Patient Safety in Hospitals 

information to compare JCAHO survey performance in detecting serious 

deficiencies with state agency survey performance. 

To determine the adequacy of CMS’s oversight of JCAHO’s hospital 

accreditation program, we reviewed relevant statutory and regulatory 

provisions regarding oversight of health care accreditation programs and 

how CMS had implemented this authority in order to provide oversight. To 

supplement our review, we conducted interviews with officials from CMS, 

state survey agencies, and JCAHO; representatives from other 

organizations active in health care accreditation and the hospital industry; 

and experts in quality of care. We conducted our work from June 2003 

through July 2004 in accordance with generally accepted government 

auditing standards. (For a complete description of our scope and 

methodology, see app. I.) 

JCAHO’s pre-2004 hospital accreditation process did not identify most of 

the hospitals found by state survey agencies in CMS’s annual validation 

survey sample to have serious deficiencies in Medicare COPs. In a sample 

of 500 JCAHO-accredited hospitals, state agency validation surveys 

conducted in fiscal years 2000 through 2002 identified 31 percent (157 

hospitals) with serious deficiencies; of these, JCAHO did not identify 78 

percent (123 hospitals) as having serious deficiencies. For the same 

validation survey sample, JCAHO also did not identify the majorityabout 

69 percentof serious deficiencies found by state agencies. Importantly, 

the number of deficiencies found by validation surveys represents 2 

percent of the 11,000 Medicare COPs surveyed by state agencies in the 

sample during this time period. At the same time, a single serious 

deficiency can limit a hospital’s capability to provide adequate care and 

ensure patient safety and health. Inadequacies in nursing practices or 

deficiencies in a hospital’s physical environment, which includes fire 

safety, are examples of serious deficiencies that could endanger multiple 

patients. 

The potential of JCAHO’s new hospital accreditation process to improve 

the detection of serious deficiencies over the pre-2004 process is unknown 

because the process was just implemented in January 2004. JCAHO plans 

to move from announced to unannounced surveys in 2006, which would 

afford JCAHO the opportunity to observe hospitals’ operations when the 

hospitals have not prepared in advance to be surveyed. In addition, the 

pilot test of the new accreditation process was of limited value in 

predicting whether it will be an improvement over the pre-2004 process in 

detecting deficiencies. Limitations in the pilot test included that hospitals 

Results in Brief 
Page 5 GAO-04-850 Medicare Patient Safety in Hospitals 

participating in the pilot were not randomly selected and that JCAHO 

evaluated the results instead of an independent entity. 

CMS has limited oversight authority over JCAHO’s hospital accreditation 

program because the program’s unique legal status effectively prevents 

CMS from taking actions, such as requiring JCAHO’s hospital accreditation 

program to submit to a direct review process or placing the program on 

probation while monitoring its performance, that it has the authority to 

take with other health care accreditation programs to ensure satisfactory 

performance. Furthermore, CMS’s existing oversight of JCAHO’s hospital 

accreditation program needs improvement. Although CMS officials said 

that validation surveys are conducted to assure Congress that JCAHO’s 

accreditation process provides a reasonable assurance that hospitals 

comply with Medicare requirements, there are limitations to the agency’s 

validation survey program. CMS has no formal written protocol for 

selecting the hospitals to include in the state agency validation survey 

sample; relies on a measure—the rate of disparity—that provides limited 

information and could mask problems with an accreditation program’s 

performance in detecting serious deficiencies; uses statistical methods 

that are insufficient to accurately portray JCAHO’s performance; and has 

reduced the percentage of validation surveys from 5 percent to 

approximately 1 percent of JCAHO-accredited hospitals, which provides 

less reliable information on the performance of JCAHO’s hospital 

accreditation program. 

We suggest that Congress consider giving CMS the same oversight 

authority over JCAHO’s hospital accreditation program that CMS has for 

all other health care accreditation programs. To improve CMS’s 

assessment of JCAHO’s hospital accreditation process, we recommend 

that CMS modify the measure it uses to indicate how well an accreditation 

program detects serious deficiencies in Medicare COPs; maximize the 

extent to which validation survey findings can be generalized to all 

JCAHO-accredited hospitals and include its survey protocol in its annual 

reports to Congress; and annually conduct validation surveys on a sample 

of JCAHO-accredited hospitals that is equal to at least 5 percent of all 

JCAHO-accredited hospitals. 

CMS and JCAHO commented on a draft of this report. In its comments, 

CMS concurred with our findings and recommendations. JCAHO stated 

that it did not object to our matter for congressional consideration that 

CMS be given the same oversight authority over JCAHO’s hospital 

accreditation program that it has over other health care accreditation 

programs. JCAHO took issue with our methodology, which it said was 
Page 6 GAO-04-850 Medicare Patient Safety in Hospitals 

incomplete and did not comprehensively assess the performance of 

JCAHO’s hospital accreditation program. Our review was not intended to 

be a comprehensive evaluation of JCAHO’s hospital accreditation 

program. Rather, we focused on the ability of JCAHO’s hospital 

accreditation program to ensure that hospitals that accept Medicare 

patients comply with Medicare COPs. In the same vein, JCAHO stated that 

the report does not sufficiently recognize JCAHO’s identification of 

deficiencies in its surveys that may be corrected before state surveyors 

arrive. We added language to the report to emphasize that our focus was 

on the serious deficiencies state survey agencies found that JCAHO did 

not because these serious deficiencies demonstrate the importance of 

CMS oversight of the hospital accreditation process. JCAHO also stated 

that we misrepresented the potential of its new accreditation process to 

detect deficiencies in Medicare COPs and provided new data for the first 

quarter of 2004 that indicate that 2004 JCAHO surveys may have detected a 

greater percentage of deficiencies related to patient care compared with 

the pre-2004 accreditation process. However, we maintain that until CMS 

validation surveys for 2004 are completed, there is no basis on which to 

determine whether the new process improves the detection of 

noncompliance with Medicare COPs. CMS and JCAHO also provided 

technical comments on the report, which we incorporated as appropriate. 

To participate in Medicare, hospitals must maintain standards of patient 

safety and health that comply with Medicare COPs. For example, the COP 

related to nursing services includes such requirements for hospitals as 

providing a 24-hour nursing service that is supervised or furnished by a 

registered nurse. There are currently 23 Medicare COPs.5

 (See app. II for a 

description of the 23 Medicare COPs.) CMS proposed revisions to all of the 

COPs in 1997, but it did not finalize them. Since then, CMS has revised 

several of the COPs, including those concerning the life safety code; 

quality assessment and performance improvement; organ, tissue, and eye 

donations; and nurse anesthetist supervision. 

Health care accreditation programs other than JCAHO’s hospital 

accreditation program may generally adopt their own requirements if CMS 

determines that an accreditation program’s requirements are at least 

 5

One of the 23 COPs cannot be deemed by an accreditation organization. CMS relies on 

organizations other than the accreditation organizations to certify that hospitals comply 

with the COP that requires hospitals to establish a utilization review plan for services 

provided to Medicare beneficiaries. 

Background 
Page 7 GAO-04-850 Medicare Patient Safety in Hospitals 

equivalent to Medicare COPs.6 If CMS also determines, among other things, 

that the accreditation program’s survey process is likely to identify any 

serious deficiencies in COPs, it must generally grant “deeming authority” 

to the accreditation program and treat entities accredited by these 

organizations as meeting Medicare COPs. CMS has the authority to review 

these programs, and it can impose a probationary period while monitoring 

performance and remove deeming authority if warranted. 

Most hospitals demonstrate compliance with standards equivalent to 

Medicare COPs through accreditation by JCAHO.7

 In 2002, JCAHO 

accredited 4,211, or 82 percent, of Medicare-participating hospitals.8

Hospitals accredited by JCAHO received payments for Medicare-covered 

inpatient services of approximately $98 billion, or 90 percent, of the $109 

billion that was spent on hospital care in 2002. JCAHO, as part of its 

accreditation-related activities, also develops survey procedures, trains its 

surveyors, and formulates performance measures. JCAHO is governed by a 

29-member board of commissioners and has a staff of over 1,000.9

JCAHO’s deeming authority for hospitals is established in statute and 

therefore can only be changed by Congress. As a result of this unique 

statutory authority, hospitals accredited by JCAHObecause they meet 

 6

Specifically, the agency’s regulations require the accreditation organization’s standards to 

be at least as stringent as the Medicare COPs, when taken as a whole. See 42 C.F.R. § 

488.6(a) (2003). 

7

Forty-nine states allow JCAHO hospital accreditation as a full or partial substitute for 

meeting health care quality standards and other requirements for state licensure. 

8

The remaining 18 percent of hospitals choose to be accredited by the American 

Osteopathic Association (AOA) or to be certified by state survey and certification agencies. 

9

The board includes seven members chosen by the American Hospital Association, seven 

chosen by the American Medical Association, three chosen by the American College of 

Physicians—American Society of Internal Medicine, three chosen by the American College 

of Surgeons, and one chosen by the American Dental Association. In addition, the board 

consists of a nurse-at-large and six public members. The president of JCAHO is an ex 

officio member of the board. 

JCAHO 
Page 8 GAO-04-850 Medicare Patient Safety in Hospitals 

JCAHO standardsare deemed to meet Medicare COPs as well.10 In 

contrast, the American Osteopathic Association (AOA)—a private, not-forprofit professional organization that offers accreditation services for 

hospitals and other health care organizations—holds deeming authority 

that is subject to CMS’s direct review and approval.11 While hospital 

accreditation is its largest program, JCAHO also has accreditation 

authority under Medicare for certain other health care providers, including 

clinical laboratories, hospices, ambulatory surgical centers, and home 

health care agencies. All of these other JCAHO accreditation programs are 

subject to CMS’s direct review and approval. 

To be accredited by JCAHO, a hospital must meet eligibility requirements, 

satisfactorily complete a triennial on-site survey process, and continue to 

maintain JCAHO’s standards between surveys. The accreditation surveys 

that JCAHO conducts every 3 years are particularly important. For most 

hospitals, the triennial survey is the only time that JCAHO conducts an onsite review of the hospital’s compliance with all quality standards and 

issues decisions on how well the hospital has complied with JCAHO’s 

standards. In 2004, JCAHO implemented a new hospital accreditation 

survey process, which, according to JCAHO, is intended to reduce the cost 

of accreditation to health care organizations and JCAHO, enhance public 

confidence that health care organizations are in continuous compliance 

with standards, increase the real and perceived value of accredited 

organizations, meet the requirements of deeming authorities and 

purchasers, and improve satisfaction for hospitals participating in the 

accreditation program. 

 10When Congress first established JCAHO’s deeming authority in 1965, it prohibited federal 

authorities from issuing standards on patient health and safety for hospitals higher than 

comparable requirements for hospital accreditation by JCAHO in deference to the 

expertise of professional accreditation organizations sponsored by medical and hospital 

associations. See Pub. L. No. 89-97, § 102(a), 79 Stat. 286, 315 (1965). Subsequent legislation 

removed the prohibition and required JCAHO to demonstrate that its standards were at 

least equivalent to any such higher standards issued by the Secretary in order to have 

deeming authority in that area. See Pub. L. No. 92-603, § 244(c), 86 Stat. 1329, 1423 (1972). 

11AOA solely accredits approximately 2 percent of hospitals and JCAHO and AOA jointly 

accredit less than 1 percent of hospitals. While JCAHO and AOA are currently the only 

hospital accrediting organizations, federal law permits CMS to approve any other national 

accreditation body that demonstrates that Medicare requirements will be met by hospitals 

it accredits. 
Page 9 GAO-04-850 Medicare Patient Safety in Hospitals 

CMS exercises oversight of JCAHO’s hospital accreditation program 

primarily through its validation surveys and annual reports to Congress. 

Under federal law, CMS must continually study the operation and 

administration of Medicare, including validating the JCAHO hospital 

accreditation process, and submit annual reports to Congress.12 CMS has 

agreements with state agencies to conduct validation surveys. There are 

different kinds of validation surveys, including traditional validation 

surveyssurveys conducted on a sample of hospitals within 60 days of 

their triennial JCAHO survey. 13 Traditional validation surveys provide the 

basis for assessing the effectiveness of JCAHO’s hospital accreditation 

process in detecting deficiencies in Medicare COPs, which JCAHOaccredited hospitals are treated as meeting. Validation surveys also 

include 18-month surveys, which monitor how well JCAHO-accredited 

hospitals are complying with Medicare COPs midway between their 3-year 

JCAHO surveys, and allegation surveys, which are triggered by complaints 

or other reports of situations that pose potential threats to patient health 

and safety in JCAHO-accredited hospitals. CMS has the authority to 

remove the deemed status of a JCAHO-accredited hospital where a state 

agency’s validation survey results in a finding that the hospital is out of 

compliance with one or more Medicare COP. 

CMS uses a rate of disparity measure to summarize the extent to which an 

accreditation program, such as JCAHO’s hospital accreditation program, 

has not found serious deficiencies identified by CMS through state agency 

validation surveys. For a hospital accreditation program, using the results 

from validation surveys, the rate of disparity for hospitals surveyed by the 

state survey agencies is calculated as the difference between the number 

of hospitals found with serious deficiencies by state agencies and the 

number of hospitals found with comparable deficiencies by the 

accreditation program, divided by the number of hospitals sampled. CMS 

regulations provide that if the validation survey results for an 

accreditation organization with deeming authority indicate a rate of 

disparity that reaches the threshold level of 20 percent disparity or greater, 

CMS will notify the organization that its deeming authority may be in 

jeopardy and that the agency is initiating a deeming authority review.14

With respect to JCAHO, CMS includes the rate of disparity in its annual 

 12See 42 U.S.C. § 1395ll(b). 

13For this report, we will refer to traditional validation surveys as validation surveys. 

1442 C.F.R. 488.8(e). 

CMS Oversight of JCAHO 
Page 10 GAO-04-850 Medicare Patient Safety in Hospitals 

reports to Congress in which it reports the results of its validation program 

for JCAHO’s hospital accreditation program. 

JCAHO’s pre-2004 hospital accreditation process often did not identify 

either hospitals with serious deficiencies or the individual serious 

deficiencies found by state survey agencies through CMS’s validation 

program. In a sample of 500 JCAHO-accredited hospitals, state agency 

validation surveys conducted in fiscal years 2000 through 2002 identified 

31 percent (157 hospitals) with serious deficiencies; of these, JCAHO did 

not identify 78 percent (123 hospitals) as having serious deficiencies. For 

the same validation survey sample, the majority of the serious deficiencies 

state survey agencies identified but JCAHO did not were in the physical 

environment COP category, which covers fire safety and prevention. 

From fiscal years 2000 through 2002, JCAHO did not identify 123 of the 157 

hospitals (78 percent) with serious deficiencies that CMS’s validation 

program identified out of a sample of 500 JCAHO-accredited hospitals. 

Table 1 shows the hospitals with serious deficiencies that state survey 

agencies identified and JCAHO did not during fiscal years 2000 through 

2002. In 343 of the 500 hospital validation surveys, state agency surveyors 

did not find serious deficiencies. Both state agency surveyors and JCAHO 

surveyors identified 34 hospitals as having a serious deficiency. 

JCAHO’s Pre-2004 

Hospital 

Accreditation Process 

Often Did Not Detect 

Serious Deficiencies 

Found by State 

Survey Agencies 

JCAHO Did Not Identify 

Three-Quarters of the 

Hospitals That State 

Survey Agencies Found to 

Have Serious Deficiencies 
Page 11 GAO-04-850 Medicare Patient Safety in Hospitals 

Table 1: Hospitals in CMS’s Validation Survey Sample with Serious Deficiencies 

that State Survey Agencies Identified but JCAHO Surveyors Did Not, Fiscal Years 

2000-2002 

 

Hospitals state survey 

agencies found to have 

serious deficiencies 

Hospitals with serious 

deficiencies identified by 

state survey agencies but 

not identified by JCAHOa

Fiscal 

year 

Number of 

hospitals 

in CMS’s 

validation 

sample Number Percent Number Percent 

2000 184 61 33 49 80

2001 204 61 30 49 80

2002 112 35 31 25 71

Total 500 157 31 123 78

Source: GAO analysis of CMS data. 

Note: Hospitals with serious deficiencies are defined as those not meeting one or more of the 

Medicare COPs. From fiscal year 2000 through 2002, JCAHO surveyed 4,666 hospitals for 

accreditation. 

a

Determined by CMS through its matching of deficient COPs found by state agency surveyors to 

JCAHO surveyors’ findings of JCAHO standards out of compliance. 

According to JCAHO, disparity between state agency and JCAHO findings 

in the 123 hospitals in part may be attributed to the timing of the two 

surveys, JCAHO’s phasing in of new requirements, different interpretations 

of the COPs by state surveyors, and inherent surveyor bias. However, in 

its comparison to determine disparity between the two surveys, CMS does 

consider whether it is reasonable to conclude that the deficiencies found 

by state survey agencies existed at the time JCAHO surveyed the hospital. 

From fiscal year 2000 through 2002, JCAHO did not detect 167 of the 241 

serious deficiencies (69 percent) identified through CMS’s validation 

program from a sample of 500 JCAHO-accredited hospitals. The number of 

serious deficiencies found by CMS’s validation program represents 2 

percent of the 11,000 Medicare COPs surveyed by state agencies in the 

sample and were found in 157 hospitals. However, one serious deficiency 

in any one of these hospitals could limit its ability to provide adequate care 

to its patients. For example, a serious deficiency in the nursing services 

COP at a hospital in Texas found by a state agency but missed by JCAHO 

in 2000 included such problems as failure to prepare and administer drugs 

in accordance with federal and state laws, inadequate supervision and 

JCAHO Did Not Detect 

Two-Thirds of the Serious 

Deficiencies Identified by 

State Survey Agencies 
Page 12 GAO-04-850 Medicare Patient Safety in Hospitals 

evaluation of the clinical activities of nonemployee nursing personnel, and 

nursing care and procedures provided to patients that were not within the 

scope of accepted standards of practice. Among hospitals with serious 

deficiencies identified by CMS’s validation program but not by JCAHO, 

there were on average 1.1 serious deficiencies per hospital, with a range 

from 1 to 6. Table 2 shows the percentage of serious deficiencies identified 

by CMS’s validation program but not by JCAHO for fiscal years 2000 

through 2002. 

Table 2: Percentage of Serious Deficiencies Identified by State Survey Agencies but 

Not by JCAHO Surveyors in CMS’s Validation Survey Sample, Fiscal Years 2000-

2002 

 

Serious deficiencies identified 

by state survey agencies but 

not by JCAHOa

Fiscal 

year 

Number of serious 

deficiencies 

identified by state 

survey agencies 

Number of serious 

deficiencies 

identified by 

JCAHO Number Percent

2000 82 12 70 85

2001 103 40 63 61

2002 56 22 34 61

Total 241 74 167 69

Source: GAO analysis of CMS data. 

Note: Hospitals with serious deficiencies are defined as those not meeting one or more of the 

Medicare COPs. 

a

Determined by CMS through its matching of deficient COPs found by state agency surveyors to 

JCAHO surveyors’ findings of JCAHO standards out of compliance. 

Of the 167 serious deficiencies identified by CMS’s validation program 

from fiscal year 2000 through 2002 but not detected by JCAHO, 87 were 

related to a hospital’s physical environment, which includes life safety 

code standards on fire prevention and safety.15 For these 3 years, JCAHO 

did not detect 81 percent of the serious physical environment deficiencies 

identified by state agency surveyors. Table 3 shows the number of serious 

deficiencies, by category, identified by state survey agencies in CMS’s 

 15Between fiscal years 2000 and 2002, JCAHO used more recent life safety code standards 

than state survey agencies performing validation surveys. CMS stated that these differences 

could account for some of the disparate findings between JCAHO’s surveys and state 

agency validation surveys. However, CMS considered these different standards in 

determining whether JCAHO had not detected serious deficiencies in the life safety code. 
Page 13 GAO-04-850 Medicare Patient Safety in Hospitals 

validation program but missed by JCAHO surveyors. The larger number of 

deficiencies in physical environment may be related to the difference in 

how state agencies generally survey separately a hospital’s compliance 

with the life safety code portion of the physical environment COP. JCAHO 

surveys assess compliance with the life safety code using a combination of 

the hospital’s self-assessment, a hospital building tour, and observations 

made by all surveyors during the survey process. Examples of deficiencies 

in physical environment that JCAHO did not identify but CMS’s validation 

program found in a hospital in Alabama in 2000 included the following: 

several exterior exits lacked emergency exit lighting; several exterior exits 

were illuminated only by single light bulbs; fire alarm system and fire 

extinguishers had not been inspected annually as required; and an 

automatic sprinkler system had not been inspected annually and 

maintained by certified personnel as required. Serious deficiencies in the 

COP on physical environment compromise patient safety and health. 
Page 14 GAO-04-850 Medicare Patient Safety in Hospitals 

Table 3: Number of Serious Deficiencies, by COP, Identified by State Survey 

Agencies but Not by JCAHO Surveyors in CMS’s Validation Survey Sample, Fiscal 

Years 2000-2002 

COP 

Number of serious 

deficiencies identified 

by state survey 

agencies 

Number of serious 

deficiencies identified 

by state survey 

agencies but not by 

JCAHOa

Physical environment 107 87

Quality of care 

Anesthesia services 3 2

Discharge planning 2 2

Emergency services 2 2

Food and dietetic services 5 4

Governing body 16 7

Infection control 15 9

Laboratory services 1 1

Medical record services 7 4

Medical staff 10 1

Nursing services 17 10

Organ, tissue, and eye procurement 5 5

Outpatient services 1 1

Patients’ rights 10 9

Pharmaceutical services 14 9

Quality assurance 18 8

Radiologic services 1 0

Rehabilitation services 1 1

Respiratory care services 1 1

Surgical services 5 4

Total quality-of-care COPs 134 80

Source: GAO analysis of CMS data. 

Note: Neither state survey agencies nor JCAHO identified serious deficiencies in two of the 

categoriescompliance with laws and nuclear medicine serviceswhich are not included in this 

table. 

a

Determined by CMS through its matching of deficient COPs found by state agency surveyors to 

JCAHO surveyors’ findings of JCAHO standards out of compliance. 

The total number of deficiencies not identified by JCAHO in the quality-ofcare COP categories—those COPs that involve the oversight and delivery 

of patient care—is similar to the number not identified by JCAHO in the 
Page 15 GAO-04-850 Medicare Patient Safety in Hospitals 

physical environment COP. While the number of serious deficiencies not 

found by JCAHO in individual quality-of-care COP categories is smaller 

than the number not found in physical environment, when these quality-ofcare COPs are combined, the proportion of serious deficiencies JCAHO 

missed is almost 60 percent of the total number of serious deficiencies 

identified by state survey agencies. The following are examples of 

hospitals found to be out of compliance with multiple quality-of-care 

COPs: 

• In 2000, CMS removed the deemed status as a Medicare provider of a 

JCAHO-accredited hospital in California for failure to comply with two 

COPs, one of which was infection control. The hospital failed to provide a 

sanitary environment to avoid sources and transmission of infections and 

communicable diseases and failed to develop a system for ensuring the 

sterilization of medical instruments. 

• Also in 2000, CMS notified a hospital in Texas that if it did not implement a 

plan of correction the hospital’s participation in the Medicare program 

would be terminated. Serious deficiencies at this hospital included lack of 

compliance with the pharmaceutical services and nursing services COPs 

because medications were administered without physician orders and a 

double dose of narcotics was given in the emergency room, with no 

explanation for the excessive dosage, to a patient who later died. 

State surveyors in CMS’s validation program also may miss serious 

deficiencies. In related work on skilled nursing facilities and home health 

agencies, we found that the number of serious deficiencies found by state 

agencies was highly variable among states and may be understated.16 State 

agencies’ detection of serious deficiencies in hospitals also varied widely 

among states for the 3 years we reviewed. For example, state survey 

agencies in California, Illinois, and Ohio found serious deficiencies in over 

45 percent of the surveys they conducted between fiscal years 2000 

through 2002. In contrast, Florida and New York found serious 

deficiencies in less than 10 percent of the surveys they conducted, and 

Louisiana did not find serious deficiencies in any of the surveys it 

conducted.17

 16U.S. General Accounting Office, Medicare Home Health Agencies: Weaknesses in Federal 

and State Oversight Mask Potential Quality Issues, GAO-02-382 (Washington, D.C.: July 

19, 2002) and U.S. General Accounting Office, Nursing Home Quality: Prevalence of 

Serious Problems, While Declining, Reinforces Importance of Enhanced Oversight, 

GAO-03-561 (Washington, D.C.: July 15, 2003). 

17All six states conducted at least 15 validation surveys from fiscal year 2000 through 2002. 
Page 16 GAO-04-850 Medicare Patient Safety in Hospitals 

The potential of JCAHO’s new hospital accreditation process to improve 

the identification of serious deficiencies is unknown because it is too soon 

after its January 2004 implementation for a meaningful evaluation; in 

addition, JCAHO’s testing of the new process was limited. CMS has not 

had the opportunity to complete its validation program for 2004 to 

determine whether JCAHO surveyors using the new process are missing 

serious deficiencies later identified by state agency validation surveys. 

While unannounced surveys, which are planned for implementation in 

2006, have the potential to improve the detection of serious deficiencies, 

other features of the new process that JCAHO did not test before 

implementation may have limitations that could affect the potential of the 

new process to identify problems with patient care. JCAHO’s pilot test of 

the new process had limitations, including using a sample of hospitals that 

volunteered for the pilot instead of using a random sample and selfevaluating the results instead of using an independent entity. 

Because JCAHO’s new accreditation process was implemented in January 

2004, it is too soon to know whether the new process is better at detecting 

serious deficiencies in Medicare COPs than the pre-2004 accreditation 

process. A JCAHO official told us the new process will aid in the detection 

of deficiencies, but we found that some of the features may have 

shortcomings that could limit their effectiveness. New features of the 

accreditation process include the hospital’s self-assessment of compliance 

with accreditation standards midway through the accreditation cycle, 

surveyor review of the care provided to specific patients to determine the 

adequacy of the hospital’s health care delivery system, and performance of 

all accreditation surveys on an unannounced basis beginning in 2006. (See 

app. III for a description of selected new features of JCAHO’s new hospital 

accreditation process.) 

Periodic performance reviews assess hospital compliance with applicable 

standards and are performed at the 18-month midpoint between 3-year onsite accreditation surveys. According to JCAHO, the periodic performance 

review will have several benefits. These include providing hospitals with a 

process to assess their ongoing compliance and requiring them to correct 

or plan to correct all deficiencies identified. Periodic performance reviews 

must be conducted either by the hospital as a self-assessment or, if the 

hospital chooses, by JCAHO through an on-site review. 

However, periodic performance reviews may not necessarily improve the 

detection of deficiencies. JCAHO did not pilot test these reviews for the 

potential to detect deficiencies and did not test whether hospitals that 

Potential of JCAHO’s 

New Hospital 

Accreditation Process 

Is Unknown, and 

Testing Was Limited 

Potential of New Process 

Is Unknown 

Periodic Performance Review 
Page 17 GAO-04-850 Medicare Patient Safety in Hospitals 

conducted reviews do a better job of continuing to comply with standards. 

In addition, for hospitals performing self-assessments, JCAHO will not 

check these self-assessments to determine whether hospitals fully and 

accurately identified quality problems and developed adequate corrective 

action plans to address the problems identified. 

According to JCAHO, the priority focus process and patient tracer 

methodology together have the potential to enhance the ability of surveys 

to detect deficiencies by directing the attention of surveyors to key patient 

care areas. The priority focus process uses a data-based formula to 

identify a limited number of areas in each hospital that are particularly 

important to patient health and safety. Priority focus areas might include 

infection control, medication management, or patient safety. Surveyors 

use the priority focus process combined with the patient tracer 

methodology to focus their surveys to specific areas for review. The 

patient tracer methodology guides their choice of current patients to 

“trace” through the experience of care within an organization. For 

example, if the hospital’s priority focus process data suggest that a patient 

with an orthopedic-related diagnosis such as a hip fracture should be 

traced, the JCAHO surveyor would review the patient’s medical record, 

noting where the patient had entered into the hospital and any services 

and transfers that occurred. Then the surveyor would retrace the steps in 

the patient’s care process by observing and talking to staff in some of the 

areas in which the patient received care. If the patient entered through the 

emergency department, was transferred to a medical/surgical unit, and 

then went to the operating room, the surveyor would go to these areas to 

interview staff about the care given to this specific patient. With 

information from patient tracers, the surveyor will assess whether any 

compliance issues exist with JCAHO standards. If the surveyor identifies a 

compliance issue while tracing one patient, the surveyor may review the 

records of similar patients to determine whether the problem is isolated or 

represents a pattern of care. 

However, JCAHO did not test the extent to which the priority focus 

process and the patient tracer methodology could help surveyors detect 

deficiencies. A JCAHO official told us these new features of the 

accreditation process were intended to help surveyors trace patients in a 

consistent way and not necessarily to improve the detection of 

deficiencies. 

Priority Focus Process and 

Patient Tracer Methodology 
Page 18 GAO-04-850 Medicare Patient Safety in Hospitals 

JCAHO plans to conduct all hospital accreditation surveys on an 

unannounced basis beginning in 2006.18 JCAHO stated that unannounced 

surveys will ensure that hospital performance is based on the observation 

of hospitals’ routine operations rather than on how they operate after they 

have the opportunity to prepare to be surveyed. A JCAHO official also 

indicated that unannounced surveys will be more likely to detect 

deficiencies. The OIG and other organizations share JCAHO’s position on 

the value of unannounced surveys of hospitals and other health care 

organizations. The value of unannounced surveys also has been 

recognized for nursing homes, which state agencies survey on an 

unannounced basis. 

JCAHO’s pilot test of its new hospital accreditation process was limited 

and therefore unable to help determine the potential of the new process to 

detect deficiencies in Medicare COPs. According to JCAHO, the pilot test 

suggests that the new process was more likely than the former process to 

find quality problems. However, the pilot test sample included hospitals 

that volunteered or were selected by JCAHO and were not randomly 

selected, pilot test surveyors were accompanied by observers from 

JCAHO’s central office, and pilot test results were not independently 

evaluated. In addition, CMS has not completed its fiscal year 2004 

validation program, which will include hospitals surveyed by JCAHO using 

the new process and thus does not yet have sufficient data on which to 

base a meaningful evaluation. 

According to JCAHO’s analysis of the pilot test, the new hospital 

accreditation process is more likely to identify quality problems since 

proportionately more hospitals under the new process received 

unfavorable accreditation decisions. JCAHO based its conclusion on a 

comparison of survey outcomes, called accreditation decisions, between 

18 hospitals in the pilot test conducted in 2002 and 2003 and the 1,524 

hospitals that had been surveyed under the pre-2004 accreditation process 

during 2003. Table 4 presents the data JCAHO used to make the 

comparison. As shown, proportionately fewer hospitals under the new 

process were accredited without having to make corrections. Although 

JCAHO provided the accreditation decision outcomes for these 18 pilot 

 18In 2004 and 2005, JCAHO will continue to conduct its accreditation surveys on an 

announced basis. 

Unannounced Surveys 

JCAHO’s Pilot Test of New 

Process Was Limited 
Page 19 GAO-04-850 Medicare Patient Safety in Hospitals 

tests, it stated it preferred to use the number of “requirements for 

improvement” as the basis for analysis. 

Table 4: Accreditation Decisions for Hospitals Surveyed Under JCAHO’s New 

Survey Process Pilot Test as Compared to Results from JCAHO’s Pre-2004 Survey 

Process 

Pilot test of new survey 

process Pre-2004 survey process 

Accreditation 

decision 

Number of 

hospitals 

surveyed

Percentage of 

hospitals 

surveyed 

 Number of 

hospitals 

surveyed

Percentage of 

hospitals 

surveyed

Accreditation 0 0 320 21

Survey findings 

with requirements 

for improvementa

13 72 1,191 78

Conditional 

accreditation 3 17 13 1

Preliminary denial 

of accreditation 2 11 0 0

Total 18 100 1,524b100

Source: JCAHO. 

Note: JCAHO reported that it conducted pilot tests of the new accreditation process in an additional 

12 hospitals in 2001. However, JCAHO was unable to provide the accreditation decisions for these 12 

pilot site hospitals. 

a

Hospitals in the pilot test with deficiencies were accredited contingent upon evidence of correcting 

deficiencies. The hospitals in the comparison group with deficiencies received accreditation with 

requirements for improvement. 

b

These 1,524 hospitals represent all those surveyed for accreditation by JCAHO during 2003. 

However, JCAHO’s pilot test analysis was limited in three respects, which 

may have accounted for the smaller number of favorable accreditation 

decisions hospitals received under the new process. 

• The hospitals participating in the pilot test were not randomly selected by 

JCAHO. As a result, these hospitals may not be representative of all 

JCAHO-accredited hospitals and therefore results cannot be generalized. 
Page 20 GAO-04-850 Medicare Patient Safety in Hospitals 

• During the pilot test, an observer from JCAHO’s central office 

accompanied each surveyor, and the knowledge that they were being 

observed may have influenced the surveyors’ actions.19 Under the pre-2004 

process, observers only rarely accompanied JCAHO surveyors. 

• JCAHO conducted its own evaluation of pilot test results. Evaluation of 

the pilot test by an entity independent of either JCAHO or the hospitals 

tested could help to ensure that survey outcomes were impartially 

interpreted. For example, CMS used an independent group to evaluate its 

redesign of the nursing home survey process. 

CMS has limited oversight authority over JCAHO’s hospital accreditation 

program, and its existing oversight activities need improvement. The 

unique status of JCAHO’s hospital accreditation program, which is 

specified in statute, does not permit CMS to take corrective action, such as 

restricting or removing its deeming authority. Additionally, CMS uses a 

measure that provides limited information to evaluate the performance of 

JCAHO’s hospital accreditation program, has significantly reduced the 

number of surveys conducted as part of CMS’s validation program, and 

does not use measures that are based on sound statistical methods to 

assess the performance of JCAHO’s hospital accreditation program. 

Because of JCAHO’s unique legal status, CMS’s oversight of JCAHO’s 

hospital accreditation program is limited in two major ways: Unlike other 

accreditation programs with deeming authority, JCAHO does not have to 

reapply to CMS to reauthorize its deeming authority, and CMS cannot take 

action to address performance problems with JCAHO’s hospital 

accreditation program. 

JCAHO’s hospital accreditation program is the only Medicare accreditation 

program for which CMS does not have to conduct an evaluation of the 

accreditation standards and the processes used to conduct surveys. 

Without this evaluation, CMS is deprived of key oversight tools it is 

authorized to use with other accreditation programs: detailed information 

 19For example, we found in our nursing home survey work in 1999 that state surveyors may 

perform their tasks more attentively when they are being observed by federal surveyors 

than they would if performing their surveys unobserved, thus masking a state surveyor’s 

typical performance. U.S. General Accounting Office, Nursing Home Care: Enhanced 

HCFA Oversight of State Programs Would Better Ensure Quality, GAO/HEHS-00-6

(Washington, D.C.: Nov. 4, 1999). 

CMS Oversight 

Authority of JCAHO’s 

Hospital 

Accreditation 

Program Is Limited 

and Needs 

Improvement 

CMS Oversight Authority 

of JCAHO Is Limited 
Page 21 GAO-04-850 Medicare Patient Safety in Hospitals 

about any proposed changes to the accreditation process and public input. 

CMS cannot require JCAHO to provide information about proposed 

changes to its accreditation requirements and hospital survey processes. 

Also, because it is not required to reapply to CMS for deeming authority, 

JCAHO does not have to provide CMS information that other accreditation 

programs must provide, such as a detailed description of its survey 

processes, a comparison of its standards to Medicare requirements, and 

the qualifications of its surveyors, which CMS reviews to ensure that the 

programs comply with Medicare requirements. For example, when 

JCAHO’s hospice accreditation program applied for deeming status in 

1999, CMS required changes to JCAHO’s hospice accreditation process, 

including requiring JCAHO to make unannounced surveys of Medicarecertified hospices. According to a CMS official, JCAHO’s hospital 

accreditation program has provided much of the information required of 

other accreditation organizations; however, CMS has no authority to 

require JCAHO to make changes to the hospital accreditation program as 

it does with other health care accreditation programs. Statutory provisions 

regarding public notice and comment do not apply to JCAHO’s hospital 

accreditation program as they do to other accreditation programs. The 

reapplication process for other accreditation programs requires affording 

the public an opportunity to provide input to CMS on an accreditation 

program’s request for deeming authority. Because JCAHO does not have to 

reapply for deeming authority, the public does not have the opportunity to 

review and comment on JCAHO’s hospital accreditation program.20

A second limitation is CMS’s inability to address any performance issues 

with JCAHO’s hospital accreditation program. Although the rate of 

disparity for JCAHO’s hospital accreditation program exceeded 20 percent 

in fiscal years 2000, 2001, and 2002 a rate that would have triggered a 

deeming authority review for any other Medicare accreditation 

programCMS was unable to take enforcement action to address 

JCAHO’s performance. When other Medicare accreditation programs have 

a rate of disparity of 20 percent or more, CMS can take steps such as 

imposing a year-long probationary period and removing deeming authority 

at the end of the probationary period if the rate of disparity remains at 20 

percent or more. For JCAHO, however, CMS’s actions toward correcting 

the program’s deficiencies are limited to including recommendations for 

 20Whenever CMS considers, approves or removes an accreditation organization’s deeming 

authority, the agency is required to publish detailed notices in the Federal Register, and 

consider public comment. See 42 U.S.C. § 1395bb(b)(3); 42 C.F.R. § 488.8(b) and (f)(7). 
Page 22 GAO-04-850 Medicare Patient Safety in Hospitals 

improvement in its annual reports to Congress and negotiating with 

JCAHO to voluntarily adopt CMS’s recommendations. 

In its annual report to Congress, CMS made recommendations in fiscal 

year 2002 aimed at improving JCAHO’s ability to detect serious 

deficiencies in the life safety code, part of the COP on physical 

environment. CMS noted that JCAHO permits hospitals to self-assess 

compliance with life safety code requirements.21 While CMS stated that it 

did not object to the concept of hospital self-assessment of life safety code 

requirements, it made five recommendations to JCAHO for improving 

implementation: 

1. Require hospitals to use qualified personnel, such as fire marshals and 

architects, to conduct self-assessments of compliance with the life 

safety code requirements. 

2. Set minimum standards for identifying and improving life safety code 

deficiencies identified by hospital self-assessments. 

3. Require hospitals to submit their self-assessments on life safety code 

issues prior to JCAHO conducting accreditation surveys to provide 

surveyors and personnel in JCAHO’s central office time to review the 

material prior to the accreditation surveys. 

4. Increase the use of JCAHO experts in the life safety code requirements 

in its central office. 

5. Address the issue of hospitals that do not make improvement within 

self-determined time frames. 

JCAHO did not adopt all of these recommendations. It disagreed with the 

first recommendation. Its response indicated that its requirement to use 

qualified personnel to complete the self-assessment, while more general, 

was sufficient. It further indicated that policies were in place for CMS’s 

second and fifth recommendations. CMS later agreed that JCAHO’s 

policies do satisfactorily address the fifth recommendation. JCAHO 

planned to examine ways to adopt CMS’s third and fourth 

recommendations. CMS however, had no authority to compel JCAHO to 

 21Beginning in 1995, JCAHO-accredited hospitals have assessed their own compliance with 

the life safety code and developed correction plans, which JCAHO must approve. If 

hospitals are in compliance with their correction plans, JCAHO’s surveyors do not record 

outstanding life safety code deficiencies. 
Page 23 GAO-04-850 Medicare Patient Safety in Hospitals 

comply with the remaining recommendations. According to CMS, it 

continues to discuss implementation of its recommendations with JCAHO. 

JCAHO stated that while its initial response to CMS’s recommendations in 

2003 reflected then current JCAHO policies, subsequent policy evolutions 

are addressing CMS’s recommendations. Specifically, JCAHO is working 

with the American Society of Hospital Engineers to develop a process for 

review by experts of hospital self-assessments on life safety code issues 

prior to JCAHO’s conducting on-site accreditation surveys and to identify 

those hospitals for which engineering expertise should be added to on-site 

surveys. 

CMS states that the goal of its validation program is to provide reasonable 

assurance to Congress that the JCAHO accreditation process ensures 

hospital compliance with Medicare COPs. However, the measure CMS 

uses to evaluate the performance of JCAHO’s hospital accreditation 

program provides limited information and could mask problems with an 

accreditation program’s performance in detecting serious deficiencies, and 

it is based on a target sample size of 1 percent of JCAHO-accredited 

hospitals. In addition, CMS does not report the extent to which its sample 

reflects the performance of the larger population of JCAHO-accredited 

hospitals. 

The rate of disparity between JCAHO’s hospital accreditation survey 

findings and state survey agency findings, as currently calculated by CMS, 

does not fully explain the performance of JCAHO’s hospital accreditation 

program in detecting serious deficiencies. CMS uses this measure in its 

reports to Congress to assess JCAHO’s hospital accreditation program and 

as the basis for making recommendations for improvement. CMS 

calculates the rate of disparity as the difference between the number of 

hospitals found with serious deficiencies by state survey agencies and the 

number of hospitals found with serious deficiencies by the accreditation 

survey, divided by the number of hospitals in the sample. For example, if 

state survey agencies conducted 200 surveys as part of CMS’s validation 

program and found 60 hospitals out of compliance with at least one COP, 

but JCAHO’s survey found that only 22 of the hospitals were out of 

compliance, the rate of disparity would be 19 percent ((60 - 22)/200). 

CMS has established in regulation a rate of disparity of 20 percent or 

greater as the threshold for taking action against an accreditation 

program. According to a CMS official, the use of 20 percent as the 

threshold is not based on empirical evidence but rather on what CMS 

believed Congress would find acceptable. Consequently, the threshold 

CMS’s Validation Program 

Needs Improvement 

Rate of Disparity
Page 24 GAO-04-850 Medicare Patient Safety in Hospitals 

may not be appropriately placed to indicate unacceptable performance by 

a hospital accreditation program. For example, if JCAHO failed to identify 

serious deficiencies in all 14 hospitals that the state agencies identified 

with serious deficiencies from a sample of 79 hospitals, the rate of 

disparity would be a satisfactory 18 percent ((14-0)/79).22

CMS’s rate of disparity measure used in isolation does not consistently 

reflect an accreditation program’s ability to detect serious deficiencies. As 

the number of hospitals with serious deficiencies detected by state survey 

agencies decreases, regardless of JCAHO’s performance in detecting them, 

it is more likely that the rate of disparity will be less than CMS’s 20 percent 

threshold. As a result, the performance of JCAHO’s hospital accreditation 

program is difficult to judge based on this measure alone. For example, if 

state survey agencies performed 200 validation surveys and found 100 

hospitals or 50 percent with serious deficiencies and JCAHO found 30 

hospitals or 30 percent of the hospitals found by state agencies, the rate of 

disparity would be 35 percent ((100-30)/200). However, if the state 

agencies found 50 hospitals, or 25 percent, of the 200 hospitals with 

serious deficiencies and JCAHO found 15 hospitals, or 30 percent of the 

hospitals that the state agencies identified, the rate of disparity would be 

almost 18 percent ((50-15)/200). The percentage of serious deficiencies 

found by state survey agencies and also by JCAHO remained the same in 

both examples, but the rate of disparity was improved significantly by the 

larger number of hospitals without serious deficiencies in the second 

example. This indicates that the rate of disparity does not consistently 

measure the accreditation program’s ability to detect serious deficiencies 

found by state survey agencies. (See table 5.) In addition to the rate of 

disparity, other components, such as the proportion of hospitals with 

serious deficiencies and the total number of serious deficiencies found by 

state agencies but missed by the accreditation program, are important 

indicators of an accreditation program’s overall performance. 

 22The example is based on the analysis of the rate of disparity in American Institutes of 

Research, Measurement and Evaluation of Revised Accredited Hospital Validation and 

Oversight (Washington, D.C.: Nov. 6, 2002). 
Page 25 GAO-04-850 Medicare Patient Safety in Hospitals 

Table 5: Hypothetical Examples of the Effect on the Rate of Disparity of a Decrease 

in the Number of Hospitals with Serious Deficiencies in a Sample of 200 Hospitals 

 Example 1 Example 2 

State agencies JCAHO State agencies JCAHO

Number of hospitals 

with serious 

deficiencies 100 30 50 15

Percentage of 

hospitals state 

agencies found with 

serious deficiencies 

that were also found 

by JCAHO 30% 30%

Percentage of hospitals 

without serious 

deficiencies identified 

by state agencies 50% 75%

Rate of disparity 35% ((100-30)/200) 18% ((50-15)/200)

Performance level Above threshold Below threshold

Source: GAO. 

Note: CMS’s rate of disparity threshold is 20 percent. 

CMS does not analyze the statistical results of its validation survey 

samples in ways that would allow it to better assess JCAHO’s ability to 

detect serious deficiencies. CMS has not documented the methods it uses 

to select hospitals for validation surveys and did not supply us with clear 

technical justification for the methods used. Further, CMS’s validation 

sample includes hospitals that, because of its sampling method, have 

varying chances of selection, but it does not take this into account when 

calculating statistics based on the sample. According to CMS’s sampling 

method, the selection of hospitals is influenced by factors such as the 

month in the fiscal year that JCAHO performed the accreditation survey 

and how many hospitals were targeted for completion that year in the 

state in which the hospital was located. Thus, some hospitals have a 

greater chance of selection than others. CMS also does not take these 

different chances of selection into account when calculating statistics for 

its annual reports to Congress, which prevents CMS from accurately 

assessing JCAHO’s performance. Moreover, CMS does not measure and 

report in its annual reports the extent to which its estimates based on the 

Statistical Analysis of 

Validation Survey Sample 
Page 26 GAO-04-850 Medicare Patient Safety in Hospitals 

validation survey sample are likely to reflect how well JCAHO detects 

deficiencies in the larger population of hospitals it accredits.23

In addition, the number of usable traditional validation surveys completed 

is smaller than the number of hospitals CMS samples for validation 

surveys. This difference may affect the accuracy of the data that CMS 

presents to Congress if the hospitals where the traditional surveys were 

completed produce different results than those where surveys are not 

completed or are not usable. During its sampling process, CMS selects a 

sample size close to the targeted number of hospitals each year. Some 

hospitals from this sample may be excluded because CMS chose to 

perform another type of survey for them that cannot be used to validate a 

JCAHO accreditation survey. In addition, state agencies are not always 

able to complete the requested traditional validation surveys within 60 

days from the JCAHO accreditation survey, as required, or a hospital may 

be excluded because it lost its deemed status or closed. The size of the 

difference between the number of hospitals sampled and the number of 

usable traditional validation surveys completed therefore varies, as it did 

during the 3-year review period (see table 6). 

Table 6: Number of Hospitals Targeted for Validation Surveys Compared with 

Usable Traditional Validation Surveys Completed 

Fiscal Year 

Hospitals targeted 

for validation 

surveysa

Hospitals sampled for 

validation surveysb

Usable traditional 

validation surveys 

completedc

2000 236 236 184

2001 227 217 204

2002 227 235 112

Source: CMS. 

a

The targeted number is set at the beginning of the fiscal year and is used for planning and resource 

allocation by CMS and the state survey agencies. 

b

The sampled hospitals are the hospitals selected for validation surveys during the year. 

c

Usable surveys exclude those not completed, those completed after the required 60-day time frame, 

and other types of surveys that can not be used to validate a JCAHO accreditation survey. 

 23For example, CMS does not measure and report the precision of the estimates from the 

sample of validation surveys through the use of confidence intervals or margins of error, 

which define the range of estimates that sample results would yield given different random 

samples for a specified level of certainty. 
Page 27 GAO-04-850 Medicare Patient Safety in Hospitals 

CMS reduced the number of validation surveys conducted by state 

agencies from a target of approximately 5 percent of the total number of 

hospitals that JCAHO accredits to a target of approximately 1 percent, 

with at least one survey in each state. Reducing the target of validation 

surveys from 5 percent to 1 percent results in the number of validation 

surveys being reduced from 227 in fiscal year 2002 to a target of 75 

validation surveys in fiscal year 2003 and 72 in fiscal year 2004. 

Reducing the targeted number of validation surveys to 1 percent provides 

less reliable information on how well JCAHO’s hospital accreditation 

program ensures compliance with Medicare COPs. For example, for a 5-

percent target, the estimate of the proportion of JCAHO-accredited 

hospitals with a particular deficiency that is derived from the validation 

survey could be as much as 6.0 percentage points higher or lower, for a 

range of 12.0 percentage points. If the 5-percent target produced an 

estimate that 50 percent of JCAHO-accredited hospitals had a particular 

deficiency, the percentage of JCAHO-accredited hospitals not complying 

could range from 44.0 to 56.0 percent. However, for a 1-percent target the 

estimate could be 11.4 percentage points higher or lower, for a range of 

about 22.8 percentage points. For example, if the 1-percent target 

produced an estimate that 50 percent of JCAHO-accredited hospitals had a 

particular deficiency, the percentage of JCAHO-accredited hospitals not 

complying with a Medicare COP could range from 38.6 to 61.4 percent.24

This reduction in the number of validation surveys is of additional concern 

because it coincides with the implementation of JCAHO’s new 

accreditation process, which has an unproven capacity to detect 

deficiencies. CMS’s target sample size for traditional validation surveys for 

fiscal year 2004 will be further reduced because the sample also includes 

18-month validation surveys. In 2004, CMS is planning to conduct 17 of 

these 18-month surveys as part of its overall validation survey target of 72. 

Thus, CMS could be using as few as 55 validation surveys to determine 

JCAHO’s performance. 

 24These estimates were developed assuming that the validation surveys are conducted on a 

simple random sample of JCAHO-accredited hospitals and a 95 percent confidence level. 

Annual Number of Validation 

Surveys 
Page 28 GAO-04-850 Medicare Patient Safety in Hospitals 

For 3 consecutive years, JCAHO’s hospital accreditation program, which 

accredits most of the hospitals participating in Medicare, exceeded CMS’s 

threshold for unacceptable performance. CMS validation surveys during 

that time period confirmed that JCAHO missed the majority of serious 

deficiencies found by state survey agencies. Yet, CMS was unable to take 

action against JCAHO’s hospital accreditation program as it can with other 

accreditation programs because it lacked the authority to do so. Although 

CMS has recommended in its annual reports to Congress that JCAHO 

make changes in its hospital accreditation program to improve its ability 

to detect serious deficiencies, some of these recommendations have not 

been implemented. Thus, it is vital for patient safety that JCAHO hospital 

accreditation surveys detect existing serious deficiencies and deny 

accreditation to hospitals that do not comply with Medicare COPs. 

CMS is unable to present to Congress an adequate assessment of JCAHO’s 

performance because of limitations in its process for selecting hospitals 

for validation surveys and analysis of the survey results. CMS does not 

consistently portray the extent to which serious deficiencies are missed 

and does not identify the limitations in reporting the estimates it makes 

from its survey sample. CMS cannot assure Congress that JCAHOaccredited hospitals meet Medicare COPs because the measure for the 

rate of disparity, which determines poor performance, allows JCAHO to 

miss the majority of serious deficiencies and still be in an acceptable range 

of performance. Further, CMS’s reduction in the number of validation 

surveys it uses to determine the performance of JCAHO’s hospital 

accreditation program will provide less reliable information at a time 

when JCAHO is implementing a new hospital accreditation process that is 

unproven in its ability to detect serious deficiencies. In light of these 

limitations in CMS’s validation of JCAHO’s hospital accreditation program, 

we believe that CMS must improve its oversight so it can provide Congress 

with more accurate information regarding JCAHO’s performance. 

Given the serious limitations in JCAHO’s hospital accreditation program 

and that efforts to improve this program through informal action by CMS 

have not led to necessary improvements, Congress should consider giving 

CMS the same kind of authority over JCAHO’s hospital accreditation 

program that it has over all other Medicare accreditation programs. 

Conclusions 

Matter for 

Congressional 

Consideration 
Page 29 GAO-04-850 Medicare Patient Safety in Hospitals 

To strengthen the ability of CMS to identify and report to Congress on 

JCAHO’s ability to ensure that the hospitals it accredits protect the safety 

and health of patients through compliance with the Medicare COPs, we 

recommend that the Administrator of CMS take the following three 

actions: 

• modify the method used to measure the rate of disparity between 

validation survey findings and accreditation program findings to provide a 

reasonable assurance that Medicare COPs are being met and consider 

whether additional measures are needed to accurately reflect an 

accreditation program’s ability to detect deficiencies in Medicare COPs; 

• provide in the annual report to Congress an estimate, based on the 

validation survey sample, of the performance of all JCAHO-accredited 

hospitals, including the limitations and protocols for these estimates based 

on generally accepted sampling and statistical methodologies; and develop 

a written protocol for these calculations; and 

• annually conduct traditional validation surveys on a sample of JCAHOaccredited hospitals that is equal to at least 5 percent of all JCAHOaccredited hospitals. 

CMS and JCAHO commented on a draft of this report. In its comments, 

CMS concurred with our recommendations. JCAHO stated it had no 

objection to our suggestion that Congress give CMS the same authority 

over its hospital accreditation program as it does over other Medicare 

accreditation programs. However, JCAHO took issue with the 

methodology we used for evaluating the performance of its hospital 

accreditation program. CMS’s and JCAHO’s specific comments and our 

response follow. CMS’s comments are reprinted in appendix IV and 

JCAHO’s comments are reprinted in appendix V. CMS and JCAHO also 

provided technical comments, which we incorporated as appropriate. 

CMS stated that it has begun to examine the need for additional or 

alternative measures for the rate of disparity calculation. CMS also stated 

it will seek additional resources to further develop and implement new 

sampling and statistical methodologies that may allow results to be 

projected to all JCAHO-accredited hospitals, and to increase the validation 

sample size. CMS specifically noted that it considers life-safety code 

compliance, on the part of all provider types, to be critically important. In 

the past 8 years, in its annual reports to Congress and its dialogues with 

JCAHO regarding its hospital accreditation program, it has identified 

physical environment as an important area where JCAHO needs to focus 

attention, and CMS noted that 68 percent of facilities that had a deficiency 

Recommendations for 

Executive Action 

Agency and Other 

External Comments 

and Our Evaluation 
Page 30 GAO-04-850 Medicare Patient Safety in Hospitals 

finding not identified by JCAHO had them in the physical environment 

area. 

JCAHO stated that our methodology for evaluating the performance of its 

hospital accreditation program was incomplete and did not provide a 

comprehensive assessment of its program’s performance. We did not 

intend to do a comprehensive evaluation of JCAHO’s overall hospital 

accreditation program. Rather, we focused our evaluation on how well 

JCAHO’s hospital accreditation program ensures hospitals’ compliance 

with Medicare participation requirements. There are four possible 

outcomes to a comparison between JCAHO’s accreditation survey and a 

state validation survey: (1) both JCAHO and state agencies identify no 

deficiencies, (2) JCAHO identifies deficiencies not found by state agencies, 

(3) both JCAHO and state agencies identify the same deficiencies, and (4) 

state agencies identify deficiencies that JCAHO does not. We limited our 

evaluation to the fourth outcome because it illustrates the need for CMS 

oversight of the hospital accreditation process. We have clarified the 

scope of our evaluation to emphasize our focus on this outcome. 

JCAHO raised a concern that our characterization of JCAHO’s missed 

deficiencies that state survey agencies found misleads readers to believe 

that JCAHO misses hospitals with deficiencies 78 percent of the time. We 

have revised language in the report to further emphasize that the missed 

deficiency rate applies to hospitals in the validation survey sample in 

which the state survey agencies found deficiencies and cannot be 

generalized to all JCAHO-accredited hospitals. JCAHO further stated that 

our report does not take into account that JCAHO’s hospital accreditation 

program detects deficiencies in hospitals that CMS does not find. 

However, it is to be expected that state survey agencies will not find all 

deficiencies found by JCAHO because hospitals may have corrected the 

deficiencies prior to the state agency surveys. 

JCAHO stated that we misrepresented the potential of the new 

accreditation process in detecting deficiencies in Medicare COPs and 

provided new data regarding its first quarter 2004 performance that 

indicate that JCAHO surveys may have detected a greater percentage of 

deficiencies related to patient care compared with the pre-2004 

accreditation process. However, we maintain that until CMS validation 

surveys for 2004 are completed, there is no basis on which to determine 

whether the new process improves the detection of deficiencies in 

Medicare COPs. In addition, JCAHO stated and we agree that evaluating 

and improving the quality of care in hospitals is not about counting 

deficiencies, it is about finding those deficiencies that, if not fixed, will 
Page 31 GAO-04-850 Medicare Patient Safety in Hospitals 

generate poor results for patients and making sure that these deficiencies 

are remedied in a timely fashion. 

JCAHO stated that we mischaracterized its response to the five 

recommendations that CMS made in 2002 to improve JCAHO’s ability to 

detect deficiencies in the life safety code and that it is involved in frequent 

and ongoing dialogue with CMS regarding the recommendations and other 

life safety code issues. We have clarified language in the report regarding 

JCAHO’s response to CMS’s recommendations. 

As agreed with your offices, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

date. We will then send copies of this report to the Secretary of Health and 

Human Services and other interested parties. We will also make copies 

available to others upon request. In addition, the report will be available at 

no charge at the GAO Web site at http://www.gao.gov. 

If you or your staffs have any questions about this report, please call me at 

(202) 512- 7119. Another contact and key contributors are listed in 

appendix VI. 

Janet Heinrich 

Director, Health CarePublic Health Issues 
Appendix I: Scope and Methodology 

Page 32 GAO-04-850 Medicare Patient Safety in Hospitals 

We examined the extent to which JCAHO’s pre-2004 survey process 

identified hospitals with deficiencies and individual deficiencies in 

Medicare COPs that were identified by state survey agencies. We chose 

these measures because they reflect performance in detecting and 

correcting serious deficiencies, which according to CMS, substantially 

limit a hospital’s capability to render adequate care and adversely affect 

the health and safety of patients. We reviewed data, provided by CMS, on 

500 traditional validation surveys conducted by state survey agencies 

during fiscal years 2000 through 2002. In these validation surveys, state 

survey agencies documented whether they found serious deficiencies in 

Medicare COPs. CMS compared state survey agency findings with 

JCAHO’s accreditation surveys that identified deficiencies in JCAHO’s 

standards. CMS then determined whether the state survey agencies’ 

findings on serious deficiencies in the 22 Medicare COPs that can be 

deemed were comparable to JCAHO’s findings on deficiencies in JCAHO’s 

standards in the following way. Two CMS experts such as nurses reviewed 

the comparability of serious deficiencies in the quality-of-care conditions 

identified in validation surveys to deficiencies in JCAHO’s accreditation 

standards identified in JCAHO’s hospital accreditation surveys. Two 

experts, such as building engineers, reviewed the comparability of serious 

deficiencies identified in the validation surveys on the condition on 

physical environment. Where there was disagreement, the two experts met 

to resolve their differences. CMS does not have written protocols for 

determining comparability. Experts are expected to use their best 

professional judgment. CMS experts also had to consider whether it is 

reasonable to conclude that the deficiencies existed at the time that 

JCAHO surveyed the hospital. For those deficiencies that CMS determines 

that JCAHO has failed to identify, it met with JCAHO to address disputed 

findings and to consider additional evidence on comparability offered by 

JCAHO. There are four possible outcomes to this comparison of survey 

findings—(1) JCAHO and state agencies both identify no deficiencies, (2) 

JCAHO identifies deficiencies not found by state agencies, (3) JCAHO and 

state agencies both identify the same deficiencies, and (4) state agencies 

identify deficiencies that JCAHO does not—we focused on the fourth 

because it highlights the need for CMS oversight of the hospital 

accreditation program. For the second outcome, there could be two 

reasons for the disparity between JCAHO’s and state survey agencies’ 

findings: hospitals corrected deficiencies identified by JCAHO prior to the 

state agency survey or the state survey agency did not identify a deficiency 

that existed. In addition, not all JCAHO findings are equivalent to 

noncompliance with a Medicare COP. 

Appendix I: Scope and Methodology 
Appendix I: Scope and Methodology 

Page 33 GAO-04-850 Medicare Patient Safety in Hospitals 

From these 500 surveys, we determined the number of hospitals with 

serious deficiencies and the total number of serious deficiencies identified 

by state agencies but that CMS determined were not identified by JCAHO. 

These data include 123 hospitals in which state survey agencies identified 

one or more serious deficiencies and JCAHO did not make comparable 

findings according to CMS. These data also include 167 serious 

deficiencies identified by state agencies but that CMS determined 

comparable findings were not identified by JCAHO. 

For fiscal years 2001 and 2002, we obtained from CMS a comparison 

between the validation surveys conducted by the state survey agencies and 

the accreditation surveys conducted by JCAHO, which identified serious 

deficiencies identified by the state agencies but not by JCAHO as 

determined by CMS. For fiscal year 2000, CMS did not supply its 

determinations of the comparability of findings in validation and 

accreditation surveys for 31 of 82 serious deficiencies. We followed a 

protocol similar to the one used by CMS to determine the comparability of 

the remaining 31 serious deficiencies, which included 29 quality-of-care 

serious deficiencies and 2 physical environment serious deficiencies. Two 

analysts with nursing backgrounds compared the findings and made 

determinations on their comparability based on their professional 

judgment. In cases of disagreement, a third analyst with a background in 

nursing made the determination. 

We did not include 1998 and 1999 data in our analysis because CMS used a 

method that undercounted the number of deficiencies identified by state 

survey agencies but not identified by JCAHO. CMS did not count as 

deficient those cases in which state survey agencies determined that a 

hospital was not meeting the COP on physical environment but JCAHO 

determined that the hospital was in compliance because the hospital was 

following correction plans approved by JCAHO. 

To determine the potential of JCAHO’s new accreditation process in 

improving the detection of deficiencies in Medicare COPs, we reviewed 

material supplied by JCAHO on development and testing of its new 

process and interviewed JCAHO officials about the steps taken to test the 

new process and to analyze results. We also examined the features of the 

new accreditation process by reviewing descriptive material obtained 

from JCAHO and interviewing experts in health care quality. Because the 

new accreditation process was implemented in January 2004, we were 

limited in our ability to determine the effectiveness of the new 

accreditation process because we were not able to perform a comparative 
Appendix I: Scope and Methodology 

Page 34 GAO-04-850 Medicare Patient Safety in Hospitals 

analysis of validation survey and JCAHO survey results under the new 

process. 

To examine the effectiveness of CMS’s oversight of JCAHO’s accreditation 

process, we analyzed the laws and regulations that define CMS’s authority 

and JCAHO’s authority. We reviewed the annual reports submitted to 

Congress on JCAHO’s performance in identifying serious deficiencies and 

reviewed correspondence between CMS and JCAHO and interviewed 

officials in both organizations. We analyzed the rate of disparity that CMS 

uses to determine the performance of JCAHO’s hospital accreditation 

process in identifying deficiencies in Medicare COPs. 

To evaluate CMS’s statistical methodology for the validation surveys, we 

interviewed CMS officials about the sampling and statistical methods. In 

the absence of written methodological documentation, we relied on 

information provided by CMS officials to evaluate the methodology. They 

gave us the following information about their sampling method. At the 

beginning of each year, CMS determines a target for the number of 

hospitals that will be sampled for validation surveys in each state. Each 

month, CMS receives a list of hospitals scheduled for a JCAHO 

accreditation survey in that month. Prior to sampling, CMS removes from 

the list those hospitals that have received a validation survey in the last 3-

year accreditation cycle and hospitals that do not participate in Medicare. 

In the first month of the year, CMS selects a random sample of hospitals to 

be surveyed from JCAHO’s list. In subsequent months, CMS removes 

hospitals in states in which the state target has been met and then selects a 

random sample of hospitals. Prior to sending the list to state survey 

agencies, CMS determines which hospitals will receive traditional 

validation surveys and which will receive other types of surveys that 

cannot be used to assess the performance of JCAHO’s hospital 

accreditation program. State survey agencies must then complete 

traditional validation surveys within 60 days of the completion of JCAHO’s 

accreditation survey for the results to be used by CMS to measure the 

performance of JCAHO’s hospital accreditation program. According to 

CMS officials, the sampling procedures CMS uses are necessary because 

they are not informed more than 1 month in advance which hospitals 

JCAHO will survey for accreditation. 
Appendix I: Scope and Methodology 

Page 35 GAO-04-850 Medicare Patient Safety in Hospitals 

In reviewing the sampling procedures they described, we determined that 

CMS initially selects a probability sample of hospitals for its state agency 

validation surveys.1

 However, hospitals have varying chances of selection 

in the sample depending on the month in the fiscal year that JCAHO 

performs the accreditation survey and the number of hospitals targeted for 

completion that year in the state in which the hospital was located. 

Additionally, the way that CMS determines which type of survey the 

sampled hospital receives is not random. Therefore, the analysis we 

performed is limited to those hospitals included in the validation survey 

sample and cannot be projected to all JCAHO-accredited hospitals. 

 1

In a probability sample, each eligible hospital accredited in a given year would have to 

have a known, nonzero chance for selection in the sample. 
Appendix II: Medicare Conditions of 

Participation 

Page 36 GAO-04-850 Medicare Patient Safety in Hospitals 

To participate in Medicare, hospitals must maintain standards of patient 

safety and health that comply with Medicare requirements. There are 

currently 23 Medicare COPs. Table 7 provides a description of each 

Medicare COP. 

Table 7: Medicare Conditions of Participation 

Medicare COP Description 

Anesthesia servicesa Anesthesia services must be well organized and directed by a qualified doctor of 

medicine or osteopathy. The service is responsible for all anesthesia 

administered. 

Compliance with federal, state, and local laws A hospital must comply with applicable federal laws on patient health and safety

and state and local laws on hospital and personnel licensing. 

Discharge planning A hospital must have a discharge planning process applicable to all patients. 

Policies and procedures must be in writing. 

Emergency servicesa If emergency services are provided they must be organized under the direction 

of a qualified member of the medical staff and have adequate medical and 

nursing personnel qualified in emergency care to meet the needs anticipated by 

the facility. 

Food and dietetic services Dietary services must be organized, directed, and staffed by qualified personnel. 

Contracted services must meet certain requirements. 

Governing body A hospital must have a legally responsible governing body or persons charged 

with the responsibilities of a governing body. 

Infection control A hospital’s sanitary environment must avoid sources and transmission of 

infections and communicable diseases. It must have an active program to 

prevent, control, and investigate infections and communicable diseases. 

Laboratory services The hospital must maintain, or have available, adequate laboratory services. 

Medical record services A hospital must have a medical record service that has administrative 

responsibility for medical records. 

Medical staff A hospital must have an organized medical staff that abides by bylaws approved 

by the governing body and is responsible for the quality of patient medical care. 

Nuclear medicine servicesa Nuclear medicine services must meet the needs of the patients in accordance 

with acceptable standards of practice. 

Nursing services An organized nursing service must provide 24-hour nursing services that are 

supervised or furnished by registered nurses. 

Organ, tissue, and eye procurement The hospital must have and implement written protocols on procurement and 

have adequate organ transplant policies. 

Outpatient servicesa Outpatient services must meet patient needs consistent with acceptable 

standards of practice. 

Patients’ rights A hospital must protect and promote patients’ rights. 

Pharmaceutical services The hospital must have pharmaceutical services that meet patient needs. 

Physical environment Hospital construction, arrangements, and maintenance must ensure patient 

safety and provide diagnostic and treatment facilities and special hospital 

services appropriate to community needs. 

Quality assessment and performance improvement A hospital must have an effective, hospitalwide quality assurance program. 

Appendix II: Medicare Conditions of 

Participation 
Appendix II: Medicare Conditions of 

Participation 

Page 37 GAO-04-850 Medicare Patient Safety in Hospitals 

Medicare COP Description 

Radiologic services The hospital must maintain, or have available, diagnostic radiologic services. 

Therapeutic services provided must meet professionally approved standards for 

safety and personnel qualifications. 

Rehabilitation servicesa Rehabilitation, physical therapy, occupational therapy, audiology, or speech 

pathology services must be organized and staffed to ensure the health and 

safety of patients. 

Respiratory servicesa Respiratory services must meet patient needs in accordance with acceptable 

standards of practice. 

Surgical servicesa Surgical services must be well organized and provided in accordance with 

acceptable standards of practice. Outpatient services must be consistent with 

inpatient care quality in accordance with the complexity of services offered. 

Utilization review Utilization review plans must provide for review of the services that a hospital 

and its medical staff provide to Medicare and Medicaid patients. 

Source: GAO summary of Medicare COPs. 

a

Optional services not required by Medicare. 
Appendix III: Features of JCAHO’s New 

Accreditation Process 

Page 38 GAO-04-850 Medicare Patient Safety in Hospitals 

In January 2004, JCAHO introduced a new hospital accreditation process 

that includes several new features. Table 8 includes a description of 

selected new features of JCAHO’s hospital accreditation process. 

Table 8: JCAHO’s Description of Features of Its New Hospital Accreditation Process 

Feature of the new 

accreditation process Description 

Periodic performance review The periodic performance review (PPR) is a new form of evaluation that is conducted by the 

organization and focuses on patient safety and quality of care issues. The organization self-evaluates 

its compliance with all standards that are applicable to the services that the organization provides, and 

develops a plan of action for all areas of performance identified as needing improvement. JCAHO will 

work with the organization to refine its plan of action to assure that its corrective efforts are on target. 

The organization will also identify measures of success for validating resolution of the identified 

problem areas when the organization undergoes its complete on-site survey 18 months later. 

Three options to the full PPR are available to organizations. The options and their requirements are: 

Option 1 

The organization performs the mid-cycle self-assessment, develops the plan of action and measures 

of success but does not submit PPR data to JCAHO. The organization attests that it has completed 

the foregoing activities but has, for substantive reasons, been advised not to submit its selfassessment or plan of action to JCAHO. 

The organization may discuss standards-related issues with JCAHO staff without identifying its 

specific levels of standards compliance. 

At the time of the complete on-site survey, the organization provides its measures of success to 

JCAHO for assessment. 

Option 2 

The organization need not conduct a mid-cycle self-assessment or develop a plan of action. 

The organization undergoes an on-site survey at the mid-point of its accreditation cycle. The survey 

will be approximately one-third the length of a typical full on-site survey and the organization will be 

charged a fee to cover survey costs. 

The organization develops and submits to JCAHO a plan of action to address any areas of noncompliance found during the on-site survey. JCAHO will work with the organization to refine its plan of 

action. At the time of the complete on-site survey, the organization provides its measures of success 

to JCAHO for assessment. 

Option 3 

The mid-cycle survey would be performed, as in Option 2, but, if the organization chooses, no written 

documentation or report of the survey would be left with the organization. Findings would be conveyed 

orally. This would eliminate the availability of a survey report for possible discovery from the 

organization, and would permit the organization, as is the case with Option 1, to control the language 

and documentation of the mid-cycle assessment activity. At the subsequent full survey, surveyors 

would not discuss with the organization, unless asked to do so, the fact that any particular standard 

had been found out of compliance at the mid-cycle assessment. Rather, they would focus on 

compliance with those standards at the time of the full survey. 

If the plan of action is approved, the organization’s accreditation decision will remain the same. 

However, if the plan of action is not approved, the organization’s accreditation decision will be 

changed to reflect the appropriate status. At the triennial on-site survey, implementation of the plan of 

action will be validated 

Appendix III: Features of JCAHO’s New 

Accreditation Process 
Appendix III: Features of JCAHO’s New 

Accreditation Process 

Page 39 GAO-04-850 Medicare Patient Safety in Hospitals 

Feature of the new 

accreditation process Description 

Priority focus process The priority focus process (PFP) is a data-driven tool that focuses survey activity on issues most 

relevant to patient safety and quality of care at the specific health care organization being surveyed. 

The PFP uses automation to gather pre-survey data from multiple sources including JCAHO, the 

hospital and other public sources. The PFP then applies rules to 1) identify relevant standards and 

appropriate survey activities, and 2) guide the selection of patient tracers. As part of the priority focus 

process, surveyors will track patients through their experience of care within an organization, 

assessing the quality and safety of care provided. The PFP does not imply that priority areas are out 

of compliance or deficient in any way. Rather, it lends consistency to the surveyor’s on-site sampling 

process. The PFP also helps to focus the surveyor’s assessment on quality and safety issues specific 

to an individual health care organization. 

The output of the PFP process will include: the top four to five priority focus areasthe processes, 

systems, or structures within a health care organization known to significantly impact the safety and 

quality of care specific to the health care organization being surveyed. 

Tracer methodology An evaluation method in which surveyors select a patient and use that individual’s record as a 

roadmap or “tracer” to assess and evaluate an organization’s compliance with selected standards and 

the organization’s systems of providing care and services. Using tracers, JCAHO surveyors will look 

at the care provided by each department within an organization, and how departments work together. 

Surveyors retrace the specific care processes that the individual experienced by observing and talking 

to staff in areas that the individual received care. As the individual’s case is examined, the surveyor 

may identify performance issues in one or more steps of the processor the interfaces between 

stepsthat affect the care of the patient. Surveyors will look for commonalities that might point to 

potential system-level issues in the organization. The tracer activity also provides several 

opportunities for surveyors to provide education to organization staff and leaders, as well as to share 

best practices from other similar health care organizations. 

Tracer patients will primarily be selected from an active patient list. Typically, individuals selected for 

the tracer activity are those who have received multiple or complex services 

Source: JCAHO. 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 40 GAO-04-850 Medicare Patient Safety in Hospitals 

Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 41 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 42 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 43 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 44 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix IV: Comments from the Centers for 

Medicare & Medicaid Services 

Page 45 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix V: Comments from the Joint Commission on Accreditation of Healthcare 

Organizations 

Page 46 GAO-04-850 Medicare Patient Safety in Hospitals 

Appendix V: Comments from the Joint 

Commission on Accreditation of Healthcare 

Organizations 
Appendix V: Comments from the Joint Commission on Accreditation of Healthcare 

Organizations 

Page 47 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix V: Comments from the Joint Commission on Accreditation of Healthcare 

Organizations 

Page 48 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix V: Comments from the Joint Commission on Accreditation of Healthcare 

Organizations 

Page 49 GAO-04-850 Medicare Patient Safety in Hospitals 
Appendix VI: GAO Contact and Staff 

Acknowledgments 

Page 50 GAO-04-850 Medicare Patient Safety in Hospitals 

Marcia A. Mann, (202) 512-9526 

In addition to the contact named above, Elaine Swift, Linda Kohn, Behn 

Kelly, Elizabeth T. Morrison, Roseanne Price, and Marie Stetser made key 

contributions to this report. 

Appendix VI: GAO Contact and Staff 

Acknowledgments 

GAO Contact 

Acknowledgments 
Related GAO Products 

Page 51 GAO-04-850 Medicare Patient Safety in Hospitals 

Medicare Home Health Agencies: Weaknesses in Federal and State 

Oversight Mask Potential Quality Issues. GAO-02-382. Washington, D.C.: 

July 19, 2002. 

Medicare: HCFA’s Approval and Oversight of Private Accreditation 

Organizations. GAO/ HEHS-99-197R. Washington, D.C.: September 30, 

1999. 

Home Health Care: HCFA Properly Evaluated JCAHO’s Ability to Survey 

Home Health Agencies. GAO/HRD-93-33. Washington, D.C.: October 26, 

1992. 

Health Care: Criteria Used to Evaluate Hospital Accreditation Process 

Need Reevaluation. GAO/HRD-90-89. Washington, D.C.: June 11, 1990. 

Related GAO Products 

(290305) 
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go to 

www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional 

Relations

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

Government Accountability Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00