Medicare: CMS Needs Additional Authority to Adequately Oversee Patient Safety in Hospitals
Published: 2004-07-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Hospitals accredited by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) are considered in compliance with Medicare participation requirements. GAO examined the extent to which JCAHO's pre-2004 hospital accreditation process identified hospitals not complying with Medicare requirements, the potential of JCAHO's new process for improving the detection of deficiencies in Medicare requirements, and the effectiveness of CMS's oversight of JCAHO's hospital accreditation program. GAO analyzed CMS data on hospitals state surveyors found to have deficiencies in Medicare requirements that JCAHO surveyors did not detect, analyzed CMS's measure of JCAHO's ability to detect noncompliance with Medicare requirements, and interviewed JCAHO officials.
section_title: What GAO Found
paragraphs: JCAHO's pre-2004 hospital accreditation process did not identify most of the hospitals found by state survey agencies in CMS's annual validation survey sample to have deficiencies in Medicare requirements. In comparing the results of the two surveys, CMS considered whether it was reasonable to conclude that the deficiencies found by state survey agencies existed at the time JCAHO surveyed the hospital. In a sample of 500 JCAHO-accredited hospitals, state agency validation surveys conducted in fiscal years 2000 through 2002 identified 31 percent (157 hospitals) with deficiencies in Medicare requirements. Of these 157 hospitals, JCAHO did not identify 78 percent (123 hospitals) as having deficiencies in Medicare requirements. For the same validation survey sample, JCAHO also did not identify the majority--about 69 percent--of deficiencies in Medicare requirements found by state agencies. Importantly, the number of deficiencies found by validation surveys represents 2 percent of the 11,000 Medicare requirements surveyed by state agencies in the sample during this time period. At the same time, a single deficiency in a Medicare requirement can limit the hospital's capability to provide adequate care and ensure patient safety and health. Inadequacies in nursing practices or deficiencies in a hospital's physical environment, which includes fire safety, are examples of deficiencies in Medicare requirements that could endanger multiple patients. The potential of JCAHO's new hospital accreditation process to improve the detection of deficiencies in Medicare requirements is unknown because the process was just implemented in January 2004. JCAHO plans to move from using announced to unannounced surveys in 2006, which would afford JCAHO the opportunity to observe hospitals' operations when the hospitals have not prepared in advance to be surveyed. In addition, the pilot test of the new accreditation process was of limited value in predicting whether it will be an improvement over the pre-2004 process in detecting deficiencies. Limitations in the pilot test included that hospitals were not randomly selected to participate; that observers from JCAHO accompanied each surveyor, thus possibly affecting surveyors' actions; and that JCAHO evaluated the results instead of an independent entity. CMS has limited oversight authority over JCAHO's hospital accreditation program because the program's unique legal status effectively prevents CMS from taking actions that it has the authority to take with other health care accreditation programs to ensure satisfactory performance. For example, requiring JCAHO's hospital accreditation program to submit to a direct review process or placing the program on probation while monitoring its performance. Further, CMS relies on a measure to evaluate how well JCAHO's hospital accreditation program detects deficiencies in Medicare requirements that provides limited information and can mask problems with program performance, uses statistical methods that are insufficient to assess JCAHO's performance, and has reduced the number of validation surveys it conducts.

section_title: Letter
paragraphs: In fiscal year 2002, nearly 7.4 million Medicare beneficiaries received  inpatient health care at hospitals that participated in Medicare. Federal  law establishes criteria for hospitals for purposes of Medicare. The  Centers for Medicare and Medicaid Services (CMS), the agency  responsible for administering Medicare, has established quality and patient  safety requirements called conditions of participation (COP) that hospitals  must meet in order to be eligible for Medicare payment. Hospitals that are  accredited by the Joint Commission on Accreditation of Healthcare  Organizations (JCAHO) are generally deemed under federal law to be  compliant with Medicare requirements for patient safety and health and  become eligible for payments from Medicare. No other health care  accreditation program has this same statutory authority.
JCAHO is a private, not-for-profit organization that accredits most of the  hospitals that participate in Medicare. JCAHO sets standards that  accredited hospitals must meet and reports that these standards are more  comprehensive than the Medicare COPs. In January 2004, JCAHO  implemented a new hospital accreditation process with goals that included  further enhancing health care quality and safety.
CMS oversight of JCAHO’s hospital accreditation program is limited  because it cannot restrict or remove JCAHO’s accreditation authority if the  agency detects problems. To oversee the program, CMS conducts on-site  validation surveys of a sample of JCAHO-accredited hospitals and reports  annually to Congress on the results of these surveys. The validation  surveys, which are performed by agencies that CMS has agreements with  in each state, help CMS determine whether Medicare quality and safety  requirements are being met. CMS compares the results of these state  surveys against survey results obtained through JCAHO’s hospital  accreditation program. CMS uses a measure called the rate of disparity  that summarizes the extent to which an accreditation program has failed  to cite deficiencies identified by state agency validation surveys. We are  using the term serious deficiency in this report to indicate a deficiency in  one or more Medicare COPs. Examples of serious deficiencies include a  hospital’s inability to provide adequate nursing services or failure to  implement and enforce infection control policies. According to CMS,  serious deficiencies substantially limit a hospital’s capability to render  adequate care and adversely affect the safety and health of patients.
Questions have been raised by the Department of Health and Human  Services’ (HHS) Office of Inspector General (OIG) and others regarding  whether accreditation by JCAHO ensures that hospitals provide adequate  care. Specifically, experts have questioned how well JCAHO’s hospital  accreditation process identifies deficiencies in hospitals that could  jeopardize patient safety and health. A comprehensive study by the HHS  OIG found that JCAHO’s surveys were not likely to identify patterns of  deficient care.
You asked that we examine the effectiveness of JCAHO’s hospital  accreditation process in ensuring that hospitals comply with Medicare  COPs to ensure the safety and health of Medicare beneficiaries.  Specifically, we (1) examined the extent to which JCAHO’s pre-2004  hospital accreditation process identified deficiencies in Medicare COPs  that were identified by state survey agencies, (2) determined whether  JCAHO’s new hospital accreditation process has potential for improving  the detection of deficiencies in Medicare COPs and whether the process  was adequately tested, and (3) examined the effectiveness of CMS’s  oversight of JCAHO’s hospital accreditation program.
To determine the extent to which JCAHO’s pre-2004 hospital accreditation  process identified deficiencies in Medicare COPs that were identified by  state survey agencies, we used data from a CMS comparison of state  validation survey findings with findings of JCAHO’s hospital accreditation  surveys, which indicated whether JCAHO found deficiencies in its  standards. Of the four possible outcomes to this comparison of survey  findings—(1) JCAHO and state agencies both identify no deficiencies, (2)  JCAHO identifies deficiencies not found by state agencies, (3) JCAHO and  state agencies both identify the same deficiencies, and (4) state agencies  identify deficiencies that JCAHO does not—we focused on the fourth  because it highlights the need for CMS oversight of the hospital  accreditation program. For the second outcome, there could be two  reasons for the disparity between JCAHO’s and state survey agencies’  findings: hospitals corrected deficiencies identified by JCAHO prior to the  state agency survey or the state survey agency did not identify a deficiency  that existed. In addition, not all JCAHO findings are equivalent to  noncompliance with a Medicare COP. To determine whether JCAHO’s  findings on deficiencies in its standards were comparable to the state  agencies’ findings, CMS staff compared the two surveys and considered  whether it was reasonable to conclude that the deficiencies found by state  survey agencies existed at the time JCAHO surveyed the hospital. For  deficiencies that CMS determined that JCAHO failed to identify, CMS met  with JCAHO to address disputed findings and consider additional evidence  on comparability offered by JCAHO. CMS provided results for a sample of  500 JCAHO-accredited hospitals from fiscal years 2000 through 2002. We  determined that the data CMS provided on serious deficiencies were  adequate for addressing the issues in this report. On the basis of this  sample of 500 JCAHO-accredited hospitals, we determined, using CMS’s  data, both the percentage of serious deficiencies and the percentage of  hospitals with serious deficiencies identified by the state survey agencies  where JCAHO surveyors did not find comparable deficiencies. The  analysis we performed on the results of the validation surveys was limited  to the hospitals included in the validation survey sample and cannot be  generalized to all JCAHO-accredited hospitals.
To determine whether JCAHO’s new hospital accreditation process has  potential for improving the detection of serious deficiencies, we identified  changes in the accreditation process and analyzed significant new  features. To determine whether JCAHO’s new hospital accreditation  process was adequately tested, we reviewed the testing procedures and  results that JCAHO used to determine the effectiveness of its new survey  process in identifying quality and safety deficiencies. Because the new  accreditation process was implemented recently, we did not have  information to compare JCAHO survey performance in detecting serious  deficiencies with state agency survey performance.
To determine the adequacy of CMS’s oversight of JCAHO’s hospital  accreditation program, we reviewed relevant statutory and regulatory  provisions regarding oversight of health care accreditation programs and  how CMS had implemented this authority in order to provide oversight. To  supplement our review, we conducted interviews with officials from CMS,  state survey agencies, and JCAHO; representatives from other  organizations active in health care accreditation and the hospital industry;  and experts in quality of care. We conducted our work from June 2003  through July 2004 in accordance with generally accepted government  auditing standards. (For a complete description of our scope and  methodology, see app. I.)
subsections: section_title: Background
paragraphs: To participate in Medicare, hospitals must maintain standards of patient  safety and health that comply with Medicare COPs. For example, the COP  related to nursing services includes such requirements for hospitals as  providing a 24-hour nursing service that is supervised or furnished by a  registered nurse. There are currently 23 Medicare COPs. (See app. II for a  description of the 23 Medicare COPs.) CMS proposed revisions to all of the  COPs in 1997, but it did not finalize them. Since then, CMS has revised  several of the COPs, including those concerning the life safety code;  quality assessment and performance improvement; organ, tissue, and eye  donations; and nurse anesthetist supervision.
Health care accreditation programs other than JCAHO’s hospital  accreditation program may generally adopt their own requirements if CMS  determines that an accreditation program’s requirements are at least  equivalent to Medicare COPs. If CMS also determines, among other things,  that the accreditation program’s survey process is likely to identify any  serious deficiencies in COPs, it must generally grant “deeming authority”  to the accreditation program and treat entities accredited by these  organizations as meeting Medicare COPs. CMS has the authority to review  these programs, and it can impose a probationary period while monitoring  performance and remove deeming authority if warranted.
Most hospitals demonstrate compliance with standards equivalent to  Medicare COPs through accreditation by JCAHO. In 2002, JCAHO  accredited 4,211, or 82 percent, of Medicare-participating hospitals.  Hospitals accredited by JCAHO received payments for Medicare-covered  inpatient services of approximately $98 billion, or 90 percent, of the $109  billion that was spent on hospital care in 2002. JCAHO, as part of its  accreditation-related activities, also develops survey procedures, trains its  surveyors, and formulates performance measures. JCAHO is governed by a  29-member board of commissioners and has a staff of over 1,000.
JCAHO’s deeming authority for hospitals is established in statute and  therefore can only be changed by Congress. As a result of this unique  statutory authority, hospitals accredited by JCAHObecause they meet  JCAHO standardsare deemed to meet Medicare COPs as well. In  contrast, the American Osteopathic Association (AOA)—a private, not-for- profit professional organization that offers accreditation services for  hospitals and other health care organizations—holds deeming authority  that is subject to CMS’s direct review and approval. While hospital  accreditation is its largest program, JCAHO also has accreditation  authority under Medicare for certain other health care providers, including  clinical laboratories, hospices, ambulatory surgical centers, and home  health care agencies. All of these other JCAHO accreditation programs are  subject to CMS’s direct review and approval.
To be accredited by JCAHO, a hospital must meet eligibility requirements,  satisfactorily complete a triennial on-site survey process, and continue to  maintain JCAHO’s standards between surveys. The accreditation surveys  that JCAHO conducts every 3 years are particularly important. For most  hospitals, the triennial survey is the only time that JCAHO conducts an on- site review of the hospital’s compliance with all quality standards and  issues decisions on how well the hospital has complied with JCAHO’s  standards. In 2004, JCAHO implemented a new hospital accreditation  survey process, which, according to JCAHO, is intended to reduce the cost  of accreditation to health care organizations and JCAHO, enhance public  confidence that health care organizations are in continuous compliance  with standards, increase the real and perceived value of accredited  organizations, meet the requirements of deeming authorities and  purchasers, and improve satisfaction for hospitals participating in the  accreditation program.
subsections: section_title: CMS Oversight of JCAHO
paragraphs: CMS exercises oversight of JCAHO’s hospital accreditation program  primarily through its validation surveys and annual reports to Congress.  Under federal law, CMS must continually study the operation and  administration of Medicare, including validating the JCAHO hospital  accreditation process, and submit annual reports to Congress. CMS has  agreements with state agencies to conduct validation surveys. There are  different kinds of validation surveys, including traditional validation  surveyssurveys conducted on a sample of hospitals within 60 days of  their triennial JCAHO survey.  Traditional validation surveys provide the  basis for assessing the effectiveness of JCAHO’s hospital accreditation  process in detecting deficiencies in Medicare COPs, which JCAHO- accredited hospitals are treated as meeting. Validation surveys also  include 18-month surveys, which monitor how well JCAHO-accredited  hospitals are complying with Medicare COPs midway between their 3-year  JCAHO surveys, and allegation surveys, which are triggered by complaints  or other reports of situations that pose potential threats to patient health  and safety in JCAHO-accredited hospitals. CMS has the authority to  remove the deemed status of a JCAHO-accredited hospital where a state  agency’s validation survey results in a finding that the hospital is out of  compliance with one or more Medicare COP.
CMS uses a rate of disparity measure to summarize the extent to which an  accreditation program, such as JCAHO’s hospital accreditation program,  has not found serious deficiencies identified by CMS through state agency  validation surveys. For a hospital accreditation program, using the results  from validation surveys, the rate of disparity for hospitals surveyed by the  state survey agencies is calculated as the difference between the number  of hospitals found with serious deficiencies by state agencies and the  number of hospitals found with comparable deficiencies by the  accreditation program, divided by the number of hospitals sampled. CMS  regulations provide that if the validation survey results for an  accreditation organization with deeming authority indicate a rate of  disparity that reaches the threshold level of 20 percent disparity or greater,  CMS will notify the organization that its deeming authority may be in  jeopardy and that the agency is initiating a deeming authority review.  With respect to JCAHO, CMS includes the rate of disparity in its annual  reports to Congress in which it reports the results of its validation program  for JCAHO’s hospital accreditation program.
section_title: JCAHO’s Pre-2004 Hospital Accreditation Process Often Did Not Detect Serious Deficiencies Found by State Survey Agencies
paragraphs: JCAHO’s pre-2004 hospital accreditation process often did not identify  either hospitals with serious deficiencies or the individual serious  deficiencies found by state survey agencies through CMS’s validation  program. In a sample of 500 JCAHO-accredited hospitals, state agency  validation surveys conducted in fiscal years 2000 through 2002 identified  31 percent (157 hospitals) with serious deficiencies; of these, JCAHO did  not identify 78 percent (123 hospitals) as having serious deficiencies. For  the same validation survey sample, the majority of the serious deficiencies  state survey agencies identified but JCAHO did not were in the physical  environment COP category, which covers fire safety and prevention.
subsections: section_title: JCAHO Did Not Identify Three-Quarters of the Hospitals That State Survey Agencies Found to Have Serious Deficiencies
paragraphs: From fiscal years 2000 through 2002, JCAHO did not identify 123 of the 157  hospitals (78 percent) with serious deficiencies that CMS’s validation  program identified out of a sample of 500 JCAHO-accredited hospitals.  Table 1 shows the hospitals with serious deficiencies that state survey  agencies identified and JCAHO did not during fiscal years 2000 through  2002. In 343 of the 500 hospital validation surveys, state agency surveyors  did not find serious deficiencies. Both state agency surveyors and JCAHO  surveyors identified 34 hospitals as having a serious deficiency.
According to JCAHO, disparity between state agency and JCAHO findings  in the 123 hospitals in part may be attributed to the timing of the two  surveys, JCAHO’s phasing in of new requirements, different interpretations  of the COPs by state surveyors, and inherent surveyor bias.  However, in  its comparison to determine disparity between the two surveys, CMS does  consider whether it is reasonable to conclude that the deficiencies found  by state survey agencies existed at the time JCAHO surveyed the hospital.
section_title: JCAHO Did Not Detect Two-Thirds of the Serious Deficiencies Identified by State Survey Agencies
paragraphs: From fiscal year 2000 through 2002, JCAHO did not detect 167 of the 241  serious deficiencies (69 percent) identified through CMS’s validation  program from a sample of 500 JCAHO-accredited hospitals. The number of  serious deficiencies found by CMS’s validation program represents 2  percent of the 11,000 Medicare COPs surveyed by state agencies in the  sample and were found in 157 hospitals. However, one serious deficiency  in any one of these hospitals could limit its ability to provide adequate care  to its patients. For example, a serious deficiency in the nursing services  COP at a hospital in Texas found by a state agency but missed by JCAHO  in 2000 included such problems as failure to prepare and administer drugs  in accordance with federal and state laws, inadequate supervision and  evaluation of the clinical activities of nonemployee nursing personnel, and  nursing care and procedures provided to patients that were not within the  scope of accepted standards of practice. Among hospitals with serious  deficiencies identified by CMS’s validation program but not by JCAHO,  there were on average 1.1 serious deficiencies per hospital, with a range  from 1 to 6. Table 2 shows the percentage of serious deficiencies identified  by CMS’s validation program but not by JCAHO for fiscal years 2000  through 2002.
Of the 167 serious deficiencies identified by CMS’s validation program  from fiscal year 2000 through 2002 but not detected by JCAHO, 87 were  related to a hospital’s physical environment, which includes life safety  code standards on fire prevention and safety. For these 3 years, JCAHO  did not detect 81 percent of the serious physical environment deficiencies  identified by state agency surveyors. Table 3 shows the number of serious  deficiencies, by category, identified by state survey agencies in CMS’s  validation program but missed by JCAHO surveyors. The larger number of  deficiencies in physical environment may be related to the difference in  how state agencies generally survey separately a hospital’s compliance  with the life safety code portion of the physical environment COP. JCAHO  surveys assess compliance with the life safety code using a combination of  the hospital’s self-assessment, a hospital building tour, and observations  made by all surveyors during the survey process. Examples of deficiencies  in physical environment that JCAHO did not identify but CMS’s validation  program found in a hospital in Alabama in 2000 included the following:  several exterior exits lacked emergency exit lighting; several exterior exits  were illuminated only by single light bulbs; fire alarm system and fire  extinguishers had not been inspected annually as required; and an  automatic sprinkler system had not been inspected annually and  maintained by certified personnel as required. Serious deficiencies in the  COP on physical environment compromise patient safety and health.
The total number of deficiencies not identified by JCAHO in the quality-of- care COP categories—those COPs that involve the oversight and delivery  of patient care—is similar to the number not identified by JCAHO in the  physical environment COP. While the number of serious deficiencies not  found by JCAHO in individual quality-of-care COP categories is smaller  than the number not found in physical environment, when these quality-of- care COPs are combined, the proportion of serious deficiencies JCAHO  missed is almost 60 percent of the total number of serious deficiencies  identified by state survey agencies. The following are examples of  hospitals found to be out of compliance with multiple quality-of-care  COPs:  In 2000, CMS removed the deemed status as a Medicare provider of a  JCAHO-accredited hospital in California for failure to comply with two  COPs, one of which was infection control. The hospital failed to provide a  sanitary environment to avoid sources and transmission of infections and  communicable diseases and failed to develop a system for ensuring the  sterilization of medical instruments.
Also in 2000, CMS notified a hospital in Texas that if it did not implement a  plan of correction the hospital’s participation in the Medicare program  would be terminated. Serious deficiencies at this hospital included lack of  compliance with the pharmaceutical services and nursing services COPs  because medications were administered without physician orders and a  double dose of narcotics was given in the emergency room, with no  explanation for the excessive dosage, to a patient who later died.
State surveyors in CMS’s validation program also may miss serious  deficiencies. In related work on skilled nursing facilities and home health  agencies, we found that the number of serious deficiencies found by state  agencies was highly variable among states and may be understated. State  agencies’ detection of serious deficiencies in hospitals also varied widely  among states for the 3 years we reviewed. For example, state survey  agencies in California, Illinois, and Ohio found serious deficiencies in over  45 percent of the surveys they conducted between fiscal years 2000  through 2002. In contrast, Florida and New York found serious  deficiencies in less than 10 percent of the surveys they conducted, and  Louisiana did not find serious deficiencies in any of the surveys it  conducted.
section_title: Potential of JCAHO’s New Hospital Accreditation Process Is Unknown, and Testing Was Limited
paragraphs: The potential of JCAHO’s new hospital accreditation process to improve  the identification of serious deficiencies is unknown because it is too soon  after its January 2004 implementation for a meaningful evaluation; in  addition, JCAHO’s testing of the new process was limited. CMS has not  had the opportunity to complete its validation program for 2004 to  determine whether JCAHO surveyors using the new process are missing  serious deficiencies later identified by state agency validation surveys.  While unannounced surveys, which are planned for implementation in  2006, have the potential to improve the detection of serious deficiencies,  other features of the new process that JCAHO did not test before  implementation may have limitations that could affect the potential of the  new process to identify problems with patient care. JCAHO’s pilot test of  the new process had limitations, including using a sample of hospitals that  volunteered for the pilot instead of using a random sample and self- evaluating the results instead of using an independent entity.
subsections: section_title: Potential of New Process Is Unknown
paragraphs: Because JCAHO’s new accreditation process was implemented in January  2004, it is too soon to know whether the new process is better at detecting  serious deficiencies in Medicare COPs than the pre-2004 accreditation  process. A JCAHO official told us the new process will aid in the detection  of deficiencies, but we found that some of the features may have  shortcomings that could limit their effectiveness. New features of the  accreditation process include the hospital’s self-assessment of compliance  with accreditation standards midway through the accreditation cycle,  surveyor review of the care provided to specific patients to determine the  adequacy of the hospital’s health care delivery system, and performance of  all accreditation surveys on an unannounced basis beginning in 2006. (See  app. III for a description of selected new features of JCAHO’s new hospital  accreditation process.)
subsections: section_title: Periodic Performance Review
paragraphs: Periodic performance reviews assess hospital compliance with applicable  standards and are performed at the 18-month midpoint between 3-year on- site accreditation surveys. According to JCAHO, the periodic performance  review will have several benefits. These include providing hospitals with a  process to assess their ongoing compliance and requiring them to correct  or plan to correct all deficiencies identified. Periodic performance reviews  must be conducted either by the hospital as a self-assessment or, if the  hospital chooses, by JCAHO through an on-site review.
However, periodic performance reviews may not necessarily improve the  detection of deficiencies. JCAHO did not pilot test these reviews for the  potential to detect deficiencies and did not test whether hospitals that  conducted reviews do a better job of continuing to comply with standards.  In addition, for hospitals performing self-assessments, JCAHO will not  check these self-assessments to determine whether hospitals fully and  accurately identified quality problems and developed adequate corrective  action plans to address the problems identified.
section_title: Priority Focus Process and Patient Tracer Methodology
paragraphs: According to JCAHO, the priority focus process and patient tracer  methodology together have the potential to enhance the ability of surveys  to detect deficiencies by directing the attention of surveyors to key patient  care areas. The priority focus process uses a data-based formula to  identify a limited number of areas in each hospital that are particularly  important to patient health and safety. Priority focus areas might include  infection control, medication management, or patient safety. Surveyors  use the priority focus process combined with the patient tracer  methodology to focus their surveys to specific areas for review. The  patient tracer methodology guides their choice of current patients to  “trace” through the experience of care within an organization. For  example, if the hospital’s priority focus process data suggest that a patient  with an orthopedic-related diagnosis such as a hip fracture should be  traced, the JCAHO surveyor would review the patient’s medical record,  noting where the patient had entered into the hospital and any services  and transfers that occurred. Then the surveyor would retrace the steps in  the patient’s care process by observing and talking to staff in some of the  areas in which the patient received care. If the patient entered through the  emergency department, was transferred to a medical/surgical unit, and  then went to the operating room, the surveyor would go to these areas to  interview staff about the care given to this specific patient. With  information from patient tracers, the surveyor will assess whether any  compliance issues exist with JCAHO standards. If the surveyor identifies a  compliance issue while tracing one patient, the surveyor may review the  records of similar patients to determine whether the problem is isolated or  represents a pattern of care.
However, JCAHO did not test the extent to which the priority focus  process and the patient tracer methodology could help surveyors detect  deficiencies. A JCAHO official told us these new features of the  accreditation process were intended to help surveyors trace patients in a  consistent way and not necessarily to improve the detection of  deficiencies.
section_title: Unannounced Surveys
paragraphs: JCAHO plans to conduct all hospital accreditation surveys on an  unannounced basis beginning in 2006. JCAHO stated that unannounced  surveys will ensure that hospital performance is based on the observation  of hospitals’ routine operations rather than on how they operate after they  have the opportunity to prepare to be surveyed. A JCAHO official also  indicated that unannounced surveys will be more likely to detect  deficiencies. The OIG and other organizations share JCAHO’s position on  the value of unannounced surveys of hospitals and other health care  organizations. The value of unannounced surveys also has been  recognized for nursing homes, which state agencies survey on an  unannounced basis.
section_title: JCAHO’s Pilot Test of New Process Was Limited
paragraphs: JCAHO’s pilot test of its new hospital accreditation process was limited  and therefore unable to help determine the potential of the new process to  detect deficiencies in Medicare COPs. According to JCAHO, the pilot test  suggests that the new process was more likely than the former process to  find quality problems. However, the pilot test sample included hospitals  that volunteered or were selected by JCAHO and were not randomly  selected, pilot test surveyors were accompanied by observers from  JCAHO’s central office, and pilot test results were not independently  evaluated. In addition, CMS has not completed its fiscal year 2004  validation program, which will include hospitals surveyed by JCAHO using  the new process and thus does not yet have sufficient data on which to  base a meaningful evaluation.
According to JCAHO’s analysis of the pilot test, the new hospital  accreditation process is more likely to identify quality problems since  proportionately more hospitals under the new process received  unfavorable accreditation decisions. JCAHO based its conclusion on a  comparison of survey outcomes, called accreditation decisions, between  18 hospitals in the pilot test conducted in 2002 and 2003 and the 1,524  hospitals that had been surveyed under the pre-2004 accreditation process  during 2003. Table 4 presents the data JCAHO used to make the  comparison. As shown, proportionately fewer hospitals under the new  process were accredited without having to make corrections. Although  JCAHO provided the accreditation decision outcomes for these 18 pilot  tests, it stated it preferred to use the number of “requirements for  improvement” as the basis for analysis.
However, JCAHO’s pilot test analysis was limited in three respects, which  may have accounted for the smaller number of favorable accreditation  decisions hospitals received under the new process.
The hospitals participating in the pilot test were not randomly selected by  JCAHO. As a result, these hospitals may not be representative of all  JCAHO-accredited hospitals and therefore results cannot be generalized.
During the pilot test, an observer from JCAHO’s central office  accompanied each surveyor, and the knowledge that they were being  observed may have influenced the surveyors’ actions. Under the pre-2004  process, observers only rarely accompanied JCAHO surveyors.
JCAHO conducted its own evaluation of pilot test results. Evaluation of  the pilot test by an entity independent of either JCAHO or the hospitals  tested could help to ensure that survey outcomes were impartially  interpreted. For example, CMS used an independent group to evaluate its  redesign of the nursing home survey process.
section_title: CMS Oversight Authority of JCAHO’s Hospital Accreditation Program Is Limited and Needs Improvement
paragraphs: CMS has limited oversight authority over JCAHO’s hospital accreditation  program, and its existing oversight activities need improvement. The  unique status of JCAHO’s hospital accreditation program, which is  specified in statute, does not permit CMS to take corrective action, such as  restricting or removing its deeming authority. Additionally, CMS uses a  measure that provides limited information to evaluate the performance of  JCAHO’s hospital accreditation program, has significantly reduced the  number of surveys conducted as part of CMS’s validation program, and  does not use measures that are based on sound statistical methods to  assess the performance of JCAHO’s hospital accreditation program.
subsections: section_title: CMS Oversight Authority of JCAHO Is Limited
paragraphs: Because of JCAHO’s unique legal status, CMS’s oversight of JCAHO’s  hospital accreditation program is limited in two major ways: Unlike other  accreditation programs with deeming authority, JCAHO does not have to  reapply to CMS to reauthorize its deeming authority, and CMS cannot take  action to address performance problems with JCAHO’s hospital  accreditation program.
JCAHO’s hospital accreditation program is the only Medicare accreditation  program for which CMS does not have to conduct an evaluation of the  accreditation standards and the processes used to conduct surveys.  Without this evaluation, CMS is deprived of key oversight tools it is  authorized to use with other accreditation programs: detailed information  about any proposed changes to the accreditation process and public input.  CMS cannot require JCAHO to provide information about proposed  changes to its accreditation requirements and hospital survey processes.  Also, because it is not required to reapply to CMS for deeming authority,  JCAHO does not have to provide CMS information that other accreditation  programs must provide, such as a detailed description of its survey  processes, a comparison of its standards to Medicare requirements, and  the qualifications of its surveyors, which CMS reviews to ensure that the  programs comply with Medicare requirements. For example, when  JCAHO’s hospice accreditation program applied for deeming status in  1999, CMS required changes to JCAHO’s hospice accreditation process,  including requiring JCAHO to make unannounced surveys of Medicare- certified hospices. According to a CMS official, JCAHO’s hospital  accreditation program has provided much of the information required of  other accreditation organizations; however, CMS has no authority to  require JCAHO to make changes to the hospital accreditation program as  it does with other health care accreditation programs. Statutory provisions  regarding public notice and comment do not apply to JCAHO’s hospital  accreditation program as they do to other accreditation programs. The  reapplication process for other accreditation programs requires affording  the public an opportunity to provide input to CMS on an accreditation  program’s request for deeming authority. Because JCAHO does not have to  reapply for deeming authority, the public does not have the opportunity to  review and comment on JCAHO’s hospital accreditation program.
A second limitation is CMS’s inability to address any performance issues  with JCAHO’s hospital accreditation program. Although the rate of  disparity for JCAHO’s hospital accreditation program exceeded 20 percent  in fiscal years 2000, 2001, and 2002 a rate that would have triggered a  deeming authority review for any other Medicare accreditation  programCMS was unable to take enforcement action to address  JCAHO’s performance. When other Medicare accreditation programs have  a rate of disparity of 20 percent or more, CMS can take steps such as  imposing a year-long probationary period and removing deeming authority  at the end of the probationary period if the rate of disparity remains at 20  percent or more. For JCAHO, however, CMS’s actions toward correcting  the program’s deficiencies are limited to including recommendations for  improvement in its annual reports to Congress and negotiating with  JCAHO to voluntarily adopt CMS’s recommendations.
In its annual report to Congress, CMS made recommendations in fiscal  year 2002 aimed at improving JCAHO’s ability to detect serious  deficiencies in the life safety code, part of the COP on physical  environment. CMS noted that JCAHO permits hospitals to self-assess  compliance with life safety code requirements. While CMS stated that it  did not object to the concept of hospital self-assessment of life safety code  requirements, it made five recommendations to JCAHO for improving  implementation:  1.  Require hospitals to use qualified personnel, such as fire marshals and  architects, to conduct self-assessments of compliance with the life  safety code requirements.  2.  Set minimum standards for identifying and improving life safety code  deficiencies identified by hospital self-assessments.  3.  Require hospitals to submit their self-assessments on life safety code  issues prior to JCAHO conducting accreditation surveys to provide  surveyors and personnel in JCAHO’s central office time to review the  material prior to the accreditation surveys.  4.  Increase the use of JCAHO experts in the life safety code requirements  in its central office.  5.  Address the issue of hospitals that do not make improvement within  self-determined time frames.
JCAHO did not adopt all of these recommendations. It disagreed with the  first recommendation. Its response indicated that its requirement to use  qualified personnel to complete the self-assessment, while more general,  was sufficient. It further indicated that policies were in place for CMS’s  second and fifth recommendations. CMS later agreed that JCAHO’s  policies do satisfactorily address the fifth recommendation. JCAHO  planned to examine ways to adopt CMS’s third and fourth  recommendations. CMS however, had no authority to compel JCAHO to  comply with the remaining recommendations. According to CMS, it  continues to discuss implementation of its recommendations with JCAHO.  JCAHO stated that while its initial response to CMS’s recommendations in  2003 reflected then current JCAHO policies, subsequent policy evolutions  are addressing CMS’s recommendations. Specifically, JCAHO is working  with the American Society of Hospital Engineers to develop a process for  review by experts of hospital self-assessments on life safety code issues  prior to JCAHO’s conducting on-site accreditation surveys and to identify  those hospitals for which engineering expertise should be added to on-site  surveys.
section_title: CMS’s Validation Program Needs Improvement
paragraphs: CMS states that the goal of its validation program is to provide reasonable  assurance to Congress that the JCAHO accreditation process ensures  hospital compliance with Medicare COPs. However, the measure CMS  uses to evaluate the performance of JCAHO’s hospital accreditation  program provides limited information and could mask problems with an  accreditation program’s performance in detecting serious deficiencies, and  it is based on a target sample size of 1 percent of JCAHO-accredited  hospitals. In addition, CMS does not report the extent to which its sample  reflects the performance of the larger population of JCAHO-accredited  hospitals.
subsections: section_title: Rate of Disparity
paragraphs: The rate of disparity between JCAHO’s hospital accreditation survey  findings and state survey agency findings, as currently calculated by CMS,  does not fully explain the performance of JCAHO’s hospital accreditation  program in detecting serious deficiencies. CMS uses this measure in its  reports to Congress to assess JCAHO’s hospital accreditation program and  as the basis for making recommendations for improvement. CMS  calculates the rate of disparity as the difference between the number of  hospitals found with serious deficiencies by state survey agencies and the  number of hospitals found with serious deficiencies by the accreditation  survey, divided by the number of hospitals in the sample. For example, if  state survey agencies conducted 200 surveys as part of CMS’s validation  program and found 60 hospitals out of compliance with at least one COP,  but JCAHO’s survey found that only 22 of the hospitals were out of  compliance, the rate of disparity would be 19 percent ((60 - 22)/200).
CMS has established in regulation a rate of disparity of 20 percent or  greater as the threshold for taking action against an accreditation  program. According to a CMS official, the use of 20 percent as the  threshold is not based on empirical evidence but rather on what CMS  believed Congress would find acceptable. Consequently, the threshold  may not be appropriately placed to indicate unacceptable performance by  a hospital accreditation program. For example, if JCAHO failed to identify  serious deficiencies in all 14 hospitals that the state agencies identified  with serious deficiencies from a sample of 79 hospitals, the rate of  disparity would be a satisfactory 18 percent ((14-0)/79).
CMS’s rate of disparity measure used in isolation does not consistently  reflect an accreditation program’s ability to detect serious deficiencies. As  the number of hospitals with serious deficiencies detected by state survey  agencies decreases, regardless of JCAHO’s performance in detecting them,  it is more likely that the rate of disparity will be less than CMS’s 20 percent  threshold. As a result, the performance of JCAHO’s hospital accreditation  program is difficult to judge based on this measure alone. For example, if  state survey agencies performed 200 validation surveys and found 100  hospitals or 50 percent with serious deficiencies and JCAHO found 30  hospitals or 30 percent of the hospitals found by state agencies, the rate of  disparity would be 35 percent ((100-30)/200). However, if the state  agencies found 50 hospitals, or 25 percent, of the 200 hospitals with  serious deficiencies and JCAHO found 15 hospitals, or 30 percent of the  hospitals that the state agencies identified, the rate of disparity would be  almost 18 percent ((50-15)/200). The percentage of serious deficiencies  found by state survey agencies and also by JCAHO remained the same in  both examples, but the rate of disparity was improved significantly by the  larger number of hospitals without serious deficiencies in the second  example. This indicates that the rate of disparity does not consistently  measure the accreditation program’s ability to detect serious deficiencies  found by state survey agencies. (See table 5.) In addition to the rate of  disparity, other components, such as the proportion of hospitals with  serious deficiencies and the total number of serious deficiencies found by  state agencies but missed by the accreditation program, are important  indicators of an accreditation program’s overall performance.
CMS does not analyze the statistical results of its validation survey  samples in ways that would allow it to better assess JCAHO’s ability to  detect serious deficiencies. CMS has not documented the methods it uses  to select hospitals for validation surveys and did not supply us with clear  technical justification for the methods used. Further, CMS’s validation  sample includes hospitals that, because of its sampling method, have  varying chances of selection, but it does not take this into account when  calculating statistics based on the sample. According to CMS’s sampling  method, the selection of hospitals is influenced by factors such as the  month in the fiscal year that JCAHO performed the accreditation survey  and how many hospitals were targeted for completion that year in the  state in which the hospital was located. Thus, some hospitals have a  greater chance of selection than others. CMS also does not take these  different chances of selection into account when calculating statistics for  its annual reports to Congress, which prevents CMS from accurately  assessing JCAHO’s performance. Moreover, CMS does not measure and  report in its annual reports the extent to which its estimates based on the  validation survey sample are likely to reflect how well JCAHO detects  deficiencies in the larger population of hospitals it accredits.
In addition, the number of usable traditional validation surveys completed  is smaller than the number of hospitals CMS samples for validation  surveys. This difference may affect the accuracy of the data that CMS  presents to Congress if the hospitals where the traditional surveys were  completed produce different results than those where surveys are not  completed or are not usable. During its sampling process, CMS selects a  sample size close to the targeted number of hospitals each year. Some  hospitals from this sample may be excluded because CMS chose to  perform another type of survey for them that cannot be used to validate a  JCAHO accreditation survey. In addition, state agencies are not always  able to complete the requested traditional validation surveys within 60  days from the JCAHO accreditation survey, as required, or a hospital may  be excluded because it lost its deemed status or closed. The size of the  difference between the number of hospitals sampled and the number of  usable traditional validation surveys completed therefore varies, as it did  during the 3-year review period (see table 6).
CMS reduced the number of validation surveys conducted by state  agencies from a target of approximately 5 percent of the total number of  hospitals that JCAHO accredits to a target of approximately 1 percent,  with at least one survey in each state. Reducing the target of validation  surveys from 5 percent to 1 percent results in the number of validation  surveys being reduced from 227 in fiscal year 2002 to a target of 75  validation surveys in fiscal year 2003 and 72 in fiscal year 2004.
Reducing the targeted number of validation surveys to 1 percent provides  less reliable information on how well JCAHO’s hospital accreditation  program ensures compliance with Medicare COPs. For example, for a 5- percent target, the estimate of the proportion of JCAHO-accredited  hospitals with a particular deficiency that is derived from the validation  survey could be as much as 6.0 percentage points higher or lower, for a  range of 12.0 percentage points. If the 5-percent target produced an  estimate that 50 percent of JCAHO-accredited hospitals had a particular  deficiency, the percentage of JCAHO-accredited hospitals not complying  could range from 44.0 to 56.0 percent. However, for a 1-percent target the  estimate could be 11.4 percentage points higher or lower, for a range of  about 22.8 percentage points. For example, if the 1-percent target  produced an estimate that 50 percent of JCAHO-accredited hospitals had a  particular deficiency, the percentage of JCAHO-accredited hospitals not  complying with a Medicare COP could range from 38.6 to 61.4 percent.
This reduction in the number of validation surveys is of additional concern  because it coincides with the implementation of JCAHO’s new  accreditation process, which has an unproven capacity to detect  deficiencies. CMS’s target sample size for traditional validation surveys for  fiscal year 2004 will be further reduced because the sample also includes  18-month validation surveys. In 2004, CMS is planning to conduct 17 of  these 18-month surveys as part of its overall validation survey target of 72.  Thus, CMS could be using as few as 55 validation surveys to determine  JCAHO’s performance.
section_title: Conclusions
paragraphs: For 3 consecutive years, JCAHO’s hospital accreditation program, which  accredits most of the hospitals participating in Medicare, exceeded CMS’s  threshold for unacceptable performance. CMS validation surveys during  that time period confirmed that JCAHO missed the majority of serious  deficiencies found by state survey agencies. Yet, CMS was unable to take  action against JCAHO’s hospital accreditation program as it can with other  accreditation programs because it lacked the authority to do so. Although  CMS has recommended in its annual reports to Congress that JCAHO  make changes in its hospital accreditation program to improve its ability  to detect serious deficiencies, some of these recommendations have not  been implemented. Thus, it is vital for patient safety that JCAHO hospital  accreditation surveys detect existing serious deficiencies and deny  accreditation to hospitals that do not comply with Medicare COPs.
CMS is unable to present to Congress an adequate assessment of JCAHO’s  performance because of limitations in its process for selecting hospitals  for validation surveys and analysis of the survey results. CMS does not  consistently portray the extent to which serious deficiencies are missed  and does not identify the limitations in reporting the estimates it makes  from its survey sample. CMS cannot assure Congress that JCAHO- accredited hospitals meet Medicare COPs because the measure for the  rate of disparity, which determines poor performance, allows JCAHO to  miss the majority of serious deficiencies and still be in an acceptable range  of performance. Further, CMS’s reduction in the number of validation  surveys it uses to determine the performance of JCAHO’s hospital  accreditation program will provide less reliable information at a time  when JCAHO is implementing a new hospital accreditation process that is  unproven in its ability to detect serious deficiencies. In light of these  limitations in CMS’s validation of JCAHO’s hospital accreditation program,  we believe that CMS must improve its oversight so it can provide Congress  with more accurate information regarding JCAHO’s performance.
section_title: Matter for Congressional Consideration
paragraphs: Given the serious limitations in JCAHO’s hospital accreditation program  and that efforts to improve this program through informal action by CMS  have not led to necessary improvements, Congress should consider giving  CMS the same kind of authority over JCAHO’s hospital accreditation  program that it has over all other Medicare accreditation programs.
section_title: Recommendations for Executive Action
paragraphs: To strengthen the ability of CMS to identify and report to Congress on  JCAHO’s ability to ensure that the hospitals it accredits protect the safety  and health of patients through compliance with the Medicare COPs, we  recommend that the Administrator of CMS take the following three  actions:    modify the method used to measure the rate of disparity between  validation survey findings and accreditation program findings to provide a  reasonable assurance that Medicare COPs are being met and consider  whether additional measures are needed to accurately reflect an  accreditation program’s ability to detect deficiencies in Medicare COPs;    provide in the annual report to Congress an estimate, based on the  validation survey sample, of the performance of all JCAHO-accredited  hospitals, including the limitations and protocols for these estimates based  on generally accepted sampling and statistical methodologies; and develop  a written protocol for these calculations; and    annually conduct traditional validation surveys on a sample of JCAHO- accredited hospitals that is equal to at least 5 percent of all JCAHO- accredited hospitals.
section_title: Agency and Other External Comments and Our Evaluation
paragraphs: CMS and JCAHO commented on a draft of this report. In its comments,  CMS concurred with our recommendations. JCAHO stated it had no  objection to our suggestion that Congress give CMS the same authority  over its hospital accreditation program as it does over other Medicare  accreditation programs. However, JCAHO took issue with the  methodology we used for evaluating the performance of its hospital  accreditation program. CMS’s and JCAHO’s specific comments and our  response follow. CMS’s comments are reprinted in appendix IV and  JCAHO’s comments are reprinted in appendix V. CMS and JCAHO also  provided technical comments, which we incorporated as appropriate.
CMS stated that it has begun to examine the need for additional or  alternative measures for the rate of disparity calculation. CMS also stated  it will seek additional resources to further develop and implement new  sampling and statistical methodologies that may allow results to be  projected to all JCAHO-accredited hospitals, and to increase the validation  sample size. CMS specifically noted that it considers life-safety code  compliance, on the part of all provider types, to be critically important. In  the past 8 years, in its annual reports to Congress and its dialogues with  JCAHO regarding its hospital accreditation program, it has identified  physical environment as an important area where JCAHO needs to focus  attention, and CMS noted that 68 percent of facilities that had a deficiency  finding not identified by JCAHO had them in the physical environment  area.
JCAHO stated that our methodology for evaluating the performance of its  hospital accreditation program was incomplete and did not provide a  comprehensive assessment of its program’s performance. We did not  intend to do a comprehensive evaluation of JCAHO’s overall hospital  accreditation program. Rather, we focused our evaluation on how well  JCAHO’s hospital accreditation program ensures hospitals’ compliance  with Medicare participation requirements. There are four possible  outcomes to a comparison between JCAHO’s accreditation survey and a  state validation survey: (1) both JCAHO and state agencies identify no  deficiencies, (2) JCAHO identifies deficiencies not found by state agencies,  (3) both JCAHO and state agencies identify the same deficiencies, and (4)  state agencies identify deficiencies that JCAHO does not. We limited our  evaluation to the fourth outcome because it illustrates the need for CMS  oversight of the hospital accreditation process. We have clarified the  scope of our evaluation to emphasize our focus on this outcome.
JCAHO raised a concern that our characterization of JCAHO’s missed  deficiencies that state survey agencies found misleads readers to believe  that JCAHO misses hospitals with deficiencies 78 percent of the time. We  have revised language in the report to further emphasize that the missed  deficiency rate applies to hospitals in the validation survey sample in  which the state survey agencies found deficiencies and cannot be  generalized to all JCAHO-accredited hospitals. JCAHO further stated that  our report does not take into account that JCAHO’s hospital accreditation  program detects deficiencies in hospitals that CMS does not find.  However, it is to be expected that state survey agencies will not find all  deficiencies found by JCAHO because hospitals may have corrected the  deficiencies prior to the state agency surveys.
JCAHO stated that we misrepresented the potential of the new  accreditation process in detecting deficiencies in Medicare COPs and  provided new data regarding its first quarter 2004 performance that  indicate that JCAHO surveys may have detected a greater percentage of  deficiencies related to patient care compared with the pre-2004  accreditation process. However, we maintain that until CMS validation  surveys for 2004 are completed, there is no basis on which to determine  whether the new process improves the detection of deficiencies in  Medicare COPs. In addition, JCAHO stated and we agree that evaluating  and improving the quality of care in hospitals is not about counting  deficiencies, it is about finding those deficiencies that, if not fixed, will  generate poor results for patients and making sure that these deficiencies  are remedied in a timely fashion.
JCAHO stated that we mischaracterized its response to the five  recommendations that CMS made in 2002 to improve JCAHO’s ability to  detect deficiencies in the life safety code and that it is involved in frequent  and ongoing dialogue with CMS regarding the recommendations and other  life safety code issues. We have clarified language in the report regarding  JCAHO’s response to CMS’s recommendations.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  date. We will then send copies of this report to the Secretary of Health and  Human Services and other interested parties. We will also make copies  available to others upon request. In addition, the report will be available at  no charge at the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please call me at  (202) 512- 7119. Another contact and key contributors are listed in  appendix VI.
section_title: Appendix I: Scope and Methodology
paragraphs: We examined the extent to which JCAHO’s pre-2004 survey process  identified hospitals with deficiencies and individual deficiencies in  Medicare COPs that were identified by state survey agencies. We chose  these measures because they reflect performance in detecting and  correcting serious deficiencies, which according to CMS, substantially  limit a hospital’s capability to render adequate care and adversely affect  the health and safety of patients. We reviewed data, provided by CMS, on  500 traditional validation surveys conducted by state survey agencies  during fiscal years 2000 through 2002. In these validation surveys, state  survey agencies documented whether they found serious deficiencies in  Medicare COPs. CMS compared state survey agency findings with  JCAHO’s accreditation surveys that identified deficiencies in JCAHO’s  standards. CMS then determined whether the state survey agencies’  findings on serious deficiencies in the 22 Medicare COPs that can be  deemed were comparable to JCAHO’s findings on deficiencies in JCAHO’s  standards in the following way. Two CMS experts such as nurses reviewed  the comparability of serious deficiencies in the quality-of-care conditions  identified in validation surveys to deficiencies in JCAHO’s accreditation  standards identified in JCAHO’s hospital accreditation surveys. Two  experts, such as building engineers, reviewed the comparability of serious  deficiencies identified in the validation surveys on the condition on  physical environment. Where there was disagreement, the two experts met  to resolve their differences. CMS does not have written protocols for  determining comparability. Experts are expected to use their best  professional judgment. CMS experts also had to consider whether it is  reasonable to conclude that the deficiencies existed at the time that  JCAHO surveyed the hospital. For those deficiencies that CMS determines  that JCAHO has failed to identify, it met with JCAHO to address disputed  findings and to consider additional evidence on comparability offered by  JCAHO. There are four possible outcomes to this comparison of survey  findings—(1) JCAHO and state agencies both identify no deficiencies, (2)  JCAHO identifies deficiencies not found by state agencies, (3) JCAHO and  state agencies both identify the same deficiencies, and (4) state agencies  identify deficiencies that JCAHO does not—we focused on the fourth  because it highlights the need for CMS oversight of the hospital  accreditation program. For the second outcome, there could be two  reasons for the disparity between JCAHO’s and state survey agencies’  findings: hospitals corrected deficiencies identified by JCAHO prior to the  state agency survey or the state survey agency did not identify a deficiency  that existed. In addition, not all JCAHO findings are equivalent to  noncompliance with a Medicare COP.
From these 500 surveys, we determined the number of hospitals with  serious deficiencies and the total number of serious deficiencies identified  by state agencies but that CMS determined were not identified by JCAHO.  These data include 123 hospitals in which state survey agencies identified  one or more serious deficiencies and JCAHO did not make comparable  findings according to CMS. These data also include 167 serious  deficiencies identified by state agencies but that CMS determined  comparable findings were not identified by JCAHO.
For fiscal years 2001 and 2002, we obtained from CMS a comparison  between the validation surveys conducted by the state survey agencies and  the accreditation surveys conducted by JCAHO, which identified serious  deficiencies identified by the state agencies but not by JCAHO as  determined by CMS. For fiscal year 2000, CMS did not supply its  determinations of the comparability of findings in validation and  accreditation surveys for 31 of 82 serious deficiencies. We followed a  protocol similar to the one used by CMS to determine the comparability of  the remaining 31 serious deficiencies, which included 29 quality-of-care  serious deficiencies and 2 physical environment serious deficiencies. Two  analysts with nursing backgrounds compared the findings and made  determinations on their comparability based on their professional  judgment. In cases of disagreement, a third analyst with a background in  nursing made the determination.
We did not include 1998 and 1999 data in our analysis because CMS used a  method that undercounted the number of deficiencies identified by state  survey agencies but not identified by JCAHO. CMS did not count as  deficient those cases in which state survey agencies determined that a  hospital was not meeting the COP on physical environment but JCAHO  determined that the hospital was in compliance because the hospital was  following correction plans approved by JCAHO.
To determine the potential of JCAHO’s new accreditation process in  improving the detection of deficiencies in Medicare COPs, we reviewed  material supplied by JCAHO on development and testing of its new  process and interviewed JCAHO officials about the steps taken to test the  new process and to analyze results. We also examined the features of the  new accreditation process by reviewing descriptive material obtained  from JCAHO and interviewing experts in health care quality. Because the  new accreditation process was implemented in January 2004, we were  limited in our ability to determine the effectiveness of the new  accreditation process because we were not able to perform a comparative  analysis of validation survey and JCAHO survey results under the new  process.
To examine the effectiveness of CMS’s oversight of JCAHO’s accreditation  process, we analyzed the laws and regulations that define CMS’s authority  and JCAHO’s authority. We reviewed the annual reports submitted to  Congress on JCAHO’s performance in identifying serious deficiencies and  reviewed correspondence between CMS and JCAHO and interviewed  officials in both organizations. We analyzed the rate of disparity that CMS  uses to determine the performance of JCAHO’s hospital accreditation  process in identifying deficiencies in Medicare COPs.
To evaluate CMS’s statistical methodology for the validation surveys, we  interviewed CMS officials about the sampling and statistical methods. In  the absence of written methodological documentation, we relied on  information provided by CMS officials to evaluate the methodology. They  gave us the following information about their sampling method. At the  beginning of each year, CMS determines a target for the number of  hospitals that will be sampled for validation surveys in each state. Each  month, CMS receives a list of hospitals scheduled for a JCAHO  accreditation survey in that month. Prior to sampling, CMS removes from  the list those hospitals that have received a validation survey in the last 3- year accreditation cycle and hospitals that do not participate in Medicare.  In the first month of the year, CMS selects a random sample of hospitals to  be surveyed from JCAHO’s list. In subsequent months, CMS removes  hospitals in states in which the state target has been met and then selects a  random sample of hospitals. Prior to sending the list to state survey  agencies, CMS determines which hospitals will receive traditional  validation surveys and which will receive other types of surveys that  cannot be used to assess the performance of JCAHO’s hospital  accreditation program. State survey agencies must then complete  traditional validation surveys within 60 days of the completion of JCAHO’s  accreditation survey for the results to be used by CMS to measure the  performance of JCAHO’s hospital accreditation program. According to  CMS officials, the sampling procedures CMS uses are necessary because  they are not informed more than 1 month in advance which hospitals  JCAHO will survey for accreditation.
In reviewing the sampling procedures they described, we determined that  CMS initially selects a probability sample of hospitals for its state agency  validation surveys. However, hospitals have varying chances of selection  in the sample depending on the month in the fiscal year that JCAHO  performs the accreditation survey and the number of hospitals targeted for  completion that year in the state in which the hospital was located.  Additionally, the way that CMS determines which type of survey the  sampled hospital receives is not random. Therefore, the analysis we  performed is limited to those hospitals included in the validation survey  sample and cannot be projected to all JCAHO-accredited hospitals.
section_title: Appendix II: Medicare Conditions of Participation
paragraphs: To participate in Medicare, hospitals must maintain standards of patient  safety and health that comply with Medicare requirements. There are  currently 23 Medicare COPs. Table 7 provides a description of each  Medicare COP.
section_title: Appendix III: Features of JCAHO’s New Accreditation Process
paragraphs: In January 2004, JCAHO introduced a new hospital accreditation process  that includes several new features. Table 8 includes a description of  selected new features of JCAHO’s hospital accreditation process.
section_title: Appendix IV: Comments from the Centers for Medicare & Medicaid Services
section_title: Appendix V: Comments from the Joint Commission on Accreditation of Healthcare Organizations
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Elaine Swift, Linda Kohn, Behn  Kelly, Elizabeth T. Morrison, Roseanne Price, and Marie Stetser made key  contributions to this report.
section_title: Related GAO Products
paragraphs: Medicare Home Health Agencies: Weaknesses in Federal and State  Oversight Mask Potential Quality Issues. GAO-02-382. Washington, D.C.:  July 19, 2002.
Medicare: HCFA’s Approval and Oversight of Private Accreditation  Organizations. GAO/ HEHS-99-197R. Washington, D.C.: September 30,  1999.
Home Health Care: HCFA Properly Evaluated JCAHO’s Ability to Survey  Home Health Agencies. GAO/HRD-93-33. Washington, D.C.: October 26,  1992.
Health Care: Criteria Used to Evaluate Hospital Accreditation Process  Need Reevaluation. GAO/HRD-90-89. Washington, D.C.: June 11, 1990.
subsections: section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs