Clinical Lab Quality: CMS and Survey Organization Oversight Should Be Strengthened
Released: 2006-06-27T00:00:00

Letter

Background

A clinical lab is generally defined as a facility that examines specimens derived from humans for the purpose of disease diagnosis, prevention, and treatment, or health assessment of individuals. While hospital and interstate labs were previously subject to regulation, CLIA strengthened federal requirements and extended them to most other clinical labs, including physician office labs. For example, CLIA strengthened personnel requirements for lab workers, strengthened proficiency testing that evaluates the accuracy of lab testing between surveys, and created a range of sanctions to enforce compliance.

Most clinical labs regulated under CLIA must obtain a certificate from CMS and pay fees every 2 years to cover the costs of administering the CLIA program, including surveys and other oversight activities. The fees vary based on the complexity and volume of testing performed. Lab tests are categorized as waived, moderate, or high complexity. Approximately 81 percent of all labs (about 157,000) are not subject to routine biennial surveys because they perform (1) “waived” tests, which are examinations and procedures that have an insignificant risk of erroneous results, including those approved for home use or determined to employ methodologies so simple or accurate that the likelihood of erroneous results is negligible; or (2) tests performed during the course of a patient visit with a microscope on specimens that are not easily transportable.

Surveyed Labs

CLIA establishes more stringent requirements for the 19 percent (about 36,000) of labs performing moderate- or high-complexity testing, including the requirement for a survey and participation in routine proficiency testing. Since the early 1990s, the number and proportion of labs subject to surveys have declined, while the number and proportion conducting waived tests have increased. Surveys examine lab compliance with CLIA program requirements in several areas including: personnel qualifications, proficiency testing, quality control, quality assurance, and recordkeeping.

Personnel: CLIA sets minimum qualifications for all persons performing or supervising moderate- or high-complexity lab tests and specifies responsibilities for each position.

Proficiency testing: Surveyed labs must participate in an approved external proficiency testing program, which evaluates the accuracy of laboratory testing. Under this requirement, labs purchase samples with unknown characteristics several times each year from an approved proficiency testing provider. The lab is required to test the samples with its routine patient testing, and the results are returned to the testing provider to be graded. A proficiency testing failure is defined as unsatisfactory performance on two consecutive or two out of three testing events. The results of proficiency testing for all inspected labs are transmitted to CMS and maintained in a database.

Quality control: Labs must have a process for routinely monitoring personnel, testing equipment, and the testing environment to ensure proper operation and accurate results.

Quality assurance: Labs must follow their plan to monitor the overall operation of the laboratory on an ongoing basis and must resolve identified problems that affect the quality of their testing.

Recordkeeping: Labs must maintain an audit trail of testing that documents specimen integrity and test performance for all phases of the test process from the test order to the test report.

Survey Organizations

In general, labs have a choice of who conducts their surveys—state survey agencies using CLIA inspection requirements or other survey organizations that use requirements CMS has determined to be at least equivalent to CLIA’s. CMS contracts with state survey agencies in most states to inspect labs against CLIA requirements. CLIA established an approval process to allow states and private accrediting organizations to use their own requirements to survey labs. As noted earlier, New York and Washington operate CLIA-exempt programs and CMS has approved six private, nonprofit accrediting organizations to survey labs—the American Association of Blood Banks (AABB), the American Osteopathic Association (AOA), the American Society of Histocompatibility and Immunogenetics (ASHI), CAP, COLA, and JCAHO. The requirements of both state CLIA-exempt programs and accrediting organizations must be reviewed by CMS at least every 6 years to ensure CLIA equivalency, but may be more stringent than those of CLIA. For example, when inspecting labs engaged in moderate- and high-complexity testing, New York and some accrediting organizations also look at the labs’ procedures for conducting “waived” tests, which is not required under CLIA. Figure 1 lists the three types of survey organizations and indicates whether they survey labs under CLIA requirements, or use their own CLIA-equivalent requirements. It also shows the percentage of labs performing moderate- to high-complexity testing surveyed by each type of organization. In general, state survey agencies, COLA, and Washington’s CLIA-exempt program survey physician office labs, while New York’s CLIA-exempt program, CAP, and JCAHO survey hospital labs.

Surveys and Complaint Investigations

Survey organizations (1) conduct surveys and complaint investigations, and (2) monitor proficiency test results submitted by surveyed labs three times a year. Surveys are typically conducted by former or current lab workers, who assess lab compliance with CLIA or CLIA-equivalent requirements. Most lab inspections are announced, that is, the lab has advance notice of when the survey will occur. Generally, surveyors verify that lab personnel are appropriately qualified to conduct testing, evaluate proficiency test records, check equipment and calibration to ensure that appropriate quality control measures are in place, and determine whether the lab has a quality assurance plan and uses it to, among other things, appropriately identify and resolve problems affecting testing quality. Surveys also include an educational component to assist labs in understanding how to comply with CLIA requirements. The duration of a survey generally depends on the size—in terms of the number of tests conducted—and complexity of a lab as well as the number of surveyors. Thus, a survey conducted at a small lab may only take a few hours to complete, while a survey at a large hospital lab may take a survey team a full week or more.

In addition to inspections, survey organizations are responsible for determining the seriousness of and investigating all complaints. For those complaints that are determined to pose immediate jeopardy—an imminent and serious threat to patient health and a significant hazard to public health—CMS requires that the investigation be initiated within 2 working days. Complaints may be investigated on-site or through communications between the survey organization and the lab. Complaint investigations for all survey organizations are unannounced.

Lab survey requirements are classified as either “standard-” or “condition-” level. Generally, condition-level requirements are made up of one or more related standard-level requirements. For example, the condition-level requirement on enrollment and testing of samples through a proficiency testing program has two related standard-level requirements: (1) enrollment, which includes requirements for the lab to provide the name of the program it has enrolled in to the Department of Health and Human Services and authorize the release of testing data to the department; and (2) testing, which specifies that the samples must be tested in the same manner as any specimen and prohibits referring the test samples to another lab for analysis.

Deficiencies are also characterized as standard- or condition-level based on the requirement in which the deficiency occurs. Deficiencies in standard-level requirements, that is, standard-level deficiencies, denote problems that generally are not serious, while condition-level deficiencies are cited when the problems are serious or systemic in nature. A serious problem is defined as an inadequacy in a lab’s quality of services that adversely affects, or has the potential to adversely affect, the accuracy and reliability of patient test results. When deficiencies are found during surveys or complaint investigations, labs are required to submit a plan of correction, detailing how and when they will address the deficiencies. Additionally, CMS can impose principal or alternative sanctions, or both. Principal sanctions include revocation of a CLIA certificate, cancellation of the right to receive Medicare payments, or limits on testing. Revocation of a CLIA certificate is equivalent to termination from the CLIA program. Alternative sanctions are less severe and include civil money penalties or on-site monitoring. For condition-level deficiencies that do not involve immediate jeopardy, labs have an opportunity to correct the deficiencies, which we refer to as a grace period, before the sanctions are imposed. If a lab is unable to correct a deficiency during this grace period, CMS determines whether to impose a sanction and the type of sanction.

CMS Oversight

CMS, including its 10 regional offices, oversees state and accrediting organization survey activities. CMS reviews and approves initial and subsequent applications from exempt-state programs and accrediting organizations to ensure CLIA equivalency. Validation reviews are one of CMS’s primary oversight tools. Federal surveyors in CMS regional offices are responsible for conducting validation reviews of state survey agency and exempt-state program inspections, but state survey agency staff conduct the validation reviews of accrediting organization inspections. An objective of these reviews is to determine if all condition-level deficiencies were identified. These reviews are conducted within 60 days of a state’s or 90 days of an accrediting organization’s survey of a lab. Starting in 1999, CMS required that at least one validation review be conducted simultaneously with an accrediting organization’s survey, a step intended to encourage an exchange of ideas and approaches among surveyors. CMS also encourages the use of simultaneous reviews of state survey agency inspections. By law, the number of labs selected for validation reviews must be sufficient to allow a reasonable estimate of the performance of each accrediting organization being assessed. CMS requires fewer validation reviews of state survey agency lab surveys (1 percent) than for those of exempt-state programs or accrediting organizations (5 percent).

Beginning in 2003, CMS regional offices began reviewing the activities of state survey agencies against a set of 13 performance standards. The standards cover areas such as the timeliness of lab inspections, surveyor personnel qualifications and training, CLIA data management, and the handling of complaints. CMS’s goal is to evaluate each state survey agency’s ability to carry out its CLIA responsibilities and to make improvements. CMS is also developing performance standards for other survey organizations that inspect labs using their own CLIA equivalent requirements.

Insufficient Data Exist to Identify Extent of Serious Lab Quality Problems

The extent of serious quality problems at labs is unclear because CMS has incomplete data on condition-level deficiencies identified by state survey agencies prior to 2004. We also found that the lack of a straightforward linkage between CLIA requirements and the CLIA-equivalent requirements of some survey organizations makes it virtually impossible to assess lab quality in a standardized manner, such as identifying the proportion of labs with condition-level deficiencies. Such deficiencies indicate serious or systemic quality problems. Proficiency testing results—the one available data source that can be used to uniformly compare lab quality across survey organizations—raise questions about whether lab quality has improved in recent years.

Limited Data Are Available on the Quality of Labs Inspected by State Survey Agencies

CMS’s OSCAR database contains limited data on the quality of labs inspected by state survey agencies and, as a result, it is not possible to analyze changes in the quality of lab testing over time. In January 2004, CMS implemented revised CLIA survey requirements and modified the existing OSCAR data—state survey agency findings—to reflect the changes. The revisions affected approximately two-thirds of the CLIA condition-level requirements. As a result of the data modifications, the findings for surveys conducted prior to 2004 no longer reflect all key condition-level requirements in effect at the time of those surveys. Based on the available 2004 OSCAR data (which represent about one half of all labs surveyed by state survey agencies), we found that 6.3 percent of labs had condition-level deficiencies (see app. II for data on all state survey agencies, including the District of Columbia). As will be discussed below, similar data are not available for labs surveyed by other survey organizations.

Quality of Labs Inspected by Other Survey Organizations Is Very Difficult to Measure in a Standardized Manner

Differences between the inspection requirements that state survey agencies use to measure lab quality and those of exempt-state programs and accrediting organizations make it virtually impossible to measure lab quality in a standardized manner. Because exempt-state programs and accrediting organizations do not classify inspection requirements and related deficiencies as either standard- or condition-level, they cannot easily identify the number of CLIA condition-level deficiencies cited at the labs they survey or the proportion of surveyed labs with condition-level deficiencies.

We asked exempt-state programs and accrediting organizations what percentage of their requirements, and any deficiencies cited for failure to meet those requirements, indicated serious problems that were equivalent to CLIA condition-level deficiencies. While only 8 percent of CLIA requirements used by state survey agencies are classified as condition- level and therefore serious, the proportion of requirements that exempt- state programs and accrediting organizations classify as serious ranged from 20 percent up to 100 percent (see table 1).

CAP and COLA crosswalked their recent survey findings to CLIA condition-level requirements. Although their analysis suggested that from about 56 to 68 percent of labs surveyed during 2004 had a deficiency in at least one condition-level requirement, they acknowledged that these proportions overstated the subset of labs with serious problems. JCAHO did not crosswalk its inspection requirements to those of CLIA because staff would have had to manually review each survey report to determine which deficiencies were equivalent to deficiencies in CLIA condition-level requirements. However, JCAHO did tell us that in 2004, about 90 percent of the labs it surveyed had a deficiency in at least one requirement and, as previously noted, JCAHO classifies all its requirements as serious.

Despite the difficulty of identifying CLIA equivalent condition-level deficiencies, two of the three accrediting organizations we reviewed have systems to identify labs they survey that have serious quality problems. COLA estimated that about 9 percent of labs it surveyed in 2004 were subject to closer scrutiny because of the seriousness of the problems identified. According to JCAHO, about 5 percent of the labs it surveyed in 2004 were not in compliance with a significant number of requirements. The third accrediting organization, CAP, has criteria for identifying labs that warrant greater scrutiny, but CAP officials told us that identifying such labs had to be accomplished on a case-by-case basis rather than through a database inquiry. As a result, CAP plans to spend in excess of $9 million during 2006 and 2007 to develop an integrated data system that pulls together multiple factors—survey results, complaints, proficiency testing, findings of other inspection bodies, and changes in lab directors— to enable it to readily identify problem labs. According to CAP officials, such labs will be targeted for greater monitoring, and CMS and other survey organizations will be notified about CAP’s actions.

Proficiency Testing Results Suggest that Quality Has Not Improved at Hospital Labs in Recent Years

Although CMS noted that proficiency testing trend data show a decrease in failures for labs as a whole, the data suggest that lab quality may not have improved at hospital labs for the period 1999 through 2003. Proficiency testing is an important oversight tool for survey organizations because it is an objective indicator of a lab’s ability to consistently produce accurate test results and is conducted more frequently than surveys—three times a year versus once every 2 years. In the absence of comparable survey data, proficiency testing results provide a uniform way to assess the quality of lab testing across survey organizations.

Our analysis of CMS proficiency testing data for 1999 through 2003 suggests that there has been an increase in proficiency testing failures for labs inspected by CAP and JCAHO, which generally inspect hospital labs, and a decrease in such failures for labs surveyed by state survey agencies and COLA, which tend to inspect physician office labs (see fig. 2). CMS defines failures as unsatisfactory performance in two consecutive or two out of three proficiency testing events. For example, the percentage of labs with proficiency testing failures surveyed by CAP and JCAHO from 1999 through 2003 increased from 4.1 percent to 6.8 percent and from 6.6 percent to 7.8 percent, respectively. It is unclear, however, whether the decrease in failures for physician office labs represents an actual improvement in lab quality or reflects the fact that some problematic labs are no longer surveyed. Specifically, many physician office labs now perform waived tests and therefore are no longer surveyed or participate in proficiency testing. Between 1998 and 2005, the percentage of labs subject to surveys and proficiency testing decreased from about 30 percent to about 19 percent.

Oversight Weaknesses Mask Quality Problems

Weaknesses in surveys, complaint processes, and enforcement mask real and potential quality problems at labs. Survey weaknesses include: (1) inspections that most organizations announce ahead of the visit, which allows labs to prepare for their inspections and portray themselves in a manner that may not accurately reflect their day-to-day quality assurance processes; (2) variability in the proportion of labs with condition-level deficiencies in 2004, which suggests surveys are not conducted in a consistent manner; and (3) the goal of educating lab workers during surveys taking precedence over, or precluding, the identification and reporting of deficiencies. Furthermore, the significant increase in complaints since CAP took steps to help ensure that lab workers know how to file a compliant suggests that some quality problems at labs inspected by other survey organizations may not be reported. Finally, sanctions are not being used effectively as an enforcement tool to promote labs’ compliance with CLIA requirements, as evidenced by the relatively few labs with repeat condition-level deficiencies on consecutive surveys from 1998 through 2004 that had sanctions imposed.

Announced Surveys May Result in Unrealistic Picture of Lab Quality

Because labs can and do prepare for surveys, CMS regional office officials and most of the state survey agencies acknowledged that announced surveys may not always provide a realistic picture of lab quality. As shown in table 2, the amount of advance notice for surveys varies from as little as 2 weeks up to 12 weeks; until recently, only the New York CLIA- exempt program conducted unannounced surveys. Survey agency officials in two states told us that surveyors had inspected labs where records documenting the implementation of periodic quality control procedures were completed in the same handwriting using the same colored pen. This degree of uniformity raises a concern about whether the quality control occurred at all, or as frequently as the records suggested. A CAP surveyor told us that the pathologist at one lab had cleaned up, and signed off on, about 3-months worth of quality control records the night before the survey. In hearings on the questionable test results at a Maryland hospital lab, a worker testified that lab staff prepared frantically for their announced inspections.

In 2006, both CAP and JCAHO began conducting unannounced inspections at most of the hospital labs they survey. Both CAP and JCAHO officials told us that the unannounced surveys will occur as early as 6 months prior to the anniversary of a lab’s prior survey. CMS and survey organizations that inspect physician office labs provided several justifications for continuing to announce inspections at such labs, including (1) ensuring that the lab is open and that appropriate personnel are available to answer surveyor’s questions, and (2) minimizing disruptions to patient care. These justifications appear to be reasonable because they reflect the operating tempo at physician office labs. It may not be appropriate, however, to provide such labs with 4 to 12 weeks advance notice, given that CMS currently limits the advance notice provided by state survey agencies to no more than 2 weeks.

Variability in Reported Survey Deficiencies Suggests that Labs Are Not Surveyed Consistently

Variability in OSCAR data for state survey agency inspections conducted in 2004 suggests that labs are not surveyed in a consistent manner, and interviews with CMS and state survey agency officials confirmed this hypothesis. As a result, available data likely understate the extent of serious quality problems at labs. In 2004, the percentage of labs that state survey agencies reported with condition-level deficiencies varied considerably by state, ranging from none in 6 states to about 25 percent of labs in South Carolina. These data only included findings for about one- half of the labs surveyed by state survey agencies. Of the 33 states that survey more than 100 labs, 16 found condition-level deficiencies at fewer than 5 percent of labs, while 6 states identified such serious deficiencies in more than 10 percent of the labs they surveyed (see app. II).

Based on interviews with CMS and 10 state survey agencies, it appears that at least some of this variability is due to differences in states’ approaches to surveys as opposed to true differences in lab quality. For example, CMS told us that, because there is not a prescriptive checklist to guide the survey process, the reliance on state surveyor judgment will result in variations in the citing of deficiencies. To compensate for the unstructured nature of the state survey process, officials we interviewed from 2 state survey agencies told us that they created checklists to help ensure that surveyors looked at all of the critical elements during lab surveys. Furthermore, while some of the state survey agencies we spoke with told us that their surveyors always cite condition-level deficiencies that are identified during surveys, officials in other states said that there are circumstances under which condition-level deficiencies would not be cited. For example, according to officials from a state survey agency we interviewed, surveyors prefer not to cite condition-level deficiencies. Rather, surveyors in this state prefer to cite multiple standard-level deficiencies instead of a condition-level deficiency because it allows the imposition of state law sanctions, avoiding what was characterized as a less efficient federal sanctions process. Additionally, officials from 2 other state survey agencies explained that surveyors consider a lab’s compliance history when determining what deficiencies to cite, while officials from a third state told us that surveyors will educate lab workers, particularly new lab workers, about the CLIA requirements rather than citing CLIA condition-level deficiencies.

Balance Between Educational and Regulatory Roles by CMS and Survey Organizations Appears to Be Inappropriate

The goal of educating lab workers sometimes takes precedence over, or precludes, the identification and reporting of deficiencies that affect the quality of lab testing. As a result, data on the quality of lab testing and trends in quality over time may be misleading. Although CLIA neither requires nor precludes an educational role for surveyors, the preamble to CMS’s implementing regulation noted that surveys are intended, in part, to provide an opportunity for on-site education regarding accepted laboratory procedures. In addition, CMS guidance and training encourage state surveyors to play an educational role. Many state survey agency officials we interviewed also told us that their surveyors play a major educational role. As noted earlier, surveyors from one state survey agency do not cite condition-level deficiencies when lab workers are new but prefer to educate the new staff. Because CMS revised its OSCAR database in 2004, it is not possible to identify states that have consistently not cited condition-level deficiencies, data that would help to quantify the extent to which an educational role is substituting for appropriate regulation of labs.

An inappropriate balance between the educational and regulatory role is also evident in some accrediting organization practices. One of the CAP surveyors we interviewed with over 30 years of lab experience estimated that the majority of pathologists—individuals who generally serve as CAP survey team leaders—view surveys as educational, rather than as assessments of compliance with lab requirements. Another surveyor told us that CAP’s survey process focuses heavily on education, and that some survey team leaders emphasize education more than others. For COLA, the process of educating labs begins even prior to a survey. For example, COLA encourages labs to submit a self-assessment for review prior to the scheduled survey so that the labs can identify COLA requirements with which they are not in compliance. About 20 percent of all labs surveyed during 2004 submitted a self-assessment (616 labs) and, compared to labs that did not submit a self-assessment, fewer deficiencies were identified at these labs during on-site surveys.

CMS appears to be inappropriately stressing education over regulation in its implementation of (1) 2003 lab quality control requirements for the CLIA program and (2) proficiency testing for lab technicians who interpret Pap smears, a test for cervical cancer. When state surveyors began assessing compliance with new lab quality control requirements in January 2004, they were instructed to note deficiencies on a cover letter to labs rather than on the survey report itself for a period of 2 years. Thus, such deficiencies are not recorded in the OSCAR database. In part because of a lack of lab “buy-in” for some of the new policies and procedures, CMS officials have extended the educational period for about another 2 years. CMS has taken a similar educational approach to Pap smear proficiency testing, which began in 2005. CMS will not cite deficiencies or impose sanctions against labs in which staff fail the new Pap smear proficiency testing in 2005 or 2006, as long as the labs and individuals involved complete such testing, including following the regulatory protocol for subsequent testing in the case of an initial failure. According to CMS, this educational focus allows labs and their staff to become familiar with the proficiency testing program and to prepare themselves for such testing, since there was about a 13-year time lag between the 1992 regulations that implemented CLIA and the 2005 implementation of Pap smear proficiency testing. This educational approach seems questionable given CMS’s concern about some of the high initial proficiency test failure rates.

Use of Volunteer Surveyors by CAP Raises Concerns

Although state survey agencies, exempt-state programs, COLA, and JCAHO employ dedicated staff surveyors, CAP relies primarily on volunteer teams consisting of lab workers from other CAP-inspected labs to conduct surveys. In contrast to the mandatory training and continuing education programs in place for the staff surveyors of other survey organizations, training for CAP’s volunteer surveyors is currently optional. CAP plans to establish a mandatory training program beginning in mid- 2006. As a condition of accreditation, labs inspected by CAP must survey another CAP-accredited lab of similar size and composition at least once every 2 years. According to data provided by CAP, two-thirds of volunteer surveyors who had recently participated in a survey had no formal training in the 3 to 5 years preceding the survey. Two CAP surveyors we interviewed told us that they had not completed any training because it was optional. Two other surveyors told us that they had never been notified about the existence of optional CAP training. While full-time surveyors employed by other survey organizations conduct from 30 to about 200 surveys per year, CAP volunteer surveyors have much less experience conducting surveys because they only survey about one lab each year.

Three of the nine CAP surveyors we interviewed stated that they believed that mandatory training was important, but some surveyors wondered when lab workers would have time to complete the courses because of their demanding work schedules. According to CAP officials, however, the required training will take only 1 to 2 days and surveyors will have a choice of live seminars and workshops or e-learning completed at their own computers. For ongoing training requirements, CAP plans to give surveyors a choice of taking additional training or passing a competency evaluation. CAP will track compliance with its new training requirements to ensure that surveyors successfully complete training and demonstrate competency within 2 years of participating in a survey. CAP’s required training is less extensive than that required by other survey organizations. For example, state survey agency inspectors must complete 5 days of basic training and periodic advanced courses afterwards while COLA staff inspectors participate in a 5-week orientation program and an annual 20 hours of continuing education.

The use of volunteer inspectors by CAP also raises concerns about the appearance of a conflict of interest. These concerns arise because of the way CAP survey teams are structured. CAP’s Commission on Laboratory Accreditation policy manual specifies that the inspection team leader is the individual responsible for the conduct of an ongoing site inspection, and must not be in a business, professional, or personal relationship that would preclude an objective inspection. Furthermore, the manual states that the inspection team leader is usually responsible for determining the size of, and assembling, the inspection team. However, until April 2006, CAP policy did not preclude competing labs from surveying one another or lab survey team members from soliciting business, such as referrals, from a lab at the conclusion of the survey. The policy was also silent about survey team members’ business, professional, or personal relationships that could cloud their independence. Typically, inspection team leaders are pathologists who direct other labs in the community, and the inspection team is comprised of several employees from the team leader’s lab.

We believe that the use of volunteers, including those from nearby labs, and the personal and professional relationships that may exist among lab staff and survey team members, creates the appearance of a conflict of interest and could undermine the integrity of the survey process. Comments from some CAP surveyors we interviewed raise a concern about having survey team leaders who are also the day-to-day supervisors of team members. For example, lack of agreement about the seriousness of a deficiency could result in the team leader instructing the team to downgrade the deficiency to a recommendation, a less serious finding that does not appear in the inspection report. Team members who are subordinates to the team leader may feel that they have no other recourse than to follow the team leader’s instructions. Recognizing that team members’ objectivity may be compromised in this situation, CAP’s revised conflict of interest policy instructs all parties to be cautious to retain objectivity in fact finding throughout the inspection process.

In discussing these findings with CAP officials, they told us that they plan to institute a number of initiatives to help ensure survey objectivity, including (1) resurveying the same lab by an independent team to assess the consistency of inspections, (2) centralizing survey team assignments performed by CAP staff, (3) not announcing surveys, and (4) not notifying labs of the survey team composition prior to the survey.

Lab Workers Who File Complaints About Quality Problems in Lab Testing Not Afforded Whistle- blower Protections

Some lab workers may not be filing complaints about quality problems at their labs because of anonymity concerns or because they may not be familiar with filing procedures. Complaints about labs can come from a variety of sources, including lab workers. Complaints are an important tool in detecting quality problems between lab surveys. For example, complaints about testing at a hospital lab were crucial because information had been concealed, complicating the detection of quality problems during the lab’s surveys. As a result of a complaint, surveyors were able to substantiate inadequate calibration of testing equipment that could adversely affect patient care.

Based on OSCAR data and data obtained from exempt-state programs and accrediting organizations for 2002 through 2004, few complaints were received about lab testing relative to the number of labs—significantly less than one complaint per lab per year. The low volume of lab complaints may be related to complainants’ concerns about anonymity and fear of retaliation for filing a complaint. It may be easy for a lab to determine the source of a complaint filed by a lab worker. For example, in some cases, either the nature of the complaint or the piece of testing equipment in question could narrow the list of possible complainants. Two CAP surveyors we interviewed commented that, in their opinion, it would be easy to determine the identity of a complainant. During congressional hearings in 2004, a Maryland hospital lab worker testified that she and her colleagues feared losing their jobs because of the complaints they filed.

Because of the difficulty of protecting the anonymity of lab workers who file complaints, whistle-blower protections for such individuals are particularly important. Two of the three accrediting organizations we interviewed have whistle-blower protections—CAP and JCAHO. For example, CAP implemented a comprehensive whistle-blower protection policy in July 2004 that includes revocation of accreditation or other appropriate action for any lab that directly or indirectly threatens, intimidates, or retaliates against a lab worker. While officials from New York and Washington’s exempt-state programs told us that whistle-blower laws in their states provide some protection for lab workers who file complaints, officials in most of the other 10 states we interviewed told us that they did not have any whistle-blower protections or were unable to identify specific protections that applied to lab workers in their state. Currently, there are no federal whistle-blower protections specifically for workers in labs covered by CLIA. In 2005, legislation was introduced to provide whistle-blower protections to workers in labs covered by CLIA.

We also found that lab workers may not know how to file a complaint. CAP experienced a significant increase in the number of complaints it received since October 2004, when it began requiring CAP-inspected labs to display posters on how to file complaints. Specifically, from October through December 2004, CAP received an average of 22 complaints per month, compared to an average of 11 complaints per month in the 9 months preceding the poster requirement. As a result, the number of complaints about the quality of lab testing more than doubled in 2004 and the number substantiated increased by more than 40 percent—even though the poster was only displayed for the last 3 months of 2004 (see table 3). In September 2005, COLA also began requiring labs to display a complaints poster similar to CAP’s. It is too early, however, to determine the impact of COLA’s new complaints poster on the number, type, and substantiation rate of complaints. Neither CMS nor JCAHO plans to require a similar complaints poster.

Lab Sanctions Are Rarely Imposed

Few labs were sanctioned by CMS from 1998 through 2004—even those with the same condition-level deficiencies on consecutive surveys— because many proposed sanctions are never imposed. Our analysis of CMS enforcement data from 1998 through 2004 found that 501 labs were sanctioned, which equates to less than 3 percent of labs inspected by state survey agencies. The most common were principal sanctions, which may result in suspension or limitation of testing or termination from the CLIA program; few labs were subjected to alternative sanctions, such as directed plans of correction or civil monetary penalties (see table 4). Appendix III shows the number of labs surveyed by state survey agencies and the number of sanctioned labs from 1998 through 2004.

Although few labs were sanctioned from 1998 through 2004, over 9,000 labs had sanctions proposed during that same time period. Before sanctions go into effect, labs are given a grace period to correct condition- level deficiencies, unless the deficiencies involve immediate jeopardy, that is, an imminent threat to patient health and significant hazard to public health. Most labs correct the deficiencies within the grace period. CMS officials told us that it was appropriate to give labs an opportunity to correct such deficiencies within a prescribed time frame and thus avoid sanctions. However, a principal objective of the enforcement process— one reflected in CMS guidance—is to motivate labs to comply with CLIA requirements, thereby helping to ensure the provision of accurate and reliable test results. Based on the large number of labs with proposed sanctions that were never imposed, it is unclear how effective the enforcement process is at motivating labs to consistently comply with CLIA requirements.

The number of labs with the same repeat condition-level deficiencies from one survey to the next also raises questions about the overall effectiveness of the CLIA enforcement process. From 1998 through 2004, 274 labs surveyed by state survey agencies had the same condition-level deficiency cited on consecutive surveys and 24 of these labs had the same condition- level deficiency cited on more than two surveys. This analysis may understate the percentage of labs with repeat condition-level deficiencies because OSCAR data prior to 2004 no longer reflect about two-thirds of condition-level requirements and associated deficiencies at the time of those surveys. We found that only 30 of the 274 labs with repeat condition- level deficiencies had sanctions imposed—either principal, alternative, or both. According to the CLIA legislative history, congressional concern about labs with repeat deficiencies led to alternative sanctions to provide an enforcement option short of principal sanctions to encourage compliance.

From 1998 through 2004, less than 1 percent of accredited labs (81) lost their accreditation; few of these labs were subsequently sanctioned by CMS and many still participate in the CLIA program. Our analysis of CMS reports on sanctioned labs found that only 9 of the 81 labs had either principal and/or alternative sanctions imposed and that 1 of the 9 still performs moderate- to high-complexity testing. Based on a review of its CLIA certificate database, CMS officials told us that about half of the 81 labs still perform moderate- to high-complexity testing but could not describe the actions taken by CMS regional offices in response to the loss of accreditation. We contacted state survey agencies or CMS regional office officials to determine why 3 labs that COLA concluded had cheated on proficiency testing by referring the samples to another lab to be tested had no sanctions imposed. The purpose of proficiency testing is to provide an objective, external evaluation of the accuracy of a lab’s test results, which is negated when another lab analyzes the sample. By statute, the intentional referral of samples to another lab for proficiency testing is a serious deficiency that should result in automatic revocation of a lab’s CLIA certificate for at least 1 year. Based on our interviews, we found that the 3 labs were allowed to continue testing because they had initiated corrective actions; in effect, these labs were given an opportunity to correct a deficiency that appears to have required a loss of their CLIA certificate for at least 1 year. A fourth lab was ultimately sanctioned for proficiency testing cheating by CMS but was allowed to continue testing for almost 2 years after having its accreditation revoked.

We also attempted to analyze data on other actions, short of revoking accreditation, used by accrediting organizations to encourage lab compliance and, in particular, how they respond to labs with serious repeat deficiencies. According to CMS, this information is dispersed across CMS regional offices. CAP officials told us that they could initiate four intermediate actions including probation (lab is closely watched to ensure correction of problems), accreditation with conditions (nonroutine inspection to be scheduled), suspension of a lab section, and cessation of a specific type of testing; suspension and probation were instituted in 2004. According to CAP, in 2005, 28 labs were on probation, 106 labs were accredited with conditions, 1 lab was suspended, and 7 labs were required to cease a specific type of test. In 2004, JCAHO awarded conditional accreditation to 3 percent of the labs it inspects because they were not in substantial compliance with its survey requirements, as evidenced by the number of requirements not met; JCAHO conducts an on-site follow-up survey at such labs. From 2002 through 2004, COLA required about 30 labs per year to cease testing due to issues identified during surveys and about 217 labs per year to cease testing certain tests or specialties due to unsuccessful proficiency testing.

CMS Oversight of CLIA Is Inadequate

CMS’s oversight is not adequate to help ensure that labs meet CLIA requirements. While CLIA requires proficiency testing quarterly, CMS only requires such testing three times each year. In addition, the agency is not meeting its responsibility to determine that accrediting organization and exempt-state requirements and processes continue to be at least equivalent to CLIA’s. CMS attributed the delay in making equivalency determinations to having too few staff. Further, ongoing CMS validation reviews do not provide an independent assessment of the extent to which surveys identify all condition-level deficiencies—primarily due to their timing. Finally, CMS does not adequately use data, such as the results of surveys, to monitor survey organization activities and processes. Realizing that its existing oversight activities need to be strengthened, CMS has begun instituting performance reviews to measure survey organization compliance with its standards and is developing protocols to ensure improved communication among survey organizations concerning complaints about lab quality.

CMS’s Implementation of Proficiency Testing Is Inconsistent with CLIA

CMS’s decision to require proficiency testing for almost all laboratory tests only three times a year is inconsistent with the statutory requirement. CLIA requires that proficiency testing be conducted “on a quarterly basis, except where the Secretary determines for technical and scientific reasons that a particular examination or procedure may be tested less frequently (but not less often than twice per year).” The committee report on the bill that forms the basis for much of CLIA indicated that “proficiency testing should be the central element in determining a laboratory’s competence, since it purports to measure actual test outcomes rather than merely gauging the potential for accurate outcomes.”

In CMS’s 1992 rule implementing CLIA, the agency provided a rationale for reducing the frequency of proficiency testing, but did not provide a technical and scientific basis for reducing the frequency for particular procedures or tests. According to CMS’s justification, experts were divided on the appropriate frequency of proficiency testing generally. In addition, requiring fewer events of proficiency testing would give laboratories more time to analyze the causes of test failures before the next event of proficiency testing and also enhance proficiency testing’s value as an educational tool. Because CLIA increased the number of labs that were required to undergo proficiency testing, CMS believed that the number of organizations that provided proficiency testing services would not have been able to meet the anticipated increase in demand for testing services. To help avoid anticipated delays in completing proficiency testing and reporting requirements, CMS reduced the frequency of testing events to three times per year.

CMS’s requirement for proficiency testing does not meet the conditions specified in the statute that must be satisfied in order to require testing less frequently than quarterly. The language of the statute, as well as relevant legislative history, indicate that a decision to reduce the frequency of testing should be in the nature of an exception made with regard to a particular test, not the norm for all tests, and must be based on “technical and scientific” considerations related to that particular test. The reasons that CMS gave for requiring only three events per year were not based on scientific and technical considerations relevant to particular tests. Instead, CMS’s decision was based on concerns of an administrative and logistical nature that CMS wanted to alleviate by reducing the frequency of testing events.

CMS Is Late in Ensuring CLIA Equivalency of Exempt States’ and Accrediting Organizations’ Inspection Requirements and Processes

We found that CMS has been late in determining that exempt states’ and accrediting organizations’ inspection requirements and processes are at least equivalent to CLIA’s. CMS must verify their equivalency and, by regulation, CMS requires such survey organizations to seek reapproval at least once every 6 years, or more frequently if deemed necessary. CMS establishes the time frames for when the next reapproval should occur, which have ranged from about 15 months to about 6 years. However, CMS has not completed its equivalency reviews within these time frames and accrediting organizations and exempt state programs have continued to operate without proper approval. Equivalency reviews for CAP, COLA, JCAHO, and Washington due to be completed between November 1, 1997, and April 30, 2001, were an average of about 40 months late. In August 1995, CMS determined that New York’s next equivalency review should be completed by June 30, 2001, but was over 4 years past due as of December 2005. Similarly, COLA’s equivalency review was about 3 years past due.

Because accrediting organizations and exempt-state programs may choose to make changes to their inspection requirements between periodic equivalency reviews (1) accrediting organizations are required to submit changes to their inspection requirements and policies 30 days prior to changing their standards and (2) exempt-state programs are required to provide notice when they change their licensure or inspection requirements. Although federal regulations require CMS to review equivalency when an accrediting organization or exempt-state program adopts new requirements, a CMS official told us that the agency is not required to review such changes before their implementation to ensure equivalency. As a result, such survey organizations may introduce changes that are inconsistent with CLIA requirements. For example, JCAHO made a significant change to its inspection requirements in January 2004 but did not receive CMS approval until 6 months later; CMS did not begin an in-depth review of JCAHO’s revised requirements until early 2005—over a year after they were implemented by JCAHO. According to CMS, their review has identified several critical areas where JCAHO standards are less stringent than those of CLIA. JCAHO acknowledged the need to make some adjustments to its revised requirements.

CMS officials attributed delays in making equivalency determinations and reviewing interim changes to having too few staff. The CLIA program, located in CMS’s Center for Medicaid and State Operations (CMSO), currently has approximately 21 full-time-equivalent positions compared to a peak of 29 such positions several years ago. The reduction occurred over time through attrition. As required by statute, the CLIA program is funded by lab fees and since its inception the program’s fees have exceeded expenses. As of September 30, 2005, the CLIA program had a carryover balance of about $70 million—far more than required to hire an additional six to seven staff members. However, CMS officials told us that because the CLIA program staff are part of CMSO, they are subject to the personnel limits established for CMSO, regardless of whether or not the program has sufficient funds to hire more staff. Although CMSO is at its authorized personnel allocation, the CLIA program could hire additional staff with approval from the Administrator. We were told that CMSO has not requested such approval.

We also noted issues that raise a question about the thoroughness of CMS equivalency reviews because some survey organizations’ procedures or policies appear to be less stringent than those required by CMS for the CLIA program. For example: Accrediting organizations provide labs more advance notice about upcoming surveys than CMS allows state survey agencies to give to the labs they inspect.

JCAHO surveyors focus their review of lab testing on the 12 months prior to the survey. CMS requires that state surveyors review the entire 24 months of testing since the last survey.

While CMS requires initial and advanced surveyor training, CAP encourages but does not require its volunteer surveyors to participate in surveyor training.

As of August 2005, CAP’s policy manual indicates that complaint investigations may be announced or unannounced. CMS guidance requires that complaint investigations be unannounced.

Prior to 2005, CMS’s equivalency determination reviews focused on the inspection requirements themselves, and not the procedures and policies used by accrediting organizations and exempt-state programs in carrying out oversight of labs; this focus on inspection requirements may explain the divergence from the policies and procedures CMS requires for state survey agencies. During 2006, CMS is simultaneously reviewing the equivalency of COLA and JCAHO inspection requirements and, for the first time, incorporating on-site observations of accrediting organization policies and systems into the review and approval process. For example, CMS is checking to ensure that accrediting organizations have adequate systems in place to track such things as (1) complaints, (2) correction of deficiencies, and (3) proficiency testing.

Many CMS Validation Reviews Lack Independence and Reviews Skip Some State Survey Agencies

CMS validation reviews that are intended to evaluate lab surveys conducted by both states and accrediting organizations do not provide CMS with an independent assessment of the extent to which surveys identify all serious—that is, condition-level or condition-level equivalent— deficiencies. CMS requires its regional offices to conduct validation reviews of 1 percent of labs inspected by state survey agencies in a year. In contrast, validation reviews of 5 percent of labs inspected by accrediting organizations during a year are conducted by state survey agency personnel. CMS does not specifically require that validations occur in each state and some states do not have validation reviews each year. Furthermore, many validation reviews occur at the same time a survey organization conducts its inspection and, in our view, the collaboration among the two teams during these simultaneous surveys prevents an independent evaluation.

Validation of State Survey Agency Lab Surveys

The requirement to validate 1 percent of labs surveyed by state survey agencies in a year—roughly 100 validation reviews each year—does not ensure sufficient oversight of state survey agencies. The validation review requirement, which is included in CMS regional office annual budget memorandums, does not specify how many validation reviews must be conducted in each state. While the 10 CMS regional offices generally validated 1 percent of the state survey agency inspections within their region, they often did not validate 1 percent of inspections within each state and, in fact, performed none in some states. From 1999 through 2003, federal surveyors: validated less than 1 percent of labs surveyed by state survey agencies in an average of about 25 percent of states, ranging from 7 states in 2002 to 17 states in 2003; and did not conduct any validation reviews in an average of 16 percent of states per year, ranging from 3 states in 2002 to 12 states in 1999.

In 11 states, no validation reviews were conducted in multiple years. For example, no validation reviews were conducted in Michigan and Washington, D.C. during 4 of 5 years from 1999 through 2003. Without validating at least some surveys in each state, CMS is unable to determine if the states are appropriately identifying deficiencies.

Seventy-five percent of validations of state lab surveys were conducted simultaneously from fiscal years 1999 through 2003. According to CMS officials, the large proportion of simultaneous validation reviews provides an opportunity for federal surveyors to share information with state surveyors, monitor their conformance with CLIA inspection requirements, and identify training and technical assistance needs. However, we found that such reviews do not provide an accurate assessment of state surveyors’ ability to identify condition-level deficiencies. Of the 13 validation reviews that identified missed condition-level deficiencies, only 1 was a simultaneous review (see table 5). Validations of state surveys typically utilize one federal surveyor for either independent or simultaneous validation reviews; therefore, increasing the proportion of independent validation reviews to strengthen CMS oversight likely would not require additional federal surveyors. Moreover, conducting independent validation reviews eliminates the extra effort required to coordinate schedules to ensure that the validation reviews occur at the same time as the state survey.

According to CMS guidance, at least one validation review of an accrediting organization’s survey of labs should be conducted simultaneously each year, but not all validation reviews should be simultaneous because a combination of simultaneous and independent reviews provides a balanced view of surveyor performance. CMS officials were unable to tell us exactly how many of the roughly 275 validation reviews conducted each year from fiscal year 1999 through fiscal year 2003 were simultaneous. However, one of the three accrediting organizations we reviewed told us that a significant proportion of their validation reviews are conducted simultaneously. JCAHO estimated that 33 percent of its validation reviews were conducted simultaneously. COLA estimated that 9 percent of validation reviews conducted in 2004 and 2005 were simultaneous. Finally, CAP officials told us that, from 2002 through 2004, 11 percent of validation reviews of CAP-accredited labs were conducted simultaneously.

Given the limitations of simultaneous reviews, conducting independent validation reviews are a more effective way of ensuring the equivalency of accrediting organization inspection requirements and processes between the equivalency determinations. CMS officials told us that the agency’s intent in instituting simultaneous reviews was for state and accrediting organization surveyors to share best practices, to promote understanding of each other’s programs, and to foster accrediting organization improvement. They indicated that they considered it a learning experience both if an accrediting organization surveyor added a deficiency noted by a state surveyor to a survey report and vice versa. However, most of the state survey agency officials we interviewed told us that simultaneous validation reviews do not provide a realistic evaluation of the adequacy of accrediting organizations’ inspection process. In fact, CMS guidance encourages surveyors to discuss the survey findings prior to concurrent conferences with lab personnel to review their findings.

From fiscal years 1999 through 2003, state survey agency surveyors found condition-level deficiencies missed by accrediting organization surveyors on 64 validation reviews, but only 6 of these validation reviews were simultaneous. In contrast, 58 (91 percent) of the validation reviews that identified serious deficiencies missed by accrediting organizations were independent validation reviews. (See table 6.)

CMS Use of Data for Oversight of CLIA Program Is Limited

CMS does not routinely collect and analyze data essential for effective oversight of the CLIA program but has initiatives to automate some available data to make them more accessible for analysis. Using data to analyze activities across survey organizations can be a powerful tool in improving CMS oversight of the CLIA program. Such analyses include identifying and addressing inconsistencies in how surveys are conducted. Although CMS tracks the most frequently cited deficiencies at labs in an effort to improve quality, it does not routinely track the proportion of labs, by state, in which state survey agencies identify condition-level deficiencies—those that denote serious or systemic problems. According to a CMS official, the agency has not evaluated variability in such deficiencies since 1999. As noted earlier in this report, variability in survey findings suggests inconsistencies in how surveys are conducted. CMS also does not require exempt-state programs and accrediting organizations to routinely submit data on serious deficiencies identified at the labs they inspect, unless the deficiencies pose immediate jeopardy to the public or an individual’s health. As noted earlier, the lack of a common vocabulary on what constitutes a serious deficiency would make it virtually impossible for CMS to analyze such data.

We also found that CMS does not effectively use available data to assess clinical lab quality in areas such as proficiency testing, sanctions, and complaints. For example, CMS’s analysis of proficiency testing data for all labs showed improvements over time. As reported earlier, proficiency testing failures have increased for labs surveyed by CAP and JCAHO. Comprehensive analysis of the proficiency testing database is particularly valuable because it provides a uniform way to assess the quality of lab testing across survey organizations, which is not currently available for survey results. CMS is now in the process of automating the annual registry of sanctioned labs, which should help it identify important trends, such as the infrequent use of alternative sanctions. Automating the registry, however, will not address the lack of data on (1) steps taken by state survey agencies and regional offices when labs have their accreditation revoked or (2) interim steps, short of revocation of accreditation, that accrediting organizations take to help encourage lab compliance. CMS also lacks a complaints database, and therefore was unable to assess the impact of CAP’s decision to require labs to prominently display a poster on how to file a complaint. CMS is developing, and plans to launch, a complaints database in March 2006.

CMS Implementing Performance Reviews for Survey Organizations

CMS has implemented performance reviews for state survey agencies and is in the process of developing such reviews for accrediting organizations. First implemented in 2004, the annual CLIA state performance reviews evaluate each survey agency’s ability to accomplish its lab oversight responsibilities. The reviews, conducted on-site by CMS regional office staff, measure performance in 13 areas, such as the timely conduct of surveys and the appropriate documentation of any deficiencies identified. According to CMS, the reviews are based on the performance- improvement model that characterizes much of the administration of the CLIA program. Consequently, the primary role of regional offices in conducting the reviews is to provide education and support for state survey agency improvement. For the 2004 reviews, 38 states were required to submit corrective action plans to their respective CMS regional offices in at least 1 of the 13 areas examined. Three areas required the most corrective action plans: principles of documentation, proficiency testing desk reviews, and survey time frames.

Principles of documentation. CMS found that some state survey agencies lacked the supervisory personnel to conduct internal reviews intended to ensure the appropriate documentation of deficiencies. It also found that some state survey agencies did not follow the protocol instructions on how to quantify such reviews.

Proficiency testing desk reviews. Because of personnel shortages, some state survey agencies were unable to perform proficiency testing desk reviews between surveys, waiting instead until the next on-site survey to address unsuccessful proficiency testing. CMS plans to provide additional surveyor training on desk review requirements.

Survey time frames. CMS regional office staff were inconsistent in scoring whether state survey agencies met the established time frames for initial surveys. While some regions were lenient if a state missed the time frame by just 1 day or provided a reasonable explanation—such as staff turnover or illness—other regions were more stringent in scoring states against the standard.

However, it is not yet clear to what extent the 2004 scores represent state survey agency shortcomings or a learning curve for the states in understanding the performance review protocols.

In partnership with the accrediting organizations, CMS is developing performance standards comparable to, but different from, those implemented in 2004 for state survey agencies. For example, both the state survey agency review protocols and those proposed for accrediting organizations measure the timeliness of the surveys, but those proposed for the latter would also focus on several areas that are unique to accrediting organizations. The performance standards would include (1) timely and consistent information sharing and (2) alerting CMS about decisions to limit or remove accreditation in a timely manner. According to a CMS official, the agency plans to phase in the performance standards, starting with a standard on complaints. For example, if the CLIA complaints database is activated in March 2006, CMS could begin to monitor accrediting organization responsiveness to, and outcomes of, complaints. Because the database will contain national lab complaint data, CMS will be able to compare the volume and outcome of complaints across survey organizations. According to CMS, implementation of the accrediting organization performance standards will be a central—not regional—office responsibility.

Conclusions

Clinical labs play a pivotal role in the nation’s health care system by diagnosing many diseases, including potentially life-threatening diseases, so that individuals receive appropriate medical care. Given this important role, lab tests must be accurate and reliable. CMS and survey organization oversight is intended to ensure that labs produce reliable test results, a key objective of CLIA. Our work demonstrated that the oversight of clinical labs needs to be strengthened in several areas.

Determining the quality of lab testing is difficult because it is virtually impossible to crosswalk inspection requirements across survey organizations. Without standardized survey findings across all survey organizations, CMS cannot tell whether the quality of lab testing has improved or worsened over time or whether deficiencies are being appropriately identified.

Lab oversight has weaknesses that make it difficult to determine the quality of lab testing because they mask quality problems. To help surveys provide a realistic picture of day-to-day operations, CAP and JCAHO began unannounced surveys of the labs they survey—generally hospital labs—in 2006. While unannounced surveys at physician office labs may not be practical, Washington’s exempt program and COLA currently give such labs more advance notice than the 2 weeks CMS prescribes for labs inspected by state survey agencies. Similarly, the greater weight that CMS and survey organizations sometimes place on their educational, as opposed to their regulatory role may lead to survey findings that do not accurately reflect lab quality. Educating labs to ensure high-quality testing should complement but not replace the enforcement of CLIA inspection requirements. The low number of lab complaints may be the result of a lack of information about how to file a complaint and lab workers’ fear of retaliation. Because protecting the anonymity of lab workers who file complaints is difficult, whistle-blower protections for such individuals are particularly important. Finally, labs with the same serious deficiencies on consecutive surveys often escape sanctions, even though Congress authorized alternative sanctions to give CMS more flexibility to achieve lab compliance. Without the threat of real consequences, labs may not be sufficiently motivated to comply with CLIA inspection requirements.

CMS’s oversight is not adequate to enforce CLIA requirements. The agency is not requiring labs to participate in proficiency testing on a quarterly basis, as required by CLIA. Furthermore CMS is not conducting CLIA- equivalency determinations within the time frames it established for such reviews, nor has it always reviewed changes to exempt-state and accrediting organizations’ inspection requirements before their implementation, even though it requires their submission to ensure continued CLIA equivalency of their requirements. Although the CLIA program has generated funds, CMS agencywide staffing limitations have prevented the program from hiring sufficient staff to complete equivalency reviews in a timely manner. Many validation reviews are conducted at the same time a survey organization conducts its survey, and such simultaneous reviews may not provide a true assessment of surveyor performance. Independent validation reviews of accrediting organization surveys are critical because CMS has not conducted equivalency reviews within the time frames it established. We also found that few validation reviews of state survey agency lab inspections are conducted each year and that none occurred in some states. Because state surveyors conduct validation reviews of accrediting organizations to ensure the continuing CLIA equivalency of their inspection requirements, conducting an appropriate number of validation reviews of state survey agency lab inspections is critical. CMS also has not yet taken the lead in ensuring the availability and use of data from survey organizations to help it monitor their performance—particularly the consistency with which surveys are conducted. CMS is creating a new complaint database, but its plan to automate the existing sanctions registry will not address the lack of data on enforcement actions taken by state survey agencies and regional offices when labs have their accreditation revoked.

Recommendations for Executive Action

To enable CMS to track the nature and extent of lab quality problems across survey organizations, we recommend that the CMS Administrator take the following action: Work with exempt-state programs and accrediting organizations to standardize their categorization and reporting of survey findings in a way that tracks to CLIA inspection requirements and allows for meaningful comparisons across organizations, such as the analysis of trends in the citation of condition-level deficiencies.

To ensure consistency in the oversight of labs by survey organizations, we recommend that the CMS Administrator take the following four actions: Ensure that the advance notice of upcoming surveys provided to physician office labs is consistent with CMS’s policy for advance notice provided by state survey agencies.

Ensure that regulation of labs is the primary goal of survey organizations and that education to improve lab quality does not preclude the identification and reporting of deficiencies that affect lab testing quality. Impose appropriate sanctions on labs with consecutive condition-level deficiencies in the same requirements.

Require all survey organizations to develop, and require labs to prominently display, posters instructing lab workers on how to file anonymous complaints.

To improve oversight of labs and survey organizations, we recommend that the CMS Administrator take the following eight actions: Consistent with CLIA, require quarterly proficiency testing, except when technical and scientific considerations suggest that less frequent testing is appropriate for particular examinations or procedures.

Ensure that evaluations of exempt-state and accrediting organization inspection requirements take place prior to expiration of the period for which they are approved in order to ensure the continued equivalency of their requirements with CLIA’s.

Ensure that changes to the inspection requirements of exempt states and accrediting organizations be reviewed prior to implementation, as required by regulation, to ensure that individual changes do not affect the overall CLIA equivalency of each organization.

Allow the CLIA program to utilize revenues generated by the program to hire sufficient staff to fulfill its statutory responsibilities.

Ensure that federal surveyors validate a sufficient number of inspections conducted by each state survey agency to allow a reasonable estimate of their performance, including a minimum of one independent validation review for each state survey agency surveyor.

Require that almost all validation reviews of each accrediting organizations’ surveys be an independent assessment of performance.

Collect and routinely review standardized survey findings and other available information for all survey organizations to help ensure that CLIA requirements are being enforced and to monitor the performance of each organization.

Establish an enforcement database to monitor actions taken by state survey agencies and regional offices on labs that lose their accreditation.

Agency and Accrediting Organization Comments and Our Evaluation

We provided a draft of this report to CMS, and to CAP, COLA, and JCAHO—the three laboratory accrediting organizations included in our review. CMS strongly endorsed our overall conclusion that quality assurance for the nation’s clinical labs should be strengthened and noted that the report provided insights into areas where it can improve, augment, and reinforce oversight of both labs and accrediting organizations to ensure quality testing. Overall, CMS concurred with 11 of our 13 recommendations. Despite this endorsement, however, CMS (1) provided an alternative assessment of lab quality, (2) disagreed that the phase-in of certain CLIA requirements inappropriately stressed education as opposed to regulation, (3) expressed concern about how to identify and sanction labs with repeat condition level deficiencies, (4) disagreed with our recommendation regarding the frequency of proficiency testing, and (5) stated that it was already meeting our recommendation to conduct almost all validation reviews of each accrediting organization independently. We continue to believe that implementation of these recommendations is necessary for the effective oversight of labs. (CMS’s comments are reproduced in app. IV.) CAP indicated that it took seriously our findings and recommendations and intended to determine if there were additional measures it could take to strengthen its own oversight. COLA said that our recommendations to improve CMS oversight of survey organizations had merit. Nonetheless, CAP, COLA, and JCAHO disagreed with some of our findings and recommendations to CMS. (CAP, COLA, and JCAHO’s comments are reproduced in app. V, VI, and VII, respectively.) Our evaluation first responds to CMS’s and related accrediting organizations’ comments and then addresses additional comments by accrediting organizations.

Assessment of Lab Quality

CMS and COLA commented that lab performance has improved since the enactment of CLIA. In particular, CMS pointed to the substantial decline— from about 80 percent to about 42 percent—in the percentage of labs nationwide with deficiencies between 1994 and 2004. It is important to note that CMS’s data (1) do not distinguish between serious condition- level deficiencies and less serious standard-level deficiencies, (2) include the early start-up period when physician office labs were first regulated, and (3) exclude deficiency data on the substantial number of labs surveyed by accrediting organizations and state CLIA-exempt programs. Due to these shortcomings, we do not believe that CMS’s data provide an accurate assessment of lab quality nationwide.

Based on the limited data available on state survey agency inspections of labs since 1998 and the lack of any comparable data on accrediting organization and exempt-state program survey findings, we concluded that insufficient data existed to identify the extent of serious quality problems at labs. CMS did not retain backup files of pre-2004 data on deficiencies identified by state survey agencies. Although CMS has determined that accrediting organization and exempt-state program lab requirements are at least equivalent to CLIA’s, there is no agreement across survey organizations on how to distinguish serious from less serious deficiencies. While CMS concurred with our recommendation to standardize the categorization and reporting of survey findings in a way that tracks to CLIA and allows meaningful comparisons across survey organizations, it noted that a straightforward linkage of requirements is limited by CMS’s authority under the statute—that is, survey organizations are permitted by statute to have different requirements—and that it will approach implementation of our recommendation cautiously. JCAHO said that it agreed with the need for a common, agreed upon, taxonomy that could be used by all survey organizations to track serious deficiencies, but commented that it thought CMS’s implementation of our recommendation would require a revamping of JCAHO’s accreditation system. That was not the intent of our recommendation and it is clear from CMS’s comments that its implementation of our recommendation would not require an overhaul of accrediting organizations’ systems. CAP acknowledged the complexity and inherent challenges in measuring the quality of lab testing, but noted that it is committed to working to develop better systems to detect labs with serious quality problems—those that impact patient care.

The statutory authority that permits standards different from CMS’s (provided they are at least as stringent) does not impair the ability to develop a crosswalk that allows for meaningful comparisons across survey organizations—such as an analysis of trends in the citation of condition- level deficiencies. In fact, CMS regulations already require accrediting organizations and exempt-state programs to submit a crosswalk—detailed comparisons of their individual accreditation or licensure approval requirements with comparable CLIA condition-level requirements—when they apply and reapply for approval from CMS. Such a comparison is possible because CMS already identifies instances when accrediting organizations have missed condition-level requirements during validation reviews. For example, CMS should require survey organizations to (1) indicate which of their requirements relate to each CLIA condition- level requirement, and (2) explain which deficiencies in their requirements, if cited, should be considered equivalent to CLIA condition- level deficiencies.

CMS also pointed to the steady increase in successful proficiency testing across all labs as an indication of improvements in lab quality. Our analysis of proficiency testing results suggested that lab quality had not improved at hospital labs in recent years. CMS correctly noted that the overall proportion of labs with no test failures increased from about 88 percent in 1998 to about 93 percent in 2003—that is, fewer labs failed proficiency testing. However, by focusing on overall proficiency testing results, CMS data mask trends in failure rates for subsets of labs such as hospital labs. For example, from 1999 through 2003, the percentage of CAP-surveyed labs with proficiency testing failures increased from 4.1 percent to 6.8 percent; CAP generally inspects hospital labs. CMS also commented that the overall improvement cannot be dismissed as a result of some labs being granted waived status because the more dramatic improvements predated the recent increase in the number of waived labs. It further commented that removing waived labs from the data would not result in improved performance rates. First, the number of waived labs— those performing waived tests or provider-performed microscopy— increased by about 26,600 from 1993 though 1998 and then increased by another approximately 33,700 labs from 1998 through 2004. Second, CMS’s comment suggested that it had conducted an analysis of the impact of removing waived labs from the proficiency testing data. However, it did not provide any data analysis when we subsequently asked to see the evidence behind its assertion. COLA also addressed this issue, and did not challenge our conclusion that the decrease in proficiency testing failures for physician office labs might not represent an actual improvement in lab quality, but instead could reflect the fact that some problematic labs are no longer surveyed.

Educational Focus of CLIA

CMS agreed that it was important to maintain an appropriate balance between its regulatory and educational approaches to CLIA implementation. While CMS noted that objective review and feedback are the bedrock of education, it emphasized that the educational approach does not preclude surveyors from identifying lab deficiencies. CAP and COLA offered similar comments. However, we found evidence that the goal of educating lab workers sometimes takes precedence over, or precludes, the identification and reporting of deficiencies and recommended that CMS take steps to ensure that regulation remains the primary goal of surveys. To address this problem, CMS stated that it will provide additional state agency surveyor training, improve guidance, develop an action plan to promote greater consistency among surveyors, and institute periodic performance and consistency reviews. CMS’s comments did not address evidence we presented that an educational emphasis may also prevent fulfillment of regulatory responsibilities by some accrediting organizations.

CMS disagreed that the extended phase-in periods for new quality control requirements and proficiency testing for lab technicians who interpret Pap smears were inappropriate. COLA noted that federal requirements in many regulated industries are phased in to allow them time to understand and effectively implement the requirements. CMS reaffirmed that, in the case of significant new requirements and for the time period specified by CMS, the educational approach may result in identified deficiencies being communicated to laboratories without a concomitant citation, as is the case with quality control and Pap smear testing requirements. As discussed in the report, we believe that CMS’s educational phase-in periods are excessive. We found that the phase-in period for new quality control requirements was extended from 2 years to about 4 years, in part because of the lack of lab “buy-in” for some of the new policies and procedures. Similarly, the phase-in period for Pap smear proficiency testing is 2 years, despite (1) CMS’s concern about some of the high initial test failure rates, (2) the consequences of inaccurate test results on patients’ diagnoses and treatment, and (3) the approximately 13-year time lag between the 1992 implementation of the CLIA regulations and the commencement of Pap smear proficiency testing.

Sanctioning Labs with Serious, Repeat Deficiencies

In commenting on our recommendation to appropriately sanction labs with repeat condition-level deficiencies, CMS acknowledged the need to carefully monitor repeat deficiencies but expressed concern that focusing on the condition cited may not indicate a true repeat deficiency because the underlying failures could have been different in the two consecutive surveys for those labs. CMS’s assertion is inconsistent with its own policy on serious, repeat deficiencies for other providers, such as nursing homes. In general, immediate sanctions must be imposed on nursing homes with consecutive serious deficiencies, regardless of whether the deficiencies are in the same care area. As we have previously reported, allowing providers to avoid sanctions by correcting serious deficiencies contributes to an up-and-down pattern of compliance and undermines the deterrent effect of sanctions. According to the CLIA legislative history, congressional concern about labs with repeat deficiencies led to the introduction of alternative sanctions such as civil money penalties as a substitute for more severe principal sanctions, which include termination from the CLIA program.

Proficiency Testing Frequency

CMS disagreed with our recommendation that it require quarterly proficiency testing except when technical and scientific considerations indicate that less frequent testing is justified for particular tests; CMS insisted that proficiency testing three times a year was “appropriate.” CMS stated that CMS and the Centers for Disease Control and Prevention had together determined that the reduced frequency was based on technical and scientific grounds. We asked for a record of the agencies’ deliberation supporting that decision. CMS supplied a brief, undated narrative, which it attributed to the Centers for Disease Control and Prevention. It was not clear to us that this narrative was contemporaneous with the decision to reduce the frequency of proficiency testing. Moreover, the narrative focused on the relative costs and benefits of proficiency testing at various intervals. There was no analysis of technical and scientific considerations with regard to particular tests that presented a basis for reducing the frequency.

Based on CMS’s response, we maintain that CMS’s decision to require proficiency testing three times a year is not authorized by CLIA. CMS did not dispute that, according to CLIA, it must base the decision to reduce the frequency of proficiency testing on scientific and technical considerations relevant to particular tests, and that the decision to reduce the frequency is in the nature of an exception to the norm of quarterly testing. CMS also acknowledged that the “public explanation” contained in the preamble to the rule setting the proficiency testing requirement at three times a year referred only to general concerns about the perceived burden associated with quarterly testing. For example, CMS stated in its final rule that the prospect of reduced frequency would provide a “needed respite” to both laboratories and proficiency test providers. In sum, CMS has not adhered to the conditions set out in the statute for reducing the frequency of proficiency testing and has implemented a policy that is not supported by statutory authority.

Equivalency of Accrediting Organization and Exempt State Programs

CMS acknowledged the need to complete timely equivalency reviews of accrediting organization and exempt-state requirements, which were an average of about 40 months late for the 3-1/2-year period we examined. Regarding interim changes made between periodic equivalency reviews, CMS agreed with our recommendation and stated it would review such interim changes for both accrediting organizations and exempt-state programs prior to implementation, as required by regulation.

Furthermore, CMS indicated that changes to accrediting organization requirements did not necessarily impact CLIA equivalency determinations because accrediting organizations may have more stringent requirements than CLIA’s. While possibly true, CMS must review the changes to determine whether CLIA equivalency is affected. For example, in 2005, when it reviewed JCAHO’s revised standards a year after they were implemented, CMS identified several critical areas where JCAHO’s standards were less stringent than those of CLIA.

CMS acknowledged that a significant increase in workload and the decline in CLIA program staff were factors which contributed to delays in making equivalency determinations and reviewing interim changes. Although CMS stated that it reserved the right to manage the work within available resources and its assessment of priorities, it also made a commitment to explore our recommendation to utilize revenues generated by the CLIA program to hire sufficient staff to fulfill its responsibilities. We believe that additional staff would not only improve the timeliness of equivalency reviews, but also their thoroughness.

Accrediting Organization Validation Reviews

CMS stated that, consistent with our recommendation, 88 percent of accrediting organization validation reviews were conducted independently in calendar year 2005. However, our recommendation was to require that almost all validation reviews of each accrediting organizations’ surveys be conducted independently. CMS’s comments do not indicate the proportion of independent validation reviews conducted for each accrediting organization. Because CMS did not begin collecting data on the number of simultaneous accrediting organization validation reviews performed until August 2003, we relied on estimates from accrediting organizations. CMS did not challenge JCAHO’s estimate that 33 percent of its validation reviews were simultaneous, compared to about 10 percent for CAP and COLA. We do not believe that performing an estimated 33 percent of JCAHO’s validation reviews simultaneously is consistent with our recommendation.

COLA commented that simultaneous validation reviews are useful in assuring consistency and in providing an understanding of processes across survey organizations. It also questioned the accuracy of most of the missed condition-level deficiencies identified by CMS during independent validation reviews. We did not assess the process CMS uses to identify such missed deficiencies but based on a discussion with CMS officials it appears that the process is thorough and time consuming. JCAHO commented that we had misinterpreted the results of simultaneous and independent validation reviews of accrediting organizations because, by JCAHO’s estimate, the proportion of missed condition-level deficiencies is roughly equivalent for both types of surveys. We did not find the assumptions behind JCAHO’s estimate convincing, given the lack of data on the actual number of simultaneous versus independent validation reviews conducted for each accrediting organization. Furthermore, most of the state survey agency officials we interviewed, whose inspectors conduct accrediting organization validation reviews, told us that simultaneous validation reviews do not provide a realistic evaluation of the adequacy of accrediting organizations’ inspection processes.

Additional Comments by Accrediting Organizations

Additional CAP comments. CAP commented that we underestimated the value of using lab professionals in the inspection process and that we provided no factual evidence that their use was less effective than other models. In contrast to CAP, other survey organizations employ dedicated staff surveyors who have mandatory and continuing education requirements. In addition, such dedicated surveyors conduct from 30 to about 200 surveys per year compared to CAP’s lab professionals who volunteer to perform about 1 survey per year. CAP partially addressed our concern about the lack of mandatory training for its volunteer surveyors. It plans to begin requiring training for survey team leaders in July 2006 and for survey team members in 2007. However, CAP’s proposed new mandatory training is much less extensive than that required by other survey organizations.

Moreover, we reported that some CAP surveyors we interviewed raised a concern about having survey team leaders who are also the day-to-day supervisors of team members. For example, lack of agreement about the seriousness of a deficiency could result in the team leader instructing the team to downgrade the deficiency to a recommendation, a less serious finding that does not appear in the inspection report. Team members who are subordinates to the team leader may feel that they have no other recourse than to follow the team leader’s instructions. CAP recently revised its conflict-of-interest policy which now instructs all parties to be cautious to retain objectivity in fact finding throughout the inspection process. We do not believe that this change in CAP’s conflict-of-interest policy addresses the concerns raised by the CAP surveyors we interviewed. In its comments, CAP indicated that it would continue to closely monitor this issue to determine if further actions were necessary.

Additional COLA comments. COLA disagreed with our assertion that announced surveys may result in an unrealistic picture of lab quality—a conclusion supported by CMS regional office staff and most state survey agency officials we interviewed. We acknowledged that unannounced surveys of the physician office labs typically surveyed by COLA and state survey agencies were not practical given the unpredictable operating hours of such labs and need to minimize disruptions to patient care. However, we recommended that the advanced notice be limited to the 2 weeks permitted by CMS for state survey agencies. COLA currently provides up to 12 weeks advanced notice. COLA contends that providing up to 6 months of advanced notice before a survey would only improve the lab’s operation more quickly if the lab took that opportunity to review COLA’s self assessment questions and correct any missing or incorrect processes or documentation. We believe that COLA’s example underscores the importance of our recommendation; such actions should be an ongoing process at labs—not a reaction to an upcoming inspections.

Additional JCAHO comments. JCAHO said that our recommendation that all survey organizations develop and require labs to prominently display posters that instruct lab workers on how to file anonymous complaints was too narrow and prescriptive and may inadvertently limit organizations from using other, more effective ways to educate lab workers on this topic. JCAHO did not explain how implementing our recommendation would limit other initiatives. In fact, CMS’s comments identified a number of promising approaches that it believed could supplement posters. JCAHO also said that our analysis of the increase in CAP complaints after it required posters in the labs it inspects failed to recognize a broad national trend. JCAHO indicated that it also experienced a dramatic increase in lab complaints between 2004 and 2005 without the use of posters. This increase may be related to JCAHO’s July 2005 requirement for labs to educate staff on how to report concerns. CAP told us that during the 3 months after they required a poster to be displayed they observed an immediate increase in the number of complaints. Thus, CAP lab complaints increased by over 100 percent in 2004 compared to 2003 and by another approximately 71 percent in 2005. We continue to believe that CAP’s experience suggests that complaint posters can be an important way to encourage lab workers to communicate their concerns.

CMS, CAP, COLA, and JCAHO also provided technical comments which we incorporated as appropriate.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the Administrator of the Centers for Medicare & Medicaid Services and appropriate congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any question s about this report, please contact me at (312) 220-7600 or aronovitzl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VIII.

Appendix I: Effects of Lab Errors on Patient Health

This appendix contains examples of lab errors and their consequences, illustrating the importance of the quality of lab testing and the effects of lab errors on patient health. The examples in table 7 are summarized from case studies in the journal Laboratory Errors and Patient Safety.

Appendix II: Labs Surveyed by State Survey Agencies and the Percentage with Condition- Level Deficiencies, by State in 2004

Appendix III: Number of Labs Subject to Surveys by State Survey Agencies in 2005 and Number of Labs with Sanctions, 1998 to 2004

Number of labs (2005)

Number of labs (2005)

Appendix IV: Comments from the Centers for Medicare & Medicaid Services

Appendix V: Comments from the College of American Pathologists

Appendix VI: Comments from COLA

Appendix VII: Comments from the Joint Commission on Accreditation of Healthcare Organizations

Appendix VIII: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Walter Ochinko, Assistant Director; Lucia P. Fort; Dan Lee; Kevin Milne; Dean Mohs; Elizabeth T. Morrison; Michelle Rosenberg; and Elizabeth Scherer made key contributions to this report.