Clinical Labs: CMS and Survey Organization Oversight Is Not Sufficient to Ensure Lab Quality
Released: 2006-06-27T00:00:00

Letter

Background

A clinical lab is generally defined as a facility that examines specimens derived from humans for the purpose of disease diagnosis, prevention, and treatment, or health assessment of individuals. Labs conduct a wide range of tests that are categorized as waived tests or as moderate- or high- complexity tests. Approximately 81 percent of all labs (about 157,000) are not subject to routine biennial surveys because they perform (1) “waived” tests, which are generally simple tests that have an insignificant risk of erroneous results, such as those approved for home use or (2) tests performed during the course of a patient visit with a microscope on specimens that are not easily transportable. CLIA establishes more stringent requirements for the 19 percent (about 36,000) of labs performing moderate- or high-complexity testing, including the requirement for a survey and participation in routine proficiency testing. Surveys examine lab compliance with CLIA program requirements in several areas including: personnel qualifications, proficiency testing, quality control, quality assurance, and recordkeeping.

Survey Organizations

In general, labs have a choice of who conducts their surveys—state survey agencies using CLIA inspection requirements or other survey organizations that use requirements CMS has determined to be at least equivalent to CLIA’s. CMS contracts with state survey agencies in most states to inspect labs against CLIA requirements. CLIA established an approval process to allow states and private accrediting organizations to use their own requirements to survey labs. As noted earlier, New York and Washington operate CLIA-exempt programs and CMS has approved six private, nonprofit accrediting organizations to survey labs—the American Association of Blood Banks (AABB), the American Osteopathic Association (AOA), the American Society of Histocompatibility and Immunogenetics (ASHI), CAP, COLA, and JCAHO. The requirements of both state CLIA-exempt programs and accrediting organizations must be reviewed by CMS at least every 6 years to ensure CLIA equivalency, but may be more stringent than those of CLIA. Figure 1 lists the three types of survey organizations and indicates whether they survey labs under CLIA requirements, or use their own CLIA-equivalent requirements. It also shows the percentage of labs performing moderate- to high-complexity testing surveyed by each type of organization. In general, state survey agencies, COLA, and Washington’s CLIA-exempt program survey physician office labs, while New York’s CLIA-exempt program, CAP, and JCAHO survey hospital labs.

Surveys and Complaint Investigations

Survey organizations (1) conduct surveys and complaint investigations and (2) monitor proficiency test results submitted by surveyed labs three times a year. Surveys are typically conducted by former or current lab workers, who assess lab compliance with CLIA or CLIA-equivalent requirements. Generally, surveyors verify that lab personnel are appropriately qualified to conduct testing, evaluate proficiency test records, check equipment and calibration to ensure that appropriate quality control measures are in place, and determine whether the lab has a quality assurance plan and uses it to, among other things, appropriately identify and resolve problems affecting testing quality. Surveys also include an educational component to assist labs in understanding how to comply with CLIA requirements.

Lab survey requirements are classified as either “standard-” or “condition-” level. Deficiencies are also characterized as standard- or condition-level based on the requirement in which the deficiency occurs. Standard-level deficiencies denote problems that generally are not serious, while condition-level deficiencies are cited when the problems are serious or systemic in nature. When deficiencies are found during surveys or complaint investigations, labs are required to submit a plan of correction, detailing how and when they will address the deficiencies. Additionally, CMS can impose principal or alternative sanctions, or both. Principal sanctions include revocation of a CLIA certificate, cancellation of the right to receive Medicare payments, or limits on testing. Alternative sanctions, authorized by Congress to give CMS more flexibility to achieve lab compliance, are less severe and include civil money penalties or on-site monitoring. For condition-level deficiencies that do not involve an imminent and serious threat to patient health and a significant hazard to public health, labs have an opportunity to correct the deficiencies, which we refer to as a grace period, before the sanctions are imposed. If a lab is unable to correct a deficiency during this grace period, CMS determines whether to impose sanctions.

CMS Oversight

CMS, including its 10 regional offices, oversees state and accrediting organization survey activities. CMS reviews and approves initial and subsequent applications from exempt-state programs and accrediting organizations to ensure CLIA equivalency. Validation reviews are one of CMS’s primary oversight tools. Federal surveyors in CMS regional offices are responsible for conducting validation reviews of state survey agency and exempt-state program inspections, but state survey agency staff conduct the validation reviews of accrediting organization inspections. An objective of these reviews is to determine if all condition-level deficiencies were identified. These reviews are conducted within 60 days of a state’s, or 90 days of an accrediting organization’s, survey of a lab.

Insufficient Data Exist to Identify Extent of Serious Lab Quality Problems

The extent of serious quality problems at labs is unclear because CMS has incomplete data on condition-level deficiencies identified by state survey agencies prior to 2004. Survey results for 2004 show substantial variability across states, which suggests that state survey agencies do not conduct surveys in a consistent manner. We also found that the lack of a straightforward linkage between CLIA requirements and the CLIA- equivalent requirements of some survey organizations makes it virtually impossible to assess lab quality in a standardized manner. CMS does not effectively use available data, such as the results of surveys and proficiency testing, to monitor and assess lab quality. Although CMS noted that proficiency testing trend data show a decrease in failures for labs as a whole, the data suggest that quality may not have improved at hospital labs for the period 1999 through 2003.

Limited Quality Data for Labs Inspected by State Survey Agencies Suggest Survey Inconsistencies

CMS’s OSCAR database contains limited data on the quality of labs inspected by state survey agencies and, as a result, it is not possible to analyze changes in the quality of lab testing over time. In January 2004, CMS implemented revised CLIA survey requirements and modified the existing OSCAR data—state survey agency findings—to reflect the changes. The revisions affected approximately two-thirds of the CLIA condition-level requirements. As a result of the data modifications, the findings for surveys conducted prior to 2004 no longer reflect all key condition-level requirements in effect at the time of those surveys.

Based on the available 2004 OSCAR data (which represent about one half of all labs surveyed by state survey agencies), we found that 6.3 percent of labs had condition-level deficiencies. However, variability in the OSCAR data suggests that labs are not surveyed in a consistent manner. In 2004, the percentage of labs that were reported to have condition-level deficiencies varied considerably by state, ranging from none in 6 states to about 25 percent of labs in South Carolina. Based on interviews with CMS and 10 state survey agencies, it appears that at least some of this variability is due to differences in states’ approaches to conducting their surveys as opposed to true differences in lab quality. For example, CMS told us that, because there is not a prescriptive checklist to guide the survey process, the reliance on state surveyor judgment results in variations in the citing of deficiencies. In fact, officials in several states said that there are circumstances under which condition-level deficiencies would not be cited, such as if the lab staff were new or if the lab had a good history of compliance. As a result, available data likely understate the extent of serious quality problems at labs.

Quality of Labs Inspected by Survey Organizations Is Very Difficult to Measure in a Standardized Manner

Differences in the inspection requirements used by survey organizations make it virtually impossible to measure lab quality in a standardized manner. Because exempt-state programs and accrediting organizations do not classify inspection requirements and related deficiencies with the same criteria used by state survey agencies—as either standard- or condition-level—they cannot easily identify the proportion of surveyed labs with condition-level deficiencies.

We asked exempt-state programs and accrediting organizations what percentage of their requirements, and any deficiencies cited for failure to meet those requirements, indicated serious problems that were equivalent to CLIA condition-level deficiencies. CAP and COLA crosswalked their recent survey findings to CLIA condition-level requirements. Although their analysis suggested that from about 56 to 68 percent of labs surveyed during 2004 had a deficiency in at least one condition-level requirement, they acknowledged that these proportions overstated the subset of labs with serious problems. JCAHO did not crosswalk its inspection requirements to those of CLIA because staff would have had to manually review each survey report to determine which deficiencies were equivalent to deficiencies in CLIA condition-level requirements.

Despite the difficulty of identifying CLIA equivalent condition-level deficiencies, two of the three accrediting organizations we reviewed have systems to identify labs they survey that have serious quality problems. COLA estimated that about 9 percent of labs it surveyed in 2004 were subject to closer scrutiny because of the seriousness of the problems identified. According to JCAHO, about 5 percent of the labs it surveyed in 2004 were not in compliance with a significant number of requirements. The third accrediting organization, CAP, has criteria for identifying labs that warrant greater scrutiny, but CAP officials told us that identifying such labs had to be accomplished on a case by case basis, rather than through a database inquiry.

CMS Use of Data for Monitoring Lab Quality Is Limited

CMS does not effectively use available data, such as the results of surveys and proficiency testing data, to monitor and assess lab quality. Although CMS tracks the most frequently cited deficiencies at labs in an effort to improve quality, it does not routinely track the proportion of labs, by state, in which state survey agencies identify condition-level deficiencies—those that denote serious or systemic problems. As noted earlier, variability in survey findings suggests inconsistencies in how surveys are conducted. CMS also does not require exempt-state programs and accrediting organizations to routinely submit data on serious deficiencies identified at the labs they inspect, unless the deficiencies pose immediate jeopardy to the public or an individual’s health.

We also found that CMS does not effectively use proficiency testing data to assess clinical lab quality. Proficiency testing is an important indicator of lab quality because it is an objective assessment of a lab’s ability to produce accurate test results and is conducted more frequently than surveys—three times a year versus once every 2 years. In the absence of comparable survey data, proficiency testing results provide a uniform way to assess the quality of lab testing across survey organizations. Although CMS’s analysis of proficiency testing data showed improvements over time, our analysis of proficiency testing data for 1999 through 2003 suggests that there has been an increase in proficiency testing failures for labs inspected by CAP and JCAHO, which generally inspect hospital labs, and a decrease in such failures for labs surveyed by state survey agencies and COLA, which tend to inspect physician office labs.

Importantly, CMS’s decision to require proficiency testing for almost all laboratory tests only three times a year is inconsistent with the statutory requirement. CLIA requires that proficiency testing be conducted “on a quarterly basis, except where the Secretary determines for technical and scientific reasons that a particular examination or procedure may be tested less frequently (but not less often than twice per year).” In CMS’s 1992 rule implementing CLIA, the agency provided a rationale for reducing the frequency of proficiency testing, but did not provide a technical and scientific basis for reducing the frequency for particular procedures or tests. CMS told us that officials from CMS and the Centers for Disease Control and Prevention had together determined that the reduced frequency was based on technical and scientific grounds and supplied a brief, undated narrative which it attributed to the Centers for Disease Control and Prevention. However, the narrative focused on the relative costs and benefits of proficiency testing at various intervals and did not include an analysis of the technical and scientific considerations with regard to particular tests that presented a basis for reducing the frequency.

CLIA Program Oversight Is Inadequate

Oversight by CMS and survey organizations is not adequate to ensure that labs meet CLIA requirements. For example, the goal of educating lab workers during surveys takes precedence over the identification and reporting of deficiencies, while the use of volunteer rather than staff surveyors by one accrediting organization raises questions about appropriate levels of training and the appearance of a conflict of interest.

The significant increase in complaints since CAP took steps to help ensure that lab workers know how to file a complaint suggests that some quality problems at labs inspected by some survey organizations may not be reported. In addition, sanctions are not being used effectively as an enforcement tool to promote labs’ compliance with CLIA requirements, as evidenced by the relatively few labs with repeat condition-level deficiencies on consecutive surveys from 1998 through 2004 that had sanctions imposed. Furthermore, CMS is not meeting its responsibility to determine that accrediting organization and exempt-state program requirements and processes continue to be at least equivalent to CLIA’s. Finally, ongoing CMS validation reviews do not provide an independent assessment of the extent to which surveys identify all condition-level deficiencies—primarily due to their timing.

Balance Between Educational and Regulatory Roles by CMS and Survey Organizations Appears to Be Inappropriate

The goal of educating lab workers sometimes takes precedence over, or precludes, the identification and reporting of deficiencies that affect the quality of lab testing. For example, surveyors from one state survey agency told us they do not cite condition-level deficiencies when lab workers are new but prefer to educate the new staff. As a result, data on the quality of lab testing and trends in quality over time may be misleading. CMS also appears to be inappropriately stressing education over regulation. For instance, in its 2005 implementation of proficiency testing for lab technicians who interpret Pap smears, a test for cervical cancer, CMS instructed state surveyors to refrain from citing deficiencies at labs whose staff fail the tests in 2005 or 2006. According to CMS, this educational focus allows labs and their staff to become familiar with the proficiency testing program; however, it is important to note that there was about a 13-year time lag between the 1992 regulations that implemented CLIA and the 2005 implementation of Pap smear proficiency testing. In addition, CMS noted that it was concerned about some of the high initial Pap smear proficiency testing failure rates. An inappropriate balance between the educational and regulatory roles is also evident in some accrediting organization practices. For instance, for COLA, the process of educating labs begins even prior to a survey, when labs are encouraged to complete a self-assessment to identify COLA requirements with which they are not in compliance. A CAP surveyor we interviewed with over 30 years of lab experience estimated that the majority of pathologists—individuals who generally serve as CAP survey team leaders—view surveys as educational, rather than as assessments of compliance with lab requirements.

Use of Volunteer Surveyors by CAP Raises Concerns

The use of volunteer inspectors by CAP raises concerns about appropriate levels of training and the appearance of a conflict of interest. Although state survey agencies, exempt-state programs, COLA, and JCAHO employ dedicated staff surveyors, CAP relies primarily on volunteer teams consisting of lab workers from other CAP-inspected labs to conduct surveys. In contrast to the mandatory training and continuing education programs in place for the staff surveyors of other survey organizations, training for CAP’s volunteer surveyors is currently optional. According to data provided by CAP, two-thirds of volunteer surveyors who had recently participated in a survey had no formal training in the 3 to 5 years preceding the survey. While full-time surveyors employed by other survey organizations conduct from 30 to about 200 surveys per year, CAP volunteer surveyors have much less experience conducting surveys because they only survey about one lab each year. CAP officials told us they plan to establish a mandatory training program for survey team leaders beginning in mid-2006. However, the required training will take only 1 or 2 days. In contrast, state survey agency inspectors must complete 5 days of basic training, while COLA staff inspectors participate in a 5-week orientation program and an annual 20 hours of continuing education.

CAP’s method for staffing survey teams also raises concerns about the appearance of a conflict of interest. Typically, inspection team leaders are pathologists who direct other labs in the community, and the inspection team is comprised of several employees from the team leader’s lab. In the event of differing opinions about survey findings, team members who are subordinates to the team leader may feel that they have no other recourse than to follow the team leader’s instructions—such as downgrading the record of an inspection finding to a less serious category. Recognizing that team members’ objectivity may be compromised in this situation, CAP’s revised conflict of interest policy instructs all parties to be cautious to retain objectivity in fact finding throughout the inspection process.

Lab Workers Who File Complaints About Quality Problems in Lab Testing Not Afforded Whistle- blower Protections

Some lab workers may not be filing complaints about quality problems at their labs because of anonymity concerns or because they may not be familiar with filing procedures. Based on OSCAR data and data obtained from exempt-state programs and accrediting organizations for 2002 through 2004, few complaints were received about lab testing relative to the number of labs—significantly less than one complaint per lab per year. We found that lab workers may not know how to file a complaint. CAP experienced a significant increase in the number of complaints it received since October 2004 when it began requiring CAP-inspected labs to display posters on how to file complaints. Specifically, from October through December 2004, CAP received an average of 22 complaints per month, compared to an average of 11 complaints per month in the 9 months preceding the poster requirement.

Because of the difficulty of protecting the anonymity of lab workers who file complaints, whistle-blower protections for such individuals are particularly important. Two of the three accrediting organizations we interviewed have whistle-blower protections—CAP and JCAHO. While officials from New York and Washington’s exempt-state programs told us that whistle-blower laws in their states provide some protection for lab workers who file complaints, officials in most of the other 10 states we interviewed told us that they did not have any whistle-blower protections or were unable to identify specific protections that applied to lab workers in their state. Although there are no federal whistle-blower protections specifically for workers in labs covered by CLIA, legislation was introduced in 2005 to provide such protections.

Lab Sanctions Are Rarely Imposed

Few labs were sanctioned by CMS from 1998 through 2004—even those with the same condition-level deficiencies on consecutive surveys— because many proposed sanctions are never imposed. Our analysis of CMS enforcement data from 1998 through 2004 found that while over 9,000 labs had sanctions proposed during these years, only 501 labs were sanctioned. This equates to less than 3 percent of the approximately 19,700 labs inspected by state survey agencies. Before sanctions go into effect, labs are given a grace period to correct condition-level deficiencies, unless the deficiencies involve an imminent and serious threat to patient health and a significant hazard to public health. Most labs correct the deficiencies within the grace period. CMS officials told us that it was appropriate to give labs an opportunity to correct such deficiencies within a prescribed time frame and thus avoid sanctions.

However, the number of labs with the same repeat condition-level deficiencies from one survey to the next also raises questions about the overall effectiveness of the CLIA enforcement process. From 1998 through 2004, 274 labs surveyed by state survey agencies had the same condition- level deficiency cited on consecutive surveys and 24 of these labs had the same condition-level deficiency cited on more than two surveys. This analysis may understate the percentage of labs with repeat condition-level deficiencies because OSCAR data prior to 2004 no longer reflect about two-thirds of condition-level requirements and associated deficiencies at the time of those surveys. We found that only 30 of the 274 labs with repeat condition-level deficiencies had sanctions imposed—either principal, alternative, or both. With respect to accredited labs, from 1998 through 2004, less than 1 percent of accredited labs (81) lost their accreditation; few of these labs were subsequently sanctioned by CMS and many still participate in the CLIA program. Moreover, CMS did not sanction 3 labs that COLA concluded had cheated on proficiency testing by referring the samples to another lab to be tested. By statute, the intentional referral of samples to another lab for proficiency testing is a serious deficiency that should result in automatic revocation of a lab’s CLIA certificate for at least 1 year. Based on our interviews, we found that the 3 labs were allowed to continue testing because they had initiated corrective actions; in effect, these labs were given an opportunity to correct a deficiency that appears to have required a loss of their CLIA certificate for at least 1 year.

CMS Is Late in Ensuring CLIA Equivalency of Exempt States’ and Accrediting Organizations’ Inspection Requirements and Processes

We found that CMS has been late in determining that exempt states’ and accrediting organizations’ inspection requirements and processes are at least equivalent to CLIA’s. Because CMS has not completed its equivalency reviews within required time frames, accrediting organizations and exempt state programs have continued to operate without proper approval. Equivalency reviews for CAP, COLA, JCAHO, and Washington due to be completed between November 1, 1997, and April 30, 2001, were an average of about 40 months late. In August 1995, CMS determined that New York’s next equivalency review should be completed by June 30, 2001, but was over 4 years past due as of December 2005. Similarly, COLA’s equivalency review was about 3 years past due. Furthermore, although federal regulations require CMS to review equivalency when an accrediting organization or exempt-state program adopts new requirements, CMS has not reviewed changes in the inspection requirements prior to use by these entities. As a result, such survey organizations may introduce changes that are inconsistent with CLIA requirements. For example, JCAHO made a significant change to its inspection requirements in January 2004; CMS did not begin an in-depth review of JCAHO’s revised requirements until early 2005—over a year after they were implemented by JCAHO.

According to CMS, its review has identified several critical areas where JCAHO standards are less stringent than those of CLIA. JCAHO acknowledged the need to make some adjustments to its revised requirements.

CMS officials attributed delays in making equivalency determinations and reviewing interim changes to having too few staff. The CLIA program, located in CMS’s Center for Medicaid and State Operations (CMSO), currently has approximately 21 full-time-equivalent positions compared to a peak of 29 such positions several years ago. As required by statute, the CLIA program is funded by lab fees and since its inception the program’s fees have exceeded expenses. As of September 30, 2005, the CLIA program had a carryover balance of about $70 million—far more than required to hire an additional six to seven staff members. However, CMS officials told us that because the CLIA program staff are part of CMSO, they are subject to the personnel limits established for CMSO, regardless of whether or not the program has sufficient funds to hire more staff.

CMS Validation Reviews Skip Some State Survey Agencies and Many Lack Independence

CMS validation reviews that are intended to evaluate lab surveys conducted by both states and accrediting organizations do not provide CMS with an independent assessment of the extent to which surveys identify all serious—that is, condition-level or condition-level equivalent— deficiencies. CMS requires its regional offices to conduct validation reviews of 1 percent of labs inspected by state survey agencies in a year. However, CMS does not specifically require that validations occur in each state. As a result, from 1999 through 2003, there were 11 states in which no validation reviews were conducted in multiple years. Without validating at least some surveys in each state, CMS is unable to determine if the states are appropriately identifying deficiencies.

Many validation reviews occur at the same time a survey organization conducts its inspection and, in our view, the collaboration among the two teams during these simultaneous surveys prevents an independent evaluation. Seventy-five percent of validations of state lab surveys were conducted simultaneously from fiscal years 1999 through 2003. According to CMS officials, the large proportion of simultaneous validation reviews provides an opportunity for federal surveyors to share information with state surveyors, monitor their conformance with CLIA inspection requirements, and identify training and technical assistance needs. However, we found that such reviews do not provide an accurate assessment of state surveyors’ ability to identify condition-level deficiencies. Of the 13 validation reviews that identified missed condition- level deficiencies, only 1 was a simultaneous review. Regarding validation reviews of accrediting organization’s survey of labs, CMS officials were unable to tell us how many of the roughly 275 validation reviews conducted each year from fiscal year 1999 through fiscal year 2003 were simultaneous. However, JCAHO estimated that 33 percent of its validation reviews were conducted simultaneously. CMS officials told us that the agency’s intent in instituting simultaneous reviews was for state and accrediting organization surveyors to share best practices, to promote understanding of each other’s programs, and to foster accrediting organization improvement. In contrast, most of the state survey agency officials we interviewed told us that simultaneous validation reviews do not provide a realistic evaluation of the adequacy of accrediting organizations’ inspection processes.

Concluding Observations

Clinical labs play a pivotal role in the nation’s health care system by diagnosing many diseases, including potentially life-threatening diseases, so that individuals receive appropriate medical care. Given this important role, lab tests must be accurate and reliable. Our work demonstrated that the oversight of clinical labs needs to be strengthened in several areas. Without standardized survey findings across all survey organizations, CMS cannot tell whether the quality of lab testing has improved or worsened over time or whether deficiencies are being appropriately identified. Using data to analyze activities across survey organizations can be a powerful tool in improving CMS oversight of the CLIA program, yet CMS has not taken the lead in ensuring the availability and use of data from survey organizations to help it monitor their performance. Furthermore, the agency is not requiring that labs participate in proficiency testing on a quarterly basis, as required by CLIA. More broadly, CMS and survey organization oversight of the lab survey process is not adequate to enforce CLIA requirements. Educating labs to ensure high-quality testing should complement but not replace the enforcement of CLIA inspection requirements. Labs with the same serious deficiencies on consecutive surveys often escape sanctions, even though Congress authorized alternative sanctions to give CMS more flexibility to achieve lab compliance. Without the threat of real consequences, labs may not be sufficiently motivated to comply with CLIA inspection requirements. By allowing validation reviews to occur simultaneously with surveys and permitting some states to go without validation reviews over a period of several years, CMS is not making full use of this oversight tool. Moreover, independent validation reviews of accrediting organization surveys are critical because CMS has not conducted equivalency reviews within the time frames it established. The recommendations we have made would help CMS to consistently identify and address lab quality problems.

Mr. Chairman, this concludes my prepared remarks. I would be happy to answer any questions that you or other Members of the Subcommittee may have.

Contact and Acknowledgments

For further information regarding this statement, please contact Leslie G. Aronovitz at (312) 220-7600 or aronovitzl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found in the last page of this statement. Walter Ochinko, Assistant Director; Jenny Grover; Kevin Milne; and Michelle Rosenberg contributed to this statement.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.