FDA's Mammography Inspections: While Some Problems Need Attention, Facility Compliance Is Growing
Published: 1997-01-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a legislative requirement, GAO reviewed the Food and Drug Administration's (FDA) program for implementing the requirements of the Mammography Quality Standards Act of 1992, focusing on: (1) the extent to which facilities are complying with the new mammography standards; (2) whether FDA's procedures for evaluating image quality at mammography facilities are adequate; and (3) whether FDA's monitoring and enforcement process ensures timely correction of mammography deficiencies.
section_title: What GAO Found
paragraphs: GAO found that: (1) GAO's work points to growing compliance by facilities with FDA's mammography standards; (2) FDA's first annual inspection began in January 1995, and by mid-1996, over 9,000 facilities had been inspected, and approximately 1,500 of these had undergone two rounds of inspections; (3) the first time these 1,500 facilities were evaluated, 26 percent had significant violations, but the second-year inspection revealed that this figure had dropped to about 10 percent; (4) the percentage of facilities with less significant deviations from quality standards had decreased; (5) while these results are positive, GAO did note some differences in how inspectors are conducting inspections that, left unaddressed, could lead to inconsistent reporting of violations, thereby limiting FDA's ability to determine the full effect of the inspection process and to identify the extent of repeat violations; (6) GAO's review of FDA's actions during the first 18 months of its inspection program showed a need for management attention to two additional aspects of the inspection program; (7) FDA's inspection procedures for an important test of mammography equipment were inadequate; (8) the way this test, called the phantom image test, was conducted was open to variability, which could have resulted in differing assessments of how well the equipment functioned; (9) in those instances in which test results showed serious problems with the phantom image quality, FDA's procedures allowed facilities to continue taking mammograms without follow-up to evaluate whether their quality was actually acceptable; (10) without such follow-up review, women are not fully protected from getting poor mammograms from facilities with potentially severe quality problems; (11) at the time of GAO's review, FDA also lacked procedures to ensure that all violations of standards were both corrected and corrected in a timely manner; (12) FDA's program lacked criteria for defining conditions constituting a serious risk to human health, which could delay enforcement of compliance and notification to women who may have received substandard mammograms; (13) for facilities with less severe but persistent violations, FDA's follow-up efforts could not always ensure corrective action was taken; and (14) delays in completing a management information system have kept FDA's compliance staff from having complete, up-to-date information about the compliance status of all mammography facilities.

section_title: Letter
paragraphs: Breast cancer is currently the second leading cause of cancer deaths among American women. One woman in eight will develop breast cancer during her lifetime, and, in 1996 alone, an estimated 44,000 women will have died from the disease. If breast cancer is detected early, however, the probability that a woman can survive is greater than 90 percent.
Currently, the most effective technique for early detection of breast cancer is mammography, an X-ray procedure that can often locate small tumors and abnormalities up to 2 years before they can be detected by touch. However, mammography is one of the most technically challenging X-ray procedures, and ensuring the quality of mammography services is difficult. To address concerns about variations in the quality of mammography services provided by the more than 10,000 facilities throughout the United States and its territories, the Congress passed the Mammography Quality Standards Act of 1992 (MQSA). This act established a number of requirements aimed at strengthening quality, including requiring accreditation and annual inspection of mammography facilities.
The act also requires us to report to the Congress on the Food and Drug Administration’s (FDA) program for implementing these requirements. Our first report, which focused on the accreditation program, found that many facilities were upgrading their procedures to meet accreditation requirements. Since that report was issued, FDA has fully implemented its annual inspection program for assessing compliance with these requirements. In our ongoing monitoring efforts, we identified several inspection-related issues that we thought important to call to your attention as you assess the act’s results. This report focuses on the extent to which facilities are complying with the new mammography standards, whether FDA’s procedures for evaluating image quality at mammography facilities are adequate, and whether FDA’s monitoring and enforcement process ensures timely correction of mammography deficiencies.
Our work is based on an analysis of results of inspections that took place at more than 9,000 mammography facilities nationwide between January 1995 and June 1996. For a better understanding of how inspections and follow-up efforts were being carried out, we also visited eight state offices that contracted with FDA to conduct inspections and three FDA field offices responsible for managing and monitoring inspections. We supplemented this information through interviews with officials from FDA and other federal, state, and private organizations involved with the program. Our work was conducted from November 1995 through September 1996 in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The best method known to reduce breast cancer mortality is early detection. Detection of breast cancer is accomplished through a combination of self-examination, physical examination by a physician, and mammography. Of these methods, mammography is the single most effective tool for detection of early-stage breast cancer. The use of mammography as a tool for detecting early or potential breast cancer continues to increase. The proportion of women aged 50 and older who had received mammograms in the previous year increased from 26 percent in 1987 to 54 percent in 1993, according to the Centers for Disease Control and Prevention. Since 1992, at least 23 million mammograms have been performed in the United States annually.
The consequences of substandard mammograms can be very serious. If the image shows an abnormality when none exists, a woman may go through unnecessary and costly follow-up procedures, such as ultrasound or biopsies. If the image is too poor to show an abnormality that is actually present, a woman may lose the chance to stop the cancer’s spread.
To help ensure the quality of images and their interpretation, MQSA required FDA to implement both an accreditation and an inspection process for mammography facilities. For the accreditation process, FDA established standards that included requirements for personnel qualifications, equipment performance, and quality assurance recordkeeping. These standards were based on those used by the American College of Radiology (ACR), a private, nonprofit professional association of radiologists, and have been endorsed by industry and government experts. As of July 1996, almost 10,000 facilities had been accredited and had received an FDA certificate to that effect.
MQSA inspection authority provides FDA with another means to ensure that facilities comply with standards on a day-to-day operating basis. While for the vast majority of facilities accreditation application and review are accomplished through the mail, all inspections are conducted on site. During an inspection, MQSA inspectors conduct various equipment tests and review the facility’s records on personnel qualifications, quality controls, and quality assurance as well as mammography reports.
FDA, which has contracted with virtually all states and territories to conduct inspections, began its first annual inspections of the nation’s mammography facilities in January 1995. It established an extensive program for training inspectors, and as of April 1996, about 220 state and FDA personnel had become certified to perform MQSA inspections. The majority of the personnel chosen to become MQSA inspectors had 5 or more years of prior experience in radiological health. FDA uses its own inspectors to conduct follow-up inspections, monitor the performance of state inspectors, and conduct inspections in states that either did not contract with FDA or lacked enough FDA-certified inspectors to do all the inspections.
FDA’s field offices are responsible for following up on inspection violations and enforcing facility compliance. For the most serious violations, FDA’s field offices issue a warning letter informing the facility of the seriousness of the violation. The facility must begin correcting its problem immediately and report the corrective action taken in writing to FDA within 15 work days of receipt of the letter. In some cases, FDA conducts a follow-up inspection of the facility to ensure that the problem is corrected. If the facility fails to correct a problem, FDA can take other enforcement actions, such as imposing a Directed Plan of Correction; assessing a civil penalty of up to $10,000 per day or per failure; or suspending or revoking a facility’s FDA certificate, which prevents a facility from operating lawfully.
section_title: Compliance With Standards Has Improved but More Consistent Reporting Is Needed
paragraphs: First-year inspections of mammography facilities showed that a significant number of facilities were not in full compliance with mammography standards. So far, second-year inspections have shown a considerable reduction in the proportion of facilities cited for violations—an indication that the inspection process is having positive results. However, inspection results vary considerably from state to state. It is not clear how much these differences reflect actual differences in the levels of quality in mammography facilities and how much they reflect varying approaches to conducting inspections and reporting the results. To gain a true picture of the full effect of the inspection process, more consistent reporting of violations is needed.
subsections: section_title: Second-Year Inspections Identified Fewer Violations Than First-Year Inspections
paragraphs: FDA’s automated inspection database contained first-year inspection results for 9,186 facilities as of June 20, 1996. Of these, 6,177 showed one or more violations of the standards. As table 1 shows, 1,849 facilities (or 20 percent) had violations that were serious enough to require the facility to provide FDA with a formal response as to the corrective actions taken. Of these, 214 facilities had violations that ranked in the most serious (or “level 1”) category, requiring FDA to send the facility a warning letter.
The most serious violations found in these inspections were mainly personnel related: 88 percent of the level 1 violations were for personnel who did not fully meet FDA’s qualification standards (see app. I for a further breakdown of the types of level 1 violations). Level 2 violations involved a greater mix of personnel-related and equipment-related problems, and the majority of level 3 violations involved missing or incomplete quality assurance records and test results as well as medical physicist survey problems.
By June 20, 1996, FDA’s database contained the results of 1,503 second-year inspections. We compared the results of first-year and second-year inspections for these 1,503 facilities and found a substantial decrease in all three categories in the proportion of facilities cited for violations (see fig. 1).
Another measure of facilities’ improvement in compliance is the extent of repeat violations, that is, violations identified in the first year’s inspection that are identified again when the facility is reinspected the following year. Facilities had a better record in not repeating the more severe violations than they did with minor findings. More specifically, our analysis of the 1,503 facilities showed the following:
None of the 50 facilities whose highest level of violation was at the level 1 category during the first-year inspection repeated one or more of the same violations in the second inspection.
Six percent of the 345 facilities whose highest level of violation was at the level 2 category during the first-year inspection repeated one or more of the same violations in the second inspection.
Twelve percent of the 669 facilities whose highest level of violation was at the level 3 category during the first-year inspection repeated one or more of the same violations in the second inspection.
section_title: Inconsistent Inspection Practices Could Lead to Underreporting of Violations
paragraphs: Our analysis of inspection results showed considerable state-by-state variation in the degree to which facilities were cited for violations of MQSA standards. For example,14 states cited no facilities for level 1 violations, while 6 states cited 5 to 12 percent of the facilities inspected for level 1 violations (see app. II for state-by-state results). We were unable to determine the reason for these differences. It may be, for example, that facilities in low-violation states really were much better at complying with standards than facilities in high-violation states. Alternatively, the differences may have been related to variations in the way inspectors conducted their inspections. In the eight states in which we observed inspections, we saw several differences in inspection practices that affected the number of violations reported. The two main differences follow.
First, inspectors’ adherence to time limits for resolving problems of missing documents was inconsistent. FDA’s current procedures allow inspectors to delay submitting their inspection reports for 5 to 30 days in order to resolve problems of missing documents. This delay is intended to avoid citing facilities for not having certain records available on site. For example, when a facility claims that its personnel meet MQSA qualification requirements but does not have the required documentation at hand, FDA guidelines instruct inspectors to either delay the transmission of the inspection report or note the “claimed items” in the inspection record. These open items are to be resolved within 30 days, at which time the inspection report is to be finalized. However, we found hundreds of cases in the inspection report database that contained open items longer than 30 days—many for over 6 months. Several inspectors we interviewed said they were not aware of the 30-day limit for resolving pending items. On the other hand, inspectors in two states we visited said they would not wait more than 5 days under any circumstances before submitting a report that a facility was in violation. Thus, a facility in one state might be reported as being in violation, while a facility with the same problem in another state would not. These differences may have resulted in inconsistent reporting of violations; moreover, these inconsistencies make it difficult to determine the full effect of the inspection process.
Second, while FDA’s policy is to cite facilities for all violations even if problems are corrected on the spot, we found that inspectors do not always adhere to this policy. For example, we observed that an inspector did not cite a facility that failed its darkroom fog test—normally a level 2 violation—because the facility immediately corrected the problem. Further, FDA’s procedures instruct inspectors to note on-the-spot corrections in the “remarks” section of the inspection software. We observed two inspections on site that involved on-the-spot corrections, but we did not see these inspectors documenting them in the remarks section.
We do not question the merit of giving inspectors time to resolve such problems as missing documents or giving facilities opportunities to correct their problems immediately. However, not documenting violations consistently creates problems in forming an accurate picture of what the inspection process is accomplishing. FDA officials told us that they had begun a program in February 1996 to review inspector performance and that, as of October 31, 1996, 65 percent of all inspectors had been audited. FDA officials expect that, when fully implemented, the audit program will help ensure that policies are consistently applied. We agree that the audit program will help identify some inconsistent inspection practices; however, we believe the inspection results should also be monitored to ensure that open items are resolved in a timely manner and that on-the-spot corrections are identified.
section_title: Procedures for Assessing Image Quality Need Strengthening
paragraphs: Although many factors can affect the quality of mammography images, one key factor is the condition of mammography equipment. We identified a need for FDA to clarify the procedures it requires for a major equipment test that evaluates image quality and to follow up when test results suggest problems with the quality of the images being produced.
subsections: section_title: Issues in Evaluating Phantom Image Test Results
paragraphs: One of the most important aspects of the inspection process is testing mammography equipment by evaluating what is called a “phantom image.” In this procedure, the inspector uses the facility’s mammography equipment to take an X-ray image of a plastic block containing 16 test objects. This block is X-rayed as though it were a breast to determine how many of the test objects can be seen on the image. The inspector evaluates aspects of the performance of the facility’s imaging system by scoring the number of objects that can be seen. We found two questions that need to be answered with regard to evaluating phantom images.
What is the impact of inconsistent phantom image scoring? FDA’s current inspection procedures instruct inspectors to score the phantom images under viewing conditions at the facilities. However, differences in inspectors’ experience and in facilities’ viewing conditions may influence the phantom image scores. For greater uniformity in scoring the images, two states we visited go beyond FDA’s standards by having their inspectors score phantom images using standardized viewing conditions (that is, away from the facility), having two or more persons read the images to ensure more consistent scoring, or both. FDA officials told us that the impact of these variations in procedure on the accuracy of image evaluation is unknown and that they are studying the problem.
How should large image receptors be evaluated? FDA procedures currently require that phantom images be checked using the receptor that is more commonly used by the facility. Since facilities use small image receptors for most mammograms, these receptors are typically tested during an inspection. Although facilities may use large image receptors for some women, FDA does not require that the large image receptor be tested and does not have specific criteria for evaluating the phantom images of the large receptor.
Inconsistent phantom image scoring and lack of standards for evaluating large image receptors can affect inspection results, as can be seen in the example of a 1995 inspection of a large mobile mammography facility headquartered in North Carolina and operating in five states. The facility is reported to perform over 20,000 mammograms a year. A state inspector cited the facility for multiple problems based on the viewing conditions at the facility and images from the small receptor. Although it was not required by FDA, the inspector also evaluated the phantom images from the large image receptor and noted in the remarks section of the inspection report that, for three of four mammography units, these images did not pass the review. An FDA inspector conducted a follow-up inspection, also using the viewing conditions at the facility and images from both the small and large image receptors. This inspector cited the facility for many violations related to both the small and large image receptors. Finally, four reviewers at FDA headquarters examined these same images away from the facility and together found fewer violations related to both the small and large image receptors than the state inspector and the FDA inspector had found. The reviewers, however, did confirm the serious violations related to the large image receptor that were found by the state inspector and the FDA inspector.
Although this facility was cited for serious violations related to the large image receptor as a result of the follow-up inspection, FDA officials told us that, because of the lack of inspection criteria, imposing strong sanctions on the basis of phantom image failures from the large receptor could prove problematic. According to FDA, standards for testing the large receptor have not yet been developed because the technical issues relating to the receptor have not yet been resolved by the scientific and medical community. We discussed this case with senior FDA officials, who said that they plan to both provide additional training and guidance to minimize the variability in phantom image scoring and study the development of standards for evaluating images from the large image receptor.
section_title: Taking Additional Steps to Better Ascertain the Seriousness of Deficiencies
paragraphs: Another issue raised by the inspections of the facility discussed above is how to proceed if the phantom image test suggests serious problems with image quality. FDA views phantom image failures as early indications of potential problems deserving further investigation. FDA’s procedures allow facilities with serious phantom image failures to continue performing mammograms while FDA investigates and the facility corrects problems. During the course of our work, we heard varying opinions on the risk of allowing facilities with serious phantom image failures to continue doing mammograms. Some people we spoke with believe the risk of patients’ getting poor mammograms from facilities with serious phantom image failures is high enough that the facilities should not be allowed to do any mammograms until their problems are corrected and those corrections are verified by a reinspection. Several states, including California, Illinois, and Michigan, have rules empowering inspectors to immediately stop facilities with level 1 phantom image failures from doing additional mammograms. However, others (including FDA officials in charge of the MQSA program) do not believe that such drastic action should be taken on the basis of phantom image test results alone. They assert that phantom image failures are an indicator of possible image system problems but are not conclusive evidence that actual mammograms are faulty.
At the time of our review, FDA did not have a follow-up system in place for reviewing the actual mammograms (called “clinical images”) of facilities with serious phantom image violations to ensure that they were not producing poor mammograms. However, in the case of the mobile facility discussed earlier, FDA did ask ACR to conduct two reviews of the clinical images produced by the facility because of image quality concerns. The more comprehensive review was conducted in July 1996, subsequent to our inquiry about FDA’s handling of the case. This review selected a total of 28 sets of images from five units operated by the facility for three different time frames over a 1-year period. In early September 1996, ACR completed the review and found most of these clinical images of unacceptable quality. On the basis of these results, FDA obtained the facility’s agreement to discontinue performing mammography until its radiologic technologists and its radiologist obtained additional training approved by FDA and ACR, which they did the following week. In addition, at FDA’s request, ACR is planning to review another sample of clinical images produced by the facility to determine to what extent patients should be notified of past quality problems at the facility.
This case clearly demonstrates the need for a procedure to review clinical images when there is sufficient evidence to suggest problems with the quality of a facility’s mammograms. Without the criteria and process in place for determining when and how follow-up review of clinical images should be conducted and patient notification should be carried out, there is no assurance that patients are protected from the risk of receiving poor mammograms. FDA officials agreed that there is a need to incorporate a follow-up clinical image review process. In its proposed final regulation dated April 3, 1996, FDA has included a provision that specifically provides FDA with authority to require clinical image review and patient notification if FDA finds that image quality at a facility has been severely compromised.
section_title: FDA’s Monitoring and Enforcement Process Does Not Ensure Timely Correction of Deficiencies
paragraphs: Although FDA has made progress in bringing facilities into compliance with mammography standards, it lacks procedures to enforce timely correction of all deficiencies found during inspections. One major problem involves the need to develop criteria for defining conditions constituting a serious risk to human health and determining when severe sanctions are warranted. Other problems that also merit attention relate to determining whether a stronger approach is needed to resolve repeated level 3 violations and establishing an effective information system for follow-up on inspection results. FDA is developing such an information system.
subsections: section_title: Developing Criteria for Defining a Serious Risk to Human Health and Determining When Severe Sanctions Are Justified
paragraphs: MQSA provides FDA a broad range of sanctions to impose against noncomplying facilities, but it emphasizes bringing facilities into compliance through those sanctions that are less severe, such as imposing a Directed Plan of Correction. FDA has the authority to impose stronger sanctions, such as an immediate suspension of a facility’s FDA certificate, if it determines that the facility’s operation presents a serious risk to human health. Since the implementation of MQSA, FDA has never done so. We found evidence that FDA needs to define those circumstances in which such actions are warranted.
In dealing with the continuing problems at the mobile facility discussed earlier, there was considerable internal debate at FDA about the level of action that should be taken. Inspections of the facility beginning in June 1995 had disclosed serious violations. (See app. III for a chronology of key events surrounding the resolution of quality assurance problems at the facility.) Several state and FDA field personnel involved in the case told us they thought the severity of violations warranted an immediate suspension of the facility’s certificate and had made such a recommendation. FDA officials decided against suspending the facility’s certificate because they thought the evidence of health risk was not clear and compelling enough to do so. In September 1996, when ACR’s review of clinical images eventually confirmed that the quality of the mammograms was unacceptable, FDA obtained the facility’s agreement to discontinue performing mammography until facility personnel received more training. Because of the agreement, FDA did not have to go through the process of imposing an immediate suspension of the facility’s certificate. Nevertheless, this incident points to the need for having criteria in place to impose such a sanction to protect patients, if necessary, from continuing to receive poor mammograms. We believe—and FDA officials agreed—that timely imposition of an appropriate sanction is in part dependent on (1) criteria for when conditions constitute a serious risk to human health, justifying immediate suspension of operations, and (2) a process for discontinuing mammography services until the problems are corrected.
section_title: Determining Whether a Stronger Approach Is Needed to Resolve Level 3 Violations
paragraphs: Another matter that also merits attention from FDA is whether more serious follow-up is needed for facilities with multiple or repeated level 3 violations. Current policy for facilities whose most serious violations are in the level 3 category requires no reporting on the facility’s part and no follow-up on the part of FDA until the next year’s inspection. However, of the facilities that had gone through two inspections, 18 percent of those whose most serious violation was in the level 3 category during the first year had five or more such violations, and 12 percent repeated one or more of the same violations in the next year. Several state inspectors we interviewed expressed concern that current procedures do not call for stronger action against such facilities. Inspectors from one state told us that their state regulations allow them to impose more serious penalties for recurring level 3 violations. Some inspectors also told us that even though level 3 violations were generally considered less serious, some level 3 violations—such as a facility’s failure to take corrective action when called for in the medical physicist’s survey report—are serious enough that they should be corrected as soon as possible to maintain quality assurance.
We did not evaluate the appropriateness of FDA’s classification of the various levels of violations. Because of the concerns expressed by the inspectors and the extent of multiple and repeat violations noted above, however, we believe that FDA should evaluate its classification of level 3 violations and the enforcement actions taken on them. If FDA believes these violations are important and need to be corrected, it could raise the violation level for facilities with multiple or repeated violations, which would ensure formal follow-up. However, if FDA views some of these violations as insignificant or having little effect on mammography, it may choose not to classify them as violations.
section_title: FDA Field Offices Need a Better Information System to Manage Inspections
paragraphs: FDA generally delegates inspection responsibility through contracts with states but remains responsible for follow-up and enforcement when violations are reported. For level 1 violations, FDA’s field offices are responsible for validating inspection results and issuing a warning letter that requires the facility to respond within 15 work days. For level 2 violations, no warning letters are sent, but facilities are required to respond in writing within 30 work days of the receipt of an inspection report. Since June 1995, FDA has been working with contractors to develop an automated compliance system that would supply its field offices with computer-based information to manage this compliance effort. Development problems have delayed the system, which is now projected to be operational early in 1997. In the meantime, FDA has been relying on field offices to maintain their own tracking systems.
Our reviews at three of FDA’s field offices showed that these interim systems were inadequate. Staff responsible for compliance follow-up had no direct access to inspection databases and were relying either on the state inspectors or on FDA headquarters to send them copies of inspection reports showing level 1 and level 2 violations that needed to be tracked. Staff said that sometimes they did not receive reports from headquarters until 2 to 3 months after the inspections and that state inspectors did not always send reports on level 2 cases. As a result, field office staff often received facility responses on corrective actions taken for level 2 violations before they even knew that violations had been cited. None of the three offices maintained case logs or prepared any status reports on their tracking efforts or the timeliness of facility responses.
Problems in these makeshift systems have stymied our attempts to determine how quickly and completely violations were being corrected. To determine whether field offices were sending out warning letters in a timely manner and whether facilities were correcting their deficiencies within required time frames, at our request, FDA headquarters in April 1996 sent all of its field offices a list of all level 1 and level 2 violations cited in their jurisdictions and asked them to compile data on facility response times for corrective actions. Field offices had difficulty responding with complete information. FDA headquarters had initially told us that these data would be available in early June, but at the time that we completed our field work, discrepancies still remained unresolved. We conducted an on-site file review at one FDA field office in August and September 1996 and found that the office had incomplete documentation for 13 of the 40 cases with level 2 violations cited between July 1, 1995, and June 20, 1996. In one case, documentation was absent altogether.
We also found problems with the timeliness of follow-up on level 1 violations. For example, while FDA guidelines require a field office to issue a warning letter for a level 1 violation within 15 to 30 business days after the inspection, the office we reviewed took up to 132 business days. Also, although FDA procedures require a facility to respond within 15 business days of receiving the warning letter, in two of the eight level 1 cases that we reviewed the facilities did not respond within the required time frame, and one case file contained no record of a facility response.
These findings highlight the importance of completing and implementing the automated compliance system as soon as possible. Until field offices have ready access to up-to-date information, it will be difficult for them to conduct effective follow-up and enforcement for facilities that violate the standards.
section_title: Conclusions
paragraphs: The results of the current inspection program of mammography facilities appear to be generally positive. Establishing this comprehensive inspection program has been a substantial effort on FDA’s part and, as mammography facilities move into their second year of inspections, violations of mammography standards are declining.
Despite these encouraging results, at the time of our review, we found indications that certain aspects of the inspection program needed attention. First, to ensure an accurate picture of how many problems were found and how well the inspection program was working, violations would need to be more consistently recorded. In addition, even though serious violations do not occur often, when they do, they have the potential for posing a serious health risk to those women affected. To ensure high quality mammography, FDA must be vigilant in its efforts to confirm that facilities promptly and adequately correct violations. As a result, FDA would need to provide an expeditious means to follow up, including notifying patients, when serious problems affecting image quality were indicated. Finally, improvements would be needed in systems and procedures for monitoring facilities with violations and for ensuring that they corrected deficiencies.
section_title: Recommendations to the Commissioner of the Food and Drug Administration
paragraphs: We recommend that FDA take action in the following areas:
Strengthening the inspection reporting process. To better reflect the extent to which inspections detect compliance problems, FDA needs to monitor its inspection results more closely to ensure that its procedures for resolving open items and documenting on-the-spot corrections are consistently followed.
Strengthening procedures for assessing image quality and protecting patients. To minimize the variability in how phantom images are scored, additional training and guidance should be provided, including guidance for evaluating phantom images using the large image receptor. Also, to minimize patients’ risk of poor quality mammograms, the final implementing regulations should include the criteria and process for requiring follow-up clinical image reviews and, when necessary, patient notification when inspections detect violations, such as serious phantom image failures, that could severely compromise image quality.
Ensuring that violations are corrected in a timely manner. Several steps are needed here. First, to help ensure that appropriate action is taken when serious problems are discovered, procedures need to be developed for (1) determining when the health risk is serious enough to justify immediate suspension of certification and (2) implementing the suspension. Second, to help ensure better performance from facilities that exhibit lingering, though less serious, deficiencies, the classification and enforcement policy on level 3 violations needs reevaluation to determine if additional follow-up is needed on facilities with multiple and repeated level 3 violations. Third, so that compliance personnel can have access to complete, up-to-date information on violations reported, all necessary steps need to be taken to ensure that the compliance tracking system currently under development is completed as soon as possible.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, FDA generally agreed with our recommendations and cited specific program enhancements and corrective actions it had recently undertaken. FDA was, however, critical of our draft on several accounts.
FDA said that the scope of our work did not address some aspects of MQSA requirements and that the draft did not adequately reflect many of FDA’s accomplishments in implementing MQSA. Moreover, FDA believed the report did not recognize changes FDA had made to improve those aspects of the inspection program that we had found in need of attention. FDA cited recent actions it had taken, including (1) establishing procedures and guidance for clinical image reviews, sanctions for failure to comply with standards, and procedures for follow-up on repeated level 3 violations; (2) implementing an inspector audit program that had evaluated 65 percent of inspectors as of October 31, 1996; and (3) making a commitment to fully implement its automated compliance tracking system in January 1997. FDA expressed concern that not acknowledging these actions would create an inaccurate impression that the program was fraught with problems, which could undermine the public confidence in mammography.
Concerning the scope of our work, this report is not intended as a vehicle for commenting on implementation of MQSA as a whole; it deals only with FDA’s inspection program. However, we think that the report speaks both to FDA’s accomplishments related to the inspection program and to those problems that we found—and that FDA has now moved to correct. The main reason that FDA’s recent actions were not reflected in the original draft was that they occurred about the same time or, in most cases, after we had provided FDA the draft for comment. We generally consider FDA’s subsequent actions and approaches to our recommendations to be responsive and believe that, if properly implemented, they should strengthen the inspection program. We recognize FDA’s concern about the importance of promoting public confidence in mammography, and, in fact, our recommendations to promote timely compliance with MQSA were made with that objective in mind.
While we generally concur with FDA’s approaches for addressing our recommendations, we continue to believe that opportunity exists for FDA to improve its reporting process. We recognize that FDA has acted to implement the inspector audit program, but we believe that FDA still needs to monitor its inspection results to ensure timely follow-up on “open items” and accurate reporting of on-the-spot corrections. As a result, we have clarified our recommendation on strengthening the inspection reporting process accordingly.
FDA also provided technical comments, which we considered and incorporated where appropriate, and cited several other areas of the report that it thought needed clarification. The full text of FDA’s comments, accompanied by our response, is contained in appendix IV.
We also received comments from the North Carolina facility that we cited in the report. The facility stated that our report addressed many of its concerns with the MQSA program. It also commented that its case demonstrates the need for an organized approach to evaluation and for all involved agencies to agree upon an appropriate standard for clinical image evaluation. The facility asserted that FDA’s process lacks these critical elements and that the facility was being held to unreasonable standards. As a result, in October 1996, the facility appealed its Directed Plan of Correction to FDA. We have updated the chronology of FDA’s enforcement actions regarding the facility to reflect the facility’s appeal and the subsequent denial of the appeal by FDA (see app. III).
We are sending copies of this report to the Secretary of Health and Human Services, the Commissioner of the Food and Drug Administration, the Director of the Office of Management and Budget, and other interested parties. We will also make copies available to others upon request.
Please contact me at (202) 512-7119 if you or your staff have any questions. Major contributors to this report are listed in appendix V.
section_title: Distribution of Level 1 Violations by Type, First-Year Inspections
paragraphs: Personnel do not meet FDA’s qualification standards Processor quality control charts not available No survey conducted by medical physicist Mammography records improperly maintained or recorded Self-referred system inadequate or not in place Less than one-half of 1 percent.
section_title: Results of States’ First-Year Inspections
paragraphs: Percentage of facilities with no violations (continued)
section_title: Chronology of FDA’s Enforcement Actions for the North Carolina Mobile Facility
paragraphs: Initial inspection revealed 1 level 1 violation for phantom image failure, 2 level 2 violations for phantom image failure, 1 level 2 violation for processor quality control problems, and 31 level 3 violations for various other problems.
ACR’s clinical image review for one unit found mammograms acceptable and resulted in ACR accreditation for that unit.
FDA issued its warning letter to the facility for the violations found in June 1995.
The facility responded by submitting new phantom images and a processor quality control chart for review.
FDA notified the facility that its response was inadequate because it did not identify the machine on which the phantom images were taken and it did not include proper paperwork for the processor.
ACR clinical image review for another unit found mammograms acceptable, and ACR accreditation was granted for that unit.
ACR, at FDA’s request, performed a clinical image review of one set of mammograms for each of two units.
The facility responded to FDA’s 11/13/95 letter by sending new phantom images and processor quality control charts.
FDA notified the facility that the 11/30/95 response was adequate.
FDA did a follow-up reinspection and found 5 level 1 and 7 level 2 phantom image failures using the large image receptor, 11 level 2 phantom image failures using the small image receptor, and numerous other level 2 and level 3 violations.
ACR notified the facility and FDA that the clinical images reviewed on 11/20/95 were acceptable.
FDA imposed a Directed Plan of Correction requiring the facility to (1) have a medical physicist complete a survey of all units within 30 days, (2) correct problems identified in the survey within 15 business days, (3) perform phantom image evaluation weekly and submit results to FDA monthly, and (4) perform other quality control tests.
FDA and state officials met with the facility’s management to discuss the Directed Plan of Correction and to review progress.
FDA reinspected the facility and found one level 2 violation involving dark room fog and two level 3 violations in other areas, but no phantom image failures for either large or small image receptors. FDA directed the facility to select a total of 28 sets of clinical images from three time periods between July 1995 and June 1996 for ACR review.
ACR review found most of the clinical images were unacceptable.
FDA imposed an amended Directed Plan of Correction and obtained agreement from the facility to discontinue performing mammography with the resident radiologic technologists and interpreting physician until they were retrained.
All but one of the facility’s radiologic technologists and the interpreting physician completed training, and a new FDA- and ACR-approved technologist was added to the facility’s staff. The facility reopened and reestablished mammography services.
FDA notified the facility that ACR would conduct additional clinical image reviews of (1) a sample of clinical images after the personnel had resumed performing mammography for about 1 month and (2) all mammograms taken between June 6, 1996, and September 9, 1996.
The facility appealed FDA’s amended Directed Plan of Correction.
FDA denied the facility’s appeal.
section_title: Comments From the Food and Drug Administration and Our Evaluation
paragraphs: The following are GAO’s additional comments on the letter received from the Food and Drug Administration dated November 18, 1996.
subsections: section_title: GAO Comments
subsections: section_title: Comment 1
paragraphs: FDA commented that the draft report did not discuss the inherent limitations of the phantom image test or the lack of scientific consensus on a test for the large image receptor. While our draft report correctly reflected FDA’s view that the phantom image test is only an indicator of image problems, we agreed to add clarifying information to recognize limitations suggested by FDA. Similarly, we have added clarification to recognize that, according to FDA, developing a standard for the large image receptor would require additional scientific testing. While we recognize that developing guidance for the large image receptor will take time, FDA is in a position to continue to provide leadership in conducting experiments and in building a scientific consensus on a particular test method.
section_title: Comment 2
paragraphs: FDA commented that our method of presenting aggregate data on the extent of all violations detected during the first and second years of inspections tended to give too much weight to level 3 violations, which FDA characterized as minor. While our report points out that all level 3 violations are not universally regarded as minor, we agree with FDA that aggregating all levels of violations could potentially be misleading. As a result, we have eliminated the aggregate totals from our final report.
section_title: Comment 3
paragraphs: While FDA acknowledged that there have been some start-up problems with the timely follow-up of violations, it asserted that it now has all necessary procedures in place to follow up on violations. We believe that the lack of an adequate compliance follow-up system has been an ongoing problem. Our contacts with FDA field offices, one as recent as late September 1996, showed the lack of a systematic approach to follow up on previous inspection violations. We agree with FDA, however, that the establishment of its automated Compliance Tracking System has significant potential to alleviate the problems with follow-up.
section_title: Comment 4
paragraphs: FDA commented that variation among the states in the number of violations reported would be expected because some states had well-developed mammography programs before MQSA and, as a result, presumably would have had fewer violations than other states. In addition, FDA stated that some states may have imposed stricter standards than those provided by MQSA. We agree that there could be variation in frequency of violations among the states attributable to the states’ pre-MQSA experiences with mammography standards. However, the violation data, in our view, are not reported in a consistent enough fashion to sustain such analysis of variation. Moreover, whether states have higher standards than MQSA should not affect violation data if they are correctly reported by the states. States that establish and enforce higher standards than MQSA should, according to FDA’s own guidance, enforce these standards outside of the MQSA process.
section_title: Comment 5
paragraphs: FDA also commented that our draft report did not accurately reflect the circumstances surrounding FDA’s enforcement in the case of the North Carolina facility. We believe our draft report provided an adequate summary of the key facts in the North Carolina case sufficient to justify our recommendations for additional enforcement procedures, guidance, and training. We note that, after reviewing our draft report, FDA took action to implement our recommendations. However, since FDA believes that additional facts are relevant to the discussion, we have added them to our final report. Specifically, we have (1) added a footnote to the body of the report to explain more fully how FDA reached its conclusion that it would not suspend the facility operation; (2) amended the appendix that contains the chronology of events related to the facility; and (3) as explained above, added information recognizing the limitations of phantom images and clarifying the lack of consensus on available tests for the large image receptor.
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, the following individuals made important contributions to this report: Sarah F. Jaggar, Special Advisor for Health Issues; Susan Lawes, Senior Social Science Analyst; Donna Bulvin, Evaluator; Stan Stenersen, Senior Evaluator; Evan Stoll, Computer Specialist; Craig Winslow and Stefanie Weldon, Senior Attorneys; and Clair Hur, Intern.
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.