Anthrax Detection: DHS Cannot Ensure That Sampling Activities Will Be Validated
Published: 2007-03-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In September and October 2001, contaminated letters laced with Bacillus anthracis were sent through the mail to two U.S. senators and members of the media. Postal facilities in New Jersey, Washington, D.C., and elsewhere became heavily contaminated. The anthrax incidents highlighted major gaps in civilian preparedness to detect anthrax contamination in buildings. GAO was asked to describe and assess federal agencies' activities to detect anthrax in postal facilities, assess the results of agencies' testing, and assess whether agencies' detection activities were validated.
section_title: What GAO Found
paragraphs: Federal agencies conducted several sampling activities, including developing a sampling strategy and collecting, transporting, extracting, and analyzing samples. They primarily collected samples from specific areas, such as mail processing areas, using their judgment about where anthrax would most likely be found--that is, targeted sampling. The agencies did not use probability sampling, which would have allowed agencies to determine, with some defined level of confidence, when all results are negative, whether a building is contaminated. The results of the agencies' testing in 286 postal facilities were largely negative--no anthrax was detected. However, agencies did not use validated sample collection and analytical methods. Thus, there can be little confidence in negative results. With a validated process, agencies and the public could be reasonably confident that any test results generated by that process would be reliable. The Department of Homeland Security (DHS) is the principal agency responsible for coordinating the federal response. Thus, in its 2005 report, GAO recommended that the Secretary of Homeland Security develop a coordinated approach to improve the overall process for detecting anthrax and increase confidence in negative test results generated by that process. DHS stated that while it has overall responsibility for coordinating the federal response during future biological attacks, other agencies have the lead responsibility for validation. Therefore, uncertainty over which agency would take the lead role--that is, who is in charge--in improving the overall process for detecting anthrax, including validation of the methods, continued after GAO issued its report. On the basis of these uncertainties, GAO recommended in its May 9, 2006, testimony that DHS's approach to validating the overall process start with a strategic plan that would include a road map outlining how individual agencies' efforts would lead to the validation of the individual activities as well as the overall process, noting that such a plan would assist DHS in monitoring progress and measuring agency performance toward improving the detection of anthrax and other prioritized threat agents. While DHS generally agreed with these recommendations, it stated that it cannot ensure validation studies would be done, since "there are legal limitations in DHS authority to direct the activities of other agencies." Also, since validation would require a sustained effort over a long period, DHS noted that it could not mandate commitment of other agencies' funds, over which it has no control. Until responsibility is accepted for ensuring that sampling activities will be validated, the fate of the validation process will remain uncertain. Without validation, if another anthrax attack were to occur tomorrow, federal civilian agencies would not be able to conclude with any given level of statistical confidence, in cases of negative results, that a building is free of contamination.

section_title: Letter
paragraphs: We are pleased to be here today to discuss our findings on anthrax  detection testing. The threat of bioterrorism has been recognized for a  considerable time. Long before the anthrax attacks of 2001, several hoax  letters indicating the presence of anthrax had been mailed to federal and  state agencies, as well as to private sector organizations. These events  raised the possibility that facilities could become contaminated and would  therefore have to be evaluated for environmental contamination. However,  federal agencies were not fully prepared to deal with environmental  contamination—that is, anthrax released through the mail—including the  potential for multiple dispersals in indoor environments.
In September and October 2001, contaminated letters laced with Bacillus  anthracis were sent through the mail to two U.S. senators, Thomas  Daschle and Patrick Leahy, and members of the media. The postal  facilities in New Jersey and Washington, D.C., that processed the senators’  letters became heavily contaminated. Other mail routed through these  facilities, as well as additional facilities in the postal network, also became  contaminated. In addition, numerous federal facilities in the Washington,  D.C., area were later found to be contaminated. The letters led to the first  cases of anthrax disease related to bioterrorism in the United States. In all,  22 individuals contracted anthrax disease in four states—Connecticut,  Florida, New Jersey, and New York—as well as in Washington, D.C. Five  of these 22 individuals died.
The anthrax incidents in September and October 2001 highlighted major  gaps in civilian preparedness to detect and respond. In today’s testimony,   I will discuss our findings concerning anthrax sampling activities,  recommendations we made, and a major issue we identified—the  Department of Homeland Security (DHS) cannot ensure and guarantee  that sampling activities will be validated.
In developing this testimony, we relied on our prior work. We conducted  our review in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: In October 2001, an American Media Incorporated employee died from  inhalation anthrax disease. In the same month, contaminated letters laced  with Bacillus anthracis, or anthrax spores, were sent through the mail to  Senators Thomas Daschle and Patrick Leahy. The response to the incident  in the American Media Incorporated building in Florida in September 2001  led to the identification of mail as the potential source of contamination;  eventually, it led to the sampling of the postal facilities. The agencies  began sampling on October 12, 2001, in Florida and stopped on April 21,  2002, when the Wallingford, Connecticut, facility was sampled for the last  time. The letters led to the first cases of anthrax disease related to  bioterrorism in the United States. In all, 22 individuals contracted anthrax  disease in Connecticut, Florida, New Jersey, and New York, as well as in  Washington, D.C., and 5 died.
The federal agencies involved in the response in the postal facilities have  different responsibilities. CDC and state and local health departments  primarily provided public health advice and assistance to USPS. CDC has  primary responsibility for national surveillance of specific diseases,  including anthrax; it also conducts epidemiologic investigations to  determine, among other things, the source of the disease, and it  participates in environmental sample collection and analysis activities.  The FBI is responsible for criminal investigations involving interstate  commerce and the mail and crimes committed on federal property. EPA is  the nation’s lead agency for responding to a release of hazardous  substances into the environment and subsequent decontamination.
On October 8, 2001, the President created the Office of Homeland Security  to develop and coordinate a comprehensive national strategy for dealing  with domestic terrorist threats or attacks. The office, which had limited  involvement in the 2001 response, was superseded by the Homeland  Security Act of 2002, which transferred many of its functions to DHS,  which became operational in 2003. DHS was created by combining many  previously separate agencies and is assigned a lead role in coordinating  the efforts of federal agencies that respond to acts of terrorism in the  United States.
section_title: Major Findings
subsections: section_title: Sampling Strategy
paragraphs: The federal agencies primarily used a targeted strategy—they collected  samples from specific areas considered more likely to be contaminated,  based on judgments. Such judgments can be effective in some situations— for example, in determining whether a facility is contaminated when  information on the source of potential contamination is definitive.
However, in the case of a negative finding, when the source of potential  contamination is not definitive, the basic question—Is this building  contaminated?—will remain unanswered.
CDC and USPS officials said that they used a targeted strategy for several  reasons, including limitations on how many samples could be collected  and analyzed. They also said that in 2001, they lacked the data from  empirical research to develop an initial sampling strategy that  incorporated probability sampling. We disagree with this interpretation.  Probability sampling is statistically based and does not depend solely on  empirical criteria regarding the details of possible contamination.
The situation in 2001 was unique, and the agencies were not fully prepared  to deal with environmental contamination. In the future, if the agencies  decide to use a targeted rather than a probability sampling strategy, they  must recognize that they could lose a number of days if their targeted  sampling produces negative test results. In this case, additional samples  would have to be collected and analyzed, resulting in the loss of critical  time for public health interventions. This was so at the Wallingford postal  facility in the fall of 2001, when about 3 weeks elapsed between the time  the first sampling took place and the results of the fourth testing, which  revealed positive results. Furthermore, about 5 months elapsed between  the time of the first sampling event and the time anthrax was found in the  Wallingford facility’s high-bay area.
Therefore, strategies that include probability sampling need to be  developed in order to provide statistical confidence in negative results.  Further, even if information on all the performance characteristics of  methods is not yet available, a probability sampling strategy could be  developed from assumptions about the efficiency of some of the methods.  And even if precise data are not available, a conservative, approximate  number could be used for developing a sampling strategy. This would give  agencies and the public greater confidence in negative test results than  was associated with the sampling strategy used in 2001.
section_title: Sampling Methods
paragraphs: CDC, EPA, and USPS, the federal agencies involved in sampling the postal  facilities in 2001 to detect anthrax, undertook several activities. These  included development of a sampling strategy followed by collecting  samples, using a variety of methods, and transporting, extracting, and  analyzing the samples. Neither these activities nor the overall process was  validated for anthrax testing. Consequently, the agencies were challenged  by limited information for reliably choosing one method over another and  by lack of information on the detection limit to use when evaluating  negative results.
Federal agencies used different methods for collecting samples. While  USPS generally used dry swabs to collect samples (the least effective  method), CDC and EPA used multiple methods—dry swabs, premoistened  swabs, wet wipes, and a high-efficiency particulate air (HEPA) vacuum— in various combinations or alone.
However, none of the agencies’ collection methods were evaluated for  anthrax detection in environmental samples. In the absence of empirical  research, agencies had no information available for reliably choosing one  method over another and no information on the limits of detection to use  when evaluating negative results.
section_title: Results of Testing
paragraphs: The majority of the samples collected from the postal facilities tested  negative. In all, federal agencies collected about 10,000 samples during  initial testing. It is interesting that of the 9,807 samples that the agencies  collected, more than 98 percent, or 9,648, were negative; a little more than  1 percent, or 159, were positive. In all, 286 facilities were tested for  anthrax contamination. Of these, Brentwood, Trenton, and Morgan were  primary facilities; that is, these 3 facilities processed the original letters  containing the anthrax.
The results of the CDC, EPA, and USPS testing in 286 postal facilities were  largely negative. Of 286 facilities, 23 tested positive. For 2 of these   23 facilities, test results were negative at first but positive on a subsequent  testing. However, in 1 of these facilities—the Wallingford, Connecticut,  facility—it was not until the fourth testing that positive results were  obtained.
Testing results differed between the primary facilities and Wallingford. In  the 3 primary facilities, results were positive each time a facility was  tested, with the important exception of the two quick tests in Brentwood.  In Wallingford, considered less likely to be contaminated, results were  positive only on the fourth sampling. These results underscore the  importance of retesting and cast doubt on the efficiency of the judgmental  sampling strategy.
Of the 263 facilities that tested negative, only 9 were sampled more than  once. A facility in West Trenton tested negative, even though an employee  had contracted cutaneous anthrax. The facility in West Trenton was tested  twice by the FBI and once by CDC, during which a total of 57 samples  were collected, with negative results.
Final, or confirmed, results will be negative if contamination is not present  in a facility. However, a result can be erroneously negative for several  other reasons, such as (1) the sampling method was not efficient enough,  (2) samples were not collected from places where contamination was  present, (3) not enough samples were collected, (4) not enough spores  were recovered from the sample material, or (5) analysis of the sample  extract was not sensitive enough to detect anthrax spores that were  present.
section_title: Conclusions
paragraphs: The agencies that sampled postal facilities in 2001—USPS, CDC, and  EPA—did not use validated sample collection and analysis methods to  perform their tests. According to these agencies, validated methods were  not available at that time. They conducted several interdependent  activities, including sample strategy development, followed by sample  collection, transportation, and analysis of the samples to detect anthrax.  Neither these activities nor the overall process had been validated for  anthrax testing.
Validation is a formal, empirical process in which an authority determines  and certifies the performance characteristics of a given method.  Therefore, investments are also needed to validate these methods, as well  as the overall anthrax detection process. Validating the overall process, as  well as the individual activities, is important because operational and  health-related decisions are made on the basis of testing results that the  process generates.
CDC and USPS officials said that they used targeted sampling; that is, they  collected samples from specific areas considered—based on agencies’  technical judgments—more likely to be contaminated. Such judgments can  be effective in some situations, for example, in determining the source of  contamination in a disease outbreak investigation, provided results are  positive. However, if the results are negative, the basic question—Is this  building contaminated?—cannot be answered with statistical confidence.
When the level of contamination is extremely high and dispersed in a  facility, the method of sampling (for example, wipes versus swabs) is not  as critical if the purpose is to find some contaminant. However, at lower  levels, a way of interpreting the significance of negative results is needed,  and this requirement emphasizes the importance of validation of the  methods and the need for statistically based sampling strategies. This  emphasizes the need for methods that have been validated, and sampling  strategies that are likely to find contamination at low levels. Probability- based sampling does allow conclusions, at specific levels of confidence,  about testing results.
Using a probability-based sampling strategy, together with validated  methods for detecting contamination, would provide a known level of  confidence with which to interpret any negative results. This would allow  agencies to be more definitive in determining necessary actions. Figure 1  shows how lack of validation could affect individual activities—including  the sampling strategy—as well as the results generated by the overall  process.
The lack of validated methods for assessing contamination in postal  facilities impeded the agencies in responding to the incidents. The  significance of the lack of validated methods was exemplified in the case  of the one postal facility where negative preliminary results were obtained  by field-based methods of analysis, with limitations that appear not to have  been well understood by some agencies. Negative results do not  necessarily mean a facility is free from contamination. As we reported,  results can be negative if (1) samples were not collected from places  where anthrax was present, (2) the detection limit of the method was  greater than the actual contamination level, (3) not enough samples were  recovered from the sample material, (4) analysis of the sample extract did  not detect spores, or (5) anthrax was not present in the facility.
In addition, while the 2001 events involved anthrax, many other biothreat  agents exist. Differences in their characteristics mean different solutions.  Accordingly, efforts to develop sampling strategies and to validate  methods should address requirements specific to those threat agents as  well. However, since addressing other agents would consume resources  and time, all these efforts should be prioritized in a long-term strategy.
The several agencies that dealt with the anthrax attacks generally worked  well together, but we have identified areas that would have benefited from  one agency’s taking the lead in coordinating the response. Given the  mission of DHS and its responsibilities, it appears that DHS is now well  positioned to take a lead role in promoting and coordinating the activities  of the various agencies that have technical expertise related to  environmental testing. In addition, it is important that all participating  agencies recognize and support DHS in that role and that they have an  effective structure for participating in identifying and addressing the  appropriate issues.
section_title: Recommendations for Executive Action
paragraphs: Given the lack of validated methods for detecting anthrax contamination  in facilities, we recommended in our 2005 report that the Secretary of  Homeland Security develop a coordinated approach to (1) improve the  overall process for detecting anthrax and (2) increase confidence in  negative test results generated by that process. This approach would  include working with agencies to ensure that appropriate validation  studies of the overall process of sampling activities, including the  methods, are conducted. Specifically, we recommended that the Secretary  1.  take a lead role in promoting and coordinating the activities of the  various agencies that have the technical expertise related to  environmental testing;  2.  ensure that a definition of validation is developed and agreed on;  3.  guarantee that the overall process of sampling activities, including  methods, is validated so that performance characteristics, including  limitations, are clearly understood and results can be correctly  interpreted;  4.  see that appropriate investments are made in empirical studies to  develop probability-based sampling strategies that take into account  the complexities of indoor environments;  5.  ensure that appropriate, prioritized investments are made for all  6.  make sure that agency policies, procedures, and guidelines reflect the  results of such efforts.
When we issued our report, CDC, DHS, and USPS agreed with our  conclusion—that methods for detecting anthrax contamination in facilities  were not validated—and with the thrust of our recommendations—calling  for a coordinated, systematic effort to validate the methods to be used for  such testing. But they (1) disagreed with or expressed concern about our  conclusions or the recommendation dealing with targeted versus  probability sampling, (2) emphasized that validated testing methods for  anthrax were not available in 2001 and that federal and state organizations  did the best they could under the circumstances, and (3) identified factors  or issues that need to be considered in validating testing methods.
section_title: Who Is Responsible for Ensuring and Guaranteeing That Anthrax Detection Methods Will Be Validated?
paragraphs: After we issued our 2005 report, it became evident that there was  uncertainty over which agency would take the lead role in improving the  overall process for detecting anthrax and how studies were to be funded.  For example, DHS stated that while it has overall responsibility for  coordinating the federal response during future biological attacks, EPA  had the “primary responsibility for establishing the strategies, guidelines,  and plans for the recovery from a biological attack” and HHS had the lead  role for any related public health response and guidelines. DHS also stated  that it coordinated regularly with EPA’s National Homeland Research  Center to exchange information on research needs and to discuss  priorities and gaps for a wide range of security-related research areas.
DHS stated that it would coordinate with EPA to ensure that appropriate  investments were made to explore improved sampling. However, it is  unclear to us how DHS would ensure that appropriate prioritized  investments are made for all biothreat agents and how such priorities and  gaps would be addressed.
On the basis of these uncertainties, we recommended in our May 9, 2006,  testimony that DHS’s approach to validating the overall process should  start with a strategic plan that includes a road map outlining how  individual agencies efforts would lead to the validation of the individual  activities as well as the overall process, noting that such a plan would  assist DHS in monitoring progress and measuring agency performance  toward improving the detection of anthrax and other prioritized threat  agents.
On May 19, 2006, DHS officials stated that while they concurred with the  recommendations from our report and accepted the overall responsibility  to ensure the methods will be validated, they stated that “there are legal  limitations in DHS authority to direct the activities of other agencies.”  They said that while they take a lead role in coordinating the meetings and  in bringing people from different agencies together, they cannot guarantee  that the overall process of sampling will be validated because different  agencies have responsibility for different aspects of validation, and DHS’s  control over other agencies actions and budgets is ultimately limited. They  stated that DHS cannot ensure and guarantee that validation studies would  be done, since this is a shared responsibility among different agencies.  Also, since validation would require a sustained effort over a long period,  DHS noted that it could not mandate commitment of other agencies’ funds,  over which it has no control.
DHS officials told us in July 2006 that they recognize that DHS is the  principal agency responsible for coordinating the federal response and  they would work with a good faith effort toward developing a strategy for  validation studies and a road map by the end of calendar year 2006  outlining how individual agencies’ efforts would lead to the validation of  the overall sampling process. On March 27, 2007, DHS told us that it had  developed a working draft of the strategic plan and the road map by  December 2006 but it could not share these with us because they were not  final.
Until responsibility is accepted for ensuring that sampling activities will be  validated, the fate of the validation process will remain uncertain. Without  validation, if another anthrax attack were to occur tomorrow, federal  civilian agencies would not be able to conclude with any given level of  statistical confidence, in cases of negative results, that a building is free of  contamination.
Mr. Chairman, this concludes my prepared remarks. I would be happy to  respond to any questions that you or other members of the subcommittee  may have at this time.
section_title: Contacts and Acknowledgments
paragraphs: For further information regarding this statement, please contact Keith  Rhodes at (202) 512-6412, or rhodesk@gao.gov, or Sushil K. Sharma, Ph.D.,  Dr.PH, at (202) 512-3460, or sharmas@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this statement. William Carrigg, Barbara Chapman, Crystal  Jones, Penny Pickett, and Elaine Vaurio made key contributions to this  statement.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.