Anthrax: Agency Approaches to Validation and Statistical Analyses Could Be Improved
Published: 2014-12-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2001, the FBI investigated an intentional release of B. anthracis , a bacterium that causes anthrax, which was identified as the Ames strain. Subsequently, FBI contractors developed and validated several genetic tests to analyze B. anthracis samples for the presence of certain genetic mutations. The FBI had previously collected and maintained these samples in a repository.
GAO was asked to review the FBI's genetic test development process and statistical analyses. This report addresses (1) the extent to which these genetic tests were scientifically verified and validated; (2) the characteristics of an adequate statistical approach for analyzing samples, whether the approach used was adequate, and how it could be improved for future efforts; and (3) whether any remaining scientific concerns regarding the validation of genetic tests and statistical approaches need to be addressed for future analyses. GAO reviewed agency and contractor documentation, conducted literature reviews, and conducted statistical analyses of the repository data. GAO's review focused solely on two aspects of the FBI's scientific evidence: the validation of the genetic tests and the statistical approach for the analyses of the results. GAO did not review and is not taking a position on the conclusions the FBI reached when it closed its investigation in 2010.
section_title: What GAO Found
paragraphs: After the 2001 Anthrax attacks, the genetic tests that were conducted by the Federal Bureau of Investigation's (FBI) four contractors were generally scientifically verified and validated, and met the FBI's criteria. However, GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process. As a result, each of the contractors developed their tests differently, and one contractor did not conduct verification testing, a key step in determining whether a test will meet a user's requirements, such as for sensitivity or accuracy. Also, GAO found that the contractors did not develop the level of statistical confidence for interpreting the testing results for the validation tests they performed. Responses to future incidents could be improved by using a standardized framework for achieving minimum performance standards during verification and validation, and by incorporating statistical analyses when interpreting validation testing results.
GAO identified six characteristics of a statistical framework that can be applied for analyzing scientific evidence. When GAO compared the approach the FBI used to this framework, it found that that the FBI's approach could have been improved in three of six areas. First, the FBI's research did not provide a full understanding of the methods and conditions that give rise to genetic mutations used to differentiate between samples of B. anthracis . Second, the FBI did not institute rigorous controls over the sampling procedures it used to build the repository of B. anthracis samples. Third, the FBI did not include measures of uncertainty to strengthen the interpretation of the scientific evidence. GAO found that since 2001 the FBI has taken some steps to build formal forensic statistical expertise. The FBI's approach to future incidents could benefit from including such expertise early in an investigation.
The lack of an understanding of how bacteria change (mutate) in their natural environment and in a laboratory is a key scientific gap that remains and could affect testing conducted in future incidents. Specifically, the significance of using such mutations as genetic markers for analyzing evidentiary samples to determine their origins is not clear. This gap affects both the development of genetic tests targeting such mutations and statistical analyses of the results of their use on evidentiary samples. The Department of Homeland Security is currently funding some research on genetic changes in bacteria and genome sequencing methods, among others. Such research is a step in the right direction since the FBI is planning to use genome sequencing methods in future investigations. However, because this research may not be complete for several more years, the extent to which it will close this gap is not known.
section_title: What GAO Recommends
paragraphs: GAO recommends that the FBI develop a framework for validation and statistical approaches for future investigations. The FBI agreed with our recommendations.

section_title: Letter
paragraphs: In September and October 2001, letters laced with Bacillus anthracis  Ames strain (B. anthracis) were mailed through the U.S. postal system to  two U.S. senators and members of the media. Public health entities,  county and state public health departments and the Centers for Disease  Control (CDC), in collaboration with the Federal Bureau of Investigation  (FBI) and the U.S. Postal Inspection Service, took the initial lead in the   The FBI worked in collaboration with public health entities  investigation.and U.S. Postal Inspection Service when a deliberate act was eventually  suspected, following the identification of the attack letters and the first  victim in Florida. In 2007, the FBI determined that the spores in the letters  were derived from a single spore-batch of the Ames strain in a flask  called “RMR-1029.”
In 2008, the FBI asked the National Research Council (NRC) of the  National Academy of Sciences (NAS) to review the scientific approaches  it had used to support its conclusions. During the NRC committee’s  deliberation, the FBI announced on February 19, 2010, that it was closing  the case, having concluded that a scientist at the United States Army  Medical Research Institute for Infectious Diseases (USAMRIID) had   In February 2011, the NAS issued its  perpetrated the attack alone.report, concluding that “it is not possible to reach a definitive conclusion  about the origins of the B. anthracis in the mailing based on the available  scientific evidence alone.” The NAS report detailed many methodological  and organizational problems in the scientific portion of the FBI’s  investigation, known by the case name Amerithrax.
Because several scientific and technical issues were not covered in the  scope of the NAS study, you asked us to conduct an independent  technical evaluation of the scientific approaches used in support of the  FBI’s investigation, focusing on certain issues. We reviewed the NAS  findings and conclusions and determined that, since NAS did not report in  depth how the genetic assays (or tests) used to screen the repository  were scientifically verified and validated, additional evaluation of the  requirements and procedures used for doing so could be informative in  developing scientific methods for future investigations. The NAS report  stated that, “Although the committee lauds and supports the effort  dedicated to the development of well-validated assays and procedures,  looking toward the future, these processes need to be more efficient.”  Further, the NAS report included several findings related to the statistical  analyses of the repository data and identified challenges concerning the  identification and collection of the repository samples. Therefore, a review  of the comprehensive statistical approach could help further clarify and  expand on the impact of the NAS conclusions and provide useful insight  for applying statistical approaches to future investigations. Consequently,  this report addresses the following three questions:  1.  To what extent were the genetic assays used to screen the FBI  repository of Ames samples scientifically verified and validated?2.  What are the characteristics of an adequate statistical approach for  analyzing the repository samples and to what extent was the  statistical approach used adequate? If not adequate, how could this  approach be improved for future efforts?  3.  What remaining scientific concerns and uncertainties, if any, regarding  the validation of genetic tests and statistical approaches will need to  be addressed in future analyses? What additional research, if any,  would be helpful in resolving such scientific uncertainties in any future  investigation?
The scope of our work was limited to a review of the scientific methods  employed to verify and validate the genetic tests used to screen the FBI’s  repository of Ames B. anthracis samples, the procedures used to identify  and collect samples of Ames B. anthracis in the creation of the FBI’s  repository, and the statistical analyses and interpretation of the results of  the genetic tests. Thus, we did not address any other scientific methods  or any of the traditional investigative techniques used to support the FBI’s  conclusions in this case, and we take no position on the conclusions the  FBI reached when it closed its investigation in 2010.
To meet our objectives, we reviewed pertinent agency and FBI contractor  documentation on the verification and validation of the genetic tests used  to screen the FBI’s repository of Ames samples. We also reviewed  scientific literature and agency and industry guidelines to determine the  essential phases in approaches to developing genetic tests. We reviewed  existing FBI standards and guidelines for verifying and validating  microbial forensics methods. To determine the extent to which the genetic  tests were verified and validated, we reviewed the results of the  contractors’ validation tests, the FBI’s requirements for the validation, and  the contractors’ approaches for developing their genetic tests against an  approach, or framework, for validation, encompassing the essential  phases that we identified. We interviewed officials and scientists at the  FBI, its contractors, and others regarding the development and validation  of the genetic tests.
To determine the adequacy of the FBI’s statistical approach, we  conducted a literature review to identify the characteristics of an adequate  statistical approach, analyzed agency and FBI contractor documentation,  and interviewed FBI and laboratory officials. We conducted an analysis of  the repository data to examine the effect of data trimming assumptions  and additional estimates of false negative rates on the conclusions of the  FBI’s statistical analyses.
To identify scientific gaps and associated research related to questions 1  and 2, we reviewed agency and contractor documents and related  scientific literature/reports and research agendas. We interviewed  officials, statisticians, and scientists from the Department of Homeland  Security (DHS), the FBI, and others regarding scientific gaps. We met  with entities that were involved in the validation of the genetic tests and  research addressing scientific gaps, such as the FBI and DHS  contractors, and others involved in the repository testing. Finally, we used  technical and scientific guidance we received from experts on the  technologies and statistical approaches used in the FBI’s investigation,  and we read their comments on our draft report. We selected these  experts for their expertise in public health and microbial forensics. Further  details of our scope and methodology are in appendix I.
We conducted this performance audit from January 2013 to November  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: The Attack History
paragraphs: In September and October 2001, at least seven envelopes containing  significant quantities of B. anthracis spores were mailed through the U.S.  postal system to two senators at their congressional offices in the District  of Columbia and to media organizations in New York City and Boca  Raton, Florida. According to the FBI, the evidence supports the  conclusion that the mail attacks occurred on two separate occasions. The  two letters of the first attack were postmarked September 18, 2001, and  sent to NBC News and the New York Post, both in New York City. Three  weeks later, two letters postmarked October 9, 2001, were mailed to two  senators—Thomas Daschle and Patrick Leahy—at their Washington,  D.C., offices. Other letters were sent to ABC, CBS, and American Media,  Inc. Hard evidence of the attacks surfaced on October 3, 2001, when  Robert Stevens, an American Media Inc. employee who worked in Boca  Raton, Florida, was diagnosed as having contracted inhalational anthrax,  from which he later died. However, because a contaminated envelope or  package was not recovered in Florida, the agencies could not initially  establish how the B. anthracis spores were delivered. According to the  Postal Service, the combination of the Florida incident and the opening of  the letter to Senator Tom Daschle on October 15 established the link to  the U.S. mail system. At least 22 victims contracted anthrax as a result of  the mailings. Eleven individuals developed inhalational anthrax, and  another 11 developed cutaneous infections. Five of the inhalational  anthrax victims died from their infections.
The attack highlighted the need for enhanced capabilities for full forensic  exploitation and interpretation of microbial evidence from acts of  bioterrorism. Ideally, forensic evidence obtained in an investigation is  sufficient to support conclusions about the culpability of a group, an  individual, or the source of material used in such an act. Forensic  evidence is used to support conclusions by classifying evidence into one  of several categories that distinguish possible sources from one another.  While classification does not unequivocally demonstrate a connection  with an individual or a single source, it can be used to reduce the number  of possible sources and thus can provide important leads in an  investigation.
The development and application of microbial forensics was essential to  the FBI’s scientific investigation, which relied heavily on genetics and  comparative genomics to classify the spore materials used in the attack,  reduce the number of possible sources and suspects, and provide  investigative leads. In fact, according to the NAS, this investigation  accelerated the development of the then nascent field of microbial  forensics.
section_title: The FBI’s Genetic Analyses of the Attack Spores
paragraphs: The FBI’s investigation, assisted by government, university, and  commercial laboratories, was an effort to develop the physical, chemical,  genetic, and forensic profiles of the anthrax spores in the letters and  envelopes used in the attacks. The investigation employed myriad  traditional and novel investigative and scientific methods. The scientific  methods involved efforts to develop the physical, chemical, genetic, and  forensic profiles of the anthrax spores and letters and envelopes used in  the attacks so as to identify the source of the spores. The FBI faced many  difficult and complex scientific challenges over the course of this  investigation, according to the NAS. New microbial forensic methods  were developed and implemented over several years, and some of them  provided valuable evidence and significant leads in the case. For  example, according to the FBI, new methods to determine the source of  the growth media for the mailed spores were inconclusive while the use of  the genetic mutations provided an investigative lead.
By October 2001, CDC had identified the microorganism used in the  attack as Bacillus anthracis (B. anthracis). This was a key step in the  classification of the microorganism used in the attack letters and was one  of the first scientific findings that allowed the FBI to begin to reduce the  number of possible sources of the spores. B. anthracis is a gram positive,  rod-shaped bacterium that causes the disease anthrax. It is a member of  the larger genus Bacillus that includes other commonly found species,  such as B. cereus, B. subtilis, and B. thuringiensis. B. anthracis, a  species of Bacillus that can be found on all continents except Antarctica,  typically shows little genetic variation among isolates. However, during  the investigation scientific methods were being developed that allowed  scientists to find some genetic differences among natural isolates of B.  anthracis.  Applying these methods allowed the FBI to refine the  classification of the spores used in the attack and further reduce the  number of possible sources of the spores.
Even in the most homogeneous species, some differences are usual in genome  sequences among populations. Although few in number, these differences are sufficient to  characterize subgroups, or “strains.” in the attack letters. In fact, this scientific evidence allowed the FBI to  focus its investigation on the limited number of laboratories that had had  access to the Ames strain before the attacks (see figure 1).
While classifying the spore material as the Ames strain was instrumental  in reducing the number of possible sources, it was not sufficient by itself  to definitively identify the source of the material used in the attack as a  single laboratory, flask, or person. The FBI then sought to identify  additional characteristics of the spores used in the attack that could  further discriminate between possible sources of the Ames strain.
Scientists from the Department of Defense (DOD) tested samples of the  spore materials found in the letters and identified several morphological  variants (or morphs).the spores from the letters over an extended period observed that a small  percentage of the colonies differed in texture, color, and growth patterns   A laboratory technician who had grown (cultured)  from those typical for the Ames strain of B. anthracis, referred to in figure  2 as the “wild type.”
In an effort to identify the source of the letters, investigators and FBI  scientists began to evaluate whether they could first identify and  characterize these morphs genetically and then determine whether any of  them were present in the repository of Ames samples. This involved  genome sequencing to identify whether specific deoxyribonucleic acid  (DNA) sequences underlay the morphs. Eventually, as shown in figure 2,  the morphs were associated with several types of genetic mutations:  duplications, single nucleotide polymorphisms (SNP), and deletions,  referred to as INDELS.
Afterward, over several years, outside contractors’ laboratories developed  and validated several genetic tests to analyze the B. anthracis samples  for the presence of certain genetic mutations. Specifically, the testing  revealed the presence or absence in a sample of a specific DNA  sequence (that is, the genetic mutation) associated with a given morph.
The FBI contractors generally referred to the tests they developed as A1,  A3, D, and E. Commonwealth Biotechnologies (CBI) developed the two A  tests (A1 and A3), which targeted two different DNA sequences; and the  Institute for Genomic Research (TIGR) developed the E test, targeting  another DNA sequence. However, unlike the others, the Illinois Institute of  Technology Research Institute (IITRI) and Midwest Research Institute  (MRI) both developed a test targeting the same DNA sequence. For  clarity, we refer to the IITRI-developed test as D-1 and the MRI- developed tests as D-2.
Genetic test A1: detects the presence of a specific duplicated DNA  sequence associated with morph A;
Genetic test A3: detects a different duplicated DNA sequence from  that targeted by A1 but also associated with morph A;
Genetic test D-1: detects the presence of a specific deleted DNA  sequence associated with morph D;
Genetic test D-2: detects the presence of the same deleted DNA  sequence associated with morph D as that targeted by the D-1 test;  and
Genetic test E: detects a deleted DNA sequence associated with  morph E.
In 2002, the FBI began collecting samples from laboratories in  possession of the Ames strain to compare them with the material used in  the attack. A grand jury issued subpoenas to 16 domestic laboratories  and the FBI requested submissions from 3 foreign laboratories that  investigators had determined possessed the Ames strain. The subpoenas  required each laboratory to identify and submit two representative  samples from each distinct stock of the Ames strain it held. The  subpoena included instructions to the laboratories on how to identify,  select, and submit samples to the FBI. Laboratories were required to ship  sample submissions to DOD scientists at USAMRIID for preparation and  entry into the FBI repository of Ames samples.
In addition to the samples submitted in response to the subpoena,  searches were conducted at three domestic laboratories to ensure that  samples were taken from each stock of Ames strain in those facilities.  The FBI assembled a repository of 1,070 Ames strain samples, of which   From 2004 through 2007, each of the 1,059 viable  1,059 were viable. repository submissions was compared to the evidentiary material using  the five genetic tests (see figure 3). The results of the genetic testing  indicated that only 8 of the 1,059 FBI repository Ames samples tested  positive for the presence of the four genetic mutations originally found in  the anthrax letter evidence.
Using submission records, investigators concluded that these 8 samples  were derived from a single source—a flask identified as RMR-1029.  According to the FBI, this information constituted a groundbreaking lead  in the development in the investigation. It allowed the investigators to  reduce drastically the number of possible suspects, because only very  few individuals had ever had access to this specific flask. Armed with this  new information obtained from the scientific evidence, the task force  focused its investigation on researchers who had had access to the  laboratory at USAMRIID where RMR-1029 was stored. In 2008, the FBI  sought to conduct statistical analyses in order to determine (1) the  probative value of genetic markers found in a sample, and (2) possible  inferences regarding the relationships of similar samples. A contractor  submitted the Final Statistical Analyses Report in October 2008.
In February 2010, the FBI closed the case concluding that a scientist at  USAMRIID had perpetrated the attack alone. Neither the case nor the  totality of the evidence, including the scientific evidence that provided the  FBI with valuable leads, was brought to trial in a court of law. The alleged  perpetrator of the attack died on July 29, 2008, from an overdose of over- the-counter medication.
section_title: Verification and Validation
paragraphs: At the start of the investigation no standards or guidelines existed for  verifying and validating microbial forensic methods, including the  polymerase chain reaction (PCR)-based tests that were eventually used  to identify the genetic mutations in the repository samples. The first  contractor, Commonwealth Biotechnologies (CBI), an established  forensics laboratory, had begun developing the A1 and A3 genetic tests in  2002. A CBI official stated that it had relied upon the National Institute of  Standards (NIST) guidance on the validation of methods for detecting  human DNA, and also on the DNA Advisory Board standard for forensics.  For the remaining three contractors, the Illinois Institute of Technology  Research Institute (IITRI), Midwest Research Institute (MRI), and The  Institute for Genomic Research (TIGR), the FBI provided “Quality  Assurance Guidelines for Laboratories Performing Microbial Forensic  Work”—guidelines that the FBI’s Scientific Working Group on Microbial  Genetics and Forensics (SWGMGF) developed and published in October  2003. These guidelines defined validation as a process by which a test  procedure is evaluated to determine its efficacy and reliability for  analysis.confidence in the ability of the test (the measurement tool) to accurately  identify the properties of interest in samples that are to be analyzed.
Verifying and validating a test method provides a level of  Verification, confirms by objective evidence, from laboratory experiments,  that the given test meets the user’s specific requirements, such as criteria  for accuracy. If the verification testing were not to produce consistent  results, then the scientist or the laboratory would have to return to the  optimization phase to further refine the method and materials and then  revise the test’s standard operating procedure (SOP) accordingly.  Verification of the acceptance criteria must include repeated testing to  account for measurement uncertainty, and confidence in performance  statistics should be reported. Depending on the intended use of a test, sensitivity, specificity, limit of detection (LOD), reproducibility, bias, and  precision may all be measures of performance (performance statistics)  that should be evaluated. The type of test (qualitative, quantitative, or  semi-quantitative) may also determine which of these performance  parameters is to be evaluated. The testing protocol and materials,  including quantities that optimize test performance are recorded in a test’s  SOP.
Validation confirms by examination, from laboratory experiments, and the  provision of objective evidence that the particular requirements for a  specific intended use are fulfilled. Successful validation offers some  assurance that a given genetic test is sufficiently robust to provide  reproducible results, regardless of the practitioner, agency, contractor, or  laboratory applying it to a sample. Validation is frequently used to connote  confidence, but it may also be thought of as defining the limitations of a  method. Studies are conducted that enable the estimation of the limits of  the procedures and the measurements of the test. To the extent possible,  the validation of a method should mimic “real world” conditions. The limits  of the method must be known, demonstrated and documented. In  essence, validation measures the uncertainty in the test output.
section_title: Reviews of the Strength of the Scientific Evidence
paragraphs: In 2007 the FBI convened a team of scientists to review selected scientific  methods used in the case. Referred to as the AMX Red Team, it was  asked to assess whether the science used was sound and to consider  what additional tests might be performed to benefit the investigation. The  team, finding no shortfalls or deficiencies in the basic methodologies it  reviewed, concluded that the “genetic signatures correlating with specific  morphs were valid tools for eliminating those repository samples not  closely related to the spores used in the attack.” However, the team also  stated that the extent of research and development of the genetic tests at  the date of its review was insufficient to determine whether the presence  or absence of one or several of the morphs in a sample was associated  with the evidence, was merely characteristic of normal culture practices,  or possibly was affected by the genetic tests’ sensitivity of detection. The  team recommended additional studies to characterize the genetic  markers as a function of growth conditions, including the influence of  growth time, growth media, and temperature. It also recommended  additional evaluation of the sensitivity of detection of each genetic test to  ensure a reliable interpretation of analyses.
In 2008, the FBI asked the National Research Council (NRC) of the  National Academy of Sciences (NAS) to review the scientific approaches  it had used to support its conclusions. In 2011, the NAS issued its report,  concluding that “it is not possible to reach a definitive conclusion about  the origins of the B. anthracis in the mailing based on the available  scientific evidence alone.” Additionally, the report included the following  findings related to the development and validation of the genetic tests:  “Specific molecular assays were developed for some of the B.  anthracis Ames genotypes (those designated A1, A3, D, and E)  found in the letters. These assays provided a useful approach for  assessing possible relationships among the populations of B.  anthracis spores in the letters and samples subsequently collected  for the  FBIR. . . . However,  more could have been done to determine the performance  characteristics of these genetic tests. In addition, the assays did  not measure the relative abundance of the variant morphotype  mutations, which might have been valuable and could be  important in future investigations. . . .”  “The development and validation of the variant morphotype  mutation assays took a long time and slowed the investigation.  The committee recognizes that the genomic science used to  analyze the forensic markers identified in the colony morphotypes  was a large-scale endeavor and required the application of  emerging science and technology. Although the committee lauds  and supports the effort dedicated to the development of well- validated assays and procedures, looking toward the future, these  processes need to be more efficient.”
Additionally, the NAS report included the following findings related to the  statistical approach taken to quantify the significance of finding the  genetic markers in a small number of repository samples:  “The results of the genetic analyses of the repository samples  were consistent with the finding that the spores in the attack  letters were derived from RMR-1029, but the analyses did not  definitively demonstrate such a relationship.”  . . . . .  “Some of the mutations identified in the spores of the attack letters  and detected in RMR-1029 might have arisen by parallel evolution  rather than by derivation from RMR-1029. This possible  explanation of genetic similarity between spores in the letters and  in RMR-1029 was not rigorously explored during the course of the  investigation, further complicating the interpretation of the  apparent association between the B. anthracis genotypes  discovered in the attack letters and those found in RMR-1029.”
section_title: Genetic Tests Generally Were Verified and Validated but Lack of a Validation Framework Limited Statistical Confidence for Interpreting Results
paragraphs: We found that the genetic tests used to screen the FBI’s repository of B.  anthracis samples demonstrated through the verification and validation  testing that they generally met the FBI’s minimum validation  requirements. However, the FBI’s validation procedure did not require  and the tests did not demonstrate a level of statistical confidence for  interpreting the validation results. Also, tests conducted after validation— although not required by the quality assurance guidelines provided to the  contractors—yielded valuable information on the performance  characteristics of the genetic tests. Therefore, by not having a  comprehensive validation approach, or framework, that sets out  consistent steps for achieving minimum performance standards, and  includes an assessment and measurement of the uncertainty in the test  performance (see table 1 for the phases of a validation framework), the  FBI cannot have statistical confidence in its validation test results.  Knowledge of uncertainty is essential for subsequent statistical analysis  that can provide quantitative measures of confidence in conclusions  drawn from tests applied to forensic samples. According to DHS, it now  validates methods and tests used to support FBI investigations and has  an established ISO-accredited program.
subsections: section_title: Three Phases in a Validation Framework for Genetic Test Development
paragraphs: In our review of scientific literature and agency and industry standards  along with guidelines regarding the verification and validation of methods  for analyzing both microbial and human DNA, we found that terminology  and the extent of verification and validation differed across industries.  However, we identified three distinct phases in genetic test development:  (1) optimization, (2) verification testing, and (3) validation testing. While  the literature and various validation standards and guidelines that we  reviewed identify the specific types of tasks for each phase, we found that  a clear boundary does not always exist between the first two phases.  That is, optimization and verification are sometimes treated as a single  continuous process. Further, we found that verification and validation are  sometimes used interchangeably to describe the same process. Thus, the  process could combine either optimization and verification or optimization  and validation. Nevertheless, what is important is that the approach, or  framework, to test development generally includes these phases and the  associated key tasks, as shown in table 1.
section_title: The FBI Set Minimum Performance Requirements and Relied on Contractors to Conduct Verification Testing
paragraphs: The FBI set limited performance requirements for the genetic tests and  relied on the contractors’ expertise to determine the processes they would  use to develop (i.e., optimize and verify) their tests (see table 1 for types  of performance parameters to be evaluated). According to the FBI, it  provided minimal direction to the four contractors on how they were to  develop their genetic tests in order to allow creative development. It  stated that, with a few exceptions, it left the development mostly to the  contractors who were experienced in developing tests. However, we  found that the contractor’s approaches differed in their (1) use of  verification and validation guidelines, (2) steps in conducting optimization  and verification testing, and (3) interpretation criteria for results generated  by the genetic tests.
The FBI required that the genetic tests detect the target mutations in an  overwhelming background of bacteria consisting of predominantly wild  type B. anthracis Ames, which had been found in the evidentiary material  (that is, the letters). Further, the FBI specified that sensitivity was to be  demonstrated by the LOD—that is, the lowest concentration level that can  be reliably detected for a qualitative and quantitative test. The FBI did  not require a specific calculation or value for the LOD. According to the  FBI, it was looking for the presence or absence of the morphs (genetic  mutations) in the repository samples and the LOD was an important  factor. Three contractors developed qualitative tests; the fourth developed  a semi-quantitative test. In this regard, the FBI wanted to know the  lowest concentration at which the genetic tests could detect the presence  of a specific genetic mutation in a sample. Specificity was to be  demonstrated by the detection of the target in a sample containing an  overwhelming background of predominantly B. anthracis Ames. We found  that the contractors evaluated other performance parameters at their  discretion (see appendix II).
We also found that standards for the verification and validation of  microbial forensics methods did not exist at the start of the investigation  and were only limited after it had begun. At that time, more was known  about verifying and validating human DNA testing methods for forensics  than about microbial forensics methods, as reflected in the revised quality  assurance guidelines. In addition, we found that the contractors’  disparate experience and the FBI’s minimal instruction to them  contributed to the differences in their expectations and approaches. Most  of the contractors had worked for other federal agencies whose  processes differed and thus their approaches to optimizing and verifying  their genetic tests differed.
While most of the contractors had developed methods for the federal  government, one contractor said that each of its federal sponsors had its  own processes for validation and that it followed a particular agency’s  processes when working with it. The contractor also stated that its own  internal quality assurance guidelines were more stringent than the  SWGMGF guidelines for validation. One contractor was a forensics  laboratory that was familiar with analyzing human DNA samples and  using associated quality assurance standards, including the DNA  Advisory Board standards. Another contractor was engaged in genomic  research. Finally, the FBI stated that it was more confident after the two A  tests were developed; it had required the contractor for the two A tests to  subject material from each polymerase chain reaction (PCR) well to  genetic sequence analysis, regardless of the result (positive or negative).  Further, it stated that after the first four genetic tests were developed, it  had been unsure as to whether it wanted to proceed with the last one— the E test.
We found that the contractors generally conducted the tasks we identified  in table 1 under the first two phases—optimization and verification—to  develop the genetic tests and determine their performance, although one  did not conduct a verification test. Specifically, CBI conducted an “internal  qualification” study that is included in the SWGMGF guidelines. CBI’s  qualification study involved multiple experiments using internally blinded  samples following an SOP to determine whether (1) the A1 and A3  genetic tests could correctly identify the targeted genetic mutations and  (2) the staff involved could be considered qualified to perform the genetic  tests. The first appeared to be equivalent to verification testing and the  second to proficiency testing. Similarly, both MRI and ITRI conducted  internal verification testing and followed it with a qualification test of  laboratory personnel. While the distinction between, interpretation of, and  expectations for the verification and any qualification testing were not  always explicit in the documentation we were provided, we found that  they were both intended to precede the validation testing. However,  TIGR—the last contractor to develop its genetic test—did not conduct the  equivalent of either a verification or a qualification study.
The FBI indicated that it believed that the contractors had conducted  verification testing but acknowledged that it was possible that one had not  been conducted for the last genetic test that was developed. Thus, the  verification testing was not consistent for all the tests—with one relying  solely on the validation testing to determine whether it met the FBI’s  requirements and also was fit for use on the repository samples.
We also found that there was no clear rationale for the lack of  complementary interpretation criteria for the results generated by the two  genetic tests that targeted the D mutation, which proved problematic  during the repository screening, after verification and validation had been  completed. Each contractor independently developed interpretation  criteria for positive, negative and inconclusive results through laboratory  experimentation, which when defined became part of its SOP. Initially, for  the A1 and A3 tests interpretation criteria were for a positive or negative  result only. For the A1 and A3 tests, validation results were reported as  the number of correct positive and negative results for the FBI-provided  samples, and excluded blind samples for which, at this stage of the  investigation, it was not yet known whether they contained the targeted  genetic mutations.were reported as the number of correct positive and negative results,  detection limit, false positive rate, and inconclusive rate.
For the D-1 and D-2 tests and the one E test, results  Criteria for an inconclusive result included several types of occurrences  that varied by the particular genetic test. The FBI stated that it reviewed  the interpretation criteria for each genetic test. However, after the  repository screening, disparate interpretation criteria for the D-1 and D-2  genetic tests determined the samples that had usable test results.  Ultimately, contradictory interpretations of the D-1 and D-2 test results  were a reason for eliminating the results of the D-1 genetic test’s  screening of the repository samples; thus, the D-1 results were not part of   This issue did  the final statistical analyses, according to the NAS report.not surface during the validation of the genetic tests.
section_title: Validation Test Results Met the FBI Requirements but Did Not Demonstrate a Level of Statistical Confidence
paragraphs: We found that (1) the genetic tests used to screen the FBI’s repository of  B. anthracis samples met the FBI’s validation requirements, (2) the  validation tests were not required to and did not demonstrate a level of  statistical confidence for interpreting the validation test results, and (3)  some information on the sensitivity and specificity of the genetic tests was  not characterized until after validation (postvalidation testing). As a result,  the performance characteristics of the genetic tests were not fully  understood when they were applied to the repository samples and more  could have been done to strengthen the quality of the data and ultimately  the validation results.
The validation test results showed that the genetic tests met the FBI’s  requirements in that they were able to detect the targeted genetic  mutation when it was present at a low level (that is, at less than 1 percent,  in a validation sample containing predominantly B.anthracis Ames),  although no measurement of statistical confidence in these results was  provided. As shown in table 2, the A3 genetic test had the lowest LOD, at  0.001 percent, while the others ranged from 0.005 percent to 0.01  percent. The validation results led the FBI to determine that there were  no false positives for any of the genetic tests, and that the inconclusive  rates were 0.12 percent and 0.02 percent for the D-1 and D-2 tests,  respectively. During validation, inconclusive rates could not all be  computed for all the genetic tests.
Since the FBI’s requirements stated that the LOD was to be used as a  measure of sensitivity, it was an important measure of the performance of  a given genetic test. LOD provides interpretations of results generated by  a genetic test with the known limitations of such data, but it is a difficult  quantity to estimate reliably. Our review of existing and current  guidelines for validation suggests that using an appropriate estimate of  LOD does provide a reliable measurement of sensitivity, but LOD  estimates, like any performance statistic, should be reported with some  measure of confidence. For example, the Environmental Protection  Agency defines the method detection limit as the “minimum concentration  of substance that can be measured and reported with 99-percent  confidence that the analyte  concentration is  greater than zero.” If test sensitivity is an important performance  criterion, then both verification and validation procedures for a genetic  test should report LOD, along with a measure of confidence. However,  the LODs for the genetic tests the four contractors performed neither  required a confidence measure nor determined one by using statistical  measurements of confidence.
Validation testing should to the extent possible simulate the conditions of  the intended use of a given genetic test, using known case samples (see  tasks under the third phase in the validation framework in table 1).  Calculating uncertainties of measurement is also an important task. All  steps in validation testing, such as sample collection, sample preparation,  transport, storage and analysis can introduce stochasticity and increase  uncertainty in the test results. Most such stochasticities (and others) will  also affect the testing of repository samples. These additional  uncertainties can be measured and understood using repeated (replicate)  experiments including all relevant steps (from collection to analysis) of  samples with known concentration levels. By designing a validation study  with a sufficient number of replicate samples, the FBI could have  quantified the level of statistical confidence in the sensitivity and  specificity of the tests.
While the SWGMGF guidelines did not require them, tests as part of  validation that examined stochastic (random) effects of the process would  have made it possible to draw more rigorous conclusions, with measures  of confidence, regarding the test results for the repository samples. addition, we found that additional information on the sensitivity and  specificity of the PCR-based genetic tests was characterized during  postvalidation testing that the FBI’s expert advisers recommended. Our  analysis of these post-validation test results suggests that the negative  rates of the genetic tests were high for samples that could be expected to  contain the genetic mutations when using the sample collection and  processing methods as required for the repository samples and that there  were stochastic (random) effects in the repository screening process.
The SWGDAM guidelines do suggest such tests for PCR-based methods, which is  important when samples contain low concentrations of the target to be detected. Sampling  fluctuations can occur in PCR-based tests. According to the 2004 SWGDAM guidelines,  for PCR-based assays, validation studies must address stochastic effects and sensitivity  levels. sequence, growth conditions would vary slightly. However, according to  the FBI, the purpose of the additional testing was to determine whether  stochastic sampling error had been introduced into the repository  preparation process as instructed in the subpoena. Therefore, the  postvalidation tests were to determine whether the procedures by which  the repository samples were processed could affect the accuracy of the  interpretation of the data. The postvalidation tests were conducted in  August and September 2007 under conditions that closely mimicked the  intended use for each of the genetic tests. According to the FBI, the  screening of the repository samples with the genetic tests was about  three-fourths complete when this testing took place.
Specifically, in the postvalidation tests, the contractors applied their  genetic tests to replicate samples derived directly from some of the  evidentiary material—including flask RMR-1029. The results revealed that the genetic tests did not always detect the genetic mutations in  samples that had been derived directly from the evidence and thus were  expected to contain all four mutations—a best-case scenario.
Our evaluation of measures of the sensitivity and specificity of the genetic  tests revealed differences between the validation and postvalidation test  results. Regarding sensitivity, under the assumption that undiluted  samples from flask RMR-1029 are positive for all four genetic mutations  (supported by the preponderance of genetic and non-genetic data), we  can estimate the negative rate as the frequency of negative results in  replicate tests of undiluted samples from RMR-1029.
Validation testing showed that for those results expected to be positive,  no negative results were observed at or above the LOD for any of the  genetic tests. However, in the postvalidation testing, the negative rates  were generally high. As shown in table 3, the negative rates for the  postvalidation tests ranged from 0 percent to 43 percent for the undiluted  samples from flask RMR-1029. (Appendix III breaks down the results of  the replicate testing for each genetic test.)
The NAS report stated that the FBI did not address false negative results  and inconclusive results, and it was concerned about the restriction of the  statistical analyses to the repository samples that had no inconclusive or  variant results.results of only D-2 were used in the FBI’s analysis of the repository  screening—that is, the analysis was restricted to the 947 samples that  contained no inconclusive or variant results, which resulted in the  exclusion of 112 samples from the analysis. Thus, the knowledge about  sensitivity and specificity obtained by the replicate testing, as well as  ensuring that these two genetic tests’ interpretation criteria were  complementary, would have been more useful if it had been completed in  the validation process.
Of the two genetic tests that targeted the D mutation, the  Regarding measures of specificity, the effect of the repeated analysis of  the undiluted nonpositive samples during the postvalidation testing  showed evidence of a nonzero false positive rate for the D-2 genetic test.  As shown in table 4, the 3.3 percent false positive rate for the D-2 genetic  test demonstrates the likelihood of a random effect in the postvalidation  tests that was not apparent from the validation results.
Although not a requirement at the time, repeated testing—such as that  conducted postvalidation—would have provided additional information on  the performance of the genetic tests. We recognize that neither the FBI  nor the SWGMGF guidelines required contractors to conduct replicate  tests of case samples to identify the stochastic (or random) effects of the  genetic tests when they were used under realistic test conditions to  further evaluate the genetic tests’ sensitivity—an important step in  validating PCR-based genetic tests. In contrast, the SWGDAM guidelines  suggested using experiments to determine the sensitivity of real-time  PCR-based tests as a part of validation. Importantly, while the LOD is a  critical performance indicator for a genetic test, LOD calculations do not  account for the data that PCR-based tests sometimes generate but that   The FBI also stated that during the development of the  are not typical.genetic tests it was concerned that stochastic effects might be a problem,  stating, for example, that it had discussed its concerns with the  contractors about evidence growth steps and the possible stochastic  effects, that is, in the context of the growth rates of the wild type cells (B.  anthracis Ames) versus the morph cells in culturing, among other things.  The postvalidation tests were able to estimate valuable performance  statistics of the genetic tests and under more realistic testing conditions  than the original validation tests.
More extensive validation testing could have reduced uncertainties in the  testing procedure. For example, the sensitivity of a given genetic test  relies on the sampling procedures, the rarity of the targeted genetic  mutation in a sample, and other factors that vary by genetic test.  Incorporating these types of tests into the validation would have resulted  in more information on the uncertainties inherent in the use of the genetic  tests and would have been a way to simulate the conditions of their  intended use. Future validation efforts would be strengthened by including  experiments designed to identify and eliminate likely uncertainties in test  performance.
The differences we have highlighted regarding the contractors’  approaches to verification and validation indicate that the use of a  comprehensive validation framework could help ensure greater  consistency. Such a framework would need to specify the defined level of  statistical confidence to be calculated for the interpretation of validation  results before they are applied to evidentiary samples. Minimally, the  statistical confidence achievable in each test should be estimated during  validation.
The development of such a framework could be facilitated by DHS’s  National Bioforensics Analysis Center (NBFAC), which validates tests  used to support FBI bioforensic investigations. According to DHS, NBFAC  will take steps to ensure that the results it generates will meet Daubert  standards for “appropriate validation” and third party review and will thus  meet admissibility requirements for evidence in federal court  proceedings. NBFAC—an ISO 17025 accredited forensic laboratory—is  experienced in working with multiple outside laboratories to verify and  validate their methods. It has an established ISO 17025 accredited  process.
The combination of limited communication among the contractors, varied  timing in the validation efforts, uncertainties the FBI faced as the  investigation unfolded, and increasing knowledge about the repository  samples made it clear, with hindsight, that the contractors’ verification and  validation approaches were likely to differ. Thus, in the future,  standardizing the approach to verification and validation testing—by the  means of a validation framework—would be more efficient, especially in  clearly communicating expectations to multiple contractors.
section_title: DHS Validates FBI’s Microbial Forensics Methods That Can Support FBI Investigations
paragraphs: In contrast to 2001, DHS’s NBFAC validates assays (or tests) that can be  used to support FBI bioforensic attribution investigations. Generally, the  NBFAC validation process involves the evaluation of methods transferred  from others, such as DOD and academic laboratories, and sometimes the  development of a new method. For forensic tests, NBFAC and the FBI  are provided with a “validation package” for each test that encompasses  data on testing previously conducted during the development stage or  before the transfer to the laboratory.have to provide information on the performance parameters (e.g.,  accuracy, LOD, precision) that they have previously verified. Next,  NBFAC conducts its own test, evaluation, and validation of the transferred  method. When evidence stemming from the use of validated methods is  needed as evidence in court, it must be defensible by meeting evidentiary  standards.
According to DHS, developers  Questions may be raised in court about the standards used for the  validation of such methods. Results generated by forensic methods,  including microbial forensics, must meet a high standard. According to  NBFAC, to ensure that results generated by a validated test will meet  Daubert standards for “appropriate validation,” the deliverables from the  Bioforensics R&D Program include SOPs for the methodologies and  technical and peer-reviewed published reports. Also, quality project  performance plans are required of researchers, who must define method  performance parameters to provide a baseline for verification and further  validation if required by law enforcement. accreditation of tests requires the demonstration of previously described  method parameters in NBFAC laboratories with trained staff followed by a  third-party review of the supporting data, procedures, equipment, and  staff training that supports ISO 17025 accreditation.
In this context, a method that has been validated elsewhere and that is transferred  would be evaluated to ensure that the NBFAC successfully used the method as intended  in the NBFAC laboratory. Performance parameters include accuracy, precision, specificity,  selectivity, LOD, limit of quantitation, linearity, ruggedness, and robustness.
section_title: Characteristics of a Statistical Framework That Would Strengthen the Significance of Microbial Forensic Evidence in Future Investigations
paragraphs: We identified six characteristics of a statistical framework that would  strengthen the significance of microbial forensic evidence. When we  compared the FBI’s statistical approach to these six characteristics, we  found that three could be improved to strengthen the significance of its  evidence for future investigations. That is, the FBI (1) could do more to  understand the methods and conditions that give rise to the chosen  genetic markers, (2) institute more rigorous controls over sample  identification and collection, and (3) include measures of uncertainty  when interpreting the results. We found that the FBI has taken some  steps to include such expertise in future investigations by building formal  forensic statistical expertise both internally and externally.
subsections: section_title: The Six Characteristics of a Statistical Framework
paragraphs: Although not always possible, an important goal of a microbial forensic  investigation is to generate meaningful comparative analyses of  evidentiary samples and suspect samples to establish their relatedness or  to exclude suspect samples from an investigation. Statistically meaningful  comparative analyses can allow the use of statistical inferences relating  to the process to produce the sample, the provenance of a sample, or the  relatedness of samples. The significance of such statistical inference  relies on the analyst’s ability to quantify both the confidence in test results  and the frequency with which results match. Confidence, in this context,  refers to the level of reliability and accuracy investigators assign to the  test results obtained from the measurement tools used to identify the  properties of interest in the samples. The frequency of the sample  properties’ presence, or generation in a relevant population of possible  sources, is a measure of how common or rare the properties are and  provides context to the probative value of the evidence. According to a  2009 NRC report, a statistical framework is needed to quantify the  probative value of forensic evidence in terms of the frequency of that  evidence in a population.
Formulating an appropriate statistical framework that is adequate for all  microbial forensic investigations is not feasible because the diversity of  many potential pathogens is unknown or, at best, difficult to describe. For  this reason, frameworks must be adapted to the specific circumstances of  each case. As shown in table 5, our review of scientific literature in  forensic science, statistics, epidemiology, and population genetics  identified the six general characteristics that a framework needs for  statistically meaningful comparative analyses of the attack material to  repository samples for the specific set of circumstances of the FBI’s  investigation.
First, a definition of what constitutes a matching type should be clearly  established. A genetic signature, or a set of genetic markers, can be  chosen to establish a genetic type (or genotype) that is used to  differentiate the samples. The genetic signature should be sufficient to  identify the target of interest at the resolution needed for an investigation.  In this case, the target of interest was the B. anthracis Ames strain,  capable of producing spores with a set of specific genetic markers linked  to morphs observed after a prolonged period of growth. The requisite  resolution was the ability to differentiate among the individual stocks (or  collections of organisms) of B. anthracis. Determining that two or more  samples have a matching type must take into account the source of the  organisms (for example, nature or the laboratory), the stability of genetic  markers, storage conditions, and conditions giving rise to the markers.  Specific growth or environmental conditions may selectively advantage or  disadvantage mutations and affect the stability of genetic markers.  Therefore, if the significance of a matching genetic signature is to be  understood, the genetic markers should be well characterized, and the  conditions giving rise to the presence of markers in a sample should be  understood.
Second, once the genetic signature has been established and a match  has been clearly defined, it is then necessary to identify and define the  population of relevant sources that may have the genetic signature in  order to understand how common or rare the genetic signature is. This  relevant source population is critical in identifying the probative value of  any match or nonmatch between samples. In a criminal investigation, a  relevant source population may be considered the population of suspects,  and it should be defined as specifically as possible to identify the smallest  population related to the evidentiary material. The definition of the  relevant source population should be based on the population related to  characteristics of the evidence and not on characteristics of a suspected  source. The relevant source population in this case is all stocks of B.  anthracis that could have been used to grow the material used in the  attack letters.
In defining the source population, the structure of the relevant source  population of bacteria should be understood. When a population is  divided into subgroups that do not mix freely, that population is said to  have structure. In this case, the relevant reference population of stocks of  B. anthracis was highly structured among the laboratories included in the  investigation. The lack of independence between stocks in a structured  population affects inferences about the evidentiary material and its most  and least likely sources.
Third, in order to quantify or estimate how common or rare a genetic  signature is in the relevant source population, a database that accurately  represents the relevant source population’s genetics should be created.  The extent to which the database reflects the population will affect the  accuracy of the match probability. The size and quality of the data in the  database will affect the power of match probability, determining the  potential probative power of the signature for distinguishing one source  from another. A large and comprehensive database is the theoretical goal  but in most cases may not be possible. However, in this case, the FBI  determined that it was possible to identify all sources of the B. anthracis  Ames strain, and it set out to create a comprehensive database. For  completeness the genetic information in the database should have  included samples from all sources of the B. anthracis Ames strain. In  such cases, the database should be complete—excluding sources results  in underrepresentation—and should avoid duplication (although  replication can be beneficial)—unknowingly including sources more than  once results in overrepresentation. Methods used to select samples from  each stock should be adequate to ensure representation of the organisms  within each stock. In an ideal situation, the database of genetic  information should be constructed to the same quality standards as the  actual evidentiary analysis. These quality standards should apply to the  selection of samples from stocks to the results of the genetic tests.
Fourth, the limitations of the measurement tools used to generate the  genetic information in the database should be identified. When  quantitative inference is attempted, care must be taken not to  overemphasize data; the limits of the methods used to generate the data  should be considered. The power and limitations of microbial forensics  methods need to be understood through validation. Validation frequently  connotes confidence, but it may be thought of as defining the limitations  of the method. This does not mean that a method must be 100 percent  accurate to be useful. Studies should allow the estimation of the limits of  the measurements. The limits of the methods must be demonstrated and  documented for all steps in the process, including sample collection,  preservation, extraction, analytical characterization, and data  interpretation.
Fifth, the choice of statistical methods should be appropriate for the data  and should properly account for the mode of inheritance of the genetic  markers and any structure in the populations. An important aspect of  computing association statistics and probability estimates is properly  accounting for the mode of inheritance. Methods appropriate for  computing probability estimates and statistical tests of significance differ  by the mode of inheritance of the genetic markers. In organisms that  reproduce asexually, such as B. anthracis, genetic diversity is driven by  mutation processes, not by random mixing. Computing match  probabilities using methods that assume independence and random  mixing within populations is not appropriate because the genetic variation  in such organisms is highly correlated. In organisms that reproduce  asexually, the frequency of a particular genetic type in the population  must be determined by direct observation. The frequency of the  evidentiary genotype in a relevant source population can be based on  counting the number of times the genotype is observed in a reference  database. The strength of this approach is affected greatly by the genetic  database and whether it has sufficiently sampled relevant populations.
Sixth and finally, the interpretation of results should include  quantifications of uncertainty. It is crucial to clarify the type of question the  analysis is addressing when evaluating the accuracy of a forensic  analysis. Although some techniques may be too imprecise to permit the  accurate identification of a specific individual, they may still provide useful  and accurate information about questions and classification. The  interpretation of results will be stronger with the proper use of statistical  and probabilistic analyses, but the strengths and weakness of any result  should be communicated. Results should indicate the uncertainty in the  measurements, and studies must be conducted that enable the  estimation of those values.
section_title: The FBI’s Statistical Approach Could Have Been Improved for Three of the Six Characteristics of a Statistical Framework
paragraphs: We believe that the six general characteristics described above make up  a comprehensive statistical framework that could have allowed the FBI to  quantify significance and probative value of the scientific evidence  collected in a statistically meaningful way and could have strengthened  the evidence it collected. However, we found that at the outset of its  investigation, the FBI did not have a comprehensive framework that  would allow for statistically meaningful comparative analyses between  samples from the attack letters and samples in the FBI repository of B.  anthracis Ames strain. Specifically, we found that the FBI’s approach to  three of the six characteristics could be improved to strengthen the  significance of evidence in future investigations.
subsections: section_title: The FBI’s Research Did Not Provide Full Understanding of the Methods and Environmental Conditions That Give Rise to Genetic Mutations
paragraphs: Although the specific genetic mutations used as genetic markers to  determine a match or exclusion were adequately characterized, the FBI  did not conduct studies to understand the methods and environmental  conditions that gave rise to the mutations. The FBI convened a team of  scientists in 2007 to review the scientific methods. Finding no shortfalls or  deficiencies in the basic methodologies they reviewed, they determined  that the usefulness of the genetic markers was sufficient. The team also  stated that the extent of research and development of the genetic tests at  the date of their review was insufficient to determine whether the  presence or absence of one or several of the genetic markers was  associated with the evidence, was merely characteristic of normal culture  practices, or possibly was affected by the sensitivity of detections of the  genetic tests. The team recommended additional studies to characterize  the genetic markers as a function of growth conditions, including the  influence of growth time, growth media, and temperature.
In response to questions from the NAS panel about this recommendation,  the FBI stated that it considered such studies academic and did not  conduct the recommended research. Consequently, experimental data  are missing that would have shown the frequency with which particular  genetic mutations occur under growth conditions that could affect their  retention or loss. In its report, NAS opined that some of the morphs used  as genetic markers might have arisen independently from RMR-1029.
According to the report and experts we spoke with, the genetic markers  might have had a selective advantage under growth conditions used for  large-scale production of spores, such as in a fermenter or in a batch  culture. If so, the presence of the genetic markers would be a function of  the growth conditions rather than direct derivation from parent material,  such as RMR-1029. This is problematic for the quantification of the rarity  of the results because it is not possible to calculate the probability of two  independent cultures having the same genetic markers if either was  subjected to growth conditions that provide selective advantage or  disadvantage.
Without the experimental data, the usefulness of the genetic markers as  an identifying signature to determine a match or exclusion was not fully  understood. For example, it is not known whether the genetic markers  could have arisen independently. To identify repository samples that  received a direct or indirect transfer from the laboratory that possessed  RMR-1029 after it was created in 1997, we examined the FBI’s  documentation of historic transfer records of B. anthracis Ames strain  between laboratories from 1981 through 2001. We supplemented this  with information from laboratory officials and researchers we interviewed.  Then, we compared the frequency of positive genetic markers in these  groups of samples to the 119 samples that we verified were independent  of transfers from the laboratory that possessed RMR-1029.
Our analysis of repository data found no evidence of independent  evolution in three of the four genetic markers (A1, A3, and E). However,  we found that repository samples with no direct or indirect relationship to  RMR-1029 tested positive for the D genetic marker at rates similar to  those of the samples that were submitted from laboratories with direct  transfers from the laboratory that possessed RMR-1029. table 6, the D genetic marker was detected in about 6.6 percent of the  repository samples submitted from laboratories with direct transfers from  the laboratory that possessed RMR-1029 compared to 6.7 percent of the  samples that were independent of the laboratory that possessed RMR- 1029.
Additionally, the NAS report found that in repository samples associated  with experiments conducted before the 2001 attacks, the D genetic  marker was the only marker detected and it occurred in about 1 percent  (3 of 296) of those samples. This provides additional evidence that the D  genetic marker may have arisen independently of RMR-1029. Additional  studies recommended to the FBI that it did not conduct could have  provided the experimental data needed to fully understand the probative  value of this genetic marker.
section_title: The FBI Established an Adequate Relevant Source Population
paragraphs: Because the FBI adequately identified the relevant source population as  all stocks of B. anthracis Ames strain, it significantly reduced the number  of possible sources. The NAS report found that the dominant organism in  the letters was correctly and efficiently identified as the Ames strain of B.  anthracis. The science performed on behalf of the FBI for identifying the  Bacillus species and B. anthracis strain was appropriate, was properly  executed, and reflected the contemporary state of the art. The correct  identification of the specific strain of B. anthracis allowed the FBI to  adequately define the relevant source population as stocks of the Ames  strain in laboratories that had the Ames strain in their inventories before  the attacks. This significantly reduced the number of possible sources.
section_title: The FBI Did Not Fully Ensure the Completeness and Accuracy of the Repository
paragraphs: We found that the FBI’s effort to create a comprehensive repository  containing samples from all known stocks of the Ames strain of B.  anthracis was appropriate for assessing the rarity of the genetic markers  in the relevant source population. Its adequacy, however, was affected by  the incompleteness and inaccuracy in the repository. The NAS report  found that the repository was not optimal for a variety of reasons. It  stated, for example, that the instructions in the subpoena issued to  laboratories for preparing samples were not precise enough to ensure  that they would follow a consistent procedure for producing samples that  would be most suitable for later comparisons.
Our analysis of FBI documents shows that FBI searches at three specific  laboratories identified hundreds of additional relevant stocks that  laboratories did not submit to the repository in response to the subpoena.  Specifically, we found that the FBI collected about 29 percent of the 1,059  repository samples through these searches. The proportions of samples  thus obtained were 34 percent, 96 percent, and 22 percent in these  laboratories (see table 7).
We were unable to determine how two of the three laboratories identified  and selected samples from relevant stocks in response to the subpoena,  but we found that individuals at one laboratory differed in interpreting the  subpoena’s instructions. Laboratory officials acknowledged differences  in interpreting the instructions on how to identify distinct Ames strains of  B. anthracis. Identifying the specific stocks to submit in response to the  subpoena at that laboratory was left up to the principal investigator  because, at that time, no one else actually working with the stocks would  have understood what was in them.
FBI officials acknowledged that the interpretation of the instructions to  determine what strains to submit to the repository varied across  laboratories, stating that the subpoena was not as precise as it needed to  be. However, they emphasized that every laboratory that submitted  samples to the repository was investigated thoroughly and that, when the  FBI conducted searches at the three laboratories, those investigations  eliminated many laboratories from being suspects. Furthermore, FBI  officials told us that the decision to conduct searches at these three  laboratories was an investigative decision, not a scientific one.
The NAS report also raised concerns that the decision to remove samples  with inconclusive or variant results contributed to the lack of  completeness of the repository data. The report stated that a major  concern was the restriction of its statistical analyses to the 947 samples  that contained no inconclusive or variant results. Notably, the report  showed that 4 of the 112 samples that were disregarded for having a  single inconclusive or variant result scored positive for the three  remaining genetic tests.
In addition, our analysis of FBI documents shows that FBI searches  contributed to inaccuracies in the repository by collecting samples from  stocks that had already been submitted to the repository. We identified 14  duplicate samples from a search conducted at one laboratory in April  2004.
FBI officials stated that they were not concerned about duplicate samples  in the repository because duplicate samples may have served other  important investigative purposes such as verifying if two samples were  related or answering other important questions related to investigative  information. They also stated that additional information collected about  the samples would allow them to reconcile duplicates. However, our  analysis of the FBI repository data indicates that known duplicates were  not removed from the repository before the statistical analysis.
As a result of these examples of incompleteness and inaccuracies in the  repository, a statistically meaningful extrapolation of the statistics and  frequencies derived from the repository to the relevant source population  was not possible. By instituting more rigorous controls over sample  identification and collection for future investigations, the FBI can improve  the completeness and accuracy of a repository.
section_title: The Statistical Analyses Did Not Account for the Genetic Markers’ Mode of Inheritance
paragraphs: The results from statistical analyses conducted in 2008 did not  adequately account for the mode of inheritance of the genetic markers,  and they added little probative value to the investigation. Many of the  methods used for the 2008 statistical analyses inappropriately relied on  the assumption of independence among the repository samples. For  example, the NAS report stated that because the repository samples  were not independent, the proportion of samples testing positive for all  four genetic markers was not a meaningful estimate of the probability of  occurrence. The FBI did not use the results of the statistical analyses and  did not quantify the confidence it had in, or the probative value of, the  repository results in its conclusions included in its final investigative  summary. An FBI official stated that the statistical analyses were viewed  from an academic standpoint and were not part of the investigation. That  official also stated that the results of the statistical analyses did not  contradict the conclusions of the investigation.
section_title: The FBI Did Not Include Uncertainty Measures in the Interpretation of Results
paragraphs: In its final investigative summary, the FBI concluded that only 8 of more  than 1,000 samples tested positive for all four genetic markers, but it did  not provide any measure of the confidence it had in this conclusion.
We found that the genetic tests show variability in the results on samples  selected from the same stock. As we previously indicated in our  assessment of the validation of the genetic tests, the additional  postvalidation tests conducted in 2007 demonstrated variability in the  results of the genetic tests when they were applied to samples under  conditions intended to mimic their use on repository samples.
Additionally, the two genetic tests for the D marker did not always give the  same result for the same sample. An analysis included in the FBI  contractor’s Statistical Analysis Report identified 24 repository samples  for which the two genetic tests yielded opposite results from the same  sample. The NAS report stated that this lack of agreement between the  two genetic tests for the D mutation illustrated the differing sensitivities  and specificities of the tests. This lack of agreement was also evident in  the eight samples that tested positive for all four genetic markers. As  shown in figure 4, our analysis of the repository data demonstrated that  one of these eight samples also tested negative using the other genetic  test for the D marker.
Further, our analysis of duplicate samples in the repository showed  differences in the results of genetic tests on samples selected from the  same stock. As shown in figure 5, only 3 of the 14 duplicate samples we  identified showed the same results across the five genetic tests. For  example, FBI repository sample number 049-004 tested positive for all  five genetic tests while a duplicate sample selected from the same stock  (066-044) tested positive for only four of the five genetic tests. In another  example, FBI repository sample number 049-016 tested positive for all  five genetic tests while the duplicate sample (047-002) tested negative for  all five genetic tests.
FBI officials stated that these results may have differed for a number of  reasons, including uncertainty from the sampling process (sampling error)  and uncertainty from the genetic test itself (stochastic error). Each step in  the process the FBI used to collect, prepare, and test repository samples  could have added uncertainty to the results of the genetic test.
As noted previously, before its searches, the FBI relied on laboratory  officials to identify and select subsamples of distinct Ames strains for  submission to the repository. The NAS report stated that the subpoena’s  instructions to laboratories for preparing samples were not precise  enough to ensure consistent procedures for producing samples that  would be most suitable for later comparisons. For example, the subpoena  instructed laboratories to select a representative sample from each stock  but did not provide guidance on how many cells or colonies to select.  Although steps were taken in the genetic tests to standardize the number  of cells being tested, the number of initial cells or colonies selected from  each stock would have affected the probability of selecting material  capable of producing the genetic markers.
This is particularly important because the mutations chosen as genetic  markers were infrequent in the evidentiary material. For example, we  interviewed the scientist who submitted the duplicate samples we  identified above as having opposite results (all five negative versus all five  positive). He told us that, in the presence of an FBI investigator, he had  not followed the subpoena instructions when he selected the sample  (047-002) that tested negative for all five genetic markers.
In addition to the selection methods we have discussed in this report, the  methods used to prepare and test the repository samples could have  introduced uncertainty to the results of the genetic tests. The NAS report  stated that replication could have been used in the design of the FBI  repository to provide measures of the uncertainty of the genetic tests.  Although laboratories were required to submit to the repository two  samples from each stock, only one of those samples was tested for the  genetic signature. Without replication, the FBI was unable to assess  uncertainty in the results of the genetic tests in the context of testing  actual repository samples.
Because the FBI did not include measures of uncertainty when presenting  the results of the genetic testing, questions have been raised about  samples that tested positive for three or fewer genetic markers. For  example, NAS stated that the FBI did not address false negative results  and raised concern regarding the restriction of the statistical analyses to  the repository samples that contained no inconclusive or variant results.  NAS further highlighted 21 samples that contained an inconclusive or  variant result and tested positive for 1, 2, or 3 genetic markers.
To illustrate the potential effect this uncertainty could have had on the  interpretation of the results, we conducted an analysis using the  estimates of false negative rates obtained from the additional replicate  testing, combined with a sensitivity analysis accounting for the decision to  restrict the statistical analyses to the 947 samples that contained no  inconclusive or variant results. We computed a range of probabilities,  given the observed results of the genetic testing, that each repository  sample was selected from a stock that could have produced all four  genetic markers. We found an additional 16 repository samples with  probabilities that exceeded a 1 percent chance of being selected from a  stock that contained all four genetic markers. We determined that 15 of  these 16 additional samples were selected from stocks held at the same  two laboratories that were the source of one or more of the 8 samples  that tested positive for all four genetic markers.
The remaining sample identified in our analysis was a sample that we had  determined was independent from RMR-1029 and tested positive for the  D marker. In addition, this sample was inconclusive for both the A1 and  A3 markers and negative for the E marker. We computed a 0 to 19  percent range of probabilities for this sample, the maximum occurring  when the model made the assumption that both inconclusive results for  A1 and A3 markers were positive.
Additionally, the results of the genetic tests for this sample further  highlight the importance of including measures of uncertainty. According  to the transfer inventory records we reviewed and the laboratory official  we interviewed, this sample was selected from a stock that was one of  four copies of the same material. As shown in figure 6, the repository  samples selected from the remaining three copies tested negative for all  five genetic markers. This demonstrates that the genetic tests could have  yielded different results for samples selected from the same material and,  as the NAS stated, replication could have been used to provide measures  of the uncertainty induced by these varying results.
section_title: The FBI Is Addressing the Need for Formal Statistical Expertise In Future Investigations
paragraphs: The FBI has taken steps to include statistical expertise in future  investigations. The NAS report stated that the FBI appeared not to have  sought formal statistical expertise early in this investigation and that  similar investigations would benefit from including statistical expertise in  their design and implementation. It noted that because many inferences  depend on the design and analysis of complex data, the FBI should  consult with expert statisticians throughout experimental design and  planning, sample collection, sample analysis, and data interpretation.  Further, the 2009 NRC report on strengthening forensic science in the  United States highlighted the importance of statistical and quantitative  proficiency for improving forensic science methods.
An FBI official told us that since the 2009 NRC report, the FBI has been  building formal forensic statistical expertise both internally and externally.
For example, he said that the FBI laboratory division had created an  internal statistical working group to examine the FBI’s statistical needs in  its forensic methods. The group included a professor of statistics visiting  for 6 months to examine the statistical questions related to patterns, such  as fingerprints, and also other science, such as chemistry and explosives.  Additionally, the FBI has established a working relationship with members  of the American Statistical Association’s Ad-Hoc Advisory Committee on  Forensic Science in order to discuss its statistical capacity. The FBI has  also worked with other agencies to identify areas of statistical research  needed for future investigations.
section_title: Scientific Gaps Remain Related to Verification, Validation, and Statistical Analyses and Research Is Ongoing
paragraphs: After the 2001 attack, the FBI did not conduct a lessons learned study but  considers the NAS report to be one. The NAS report identified some  scientific gaps related to the development of genetic tests and statistical  analyses. In addition, we identified a key scientific gap that is related to  the verification and validation of the genetic tests and the statistical  analyses—that is, the significance of using genetic mutations in B.  anthracis as genetic markers for analyzing evidentiary samples. DHS has  funded some research on this gap but this research is not yet complete,  and it is not yet known whether it will fully address the gap.
subsections: section_title: The NAS Report Identified Scientific and Technical Gaps
paragraphs: The FBI has not conducted a formal lessons-learned study of the  scientific and technical methods it used in the investigation and thus has  not specifically identified any scientific gaps in research related to the  validation of genetic tests and statistical approaches. An FBI official  stated that such a study was not needed because the 2001 incident was  unique and the case is closed. This FBI official also told us that he  considered the NAS report to be the lessons-learned study because it  had identified several scientific gaps. For example, the NAS report  indicated that the investigation lacked  1.  a method for interpreting the genetic similarity between the attack  spores in the letters and those in RMR 1029; and  2.  an experimental design that included statistical input in the early  stages of the investigation.
Nevertheless, the FBI does not necessarily agree with the scientific gaps  that NAS highlighted in that report. However, the FBI stated in 2010 that  the active dynamics of the microbial genome for any given species need  to be understood—for example, the location on the genome of “hot spots”  for mutation and diversity and whether there is a high rate of genetic  mobility and change in any given species. Further, in September 2014,  according to an FBI official, technology has changed since the  investigation, and in the future genome sequencing will be used to  analyze evidence samples.
section_title: A Key Gap Remains on the Significance of Using Genetic Mutations as Markers In Analyzing Evidentiary Samples
paragraphs: In addition to the gaps identified in the NAS report, we identified a key  scientific gap that has not been fully addressed. This gap is related to the  significance of using genetic mutations as genetic markers for analyzing  evidentiary samples to determine their origins. Recognized by NAS, this  issue is associated with the gaps it identified. With respect to verification  and validation, the genetic tests targeted specific DNA sequences of  certain genetic mutations in their screening of the repository samples.  The FBI used the results of the analysis of the repository screening by  those tests to narrow the source of the attack spores. However, during  the investigation, it was not known how stable genetic mutations were in a  microbial genome or how significant they were as genetic markers.
We found that conditions causing the rise of the genetic mutations in the  evidence were not known before or after validation or during the  subsequent statistical analysis of the results of the repository screening.  During the investigation, it was not known what conditions would have  promoted or inhibited the presence of the genetic mutations at detectable  levels. Such knowledge would have indicated whether they were  associated with the evidence itself or with the culture practices normally  used in a laboratory. Although FBI expert advisers recommended  experiments, none were conducted at that time to attempt to obtain this  information. Such experiments could have helped in understanding the  evolution of these particular genetic mutations.
section_title: DHS-Funded Research on the Evolution of Morphological Variants is Ongoing
paragraphs: DHS has recognized the need for a methodology to determine how a  material has been grown and produced and for obtaining information on  the biology of agents, including their mutation rates and genome  “hotspots” for mutation, so that their “relatedness” can be measured. In  this context, an expert who reviewed this report stated that computational  methods are also needed to reconstruct (or assemble) genome  sequencing data so that the relationship between markers that are not  independent, as is common in asexually reproducing bacterial genomes,  can be inferred.
As a result, DHS has funded research that is intended to provide a better  understanding of how morphological variants, or mutations, could emerge  and evolve in bacterial genomes. Some of the technologies involved in  DHS’s research, such as whole genome sequencing, are still evolving.  DHS-funded research includes studies of the population genetics of  bacterial agents, including B. anthracis, at Northern Arizona University   This research involves studies of diversity that include mutations  (NAU).among these agents. DHS’s NBFAC is also studying genome sequencing  methods. The purpose of these studies is to develop the capability to  perform a metagenomic analysis of an entire sample using a hybrid- assembly. According to DHS, the field of “metagenomics,” is broad but  unified by its focus on a community of genomes rather than individual  isolates. Such research is a step in the right direction, since the FBI has  indicated that it is likely to use genome sequencing methods in future  investigations to analyze evidence. However, since this research is  ongoing it is not clear when it will close the gap or whether it can do so  alone.
section_title: Conclusions
paragraphs: Although we identified several aspects of the FBI’s scientific methods we  reviewed that could be improved in a future investigation, we recognize  that in 2001, the FBI was faced with an unprecedented case. Determining  the source of the spores in the envelopes was complicated by many  factors, including the uncertain provenance of samples in the FBI  repository, an unknown mutation rate for B. anthracis under laboratory  growth conditions, and the performance of the genetic tests under “real- world” conditions.
The genetic tests were generally verified, validated and demonstrated  through the validation testing that they met the FBI’s acceptance criteria,  but the lack of a comprehensive approach—that is, a validation  framework—allowed for differences in the contractors’ approaches.  Further, the results of the postvalidation testing raise questions about  whether additional information could have been obtained during  verification and validation and, thus, whether the validation testing could  have been more rigorous. The use of a standardized approach to  verification and validation from the beginning could have more definitively  established the performance of all the genetic tests. It could have helped  in communicating expectations clearly, ensuring confidence in results  generated by any genetic tests developed.
DHS could be instrumental in developing a validation framework and  future efforts using a framework could help achieve minimum  performance standards during verification and validation, particularly  under multiple contracts. Also, incorporating statistical analyses in the  framework would allow the calculation of statistical confidence for  interpreting the validation testing results.
The FBI’s statistical approach to its study design and plan, sample  collection and analysis, and interpretation of data and scientific evidence  lacked several important characteristics that could have strengthened the  significance of that evidence. Although the complexity and novelty of the  scientific methods at the time of the FBI’s investigation made it  challenging for the FBI to adequately address all these problems, the  agency could have improved its approach by including formal statistical  expertise early in the investigation and establishing a statistical  framework that could identify and account for many of the problems. In  future investigations, statistical expertise early in the investigation will  help identify the importance and role of fully understanding the (1)  evolution of the genetic markers, (2) sources of dependence between  samples, and (3) uncertainty in the measurement tools used to identify a  genetic signature. This expertise could influence an investigation’s  methods and strengthen the significance of scientific evidence.
A key scientific gap—how stable genetic mutations are in a microbial  genome and thus their suitability as genetic markers—remains an issue.  Lack of this knowledge has implications for both the development of  genetic tests, or other investigative approaches and technologies, and the  analysis of the results they generate. For example, how likely it is that the  same genetic mutations will arise independently in separate cultures is  currently unknown, and so is whether different culture conditions can  change the ratio of the mutations significantly enough to provide a  negative rather than a positive result. DHS-funded research into the  evolutionary behavior of variants in the genome of B. anthracis and other  microbial agents and the use of genome sequencing is a step in the right  direction because the FBI is planning to use sequencing in future  investigations to analyze all the material in evidence samples. However,  in determining the significance of using mutations as genetic markers, an  understanding is still needed about the stability of genetic mutations.  DHS’s ongoing research is likely to take several years and some of the  technologies it entails, such as whole genome sequencing, are still  evolving. Therefore, it is not clear when and whether this research alone  will address this gap.
section_title: Recommendations
paragraphs: To ensure that a structured approach guides the validation of the FBI’s  future microbial forensic tests, we recommend that the Director of the  Federal Bureau of Investigation work with the Secretary of Homeland  Security to develop a verification and validation framework. The  framework should be applied at the outset of an investigation involving an  intentional release of B. anthracis, or any other microbial pathogen. It  should (1) incorporate specific statistical analyses allowing the calculation  of statistical confidence for interpreting the results and specifying the  need for any additional testing to fully explore uncertainties relative to the  type of genetic test being validated and (2) applied and adapted to a  specific scenario and employs multiple contractors.
In addition, we recommend that the Director of the FBI establish a general  statistical framework that would require input from statistical experts  throughout design and planning, sample collection, sample processing,  sample analysis, and data interpretation that can applied and adapted to  address a specific scenario involving an intentional release of B. anthracis  or any other microbial pathogen.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the FBI and DHS for review and  comment. The FBI provided written comments, which are reprinted in  appendix IV. In its comments, the FBI agreed with our recommendations  and stated that it had taken significant steps toward addressing them.  In  addition, the FBI provided technical comments that we have addressed in  the body of our report as appropriate. DHS stated that it had no  comments on the draft report.
With respect to the first recommendation, the FBI stated that “NBFAC  programs have developed analytical capabilities in microbial forensics for  numerous biological agents” in “support of investigations of the use or  suspected use of biological weapons.” It stated that “these assays are  validated and accredited under international standards (ISO17025) . . . .”  According to the FBI, these capabilities, and those still being developed,  “address part 2” of our recommendation  “…applied and adapted to a  specific scenario…” in as much as they represent capabilities addressing  numerous biological agents and toxins. Further, the FBI stated that the  NBFAC is pursuing the most current techniques of microbial genetic  analyses and that some of these may soon be accredited.
The FBI added that it actively participates in the National Strategy for  Countering Biological Threats, under which the agency has helped in  “Establishing a National level research and development strategy and  investment plan for advancing the field of microbial forensics.” Further, it  stated that it is helping to maintain “the National Biological Forensics  Analysis Center (NBFAC) as the Nation’s lead Federal facility for forensic  analysis of biological material in support of law enforcement  investigations,” which advances the field of microbial forensics through  scientific workshops sponsored by the FBI.  According to the FBI, such  workshops have included work on interpreting microbial genetic data  acquired by next generation sequencing platforms. The FBI stated that  this work has included “statistical analyses of the confidence in base  calling” using these platforms and “bioinformatic software.”   We  recognize the importance of the FBI’s active participation in microbial  forensic research and scientific workshops that address key issues  related to the performance of emerging microbial forensic tests. We also  recognize that establishing the error rates of genome sequencing  platforms, which the FBI stated it may use in future investigations, would  be an important step in verification and validation. Further, as we state in  this report, developing a framework for verification and validation when  employing multiple contractors in the same investigation could help  standardize the process with minimum performance standards. Thus, we  believe that the FBI’s continued work with DHS could help ensure the  development of such a framework and improve its approaches to future  investigations. A written plan could assist in the development of the  framework.
With respect to the second recommendation, the FBI stated that scientists  from the FBI and NBFAC participate in the Food and Drug  Administration’s related efforts, the “Global Microbial Identifier”  symposiums, “whose activities include statistical analyses for interpreting  microbial genetic data in investigations of food-borne illness.” We  recognize the importance of the FBI’s continued participation in research  on the statistical interpretation of microbial genetic data. The evidence we  present in this report suggests that if statistical expertise had been  included early in the FBI’s investigation, it could have improved the  significance of the collected microbial forensic evidence. By establishing a  general statistical framework, the FBI will be able to provide some  assurance that input from statistical experts will be included in future  investigations so that they will benefit from statistical expertise.  Developing such a framework could also be facilitated by a written plan.  We believe that the actions that the FBI states it has taken are a step in  the right direction toward addressing our two recommendations.
We are sending copies of the report to the FBI and DHS, appropriate  congressional committees, and other interested parties. The report is also  available at no charge on the GAO website at www.gao.gov.
If you or your staff have any questions about this report, please contact  Timothy M. Persons, Ph.D. at (202) 512-6412 or personst@gao.gov.  Contact points for our Office of Congressional Relations and Office of  Public Affairs appear on the last page of this report. Key contributors to  the report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The scope of our work was limited to a review of the scientific methods  employed to validate the genetic tests used to screen the FBI’s repository  of Ames B. anthracis samples, the procedures used to identify and collect  samples of Ames B. anthracis in the creation of the FBI’s repository, and  the statistical analyses and interpretation of the results of the genetic  tests. We did not address any other scientific methods or any of the  traditional investigative techniques used to support the FBI’s conclusions  in this case, and we take no position on the FBI’s conclusions when it  closed its investigation in 2010.
Our objective for this performance audit was to answer the following  questions:  1.  To what extent were the genetic assays used to screen the FBI  repository of Ames samples scientifically verified and validated?  2.  What are the characteristics of an adequate statistical approach for  analyzing the repository samples and to what extent was the  statistical approach used adequate? If not adequate, how could this  approach be improved for future efforts?  3.  What remaining scientific concerns and uncertainties, if any, regarding  the validation of genetic assays and statistical approaches will need to  be addressed in future analyses? What additional research, if any,  would be helpful in resolving such scientific uncertainties in any future  investigation?
To determine the extent to which the genetic tests were verified and  validated, we collected and reviewed data regarding (1) the FBI’s  requirements for validation, (2) documentation from the FBI’s contractors  on their verification and validation testing, and (3) documentation from the  FBI on the contractors’ efforts to develop their genetic tests as well as  results from the validation testing. We also reviewed related scientific  literature and agency and industry standards and guidelines regarding the  verification and validation of analytical methods, including real-time PCR- based tests for detecting B. anthracis, among others. We developed  criteria for assessing the extent of the validation. We used references  from agency standards, reports, and guidelines for validation and from  scientific literature to identify the essential phases in an approach, or  framework, for developing genetic tests. We compared what the FBI and  its contractors had done to verify and validate the genetic tests against  these phases and tasks.
Specifically, we reviewed the FBI’s and its contractors’ laboratory  documentation to determine for each genetic test (1) the steps each took  to verify the genetic tests’ performance and conduct the FBI-administered  validation, (2) whether the validation test results met the FBI’s acceptance  criteria and minimum requirements, and (3) whether the FBI’s  postvalidation testing of the genetic tests on the flask RMR-1029 provided  further insights into the sensitivity and specificity of the genetic tests  beyond those obtained by the validation. We also determined whether the  processes the contractors’ laboratories followed for verifying and  validating their genetic tests were consistent. Finally, we reviewed the  NAS report’s observations on the performance of the genetic tests in  screening the FBI’s repository samples. We interviewed officials and  scientists at the FBI contractors, the FBI, and elsewhere on how the  genetic tests had been verified and validated, standards or guidelines had  been applied, and the FBI’s rationale for its requirements and acceptance  of the five genetic tests as validated.
We also compared the validation test results with the results of the  additional testing that was conducted after validation to determine if any  additional information was provided on the performance characteristics of  the genetic tests. We did not independently verify whether the contractors  followed their quality assurance guidelines in developing, verifying, and  validating their genetic tests, but we assumed that they did so from the  documentation provided.
To determine the extent to which the statistical approach used for  analyzing the repository samples was adequate, we used three  approaches. First, we collected and analyzed documentation from the  FBI, the three domestic laboratories searched by the FBI, and the  contractor who did the statistical analyses. We reviewed contract records  and conducted interviews with the FBI and laboratory officials. We  conducted a literature review to collect relevant references from forensic  science, statistics, epidemiology, and population genetics. Informed by  the relevant literature, we identified and developed the set of  characteristics that would be a statistical approach adequate to achieve  the stated purposes of the FBI’s statistical analyses.
We submitted the set of desirable characteristics described in this report  to our experts and a subcommittee of the American Statistical  Association’s (ASA) Ad Hoc Advisory Committee on Forensic Science for  their review and comment. To obtain information about how samples  were selected from stocks and submitted to the repository, we reviewed  the FBI subpoena protocols, conducted semi-structured interviews with  officials, and collected relevant laboratory documentation from the three  laboratories that the FBI searched.
Second, to obtain information about samples collected through the three  follow-up searches, we interviewed FBI officials and reviewed the  agency’s documentation, conducted semi-structured interviews with  officials from the three laboratories that the FBI searched, and reviewed  relevant laboratory documentation. To identify duplicate samples in the  repository, we compared the documentation of samples obtained through  the searches to samples submitted through the subpoena process.
Third, to demonstrate the impact of the sensitivity of the genetic tests and  data trimming assumptions made in the statistical analyses, we analyzed  the FBI repository data and estimated false negative rates for each  genetic test under repository conditions, using the post-validation results  from replicate testing of RMR-1029 and evidentiary material. We  conducted sensitivity analyses to examine the impact of data trimming  assumptions made in the FBI’s statistical analysis by varying the  assumptions made to remove all inconclusive, no-growth, and variant  results from the analysis. We computed conditional probabilities that a  repository sample was selected from a stock containing all four morphs,  given the observed combinations of genetic test results. We combined the  probability analysis with the data trimming sensitivity analysis to compute  a range of conditional probabilities for each repository sample. We  identified the samples that had a maximum conditional probability of  greater than 1 percent (nontrivial).
To assess the reliability of the FBI repository data, we summarized the  data and compared the results to the contractor’s final report on the  statistical analysis and to published reports by the FBI and the National  Academies to ensure external validity of the data. From the results of this  testing, we found the data to be sufficiently reliable for the purposes of  our review.
To determine any remaining scientific concerns and uncertainties  regarding the validation of the genetic tests and statistical approaches  that would need to be addressed in future analyses, we reviewed relevant  federal agencies’ and their contractors’ documents, published literature,  and industry documentation on the validation of polymerase chain- reaction based tests, such as those for detecting rare variants, and  related scientific concerns and uncertainties that could affect a future  investigation. We reviewed the contractors’ final reports on the statistical  analysis, reviewed contract documents, and interviewed FBI officials to  identify where improvements to the approach could be made. In addition,  we reviewed the Centers for Disease Control and Prevention’s (CDC), the  Animal and Plant Health Inspection Service’s (APHIS), and the  Department of Defense’s (DOD) select agent requirements for storing,  handling, shipping, and maintaining inventory controls. We interviewed  agency officials to determine if gaps exist in documenting important  information about the provenance of B. anthracis stocks.
Further, to identify scientific concerns arising during the FBI’s  investigation of the validation of the genetic tests and statistical  approaches, we reviewed pertinent documentation on scientific issues or  problems the FBI and NAS had identified and their effect on the FBI’s  ability to validate the genetic tests or develop appropriate statistical  approaches. Assisted by experts, we determined which gaps were  significant and their potential effect on a future investigation with a similar  scenario. We also interviewed officials and scientists at the contractors,  the FBI, DHS, the National Bioforensic Analysis Center (NBFAC), DOD  (at Dugway and USAMRIID), the Department of Energy’s (DOE)  Lawrence Livermore National Laboratory (LLNL), the Joint Genome  Institute (JGI), EurekaGenomics, and the Executive Office of the  President’s Office of Science and Technology Policy (OSTP), regarding  scientific challenges to genetic test validation, statistical analyses of the  repository data, scientific gaps related to the FBI’s investigation, and any  federal research being conducted, or planned, to fill those gaps.
To determine additional research that would be helpful in resolving such  scientific uncertainties in any future investigation, we reviewed  documentation on research DHS is conducting to address any scientific  gaps we found related to the validation of the genetic assays and issues  related to the statistical analyses of the results of the repository  screening. We reviewed the identified gaps and DHS’s research and  determined the progress that had been made to close them. Further,  following on interviews with scientists and agency officials, and input by  our experts, we determined whether any additional research is needed.
We asked scientists with expertise in public health and microbial forensic  investigations to review and comment on a draft of our report. They  included Jim Bristow, M.D., Deputy Director for Scientific Programs, DOE  Joint Genome Institute; Karin S. Dorman, Associate Professor,  Departments of Statistics and Genetics, Development, and Cell Biology,  Iowa State University; George V. Ludwig, Ph.D., Deputy Principal  Assistant for Research and Technology, U.S. Army Medical Research  and Materiel Command; Jack Melling Ph.D., Director (retired), U.K.  Centre for Applied Microbiology and Research, Porton Down, U.K.; Jeff  Mohr, Ph.D., Chief (retired), Life Sciences Division, U.S. Army, Dugway  Proving Grounds; and Stephen Velsko, Ph.D., Senior Scientist and  Associate Program Leader, Lawrence Livermore National Laboratory.
Finally, we asked a subcommittee of the American Statistical  Association’s (ASA) Ad-hoc Advisory Committee on Forensic Science for  its review and comment on the statistical aspects of a draft of our report.  The subcommittee provided us with detailed comments that expressed  general agreement with the statistical aspects of the draft, suggested  changes to terminology related to the frequency with which microbial  properties are present in a population, and suggested appropriate  caveats and limitations to analyses we conducted. We incorporated these  comments as appropriate throughout the report.
We conducted this performance audit from January 2013 to November  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Performance Parameters Evaluated by Genetic Test
paragraphs: Appendix II: Performance Parameters  Evaluated by Genetic Test n.a. n.a. n.a. n.a.  n.a.
Legend:  = evaluated; ο = not evaluated; n.a. = not applicable for qualitative tests.
section_title: Appendix III: The Effect of Assay Sensitivity and Data Trimming Assumptions in the Statistical Analyses
paragraphs: To illustrate the potential effect of the sensitivity of the genetic tests and  data trimming assumptions made in the statistical analyses, we analyzed  the FBI repository data and estimated false negative rates for each assay  under repository conditions using the results from postvalidation replicate  testing of RMR-1029 and evidentiary material. We conducted a sensitivity  analysis to examine the effect of data trimming assumptions in the FBI’s  statistical analysis by varying the assumptions to remove all inconclusive,  no-growth, and variant results. We computed the conditional probabilities  that a repository sample was selected from a stock containing all four  genetic markers, given the observed combinations of results. We  combined the probability analysis with the data trimming sensitivity  analysis to compute a range of conditional probabilities for each  repository sample. We then identified the samples that had a maximum  conditional probability of greater than 1 percent (nontrivial).
To build a model to compute this probability, we defined the sample  space of possible outcomes. There are 16 combinations for a binary  measure of the presence (+) or absence (-) of each of the four genetic  markers. Therefore we defined the possible outcomes for the four genetic  markers (A1, A3, D, and E) as S1 through S16, as shown in figure 7.
The observed assay results have the 16 possible outcomes listed in  figure 7. Since the goal of this analysis was to compute the probability  that a repository sample had been selected from a stock that contained all  four genetic markers, given the observed test result, we are interested in  the probability of S, given the observed test result for a repository  sample, 𝑃(𝑆1 |𝑜𝑏𝑠). Using Bayes’ theorem, this can be written as a  , where  posterior probability, (𝑆1|𝑜𝑏𝑠)= 𝑃(𝑜𝑏𝑠|𝑆1)𝑃(𝑆1)
𝑃(𝑜𝑏𝑠|𝑆𝑖)𝑃(𝑆𝑖)
𝑃(𝑜𝑏𝑠|𝑆𝑖)
𝑃(𝑜𝑏𝑠|𝑆1)=𝑃𝐴1(𝑜𝑏𝑠𝐴1|+)∗𝑃𝐴3(𝑜𝑏𝑠𝐴3|+)𝑃𝐷(𝑜𝑏𝑠𝐷|+)∗𝑃𝐸(𝑜𝑏𝑠𝐸|+)
We used statistics derived from the results of postvalidation replicate  testing conducted on RMR-1029 and letter material to estimate false  negative rates.
Figure 8 shows the breakdown of the results of the replicate testing.
The sensitivity analysis examined the effect of two data trimming  decisions made in the FBI’s statistical analysis of the repository  samples─the choice of D assay results and the treatment of inconclusive  results. The D marker was typed by two assay procedures (D-1 and D-2),  only one of which (D-2) the FBI used in its analysis. The Statistical  Analysis Report was restricted to the analysis of 947 samples that  contained no inconclusive or variant results and, therefore, excluded 112  samples. To explore the potential effect of the inconclusive exclusion on  the probabilities of observing all four morphs, we explored three possible  outcomes for inconclusive results. We treated all inconclusive results as  first positive and then negative, and then we excluded the inconclusive  results from the analysis. The sensitivity analysis examined the six  different combinations of outcomes, the two D assay possibilities, and the  three potential outcomes of the inconclusive data.
The computation included all 1,059 repository samples and varied the  assumptions made around data trimming from most to least conservative.  The results for each set of estimated false negative rates show that 7 of  the 16 possible outcomes of the genetic testing had a range of  probabilities that included values exceeding a 1 percent chance of being  selected from a stock that contained all four genetic markers (table 8).
Further, when we computed the probabilities for the repository samples,  we found that only a small subset of the 1,059 repository samples had a  range of probabilities that included values that exceeded a 1 percent  chance of being selected from a stock that contained all four genetic  markers. Specifically, we identified 24 repository samples, including the 8  that tested positive for all four genetic markers, which had a nontrivial  chance of being selected from a stock that contained all four genetic  markers.
By using estimates of false negative rates from the results of the  postvalidation replicate tests on RMR-1029 and the letter material, we  have made an assumption that the genetic variants in all samples in the  FBI repository were at least as concentrated as in RMR-1029 or the letter  material. Additionally, since these replicate samples were selected in a  controlled environment, false negative rates may have been  underestimated because they are not affected by variation in test results  caused by the sampling procedures used to submit samples to the  repository. These assumptions contribute to a conservative estimate of  the probability of a source matching all four genetic markers.
section_title: Appendix IV: Comments from the FBI
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Timothy M. Persons (Chief Scientist), (202) 512-6412 or  personst@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Sushil Sharma, Assistant  Director, Pille Anvelt, James Ashley, Hazel Bailey, Amy Bowser, Mae  Liles, Jan Montgomery, Penny Pickett, and Elaine Vaurio also made key  contributions to this report.
section_title: Related GAO Products
paragraphs: Anthrax: DHS Faces Challenges in Validating Methods for Sample  Collection and Analysis GAO-12-488 (Washington D.C.: July 31, 2012).
Federal Agencies Have Taken Some Steps to Validate Sampling  Methods and to Develop a Next-Generation Anthrax Vaccine,   GAO-06-756T (Washington D.C.: May 9, 2006).
Anthrax Detection: Agencies Need to Validate Sampling Activities in  Order to Increase Confidence in Negative Results, GAO-05-251  (Washington D.C.: March 31, 2005).
U.S. Postal Service: Better Guidance Is Needed to Ensure an Appropriate  Response to Anthrax Contamination, GAO-04-239 (Washington D.C.:  September 9, 2004).
U.S. Postal Service: Issues Associated with Anthrax Testing at the  Wallingford Facility, GAO-03-787T (Washington D.C.: May 19, 2003).
U.S. Postal Service: Better Guidance Is Needed to Improve  Communication Should Anthrax Contamination Occur in the Future,  GAO-03-316 (Washington D.C.: April 7, 2003).