Bioforensics: DHS Needs to Conduct a Formal Capability Gap Analysis to Better Identify and Address Gaps
Released: 2017-02-10T00:00:00

Letter

Background

DHS coordinates the federal government’s overall response to or recovery from terrorist attacks. The Centers for Disease Control and Prevention (CDC) within the U.S. Department of Health and Human Services is the primary agency for the public health response to a biological terrorism attack or naturally occurring outbreak. The FBI within DOJ is the primary agency for the criminal investigation of incidents of bioterrorism. In its recommended guidelines for laboratories engaged in microbial forensic analyses, the Scientific Working Group on Microbial Genetics and Forensics (SWGMGF) defines attribution as “the information obtained regarding the identification or source of a material to the degree that it can be ascertained.”

As part of the effort to deter biological terrorism and strengthen the law enforcement response to such an act, Homeland Security Presidential Directive (HSPD) 10, “Biodefense for the 21st Century” established within DHS a dedicated central microbial forensic laboratory known as the NBFAC to provide bioforensics analysis of evidence associated with the event. This Presidential Directive established the NBFAC as “the lead federal facility to conduct and facilitate technical forensic analysis and interpretation of materials recovered from biocrime and bioterror investigations in support of the appropriate lead federal agency.”

DHS Science and Technology (S&T) is to accelerate the delivery of enhanced technological capabilities to meet the requirements and fill capability gaps to support DHS agencies in accomplishing their mission. Pursuant to this mission the DHS Chemical and Biological Defense (CBD) Division seeks technologies to defend against a chemical and biological attack. In addition, the division is charged with pursuing research to improve response and restoration, conduct threat risk assessments, and invest in bioforensics R&D. In this regard, the Bioforensics R&D Program, according to DHS, supports NBFAC operational threat agent identification and characterization through investments in bioforensics research and next generation technologies to include molecular biology, genomic comparison techniques, genotyping assays and physical and chemical analysis of sample matrix to better understand the origin, evolutionary history, production method and dissemination mechanism associated with the malicious use of biological agents.

Bioforensics has been defined as an interdisciplinary field of microbiology devoted to the development, evaluation, validation, and application of methods to detect and fully characterize microbial samples containing a biological agent or its components for the purpose of making statistically meaningful comparative analyses. Attributing something to a perpetrator requires different types of information and analysis—both traditional and bioforensics. Information produced by forensic examination can result in an investigative lead or provide support for the investigation.

Bioforensics capabilities used to provide analyses of evidence may show how, when, and where microorganisms were grown and potential methods for dissemination, which assists attribution.

Bioforensics evidence could include the agent that was released, toxins, nucleic acids, and protein signatures. It could also include contaminants, additives, and evidence of preparation methods.

Traditional evidence could include fingerprints, hair, fibers, documents, photos, firearms, and body fluids.

In a bioforensics case, the intent would likely be to gather sufficient information to allow a comparison of an evidentiary sample with a known reference sample to assist in supporting source attribution. Evidence from a bioforensics investigation must also meet the scientific community’s standards for evidence as well as a criminal court’s standards for legal admissibility.

DHS Has Not Performed a Capability Gap Analysis to Help Focus Its Resources for Addressing Bioforensics Capability Gaps

DHS has developed strategic plans and goals related to bioforensics attribution and identified some key bioforensics capability needs. However, according to DHS officials, DHS did not perform a bioforensics capability gap analysis, but, rather, used an informal approach to identify capability needs and gaps. DHS officials stated that they did not document the process DHS used or the results of its informal approach. Further, DHS officials told us that there is not a complete list of the gaps identified using the informal approach. Finally, although they indicated that DHS had focused resources toward addressing those gaps, they could not provide documentation of the bioforensics capability requirements and other relevant information to support the capability gap identification and resource allocation decisions that were made. In the absence of documentation of the processes, discussions, analyses, decisions, or any other activities performed to identify and prioritize bioforensics gaps, DHS’s rationale for the identification and prioritization of needs and gaps on which to focus its resources is unclear.

DHS Did Not Use a Systematic, Documented Approach to Identify Its Bioforensics Capability Needs and Gaps

According to DHS officials, DHS relies on the DHS S&T-managed NBFAC and bioforensics R&D programs to identify bioforensics capability needs and gaps. However, DHS does not have a complete list of its bioforensics capability gaps because it has not performed a bioforensics capability gap analysis. According to the DHS Systems Engineering Life Cycle guide, a gap analysis is a best practice that is essential to understanding whether capabilities exist that can meet requirements, or if they must be developed. In addition, some DHS officials told us that performing a capability gap analysis is a best practice that DHS programs should follow, even in the absence of DHS guidance to do so. In interviews and written responses, DHS officials described generally how DHS identified and documented capability needs and gaps. They told us that they identify priorities each fiscal year, develop projects to meet these priorities, and develop the NBFAC Annual Plan to address these priorities. However, they told us that there is no documentation of the process or results of the informal approach they used to identify bioforensics capability needs and gaps. According to federal standards for internal control, documentation is necessary for the effective design, implementation, and operation of an entity’s internal control system. Lacking documentation of the processes, discussions, analyses, decisions, or any other activities performed to identify and prioritize capability gaps, it is unclear what DHS’s rationale was for the identification and prioritization of bioforensics capability needs and gaps. Identifying and prioritizing capability gaps enables the proper allocation of resources to the highest priority needs. Thus, without a capability gap analysis, DHS may not have identified and prioritized all capability needs and gaps, and so may not be allocating resources to address the most significant gaps to meet its mission needs.

DHS officials told us that no complete list of bioforensics capability gaps has been created since 2010. However, they told us that DHS had developed a document from 2013-2014—the Bioforensics Roadmap (Roadmap)—as a means to identify and achieve consensus from stakeholders on the key bioforensics capability needs on which to focus DHS resources. DHS officials said that the Roadmap lays out the Bioforensics R&D Program execution and also lists the key needs on which DHS has focused, or will focus, resources, along with the associated programs to address the needs. DHS developed strategic plans for its National Biodefense Analysis and Countermeasures Center (NBACC) in 2012 and Chemical and Biological Defense in 2013. It documented NBFAC strategic goals in 2013. These documents include strategic objectives related to bioforensics that could be used to guide a capability gap analysis.

A former DHS official who had participated in DHS’s process for the identification of bioforensics gaps told us that the process was informal and also that there is no documentation of the process. This official summarized it as generating a list of topics and issuing BAAs to address them. Other DHS officials confirmed that the process was informal and that there was no documentation of the results. They told us that they are unaware of the details of the processes and activities performed to identify capability needs and gaps. However, they did describe generally the informal process that DHS used.

They indicated that the process included working with key interagency partners and other stakeholders—such as the FBI—and engaging in discussions, exchanging emails, and holding periodic meetings. DHS officials also stated that they met informally as needed with the FBI and some intelligence agencies to discuss needs and gaps. These officials explained that the discussions with DHS’ interagency partners were part of a larger process to develop and manage NBFAC and bioforensics R&D programs. They said that DHS coordinated with the FBI and the Intelligence Community to focus these programs’ activities to meet the needs of these end users. Further, they said that the Roadmap was vetted by other agencies and researchers. In addition, according to FBI officials, the FBI conducted assessments of its capabilities by working with the DHS S&T and providing direction to DHS about its capability needs. FBI officials stated that sometimes the FBI does not know there is a bioforensics capability gap until it encounters one during an investigation.

Independent assessments of DHS’s S&T Bioforensic R&D program have raised similar concerns about how DHS has identified and prioritized bioforensics capability gaps. For example, external assessments of the CBD portfolio from 2012 and 2014 found a lack of clarity about how the Bioforensics R&D program identified and prioritized capability gaps and why some projects were chosen. The reviewers recommended that the program manager describe the program’s basis for identifying capability gaps and selecting projects in future reviews. Specifically, a November 2014 review acknowledged the need for enhancing bioforensics capabilities but questioned the lack of information on how gaps in capability or knowledge guiding R&D investments were identified and prioritized. A 2012 review stated that it was unclear why some research studies were chosen over others, as well as how the selection of projects was linked to, or justified against, the risk assessment.

DHS and the FBI Have Identified Several Bioforensics Capability Needs and Gaps

Through the Roadmap, key bioforensics efforts have been identified, which DHS officials characterized as gaps. The efforts listed in the Roadmap include (1) operational infrastructure, (2) sample collection and preservation, (3) sample extraction, (4) identification and characterization, and (5) data analysis and integration. The Roadmap also includes existing and future CBD and other agency programs, as well as commercial development, linking them to the particular capability gap they address. The Identification and Characterization effort in the Roadmap includes developing capabilities to characterize unique, novel, and engineered agents; characterize unknowns (emerging or synthetic organisms); identify and characterize toxins, such as ricin; and quantify and communicate uncertainty, which is of particular significance when using metagenomics and proteomics capabilities. A DHS official and FBI officials also told us that other items they considered to be gaps include the difficulties in interpreting metagenomics data, limited sequences for select organisms in its reference database, and the need for a greater ability to examine proteins. However, the Roadmap provides no additional details about the bioforensics capability gaps other than the projected timeframe of 2014 to 2020 for completion of the agency programs. Figure 1 shows the Roadmap. According to DHS officials, to bridge the broad gap areas set out in the Roadmap would require resources far beyond those available to DHS.

Experts Have Identified Several Capability Needs for Bioforensics

In addition to DHS and the FBI, other organizations, such as the NRC of the NAS and the NSTC of the Office of Science and Technology Policy (OSTP), have been involved in identifying bioforensics capability needs. The NRC Committee on Science Needs for Microbial Forensics was an international group of experts that identified scientific challenges that must be met to improve the capability of bioforensics to investigate suspected outbreaks and to provide evidence of sufficient quality to support responses, legal proceedings, and the development of government policies. Similarly, OSTP’s National Research and Development Strategy for Microbial Forensics was established to guide and focus U.S. Government research efforts to advance the discipline of bioforensics. With the assistance of the NAS, we convened our own meeting of experts in April 2016 to review and update capability needs that the NRC and OSTP identified and to identify additional needs that might be useful for DHS and the FBI to consider when they identify their capability needs as part of a bioforensics capability gap analysis. Some of the experts provided alternative views about certain aspects of the identified capability needs. While some of the bioforensics capability needs identified overlap with efforts listed in the DHS Roadmap, they were not formulated specifically considering DHS requirements so may not be relevant to DHS. However, we believe that this information could help inform the DHS’s and FBI’s efforts to identify capability needs and prioritize gaps.

Starting with the capability needs identified by the NAS and OSTP, the experts that participated in the GAO meeting identified and generally agreed upon the capability needs listed in Table 1. Some of the needs the experts confirmed as still relevant were similar to those identified by the DHS and the FBI, and some were different. For example, like DHS and the FBI the experts agreed that an ability to characterize genetically engineered agents was needed, but they also suggested that evaluating existing protocols, such as those for DNA sequencing, to determine whether they were validated, was needed. The identified needs in table 1 can generally be grouped into three broad areas: (1) science, (2) technology and methods, and (3) bioinformatics and data. There are six needed capabilities within the science area; five within technology and methods; and three within the bioinformatics and data areas.

While the majority of the experts agreed generally with the bioforensics capability needs in the three broad areas listed in table 1, some experts also had alternative views about some of the needs. For example, some experts thought that some of the needs should have a different focus or should be given a lower priority than others. In addition, some experts suggested that because it may be impossible to characterize all microbes, the first science capability need—the identification, monitoring, and characterization of microbial species—should instead focus on (1) developing a dynamic process and infrastructure for rapid collection and typing when an event occurs or (2) using a species-agnostic approach to identify both natural and synthetic microbes, such as focusing on genetic mechanisms rather than organisms.

Additionally, some experts stated that limited emphasis should be placed on the third science need—developing methods to distinguish among natural, accidental, and deliberate outbreaks. They indicated that other investigatory data would be available that would be better suited for making this determination. Instead, they said the focus should be on identifying introductions of additional virulence or genetic elements into an organism and determining other elements that suggest that somebody has modified the organism. There was also disagreement on which microbes DHS and the FBI should focus. Some experts stated that, because new pathogens are difficult to create, the greater concern is naturally-occurring or modified microbes. Further, they also said that distinguishing among existing organisms already presents a difficult enough challenge. Finally, some experts said that the focus should be on microbes in laboratories because they are the most relevant to bioforensics and are not typically studied by the larger scientific community.

The experts disagreed on whether the sixth science capability need— metagenomics research—was important for bioforensics. Some stated that metagenomics is worth exploring as a future capability but there are easier problems that need to be solved. Others said that a metagenomics capability is not necessary for analyzing simple samples but it might be useful for analyzing complex samples. In addition, one expert questioned the fifth technology and methods need—the need for nongenetic orthogonal methods—indicating that it is not a requirement in court to have two different methods to determine a result.

For some of the bioforensics capability needs, experts indicated that other groups would develop the capabilities so that DHS or the FBI would not need to invest in them. For example, some experts said that the FBI should not focus its effort on the second science need—researching the mechanisms of pathogenicity—because it is unlikely to be closed quickly, and other groups are already addressing it. Regarding the second technology and methods need—adapting assay and sequencing technologies—some experts indicated that the commercial market will drive the development of improved sequencing technologies. Similarly, some experts said that agencies, such as the FBI and CDC, are working to address the first bioinformatics and capability need—the creation of data repositories and reference collections for pathogens and other microorganisms.

The specific bioforensics casework requirements that formed the basis for the DHS’s efforts were not known to the experts that participated in the meeting and, consequently, the list of capability needs cannot be directly compared to the efforts in the Roadmap. These capability needs, along with the alternative views presented, could help inform the DHS’ and FBI’s efforts to identify and prioritize bioforensics capability gaps. These agencies could consider this information as part of any capability gap analysis.

DHS and the FBI Have Acted to Enhance Bioforensics Capabilities but Face Numerous Challenges

DHS and the FBI have taken actions to enhance some bioforensics capabilities but face numerous challenges before they can achieve the desired enhancements. Actions include not only the concrete steps that DHS has taken to enhance its capabilities, such as funding R&D activities, but also key strategic decisions underlying those actions. In this context, DHS actions include (1) developing methods-based capabilities to provide a broader bioforensics capability; (2) funding R&D activities to enhance its capabilities; (3) developing capabilities for short-term casework needs; (4) establishing an in-house reference database; and (5) developing capabilities for characterizing genetically engineered and unique, novel, or unknown (emerging or synthetic) agents. However, to achieve the capability enhancements they are pursuing, DHS and the FBI must overcome numerous challenges. These include (1) achieving the ability to interpret and communicate results from the bioforensics capabilities with a statistical confidence, (2) developing statistical frameworks, quantitative measures, and quality reference collections, (3) ensuring that its Bioforensic Repository Collection (BRC) contains quality data and appropriate agent strains, and (4) determining future casework needs relative to views of the evolving threat landscape. In addition, experts at our meeting and those we interviewed identified challenges regarding reference databases, the use of statistical frameworks, and the communication of results.

DHS and the FBI are Focusing on Methods- Based Capabilities to Provide a Broader Bioforensics Capability

DHS has taken several actions to enhance some of NBFAC’s bioforensics capabilities for use on FBI casework. For example, we found that since 2010, DHS, with FBI input, made a strategic decision to focus on the development of methods-based capabilities rather than agent- based capabilities for identifying and characterizing biological agents. This strategy is reflected in the 2012 NBACC strategic plan and its goals for NBFAC. Methods-based capabilities, according to DHS’s written responses to our questions, include genomics (whole genome sequencing and bioinformatics analysis), and analytical chemistry (mass spectrometry and scanning and transmission electron microscopy). In addition, DHS will maintain and enhance its agent-based capabilities in the interim—which include molecular biology, virology, bacteriology and toxinology—some of which will always be necessary for certain types of casework. Both types of capabilities will reside at NBFAC.

The FBI agrees that such enhanced capabilities are needed. In responding to our questions, the FBI stated that DHS’s approach will provide “an adaptive and agile capability to characterize unique, novel, engineered or emerging biological agents.” While agreeing with the need to develop methods-based capabilities, however, the FBI also acknowledged in its responses that some agent-specific capabilities will always be needed for its investigations.

Methods-based capabilities, according to DHS’s written responses to our questions, can potentially provide NBFAC with a broader bioforensics capability. For example, DHS stated that genomic analysis can use unique features as signatures to differentiate a particular isolate from others. DHS also responded that such features could include single nucleotide polymorphisms (SNP), rare variants, and epigenetic variation. Further, DHS stated that genomics-based characterization—including the ability to characterize background nucleic acids that may be derived from the environment in which the sample originated—represents a unique investigative signature that agent-based bioforensics procedures would miss. According to experts at our meeting, signatures range from anything that aids an investigation, to genetic signatures, syndromic signatures, metadata, and proteins, as well as other molecular signatures.

DHS officials told us that the use of methods-based approaches, such as genomics, have dramatically reduced investigation timeframes. They said that DHS can now detect and sequence not only select agents, but a number of other biological agents (even bioengineered ones) in a fraction of the time. What took years to complete in the 2001 Amerithrax case can now happen much more quickly with such improvements. An FBI official further elaborated, stating that improvements in techniques and technologies have led to potential increases in obtainable information and significant reductions in analysis times supporting bioterrorism investigations.

In contrast, based on our review, prior to 2010, we found that NBFAC’s bioforensics capabilities focused on identifying biological agents on the CDC and USDA select agent lists. In this regard, in its written responses to our questions, DHS stated that it has established International Organization for Standardisation (ISO) 17025 accredited, complementary assays such as culture, real-time polymerase chain reaction (PCR), and immunoassays for most traditional bacterial, viral, and toxin agents. However, unlike methods-based capabilities, DHS stated that these require prior knowledge of an organism and the maintenance of agent‐ specific reagents. Further, agent-based capabilities do not cover a wide array of potential threats, including genetically modified or de novo agents, and have not been developed for known human pathogens, especially those that may not be cultivable. Thus, according to DHS responses, a methods-based approach will ultimately provide NBFAC with capabilities not only for analyzing challenging samples but also with a broader, more comprehensive bioforensics capability for characterizing unique, engineered, or emerging biological agents.

Table 2 shows these two types of capabilities and the types of analyses they could be used to perform on evidentiary samples. For example, bacteriology involves culturing and deriving phenotypic information on an agent to identify and characterize it.

However, toxinology, another agent-based capability, could involve identifying and characterizing protein toxins such as ricin using an immunoassay—such as ELISA, an enzyme-linked immunoabsorbant assay. In addition, analytical chemistry, a methods-based capability, could be used to characterize toxins by using mass spectrometry, which also supports proteomics analysis. That is, both types of capability could be involved in analyzing toxins. According to DHS responses to our questions, use of each of its mass spectrometry methods function independently and provide complementary information to confirm results derived from immunoassays and biological activity assays for protein toxins. Electron microscopy—a methods-based capability—involves nonbiological analysis of evidence samples. For example, it could provide elemental analysis of an agent. Transmission electron microscopy (TEM) and Scanning Electron Microscopy (SEM) can be used to provide images of nonspore forming bacteria and viruses, and castor bean products, among others.

DHS Has Funded R&D for Bioforensics

DHS has solicited and funded R&D projects to enhance NBFAC’s bioforensics capabilities—completion of which DHS anticipates will extend beyond 2025. The R&D is related to areas in which DHS has stated there are capability gaps, or it is linked to some of the program responses listed in the 2014 Roadmap. It also reflects DHS’s shift toward methods-based approaches, such as genomics and proteomics.

Using a BAA mechanism, DHS solicited research proposals for R&D related to enhancing its bioforensics capabilities. To more clearly describe the type of research sought, the BAAs specified not only broad topic areas as well as technical topic areas—more specific, technical details about the type of research being solicited. Subsequently, DHS awarded about 36 contracts for solutions or products addressing areas related to bioforensics. According to the FBI’s response to our questions, it is involved in the process from start to finish, including assisting in drafting the BAA, the proposal evaluation and selection process, and meeting with DHS and the contractors throughout the course of the contract.

Before being used in FBI casework, SOPs and other deliverables from the funded research would have to make a transition to NBFAC operations and potentially be accredited under ISO 17025, as appropriate. In responding to our questions, DHS stated that both ISO 17025 accreditation and deliverables such as publications provide the data necessary to support the general acceptance of a method within the scientific community and to meet the Daubert standards for admissibility of analysis for a federal prosecution. DHS officials explained that, to the extent possible, they publish their research results so that NBFAC’s bioforensics techniques can be peer-reviewed, validated, and supported in court. Figure 2 shows the broad topic areas and the years in which research was solicited through the BAAs.

Broad and technical topic areas: Based on our review, both the broad topic areas in figure 2 and the underlying technical topic areas in the BAAs reflect the long-term, methods-based enhancements, and also the enhancements to toxin analysis capabilities for the FBI’s current casework needs sought by DHS. For example, the following technical topics were included as part of the 2015 solicitation for Bioforensics research: products to identify select agents including toxins, with high next generation and novel technologies to characterize biological threat agents for source attribution, bacterial populations of select agents with critical knowledge gaps, including C. botulinum and B. anthracis (North Africa, Middle East), high-confidence methods for metagenomics analysis of complex biologicals in complex samples to support whole genome sequencing, and informatics and statistical tools.

DHS-funded R&D: In line with the broad topic areas indicated in the figure—bacterial population genetics, sequence-based approach to bioforensics, and bioforensics research—we found that DHS-funded R&D contracts include the following areas: population genetics for forensics, biological toxin identification, metagenomics sequence data, statistical confidence in evidentiary materials based on bacterial forensic proteomics of virus production,

Bayesian taxonomic assignment for next-generation sequencing, and sequencing-based bioforensics analyses.

R&D supports DHS’s efforts to develop methods-based capabilities, including sequencing methods to enable genomic analysis of any organism in any sample, as well as bioinformatics methods for de novo assembly, metagenomic classification, comparative analysis, identification of genetic engineering signatures, and the inference of biological function. For example, according to DHS’s responses to our questions,

Population genetics: research into population genetics, a 5-year timeline project, is published in the open scientific literature, in the sequence data in GeneBank; NBFAC used the information to better understand the genetic diversity of the organisms studied in that project. According to documentation we reviewed, results from such studies will refine understanding of the population genetics of certain select agents to better calculate match statistics in a forensic setting.

Biological toxin characterization: Regarding toxins, DHS has funded contracts to develop SOPs for protein toxin characterization using mass spectrometry, among other projects.

Metagenomics sequencing: DHS research into genetic issues is ongoing. DHS is seeking a means for future use of metagenomics analyses on complex samples. Regarding metagenomics, DHS plans research into high-confidence metagenomics analysis of complex biological samples, as well as developing statistical models and software to identify the organisms in a complex sample and estimate their relative abundance, including developing an existing system for probabilistic reconstruction of the taxonomic structure present in a metagenomic sample.

Bioforensic proteomics: DHS has also funded research on proteomics—including proteomics of virus production—and analysis of proteins and metabolites of unknown samples to complement genetic characterization.

Ricin Ricin, or Ricinus communis, is one of the most poisonous naturally occurring substances. Ricin is derived from the beans of the castor plant. Ricin is toxic to cells and damages all human organs. It is considered a select agent (toxin). No antidote is available.

DHS and the FBI are Enhancing NBFAC’s Biological Toxin Analysis Capabilities for Current Bioforensics Casework: Based on our review and DHS’s responses, DHS’s primary focus is on bioforensics capabilities in the short term to address the FBI’s current casework needs. Such casework has involved the FBI’s investigation of multiple biocrimes involving the use of ricin, including a case in 2013 in which ricin was sent to the U.S. President. NBFAC analyzed some of the samples in that case, according to the FBI’s responses to our questions.

FBI casework carried out by NBFAC involves the FBI’s transporting evidentiary samples to NBFAC, which (1) develops a sample analysis plan (which could involve traditional as well as bioforensics analyses) for FBI approval, (2) conducts analyses, and (3) reports the results to law enforcement, which uses them to inform the bioforensics investigation.

Based on our review, for prosecution in a case involving ricin, the scientific evidence may need to establish that the toxin is present in an evidentiary sample. We found that a combination of analytical capabilities may be used to confirm this, with each detecting a specific target. For example, to confirm the presence of ricin in a sample, antibody tests, such as ELISA, and mass spectrometry can be used for detecting the presence of ricin, and examining the protein, respectively. Added to them can be cell-free translation assays, another type of antibody test, which also detects ricin. We also found that NBFAC’s capabilities for analyzing ricin toxins initially included all the independent capabilities above, with the exception of an accredited mass spectrometry capability for characterizing ricin and other toxins. Based on our review, we found that NBFAC had contracted with a laboratory to examine protein toxins by mass spectrometry when it did not yet have that capability. Doing so, according to their responses, resulted in a 2—3 day delay, and the laboratory was also not accredited under ISO 17025. As a result, the FBI further responded that it requested that DHS develop an in-house ISO 17025 accredited toxin analysis capability at NBFAC. The FBI provided the equipment and funding for this transition to NBFAC.

Enhancing Genomics and Proteomics Is a Long-Term Effort: According to a DHS official, DHS is continuing to enhance both genomics and proteomics capabilities, which is expected to provide a complementary capability that will link proteomic analysis to metagenomics analysis of complex samples, thereby providing additional information about an agent. Further, according to this official, genomics and mass spectrometry will support developing metagenomics and proteomics. Based on our review, some of the ways in which metagenomics capabilities may be used are as follows:

Metagenomics: It allows sampling of the genomes of microbes without culturing them; rather, the DNA is directly isolated from the sample before genome sequencing. A DHS official stated that DHS plans to provide comprehensive metagenomics analysis of complex evidentiary samples. These types of samples may contain both microbial and human DNA as well as mixtures that derive from possible processing steps (growth media, etc.), which could provide links to a possible source.

In the context of metagenomics, according to a DHS official, “complex samples may be from any environment and can be mixtures of many types of organisms. The simplest of metagenomics samples may be viruses in the tissue culture which contains the genomes of two organisms, the cell line, and the virus. The most complex metagenomic samples are from soil samples. Soil samples may contain an organism of interest, at low concentration, but also will likely have DNA and other biological materials from things such as plants, animals, fungi, bacteria and viruses. The ability for the forensic laboratory to collect metagenomic data and analyze it relies on the development of tools for metagenomics.” In this regard, according to an expert who we contacted, metagenomics or the evaluation of environmental samples for genetic information is a task that may not give DHS the returns from its investment. It is a very time consuming technique and should probably be left to academia and/or industry. Once these methods are developed, DHS would be able to apply the most applicable techniques, according to this expert.

Figure 3 illustrates the possible composition of a complex sample.

Metagenomics analysis of a complex sample could reveal the presence of DNA and other types of material at different percentages, including eukaryotic nucleic acids. However, because evaluating metagenomic sequence data is based on relative abundances, large amounts of data may be generated. Interpreting these data and their meaning in terms of an agent’s source will be necessary.

Based on our review, some of the ways in which proteomics capabilities may be used are as follows:

Proteomics: Proteomics is the study of proteomes. A proteome is a set of proteins produced in an organism, system, or biological context. In response to our questions, DHS advised us that it plans to establish a proteomics capability for NBFAC sometime in the future. Mass spectrometry is being used for proteomics analysis and is able to provide information indicative of a particular protein.

While proteomics does not replace genomic analysis, it may provide additional information if the microbial DNA is too damaged for analysis, according to an expert who attended our April meeting. In addition, according to this expert, there are differences in microbes between those naturally occurring and those grown in laboratories, including differences in growth patterns. Proteins express themselves based on different food sources. Consequently, according to this expert, analyzing the microbes to determine the growth medium used could be useful for bioforensics. Further, protein profiles have the potential to provide information on the environment that the microorganism has experienced. For example, cultivation might provide information about the skills of the people who grew the organisms. Thus, proteomics provides a different level of discrimination from that of genomics. Finally, according to another expert who we contacted, proteomics analysis should become a valuable tool for bioforensics and may rival genetic information when methods have matured.

Based on DHS’s responses to our questions, achieving a genomics and proteomics capability will also require (1) a bioinformatics and a statistical framework for inference and analysis of unknowns in microbial isolates, and (2) significantly expanded genome databases and an understanding of the underlying determinants of various pathogenic traits. DHS responded that both of these are currently funding priorities. In this regard, DHS stated that NBFAC continues to expand a major genomics capability that includes multiple, complementary sequencing platforms and advanced bioinformatics within high-performance computing environment. In its responses, DHS termed this approach as “agent- agnostic” as the analytical procedures require no knowledge of which agent might be present in a sample. However, while DHS also stated that it provides confidence estimates for aspects of its genome sequencing and continues its incremental development of its genomics capability, it also acknowledges the need for statistical frameworks. For example, it stated that “the issues regarding statistical uncertainty require the development of statistical frameworks to ensure that attribution signatures are clearly defined and understood; that there is standardization, validation and verification of the signatures; that relevant source populations are fully characterized and understood; the limitations of measurement tools are known, and the statistical methods being used are appropriate for the signatures data.” Finally, DHS stated in its response to our questions, that understanding and communicating of the uncertainty is of particular significance when using metagenomics and proteomics on complex sample types.

Other experts we interviewed also agreed that there is a need for more flexible bioforensics capabilities. For example, an expert from our meeting stated that currently, characterizing an agent is achieved by using sequence data. Learning what can be exploited for this purpose is in its early stages. In addition, a U.K. official we interviewed said that while a priority list of organisms will still be needed for responding to emerging pathogens and diseases or synthetic biological agents, now a more agnostic or “horizon spanning approach” will be used. Nevertheless, not all experts were in agreement that DHS should pursue metagenomics for bioforensics purposes, at least not in the short-term. For example, in a 2016 independent assessment of DHS Bioforensics R&D program, reviewers recommended a “more measured investment” in metagenomics and expressed doubt that an operational metagenomics capability was likely to be available at NBFAC in 5 years. Instead, they suggested that DHS take a more proactive investment stance by following developments in the field that were occurring elsewhere.

Completion of Capability Enhancements: Based on our review of the roadmaps that DHS provided to us regarding the bioforensics enhancements, DHS estimates that most of the R&D tasks associated with capability enhancements will be completed by 2025 or later, with some exceptions. For example, in July 2016, a DHS official indicated that DHS’s new mass spectrometry casework capability may be available after it has been accredited under ISO 17025 over the next 12 months (in 2017). In addition, DHS has a 3-5 year focus for developing metagenomics so that it can be used on casework. Completion of more advanced enhancements will likely extend beyond 2025, according to DHS responses to our questions, such as for genomics and proteomics, areas that are evolving. Activities include establishing (1) integrated processes within metagenomics analyses to facilitate high resolution characterization of all agents and nucleic acids in complex samples; (2) a bioinformatic and statistical framework for phenotypic inference and analysis of “unknown unknown” microbial isolates; and (3) increased capabilities to support large-scale proteomic analysis integrated with inferential analyses. See appendix II for more details on the BAAs.

DHS is establishing an In- House Reference Collection

DHS is also taking actions to establish an in-house reference collection of biological materials—the NBFAC BRC—which will provide materials for comparative forensic analyses, assay development and evaluation, and proficiency testing. According to DHS responses to our questions, the BRC is a long-term storage site for materials acquired from other institutions (government, academia, commercial and international sources) and NBACC projects. Housed at NBACC, it includes select and nonselect agent bacteria and viruses, toxins, and their near neighbors.

The BRC supports characterization of bacterial and viral agents by determining phylogenetic relatedness of different bacterial and viral isolates and enabling isolate-level characterizations, which according to DHS, is important for isolates that have never been fully characterized or sequenced. DHS began obtaining a variety of biomaterials such as select agents and toxins through subcontracts with government agencies, which were stored in external laboratories. In fiscal year 2010, the new NBACC laboratory opened, after which the collection was consolidated within the biocontainment facilities at NBACC and became available for use as a reference material. DHS states that it is working with the FBI to expand the number of strains of interest in the BRC. In addition, DHS projects that develop information on biological organisms are published in the open, peer-reviewed literature; sequence data are published in GeneBank and are available to the larger community and NBFAC.

DHS is Developing a Capability to Identify and Characterize Engineered, Novel, and Unknown Agents

DHS’s actions also include developing incrementally a capability for identifying and characterizing genetically engineered novel, and unknown (emerging or synthetic) agents that uses methods-based capabilities. In this regard, NBFAC has developed a genomics capability that DHS asserts can be used to infer genetic engineering from DNA sequencing and protein sequences. Genetic engineering involves inserting a foreign sequence of genetic codes into an existing sequence of genetic codes in a target organism with a view to altering some of its functions. DHS states it can identify genetic modifications by screening against genes of interest (for example, virulence factors or antibiotic resistance genes), comparing genome alignments, and comparing regions with unusual sequence composition to those typically found in nature.

In the past, restriction enzymes have been used to cut DNA and insert specific genes from a different organism to produce a desired effect (for example, producing human insulin using bacterial cells), which results in “scarring” at the restriction sites. Genome characterization and analysis according to DHS’s and the FBI’s responses to our questions, respectively, would be able to detect such scarring. However, gene editing techniques are evolving and may be harder to detect. Clustered, regularly interspaced, short palindromic repeats (CRISPR) Cas-9—is now available to researchers. It engineers microbes by inserting genes, although unlike previous methods the restriction sites may not be evident, and the enzymes used will not cause scarring. Figure 4 is a simple illustration of genetic engineering using restriction enzymes.

Experts from our meeting, as well as other experts we interviewed, indicated that identifying genetic engineering could be approached by determining an agents’ virulence and then using capabilities such as mass spectrometry to identify if elements exist that suggest modification. Nevertheless, it was thought that a genetically engineered agent would have some parts that remain unchanged, which would help to determine its characteristics. For example, according to an expert at our meeting, the focus should be on (1) identifying those introductions of additional virulence or genetic elements into an organism, which can be done fairly quickly, and then determining if there are other elements that suggest somebody has modified the organism and (2) using methods like mass spectrometry, microscopy, and other methods that can identify the means of production or culture and dissemination or delivery of the organism. In addition, according to a U.K. expert we interviewed, the core part of the genome in an engineered agent would not be changed, and it also must reproduce and metabolize. If it is an engineered virus, it would have some similarities to other viruses, such as in how it attaches itself to a cell to propagate its genome. So within its genome some signatures would be available for comparison. Further, this expert stated that even if the organism was a synthetic one and CRISPR Cas-9 had been used, he would still look to see whether any scarring was present (see figure 4).

Regarding synthetic agents, DHS asserts that they can be analyzed similarly to those that are genetically engineered, with the addition of NBFAC’s inferential analysis capability, whose analysis will provide clues to the functionality of a synthetic agent. DHS is developing a capability that will allow NBFAC to characterize unique, novel agents, “unknowns” (emerging or synthetic organisms) and “unknown, unknowns” (de novo synthetic organisms). However, achieving this capability will also require a bioinformatics and a statistical framework for inference and analysis of unknowns in microbial isolates and expanded genome databases, according to DHS. DHS indicates that it is developing a “multi-layered inferential analysis capability” that would include establishing comparative methods for the analysis of any DNA or protein sequence to identify such things as peptides, restriction sites, and a statistical model that allows confidence estimates to be placed on these analyses.

DHS Faces Numerous Challenges in Enhancing Its Bioforensics Capabilities

DHS faces numerous challenges as it attempts to enhance its bioforensics capabilities. Our review of agency documentation and related literature, and interviews with agency officials, scientists, and subject matter experts at our meeting and elsewhere, as well as our prior work, indicate that challenges must be overcome if DHS is to develop enhanced capabilities suitable for bioforensics. These include capabilities that not only can be relied on for identifying and characterizing known agents but also those that have been genetically engineered, or are unique, novel or unknown (emerging or synthetic). Challenges that DHS faces include (1) the ability to interpret and communicate results with defined statistical confidence, (2) obtaining access to quality references and databases for bioforensics analysis, and (3) the effect of the evolving threat landscape on future casework needs. Further, their results must be also able to stand up to court scrutiny.

DHS Faces Challenges in Its Ability to Interpret and Communicate Analyses Results with Statistical Confidence

DHS plans to develop advanced metagenomics and proteomics capabilities. However, it is not clear to what extent or when DHS will be able address key challenges related to enhancing its bioforensics capabilities that include interpreting results from metagenomics and proteomic analyses, with a defined statistical confidence, according to both DHS and the FBI officials. Further, communicating the uncertainty associated with the results will be particularly important when using these capabilities on complex sample types.

Without a defined level of statistical confidence, the probative value of inferences made from the results of such analyses may not be known. In general, as the level of statistical confidence increases in these inferences—signifying a higher degree of scientific certainty—the probative value of the inference also increases. In figure 5, we have extracted and expanded on one dimension of “the forensic continuum” that has been used to represent the evaluation and analysis of a bioforensics sample and its probative value. As indicated in the figure, probative value depends on confidence in analysis and the interpretation and meaning of evidence.

Investigative leads or bioforensics data may rely on the use of bioinformatics and data and inferences using those data. In this regard, according to a DHS official, an issue DHS continues to struggle with is how to interpret metagenomics analysis—whether it is possible to define with certainty whether a piece of the genome of an agent is present— versus defining the error rates for each sequencing base call, which DHS can do.

DHS solicitations for R&D reflect some of these challenges, including the following extract from a related 2012 BAA solicitation regarding interpreting the results of metagenomics analyses: “Currently, it is difficult to assign confidence to the results of metagenomic analyses. For example, in metagenomic sequencing, what do a small number of reads that match a particular organism say about the probability that the organism is actually present in the sample? New methods are needed to assess the likelihood that an organism is present in a metagenomic sample and to provide confidence intervals on abundance estimates.

Bioforensics R&D is looking to invest in the development and application of mathematical models for (1) estimating the likelihood of a genome being present in a metagenomic sample, and (2) the most likely composition of a metagenomic sample including a list of genomes and their relative abundance. The system should go beyond metagenomic classification to provide a statistically supported estimate of sample composition that could be used in a biothreat agent detection context.”

Based on our review, we found that analysis of metagenomics data sets will rely on advanced bioinformatics analyses that involve a large statistical component. However, forensic casework may involve mixtures, and separating these into individual components may be difficult—a problem that may also apply to metagenomics. In this regard, challenges involve the development and applications of appropriate bioinformatics and data that will provide the ability to not only to describe the relative abundance of sequence data but also to make inferences using those data to provide either an investigative lead or to support attribution. Efforts to achieve this are complex and will be conducted over multiple years. For example, according to NBACC’s 2015 annual plan, “a sequence-based, bioinformatics-driven genomics approach is a complex endeavor that requires incremental implementation of critical technologies over multiple years.”

Regarding proteomics, challenges remain in interpreting data. For example, according to experts at our meeting, a quantitative measure for proteomics needs to be available so that an informed decision can be made. However, this is complicated by the lack of a framework for expressing confidence in a result. Further, related to data analysis and interpretation, for example, the potential for rapidly expanding protein databases to result in false matches exists and the lack of standardized approaches to proteomic data analysis is problematic.

DHS is Challenged by Data Quality and Accessing Agent Strains for the BRC

DHS must address several challenges related to its reference materials that could affect NBFAC’s comparative analysis of evidentiary samples. According to DHS’s responses to our questions, these include access to reference strains of interest, international agents, and ensuring the quality of the data in the BRC. During our review we found that in contrast to human DNA—a single species—the challenges for bioforensics involve a multitude of species. Further, the quality of the data entered into a particular database, including the metadata, and whether the database is kept up to date could affect analysis if NBFAC uses that database. In addition, ensuring that agents of interest are available for comparative analyses is necessary.

Regarding the BRC specifically, not all strains are readily available and obtaining agents internationally raises issues, according to DHS responses. Further, not all researchers are willing to share their strains. As a result, DHS is working with the FBI to develop an acquisition and curation plan to expand the number of strains of interest in the BRC. A DHS official stated that replacing agent-specific assays with DNA sequencing methods will require DHS to have a comprehensive, sophisticated database, which it currently does not have. Therefore, ensuring the usefulness and quality of its reference collection and its ability to obtain the strains of interest will continue to be a challenge for DHS.

Experts Identified Two Key Challenges to Enhancing Bioforensics Capabilities

Experts at our meeting and others we interviewed identified two key challenges associated with enhancing bioforensics capabilities: (1) accessing and maintaining quality data on global microbial species in databases and (2) implementing statistical frameworks and acceptable communications of statistical analyses in court.

Reference databases and quality data: Experts and officials in both the United States and the United Kingdom whom we interviewed had differing opinions about the challenges associated with obtaining access to global microbial species and maintaining quality data for comparative analyses of samples. For example, some indicated the need to establish and maintain and a central database, whereas others considered it necessary only to have the ability to access one that is relevant when an incident occurs. Experts at our meeting also expressed reservations about whether a centralized system is the best solution. They stated that it is possible that a hybrid system in which each organization would own its own dataset but would allow it to be searched by others might be a possible solution. Alternatively, organizations could be required to send their data to a central database in addition to storing it locally.

Questions have been raised about how reference data can be used effectively for bioforensics. For example, regarding the use of population genetics, it has been observed in the literature that a more useful database for each pathogen would consist of a detailed record of human and enzootic outbreaks noted through international outbreak surveillance systems, and “representative” genetic sequences from each outbreak. Another expert suggested that if necessary one could individually forage and collect organisms of interest in relevant areas or countries. Further, according to these experts, much is known about Bacillus anthracis, but other organisms like Burkholderia are much more challenging. To solve this problem would involve first going back to close the gaps in the reference databases and the population genetics.

Another challenge involved concerns about managing the quality of the data entered into the database to ensure it meets quality standards. Regarding the quality of such references for bioforensics, standards are needed for data repositories and reference collections of pathogens and other microorganisms, according to the experts at our meeting. Also, because of the uncertainty about the reference data, the meeting’s experts stated that the raw data should be maintained for further analyses. In addition, according to these experts, questions about the meaning of the data for these applications and the confidence value of the data need to be resolved before focusing on them for bioforensics purposes. According to a U.K. official we interviewed, the level of uncertainty in matching microbes cannot be quantified, and attribution depends on a reference set, which is incomplete for microbes. It could be concluded that the microbe in question has the same DNA as that of a microbe in the reference database, but not with certainty that it would not match another microbe that is not in the reference database. Because of the limits of using one approach, it is important to also use traditional forensics to build an evidence base. In court, traditional forensics, in addition to expert testimony on bioforensics, would therefore be used for attribution.

Statistical frameworks and communicating results: As we reported in 2014, a statistical framework allows for statistically meaningful comparative analyses; it is a set of concepts and organizing principles that support the compilation and presentation of a set of statistics. Experts at our meeting expressed the view that a gap that permeates science, capabilities, and bioinformatics is the lack of a formulation or framework for expressing confidence in genomics results as well as similar challenges with non-genetic results. This is especially true with mixed, metagenomics samples. Further, how to combine or communicate uncertainties and error rates associated with the analytical and collection processes needs greater clarity, according to the experts at our meeting. They stated that to have a more robust statistical foundation, it is critical to do enough experiments to assess the various contributions of these sources of variability. NBFAC is moving into the realm of metagenomics, which has problems with the statistical unknowns associated with it. Metagenomic samples may contain mixtures as we stated previously. In this regard, these experts stated that the problem of mixtures is an opportunity for statistical methods to improve the results for different kinds of evidence. For example, is the evidence confirmatory; is it consistent with what’s in the database, and what are some possible alternatives that could have given rise to the evidence? In addition, these experts stated that challenges in a bioforensics context include the need for a quantitative measure for genetics, proteomics, or other methods so that an informed decision can be made.

Communicating results using statistical probabilities may not always be acceptable by courts, despite the need for statistical frameworks to assist in interpreting bioforensics analyses. Even if accepted, such statistical information may not be understood. This issue is important because statistics could play a large part in some types of analyses. For example, using statistics when communicating the results of human DNA analysis are generally accepted by courts in the United States. For interpretation of bioforensic results, according to the experts at our meeting, the question should be: What is the confidence you have achieved with the data or information that you have? How that confidence is communicated is important. It is forensic evidence, a piece of the puzzle. It adds value, but the confidence for it may be low, whereas the confidence for some other evidence may be high. The level of uncertainty for that result should also be indicated. Thus, bioforensic analyses of microbial DNA, and its associated statistical elements, may have to overcome many obstacles before they reach a similar level of acceptance in a U.S. federal court.

Other experts have acknowledged such challenges. For example, regarding the U.S. legal system, issues may arise when new methods are applied for the first time in bioforensics that have not undergone the depth of scrutiny undergone by traditional forensic techniques. Also, reference databases used for comparisons take time to develop. Our interviews with U.K. scientists and government officials provided some insights into issues associated with human DNA analysis results, which are generally accepted by courts and that could have implications for bioforensics in both the United Kingdom and the United States. For example, regarding the communication of probability data, according to a U.K. official we interviewed, while academics say there is some set level of probability to achieve “beyond a reasonable doubt,” the court is concerned with the baseline probability. However, this official stated that human DNA is the only science in which the baseline probability data is considered incontrovertible. Further, he said that in almost every other science, the legal system would want assurances from expert witnesses regarding the analysis results—not the numerical scientific results themselves.

In light of the discussion above, it is not clear how long it will take for the results of metagenomics and proteomics analyses to be acceptable to courts. Nevertheless, what is clear is that the ability to quantify statistical uncertainty will require the use of comprehensive databases that contain characteristics of signatures and information on the variations in the population of the agent in question.

The Evolving Threat Landscape Raises Questions about the Bioforensics Capabilities That Will Be Needed

A long-term challenge facing NBFAC, according to DHS’s responses to our questions, is the increasingly complex biological threat landscape: New infectious disease agents emerge every year, and advances in genetic engineering and “do it yourself” biology methods make the nefarious use of enhanced and biological agents a possibility. DHS further responded that as a result NBFAC must regularly establish new methods and assays to support bioforensics casework that may involve future threats. Further, we found it is still challenging to distinguish between a natural and a deliberately released organism. However, according to the experts at our meeting, when using “natural,” “accidental” and “deliberate,” the issue could be more to do with the means by which an agent is used, or about the characteristics of the agent itself. Determining intent is likely to rely on information beyond the science alone. In addition, while epidemiologic tools determine whether something is unusual, what is also needed is a defined and validated tool that will determine whether a microbe is unusual, made by humans, cultured, or engineered.

Although DHS is developing capabilities to detect manipulated agents, it faces several challenges related to the perceived potential for the creation of agents that could cause harm accidentally or intentionally. To identify and characterize novel synthetic agents, these challenges go beyond identifying changes in an agent’s genomics (such as its antibiotic resistance). DHS, in its responses to our questions, stated that identifying more complex traits is more difficult because of the current scientific understanding of how these processes work at the molecular level. DHS also responded that both genetic engineering and NBFAC’s ability to infer its intended effects require a deeper understanding of the physiology of the biological agent as well as its interaction with a human host. Concerns have also been raised about the potential for gain of function research to result in manipulation of microbial agents with the potential for causing harm. Such manipulations could involve, for example, agents or toxins in which harmful consequences have been enhanced, such as making them antibiotic resistant, more virulent, or more transmissible to humans. The use of CRISPR Cas-9 also raises other issues because it may be more difficult to detect than were previous gene editing approaches. However, not all agree that the risk of possible misuses of biology is significant.

Regarding the use of genetically engineered agents to cause harm and the likelihood of this becoming a problem, we found differences in the views of the experts we spoke to both here and in the United Kingdom. Some said that it is difficult to create new pathogens so the use of naturally-occurring microbes is of the greatest concern. While acknowledging there are many technically possible misuses of biology, they concluded that it is far more likely that minor modifications would be made to existing organisms rather than the creation of new ones.

According to an expert we contacted, developing a new microbe with novel pathogenic characteristics or antibiotic resistance significantly more difficult than introducing these characteristics by gene manipulation. Thus, one of the challenges DHS faces is to consider the risks in relation to not only the bioforensics capabilities it needs but also its strategy for addressing current and potential threats.

Conclusions

DHS has identified some bioforensics capability gaps since 2010 using an informal, undocumented process but has not systematically identified the gaps or performed a bioforensics capability gap analysis. In the absence of a bioforensics gap analysis demonstrating the existence of gaps, it is difficult to determine whether DHS has identified all its capability needs and gaps. Identifying gaps and prioritizing bioforensics capability needs and gaps can help guide the proper allocation of resources to the highest priority needs. Therefore, without a capability gap analysis and documentation of the results of its process for identifying gaps, the rationale for DHS’s resource allocations and its plans for future enhancements to its existing capabilities are not clear.

Recommendation for Executive Action

We recommend that the Secretary of Homeland Security—in consultation with the Federal Bureau of Investigation—conduct a formal bioforensics capability gap analysis to identify scientific and technical gaps and needs in bioforensics capabilities to help guide current and future bioforensics investments and update its analysis periodically.

Agency Comments and Our Evaluation

We provided a draft of this report for review and comment to DHS and the FBI. DHS provided written comments, which are reproduced in appendix IV. DHS concurred with our recommendation. The FBI did not provide comments. Neither DHS nor the FBI provided technical comments. In its response, DHS described actions it plans to take to address the recommendation. Specifically, according to DHS, S&T's Homeland Security Advanced Research and Projects Agency's Chemical and Biological Defense (CBD) Division has initiated a formal, well- documented capability analysis of its Bioforensics R&D program. Further, DHS stated that CBD will leverage this analysis to conduct a parallel capability analysis of the Chemical Forensics and Attribution program that addresses similar analytical and attribution needs for chemical threat agents. DHS states that the CBD Division staff has prepared newly updated Operational Requirements Documents and Strategic Plans (Fiscal Years 2017-2021) for both programs, although we have not reviewed these documents. According to DHS, the CBD Division initially identified and compiled a number of bioforensics capability needs from a review of external programs and meetings with end-users, such as the FBI, and it is identifying and grouping additional needs under three areas (science, technology and methods, and bioinformatics and data) through reviews of documents, such as the National Research Council, Science Needs for Microbial Forensics, 2014, and GAO’s report, Anthrax: Agency Approaches to Validation and Statistical Analyses Could be Improved (GAO-15-80), among others. According to DHS, the CBD Division is conducting the formal capabilities analysis using methods and best practices identified in the documents that include the DHS Instruction Manual 107-01-001-01, DHS Manual for the Operation of Joint Requirements Integration and Management System, April 21, 2016; DHS S&T "Requirements Development Guide" April 2008; and GAO’s reports, Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research (GAO-11-176) and Chemical, Biological, Radiological, and Nuclear Risk Assessments: DHS Should Establish More Specific Guidance for Their Use (GAO-12-272). Finally, according to DHS, the CBD Division is consolidating and prioritizing these needs to ensure that they are in alignment and harmonized with current research goals and strategic plans within DHS, S&T, Homeland Security Advanced Research and Projects Agency, and the CBD Division. DHS plans to complete these efforts by June 30, 2017, and states that the CBD Division will ensure that the formal analysis is updated on an annual basis and is used to guide current and future bioforensics investments.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Homeland Security and the Director of the FBI, appropriate congressional committees, and other interested parties. The report is also available at no charge on GAO’s website at http://www.gao.gov.

If you and your staff have any questions about this report, please contact Timothy M. Persons, Ph.D. at (202) 512-6412 or personst@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs may be found on the last page of this report. Key contributors the report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

For this report, we evaluated (1) the extent to which DHS and the FBI have identified gaps in their bioforensics capabilities since 2010, (2) bioforensics needs experts have identified, and (3) any actions DHS and the FBI have taken to enhance their bioforensics capabilities, including those for characterizing a novel synthetic biological weapon, and any challenges they have experienced in enhancing bioforensics capabilities.

To determine the extent to which DHS and the FBI have identified gaps in their bioforensics capabilities, we reviewed agency documents and interviewed relevant agency officials about their efforts to identify such gaps since 2010, which is when the Department of Justice closed the FBI’s investigation into the 2010 anthrax case. We examined agency planning documents, such as DHS’s Strategic Plan 2015-2019 and NBFAC’s Bioforensics Roadmap for research, among others. We reviewed DHS policy and guidance, such as DHS’s Joint Requirements Integration and Management System, which formed the basis for the criteria we used to compare and assess the extent to which DHS had identified capability gaps or conducted a capability gap analysis of its bioforensics capabilities. We also interviewed agency officials, including those with DOD, to determine whether any gaps had been identified that related to bioforensics and their interactions with DHS in this regard.

We developed a list of bioforensics needs that experts had identified. To do this, we identified capabilities that might be needed for bioforensics purposes from a 2014 NRC publication entitled Science Needs for Microbial Forensics: Developing Initial International Research Priorities and the 2009 National Research and Development Strategy for Microbial Forensics from the National Science and Technology Council (NSTC). We excluded capability needs in the literature that were not related to science and technology development as these would have been beyond our scope. We grouped the remaining capability needs into three broad areas: (1) science, (2) technologies and methods, and (3) bioinformatics and data. We then convened, with the assistance of the National Academy of Sciences (NAS), a 2-day meeting of 16 experts to discuss and update the capability needs we identified, including identifying issues related to these needs. To identify the experts appropriate for the meeting, we worked iteratively with NAS staff to identify and review biographical information and relevant qualifications of experts, as well as factors such as representation from academia, industry, and expertise in a range of areas. The Board on Life Sciences of NAS solicited nominations for the expert panel from its extensive contacts in the microbial forensics area. From this initial list, NAS selected experts based on their knowledge and expertise in forensics, microbiology, molecular genetics, non-genetic methods, genetic engineering, bioinformatics, statistics, and legal issues related to bioforensics. Once we came to agreement with NAS on the final list of 16 experts for the meeting, these experts were evaluated for any conflicts of interest. A conflict of interest was considered to be any current financial or other interest that might conflict with the service of an individual because it (1) could impair objectivity and (2) could create an unfair competitive advantage for any person or organization. We discussed internally all potential conflicts. The experts were determined to be free of conflicts of interest, and the group as a whole was judged to have no inappropriate biases. See appendix III for a list of the experts. The meeting was recorded and transcribed to ensure that we accurately captured the experts’ statements, and we reviewed and analyzed the transcripts as a source of evidence. We developed the session topics based on our researchable objectives and issues that were identified in our audit work. The session topics were gaps in the science underpinning bioforensics capabilities, gaps in capabilities (technologies) and methods for attribution, and gaps in bioinformatics, data and statistical Interpretation of bioforensics. We subsequently obtained their comments on the list of capability needs identified during the April 2016 meeting to update and amend it based on their input.

To determine the actions DHS and the FBI had taken to enhance their bioforensics capabilities since 2010 and any challenges they encountered, we reviewed agency documents, including planning documents and research and development (R&D) efforts. We also examined DHS’s actions to enhance NBFAC’s capabilities for the long term as well as for the FBI’s casework. We reviewed DHS’s Broad Area Announcements (BAA) and Open Broad Area Announcements (OBAA) from 2008 to 2016. These are the mechanisms by which DHS solicits research to develop its bioforensics capabilities. We obtained details on contracted external R&D efforts. Deliverables included statistical models, SOPs, and genetic sequences from external researchers. To determine any challenges to enhancing bioforensics capabilities, we reviewed agency documentation, including planning and contract documentation, related literature, and our prior work on bioforensics. We interviewed agency officials and scientists, including those at DHS, DOD, and the FBI and obtained the opinions of experts in the United Kingdom, which collaborates with DHS and the FBI on bioforensics-related issues, as well as those in the United States regarding bioforensics-related challenges. We also discussed potential challenges with experts present at our expert meeting. We conducted site visits to national laboratories and academic institutions conducting research on bioforensics-related issues, including issues related to synthetic biology. These included discussions with DHS contractors, scientists in academia, officials from the U.K. Home Office, Public Health England at Porton Down, and scientists in academia regarding challenges related to bioforensics capabilities. We also interviewed some of the scientists involved in conducting research for DHS.

We conducted this performance audit from July 2015 to January 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: DHS R&D to Enhance Bioforensics Capabilities, 2008 – 2016

Bacterial and viral bioforensics research and development

Sequence-based approach to bioforensics analysis Whole genome approach to microbial forensics Bioforensics research R&D: Whole genome approach to microbial forensics and the identification of adaptive mutations that can have forensic utility Establishing match criteria for discriminating “difference” or “sameness” in sample comparisons Developing statistically rigorous sampling strategies to acquire spatially referenced genetic information on reservoirs of these pathogens Developing bioinformatics-based analytical tools for supporting hypotheses testing regarding pathogen origin that go beyond current phylogeny-based inferential methods and can meet forensic (legal) admissibility Develop novel techniques to culture threat agents from complex environmental samples Improve dry collection and extraction strategies for forensic samples Develop detection methods for rare variant detection in a bacterial sample using ultra- high throughput next generation sequencing technology Understand dynamics of mobile elements in select agent bacteria Develop forensic genotyping methods for select agent viruses Develop novel applications of orthogonal methods to genetic characterization of biological threat agent signatures and their sample matrices Develop biased primer set design to amplify biological threat agents from complex backgrounds Production methods for ultraclean reagents Sequence data error model for next-generation and single molecule sequencing platforms Taxonomic classification of metagenomic sequences Develop and apply mathematical models for statistical confidence measurements in metagenomic analysis Develop a procedure to transport agents from BSL-3 to BSL-2 laboratories Produce whole-genome sequencing to capture the global biodiversity of human, plant, and animal pathogens (bacterial, viral, and fungal) in support of microbial forensics analysis Development and population of comparative genomic database with pathogen sequence data at the National Center for Biotechnology Information Center Products to identify select agents including C. botulinum toxins, with high confidence Next generation and novel technologies to characterize biological threat agents (the organism, the agent, or the sample matrix) for source attribution Research on the bacterial populations of select agents with critical knowledge gaps, including C. botulinum and B. anthracis (North Africa, Middle East)

Year 2016

Appendix III: List of Experts

The names and affiliations of the experts who participated in the group meeting held April 20-21, 2016, in Washington, D.C. are as follows:

Christopher Bidwell, J.D., Senior Fellow for Nonproliferation Law and Policy at the Federation of American Scientists.

Bruce Budowle, Ph.D., Professor, Executive Director of Institute of Applied Genetics Molecular and Medical Genetics, University of North Texas Health Science Center.

Rockne Harmon, J.D. Consultant, Instructor at U.C. Davis in the Masters in Forensic Science program.

Dag Harmsen, MD, Ph.D., Professor, Head of Research, Center for Oral and Maxillofacial Surgery, Department of Periodontology, University of Munster.

Molly Isbell, Ph.D., Director of Quality Assurance and Statistical Sciences. Signature Science, LLC.

Dana Kadavy, Ph.D., Director of Biological Services, Signature Science, LLC.

Karen Kafadar, Ph.D., Commonwealth Professor and Chair of Statistics, University of Virginia.

Paul Keim, Ph.D., Regents’ Professor in Biology Cowden Endowed Chair in Microbiology Northern Arizona University’s Microbial, Genetics and Genomics Center. Northern Arizona University.

Jack Melling, Ph.D. (via phone), Consultant.

Stephen S. Morse, Ph.D., Professor, Epidemiology, Founding Director and Senior Resident Scientist, Center for Public Health Preparedness, Columbia University,

Karen Nelson, Ph.D., President, The J. Craig Venter Institute.

David Relman, MD, Thomas C. and Joan M. Merigan Professor in Medicine, and Microbiology and Immunology, Co-Director of the Center for International Security and Cooperation, Stanford University, and Chief of Infectious Diseases, the Veterans Affairs Palo Alto Health Care System.

Tom Slezak, Ph.D., Associate Program Leader for Informatics for the Global Security Program Efforts, Lawrence Livermore National Laboratory.

Stephen Turner, Ph.D., Assistant Professor of Public Health Sciences, University of Virginia School of Medicine.

Stephan Velsko, Ph.D., Senior Scientist and Associate Program Leader Lawrence Livermore National Laboratory.

Karen Wahl, Ph.D., Chemist, Pacific Northwest National Laboratory.

The comments of most of these experts represented the views of the experts themselves and not the agency, university, or company with which they are affiliated.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Timothy M. Persons, (202) 512-6412 or personst@gao.gov.

Staff Acknowledgments

In addition to the individuals named above, Sushil Sharma (Assistant Director), Pille Anvelt, James Ashley, Hazel Bailey, Amy Bowser, Caitlin Dardenne, Jack Melling, Jeff Mohr, Penny Pickett, Amber Sinclair, Maria Stattel, Elaine Vaurio, and Elizabeth Wood made key contributions to this report.