Bioforensics: DHS Needs to Conduct a Formal Capability Gap Analysis to Better Identify and Address Gaps
Published: 2017-01-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The ability to attribute the source of an intentionally released biological threat agent and quickly apprehend and prosecute the perpetrator is essential to our nation's safety. However, questions remain about whether DHS's and the FBI's capabilities have improved since the 2001 anthrax attack. GAO was asked to report on DHS's and the FBI's bioforensics capabilities.
This report examines the (1) extent to which DHS and the FBI have identified gaps in their bioforensics capabilities since 2010, (2) bioforensics needs experts have identified, and (3) actions, if any, DHS and the FBI have taken to enhance their ability to attribute the source of a biological attack, and to identify any challenges to enhancing bioforensics capabilities. GAO's review focused on the agencies' efforts since 2010, when the FBI's investigation of the 2001 anthrax attack was closed. GAO analyzed relevant agency documents and interviewed agency officials and scientists on issues related to bioforensics. GAO also convened a meeting of experts with NAS's assistance to discuss potential bioforensics needs.
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) and the Federal Bureau of Investigation (FBI) have identified some gaps in their bioforensics capabilities, but DHS has not performed a formal bioforensics capability gap analysis. It is therefore not clear whether DHS and the FBI have identified all of their capability gaps. A capability gap analysis can help identify deficiencies in capabilities and can help support the validation and prioritization of how to address the gaps. DHS and the FBI have identified capability gaps using an informal undocumented process. For example, DHS held informal meetings to seek FBI input on capability gaps associated with recent casework. Gaps identified through this informal process include the inability to (1) characterize unique, novel, and engineered agents and “unknowns” (emerging or synthetic organisms) and (2) understand and communicate uncertainty associated with analyzing complex biological samples, among other things. In the absence of a well-documented bioforensics capability gap analysis, the rationale for DHS's resource allocations, or its plans for future enhancements to existing capabilities are not clear and thus cannot ensure that resources are being targeted to the highest priority gaps.
In addition to DHS and the FBI, other organizations, such as the National Research Council (NRC) of the National Academy of Sciences (NAS), and the National Science and Technology Council (NSTC) of the Office of Science and Technology Policy (OSTP), have identified potential bioforensics capability needs. These needs can generally be grouped into three areas: science, technology and methods, and bioinformatics and data. GAO also convened a meeting of experts, with the help of NAS, and these experts updated a list of potential bioforensics capability needs that NAS and OSTP had previously identified within each of these areas. Some of the needs these experts confirmed as still relevant were similar to those DHS and FBI officials have identified, while others were different. For example, like DHS and the FBI, the experts agreed that an ability to characterize genetically engineered agents was needed, but they also suggested that evaluating existing protocols, such as those for DNA sequencing, to determine whether they were validated, was needed. GAO believes that this information may be helpful to DHS and the FBI as part of any future bioforensics capability gap analysis they undertake.
Since 2010, DHS has enhanced some of its bioforensics capabilities, with FBI input, by focusing on developing methods-based capabilities while maintaining agent-based capabilities. DHS has funded research and development projects addressing areas such as genome sequencing approaches, which underpin many methods-based bioforensics capabilities. DHS is also developing an in-house reference collection for use in investigations. In addition, DHS is developing the ability to characterize unique, novel agents as well as “unknowns,” such as synthetic organisms. DHS projects that some enhanced capabilities will be complete in about 2025. However, in pursuing enhancements, DHS faces several challenges, including establishing a statistical framework for interpreting bioforensics analyses and associated inferences and communicating them in a court setting, as well as obtaining suitable biological agents and DNA sequences to ensure quality references for use in investigations.
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS—in consultation with the FBI—conduct a formal bioforensics capability gap analysis and update it periodically. DHS concurred with GAO's recommendation.

section_title: Letter
paragraphs: According to the Department of Homeland Security (DHS), the threat of a  terrorist or criminal use of pathogenic organisms and their toxins is a  great concern in the United States. Further, with the advent of synthetic  biology, concerns have been raised that more virulent genetically  engineered pathogens may be created and that completely new  pathogens are being created synthetically, to be used nefariously. While  not all are convinced that such threats will result in an attack, DHS’s  mission requires it to be prepared for one. Therefore, an ability to attribute  the source of a biological attack and quickly apprehend and prosecute the  perpetrator is essential to our nation’s safety. However, concerns have  been raised about whether that ability has improved since the 2001 case  concerning the anthrax-contaminated letters sent through the postal  facilities to members of Congress and the media (Amerithrax), in which  attribution took several years.  DHS and the Federal Bureau of  Investigation (FBI) continue to play a role not only in responding to  bioterrorism but also biocrimes, including the FBI’s investigation of  multiple biocrimes involving the use of ricin—one of the most poisonous,  naturally occurring substances—such as a case in 2013 in which ricin  was sent to the U.S. President.
Attribution relies on many facets of an investigation—one of which is  bioforensics. In its recommended guidelines for laboratories engaged in  microbial forensic analyses, the Scientific Working Group on Microbial  Genetics and Forensics (SWGMGF) defines attribution as “the  information obtained on the identification or source of a material to the  degree that it can be ascertained.” The FBI has also described attribution  as “the act of attributing, the ascribing of a crime or act of terrorism to an  actor.” It requires the totality of investigative information—one aspect of  which may be forensic evidence—according to the FBI. However, when  using scientific analyses in an investigation, “scientific attribution” has  been defined as the assignment of a sample of questioned origin to a  source, or sources of known origin, to the highest possible degree of  scientific certainty—while excluding origination from other sources.  Therefore, attribution to a perpetrator—the ultimate goal—is likely to be  supported by both traditional forensics (for example, fingerprints) and  bioforensics.
Attribution for the 2001 anthrax case took about 9 years. On February 19,  2010, the FBI announced that that it was closing the case, having  concluded that a scientist at the United States Army Medical Research  Institute for Infectious Diseases had perpetrated the attack alone. The  scientific analyses were a key investigative lead, according to the FBI.
However, experts have noted that U.S. bioforensics capabilities at that  time were “initially limited to detection and identification and did not  include detailed characterization and comparative analyses.” Therefore,  because the case was never tried in a court of law, it is not clear whether  the scientific analyses the FBI conducted were sound enough to support  its conclusions.
In 2008, the FBI asked the National Research Council (NRC) of the  National Academy of Sciences (NAS) to review the scientific approaches  it had used to support is conclusions. NRC issued its report in 2011. NRC  found that “it is not possible to reach a definitive conclusion about the  origins of the B. anthracis in the mailing based on the available scientific  evidence alone,” and it also details many methodological and  organization problems in the scientific portion of the FBI’s investigation.  Our prior work also revealed several gaps in the FBI’s validation of a key  set of genetic assays for the 2001 investigation as well as challenges  related to characterizing microbial deoxyribonucleic acid (DNA) in an  effort to identify its source—which could be important in future  investigations. Specifically, we reported on the need for a statistical  framework for analyzing the results the assays generated. We also  identified challenges related to the use of signatures, or genetic markers,  and their significance in an investigation.
Agency responsibilities changed after the 2001 anthrax attack with DHS’s  establishment. Today, DHS and the FBI coordinate tasks to attribute the  source of a released biological agent—while the FBI alone investigates  actual criminal acts and DOJ prosecutes alleged perpetrators. DHS’s  National Bioforensics Analysis Center (NBFAC)—a dedicated  bioforensics laboratory—is responsible for analyzing evidentiary samples  in a bioforensics investigation.
In addition, capabilities that could enhance bioforensics continue to  evolve. The Bioforensics Research and Development (R&D) program  supports both customers within DHS (for example, Customs and Border  Patrol) as well as other agencies, including the FBI, and collaborates with  federal and international partners. DHS has funded R&D of new  technologies and associated processes that are intended to achieve a  range of homeland security goals, including those for NBFAC’s  bioforensics analyses for FBI casework. These include the ability to  examine biological agents and nonbiological materials submitted as  evidence to provide relevant information and intelligence that supports an  investigation. DHS also works with the FBI and others to identify potential  gaps in NBFAC’s bioforensics capabilities, in an attempt to ensure that it  is prepared for FBI’s casework needs.
In this context, you asked us to evaluate issues related to the status of  DHS’s and the FBI’s bioforensics capabilities for attributing a biological  attack and whether those capabilities have any scientific and technical  gaps. For this report, we evaluated (1) the extent to which DHS and the  FBI have identified gaps in their bioforensics capabilities since 2010,   (2) bioforensics needs experts have identified, and (3) any actions DHS  and the FBI have taken to enhance their bioforensics capabilities,  including those for characterizing a novel synthetic biological weapon,  and any challenges they have experienced in enhancing bioforensics  capabilities.
To determine the extent to which DHS and the FBI have identified gaps in  their bioforensics capabilities, we reviewed agency documents and  interviewed relevant agency officials about their efforts to identify such  gaps since 2010, when the Department of Justice closed the FBI’s  investigation of the 2001 anthrax attack. We examined agency planning  documents, such as DHS’s Strategic Plan 2015-2019 and NBFAC’s  Bioforensics Roadmap for research, among others. We reviewed DHS  policy and guidance, such as DHS’s Joint Requirements Integration and  Management System, which formed the basis for the criteria we used to  compare and assess the extent to which DHS had identified capability  gaps or conducted a capability gap analysis of its bioforensics  capabilities.
To develop a list of bioforensics needs that experts have identified, we  identified capabilities that might be needed for bioforensics purposes from  a 2014 NRC publication entitled Science Needs for Microbial Forensics:  Developing Initial International Research Priorities and the 2009 National  Research and Development Strategy for Microbial Forensics from the  National Science and Technology Council (NSTC). We extracted potential  capability needs from these publications. We grouped the remaining  bioforensics capability needs into three broad areas: (1) science,   (2) technologies and methods, and (3) bioinformatics and data. We then  convened, with the assistance of the National Academy of Sciences, a  meeting of experts to discuss and update the capability needs we  identified, including identifying issues related to these needs. These  experts represented industry, academia, and government, and had  experience in, among other things, microbiology, molecular genetics, non- genetic methods, genetic engineering, bioinformatics and statistics, and  legal issues related to bioforensics. We also interviewed agency officials,  including those with DOD to determine whether any gaps had been  identified that related to bioforensics and their interactions with DHS in  this regard.
To determine the actions DHS and the FBI have taken to enhance their  bioforensics capabilities since 2010 and any challenges they  encountered, we reviewed agency documents, including planning  documents and R&D efforts. We also examined DHS’s actions to  enhance NBFAC’s capabilities for the long term as well as the FBI’s  casework needs. We reviewed DHS’s Broad Area Announcements (BAA)  and Open Broad Area Announcements (OBAA) from 2008 to 2016. These  are the mechanisms by which DHS solicits research to develop its  bioforensics capabilities. We obtained details on contracted external R&D  efforts. These included statistical models, standard operating procedures  (SOP), and genetic sequences from external researchers. To determine  any challenges to enhancing bioforensics capabilities, we reviewed  agency documentation, related literature, and our prior work on  bioforensics. We also interviewed agency officials and scientists, and  obtained the opinions of experts in the United Kingdom, which  collaborates with DHS and the FBI on bioforensics-related issues, as well  as those in the United States, including those at our expert meeting.  These included officials from the U.K. Home Office, Public Health  England at Porton Down, and academia, regarding challenges related to  bioforensics capabilities, including synthetic biology.
We conducted this performance audit from July 2015 to January 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DHS coordinates the federal government’s overall response to or  recovery from terrorist attacks. The Centers for Disease Control and  Prevention (CDC) within the U.S. Department of Health and Human  Services is the primary agency for the public health response to a  biological terrorism attack or naturally occurring outbreak. The FBI  within DOJ is the primary agency for the criminal investigation of incidents  of bioterrorism. In its recommended guidelines for laboratories engaged  in microbial forensic analyses, the Scientific Working Group on Microbial  Genetics and Forensics (SWGMGF) defines attribution as “the  information obtained regarding the identification or source of a material to  the degree that it can be ascertained.”
As part of the effort to deter biological terrorism and strengthen the law  enforcement response to such an act, Homeland Security Presidential  Directive (HSPD) 10, “Biodefense for the 21st Century” established within  DHS a dedicated central microbial forensic laboratory known as the  NBFAC to provide bioforensics analysis of evidence associated with the  event. This Presidential Directive established the NBFAC as “the lead  federal facility to conduct and facilitate technical forensic analysis and  interpretation of materials recovered from biocrime and bioterror  investigations in support of the appropriate lead federal agency.”
DHS Science and Technology (S&T) is to accelerate the delivery of  enhanced technological capabilities to meet the requirements and fill  capability gaps to support DHS agencies in accomplishing their mission.  Pursuant to this mission the DHS Chemical and Biological Defense (CBD)  Division seeks technologies to defend against a chemical and biological  attack. In addition, the division is charged with pursuing research to  improve response and restoration, conduct threat risk assessments, and  invest in bioforensics R&D. In this regard, the Bioforensics R&D Program,  according to DHS, supports NBFAC operational threat agent identification  and characterization through investments in bioforensics research and  next generation technologies to include molecular biology, genomic  comparison techniques, genotyping assays and physical and chemical  analysis of sample matrix to better understand the origin, evolutionary  history, production method and dissemination mechanism associated with  the malicious use of biological agents.
Bioforensics has been defined as an interdisciplinary field of microbiology  devoted to the development, evaluation, validation, and application of  methods to detect and fully characterize microbial samples containing a  biological agent or its components for the purpose of making statistically  meaningful comparative analyses. Attributing something to a perpetrator  requires different types of information and analysis—both traditional and  bioforensics. Information produced by forensic examination can result in  an investigative lead or provide support for the investigation.
Bioforensics capabilities used to provide analyses of evidence may show  how, when, and where microorganisms were grown and potential  methods for dissemination, which assists attribution.
Bioforensics evidence could include the agent that was released,  toxins, nucleic acids, and protein signatures. It could also include  contaminants, additives, and evidence of preparation methods.
Traditional evidence could include fingerprints, hair, fibers,  documents, photos, firearms, and body fluids.
In a bioforensics case, the intent would likely be to gather sufficient  information to allow a comparison of an evidentiary sample with a  known reference sample to assist in supporting source attribution.  Evidence from a bioforensics investigation must also meet the  scientific community’s standards for evidence as well as a criminal  court’s standards for legal admissibility.
section_title: DHS Has Not Performed a Capability Gap Analysis to Help Focus Its Resources for Addressing Bioforensics Capability Gaps
paragraphs: DHS has developed strategic plans and goals related to bioforensics  attribution and identified some key bioforensics capability needs.  However, according to DHS officials, DHS did not perform a bioforensics  capability gap analysis, but, rather, used an informal approach to identify  capability needs and gaps. DHS officials stated that they did not  document the process DHS used or the results of its informal approach.  Further, DHS officials told us that there is not a complete list of the gaps  identified using the informal approach. Finally, although they indicated  that DHS had focused resources toward addressing those gaps, they  could not provide documentation of the bioforensics capability  requirements and other relevant information to support the capability gap  identification and resource allocation decisions that were made. In the  absence of documentation of the processes, discussions, analyses,  decisions, or any other activities performed to identify and prioritize  bioforensics gaps, DHS’s rationale for the identification and prioritization  of needs and gaps on which to focus its resources is unclear.
subsections: section_title: DHS Did Not Use a Systematic, Documented Approach to Identify Its Bioforensics Capability Needs and Gaps
paragraphs: According to DHS officials, DHS relies on the DHS S&T-managed  NBFAC and bioforensics R&D programs to identify bioforensics capability  needs and gaps. However, DHS does not have a complete list of its  bioforensics capability gaps because it has not performed a bioforensics  capability gap analysis. According to the DHS Systems Engineering Life  Cycle guide, a gap analysis is a best practice that is essential to  understanding whether capabilities exist that can meet requirements, or if  they must be developed.  In addition, some DHS officials told us that  performing a capability gap analysis is a best practice that DHS programs  should follow, even in the absence of DHS guidance to do so. In  interviews and written responses, DHS officials described generally how  DHS identified and documented capability needs and gaps. They told us  that they identify priorities each fiscal year, develop projects to meet  these priorities, and develop the NBFAC Annual Plan to address these priorities. However, they told us that there is no documentation of the  process or results of the informal approach they used to identify  bioforensics capability needs and gaps. According to federal standards  for internal control, documentation is necessary for the effective design,  implementation, and operation of an entity’s internal control system.  Lacking documentation of the processes, discussions, analyses,  decisions, or any other activities performed to identify and prioritize  capability gaps, it is unclear what DHS’s rationale was for the  identification and prioritization of bioforensics capability needs and gaps.  Identifying  and prioritizing capability gaps enables the proper allocation of  resources to the highest priority needs. Thus, without a capability gap  analysis, DHS may not have identified and prioritized all capability needs  and gaps, and so may not be allocating resources to address the most  significant gaps to meet its mission needs.
DHS officials told us that no complete list of bioforensics capability gaps  has been created since 2010. However, they told us that DHS had  developed a document from 2013-2014—the Bioforensics Roadmap  (Roadmap)—as a means to identify and achieve consensus from  stakeholders on the key bioforensics capability needs on which to focus  DHS resources.  DHS officials said that the Roadmap lays out the  Bioforensics R&D Program execution and also lists the key needs on  which DHS has focused, or will focus, resources, along with the  associated programs to address the needs. DHS developed strategic  plans for its National Biodefense Analysis and Countermeasures Center  (NBACC) in 2012 and Chemical and Biological Defense in 2013.   It documented NBFAC strategic goals in 2013. These documents  include strategic objectives related to bioforensics that could be used to  guide a capability gap analysis.
A former DHS official who had participated in DHS’s process for the  identification of bioforensics gaps told us that the process was informal  and also that there is no documentation of the process. This official  summarized it as generating a list of topics and issuing BAAs to address  them. Other DHS officials confirmed that the process was informal and  that there was no documentation of the results. They told us that they are  unaware of the details of the processes and activities performed to  identify capability needs and gaps. However, they did describe generally  the informal process that DHS used.
They indicated that the process included working with key interagency  partners and other stakeholders—such as the FBI—and engaging in  discussions, exchanging emails, and holding periodic meetings. DHS  officials also stated that they met informally as needed with the FBI and  some intelligence agencies to discuss needs and gaps. These officials  explained that the discussions with DHS’ interagency partners were part  of a larger process to develop and manage NBFAC and bioforensics R&D  programs. They said that DHS coordinated with the FBI and the  Intelligence Community to focus these programs’ activities to meet the  needs of these end users. Further, they said that the Roadmap was  vetted by other agencies and researchers. In addition, according to FBI  officials, the FBI conducted assessments of its capabilities by working  with the DHS S&T and providing direction to DHS about its capability  needs. FBI officials stated that sometimes the FBI does not know there is  a bioforensics capability gap until it encounters one during an  investigation.
Independent  assessments of DHS’s S&T Bioforensic R&D program have  raised similar concerns about how DHS has identified and prioritized  bioforensics capability gaps. For example, external assessments of the  CBD portfolio from 2012 and 2014 found a lack of clarity about how the  Bioforensics R&D program identified and prioritized capability gaps and  why some projects were chosen. The reviewers recommended that the  program manager describe the program’s basis for identifying capability  gaps and selecting projects in future reviews. Specifically, a November  2014 review acknowledged the need for enhancing bioforensics  capabilities but questioned the lack of information on how gaps in  capability or knowledge guiding R&D investments were identified and  prioritized. A 2012 review stated that it was unclear why some research  studies were chosen over others, as well as how the selection of projects  was linked to, or justified against, the risk assessment.
section_title: DHS and the FBI Have Identified Several Bioforensics Capability Needs and Gaps
paragraphs: Through the Roadmap, key bioforensics efforts have been identified,  which DHS officials characterized as gaps. The efforts listed in the  Roadmap include (1) operational infrastructure, (2) sample collection and  preservation, (3) sample extraction, (4) identification and characterization,  and (5) data analysis and integration. The Roadmap also includes  existing and future CBD and other agency programs, as well as  commercial development, linking them to the particular capability gap  they address. The Identification  and Characterization effort in the  Roadmap includes developing capabilities to characterize unique, novel,  and engineered agents; characterize unknowns (emerging or synthetic  organisms); identify and characterize toxins, such as ricin; and quantify  and communicate uncertainty, which is of particular significance when  using metagenomics and proteomics capabilities. A DHS official and FBI  officials also told us that other items they considered to be gaps include  the difficulties in interpreting metagenomics data, limited sequences for  select organisms in its reference database, and the need for a greater  ability to examine proteins. However, the Roadmap provides no additional  details about the bioforensics capability gaps other than the projected  timeframe of 2014 to 2020 for completion of the agency programs. Figure  1 shows the Roadmap. According to DHS officials, to bridge the broad  gap areas set out in the Roadmap would require resources far beyond  those available to DHS.
section_title: Experts Have Identified Several Capability Needs for Bioforensics
paragraphs: In addition to DHS and the FBI, other organizations, such as the NRC of  the NAS and the NSTC of the Office of Science and Technology Policy  (OSTP), have been involved in identifying bioforensics capability needs.  The NRC Committee on Science Needs for Microbial Forensics was an  international group of experts that identified scientific challenges that  must be met to improve the capability of bioforensics to investigate  suspected outbreaks and to provide evidence of sufficient quality to  support responses, legal proceedings, and the development of  government policies. Similarly, OSTP’s National Research and  Development Strategy for Microbial Forensics was established to guide  and focus U.S. Government research efforts to advance the discipline of  bioforensics. With the assistance of the NAS, we convened our own  meeting of experts in April 2016 to review and update capability needs  that the NRC and OSTP identified and to identify additional needs that  might be useful for DHS and the FBI to consider when they identify their  capability needs as part of a bioforensics capability gap analysis. Some of  the experts provided alternative views about certain aspects of the  identified capability needs. While some of the bioforensics capability  needs identified overlap with efforts listed in the DHS Roadmap, they  were not formulated specifically considering DHS requirements so may  not be relevant to DHS. However, we believe that this information could  help inform the DHS’s and FBI’s efforts to identify capability needs and  prioritize gaps.
Starting with the capability needs identified by the NAS and OSTP, the  experts that participated in the GAO meeting identified and generally  agreed upon the capability needs listed in Table 1. Some of the needs the  experts confirmed as still relevant were similar to those identified by the  DHS and the FBI, and some were different. For example, like DHS and  the FBI the experts agreed that an ability to characterize genetically  engineered agents was needed, but they also suggested that evaluating  existing protocols, such as those for DNA sequencing, to determine  whether they were validated, was needed. The identified needs in table 1  can generally be grouped into three broad areas: (1) science,   (2) technology and methods, and (3) bioinformatics and data. There are  six needed capabilities within the science area; five within technology and  methods; and three within the bioinformatics and data areas.
While the majority of the experts agreed generally with the bioforensics  capability needs in the three broad areas listed in table 1, some experts  also had alternative views about some of the needs. For example, some  experts thought that some of the needs should have a different focus or  should be given a lower priority than others. In addition, some experts  suggested that because it may be impossible to characterize all microbes,  the first science capability need—the identification, monitoring, and  characterization of microbial species—should instead focus on   (1) developing a dynamic process and infrastructure for rapid collection  and typing when an event occurs or (2) using a species-agnostic  approach to identify both natural and synthetic microbes, such as  focusing on genetic mechanisms rather than organisms.
Additionally,  some experts stated that limited emphasis should be placed  on the third science need—developing  methods to distinguish among  natural, accidental, and deliberate outbreaks. They indicated that other  investigatory data would be available that would be better suited for  making this determination. Instead, they said the focus should be on  identifying introductions of additional virulence or genetic elements into an  organism and determining other elements that suggest that somebody  has modified the organism. There was also disagreement on which  microbes DHS and the FBI should focus. Some experts stated that,  because new pathogens are difficult to create, the greater concern is  naturally-occurring or modified microbes. Further, they also said that  distinguishing among existing organisms already presents a difficult  enough challenge. Finally, some experts said that the focus should be on  microbes in laboratories because they are the most relevant to  bioforensics and are not typically studied by the larger scientific  community.
The experts disagreed on whether the sixth science capability need— metagenomics research—was important for bioforensics. Some stated  that metagenomics is worth exploring as a future capability but there are  easier problems that need to be solved. Others said that a metagenomics  capability is not necessary for analyzing simple samples but it might be  useful for analyzing complex samples. In addition, one expert questioned  the fifth technology and methods need—the need for nongenetic  orthogonal methods—indicating that it is not a requirement in court to  have two different methods to determine a result.
For some of the bioforensics capability needs, experts indicated that other  groups would develop the capabilities so that DHS or the FBI would not  need to invest in them. For example, some experts said that the FBI  should not focus its effort on the second science need—researching the  mechanisms of pathogenicity—because it is unlikely to be closed quickly,  and other groups are already addressing it. Regarding the second  technology and methods need—adapting assay and sequencing  technologies—some experts indicated that the commercial market will  drive the development of improved sequencing technologies. Similarly,  some experts said that agencies, such as the FBI and CDC, are working  to address the first bioinformatics and capability need—the creation of  data repositories and reference collections for pathogens and other  microorganisms.
The specific bioforensics casework requirements that formed the basis for  the DHS’s efforts were not known to the experts that participated in the  meeting and, consequently, the list of capability needs cannot be directly  compared to the efforts in the Roadmap. These capability needs, along  with the alternative views presented, could help inform the DHS’ and  FBI’s efforts to identify and prioritize bioforensics capability gaps. These  agencies could consider this information as part of any capability gap  analysis.
section_title: DHS and the FBI Have Acted to Enhance Bioforensics Capabilities but Face Numerous Challenges
paragraphs: DHS and the FBI have taken actions to enhance some bioforensics  capabilities but face numerous challenges before they can achieve the  desired enhancements. Actions include not only the concrete steps that  DHS has taken to enhance its capabilities, such as funding R&D  activities, but also key strategic decisions underlying those actions. In this  context, DHS actions include (1) developing methods-based capabilities  to provide a broader bioforensics capability; (2) funding R&D activities to  enhance its capabilities; (3) developing capabilities for short-term  casework needs; (4) establishing an in-house reference database; and   (5) developing capabilities for characterizing genetically engineered and  unique, novel, or unknown (emerging or synthetic) agents. However, to  achieve the capability enhancements they are pursuing, DHS and the FBI  must overcome numerous challenges. These include (1) achieving the  ability to interpret and communicate results from the bioforensics  capabilities with a statistical confidence, (2) developing statistical  frameworks, quantitative measures, and quality reference collections,   (3) ensuring that its Bioforensic Repository Collection (BRC) contains  quality data and appropriate agent strains, and (4) determining future  casework needs relative to views of the evolving threat landscape. In  addition, experts at our meeting and those we interviewed identified  challenges regarding reference databases, the use of statistical  frameworks, and the communication of results.
subsections: section_title: DHS and the FBI are Focusing on Methods- Based Capabilities to Provide a Broader Bioforensics Capability
paragraphs: DHS has taken several actions to enhance some of NBFAC’s  bioforensics capabilities for use on FBI casework. For example, we found  that since 2010, DHS, with FBI input, made a strategic decision to focus  on the development of methods-based capabilities rather than agent- based capabilities for identifying and characterizing biological agents.  This strategy is reflected in the 2012 NBACC strategic plan and its goals  for NBFAC. Methods-based capabilities, according to DHS’s written  responses to our questions, include genomics (whole genome  sequencing and bioinformatics analysis), and analytical chemistry (mass  spectrometry and scanning and transmission electron microscopy). In  addition, DHS will maintain and enhance its agent-based capabilities in  the interim—which include molecular biology, virology, bacteriology and  toxinology—some of which will always be necessary for certain types of  casework. Both types of capabilities will reside at NBFAC.
The FBI agrees that such enhanced capabilities are needed. In  responding to our questions, the FBI stated that DHS’s approach will  provide “an adaptive and agile capability to characterize unique, novel,  engineered or emerging biological agents.” While agreeing with the need  to develop methods-based capabilities, however, the FBI also  acknowledged in its responses that some agent-specific capabilities will  always be needed for its investigations.
Methods-based capabilities, according to DHS’s written responses to our  questions, can potentially provide NBFAC with a broader bioforensics  capability. For example, DHS stated that genomic analysis can use  unique features as signatures to differentiate a particular isolate from  others. DHS also responded that such features could include single  nucleotide polymorphisms (SNP), rare variants, and epigenetic variation.  Further, DHS stated that genomics-based characterization—including the  ability to characterize background nucleic acids that may be derived from  the environment in which the sample originated—represents a unique  investigative signature that agent-based bioforensics procedures would  miss. According to experts at our meeting, signatures range from  anything that aids an investigation, to genetic signatures, syndromic  signatures, metadata, and proteins, as well as other molecular signatures.
DHS officials told us that the use of methods-based approaches, such as  genomics, have dramatically reduced investigation timeframes. They said  that DHS can now detect and sequence not only select agents, but a  number of other biological agents (even bioengineered ones) in a fraction  of the time. What took years to complete in the 2001 Amerithrax case can  now happen much more quickly with such improvements. An FBI official  further elaborated, stating that improvements in techniques and  technologies have led to potential increases in obtainable information and  significant reductions in analysis times supporting bioterrorism  investigations.
In contrast, based on our review, prior to 2010, we found that NBFAC’s  bioforensics capabilities focused on identifying biological agents on the  CDC and USDA select agent lists. In this regard, in its written responses  to our questions, DHS stated that it has established International  Organization for Standardisation (ISO) 17025 accredited, complementary  assays such as culture, real-time polymerase chain reaction (PCR), and  immunoassays for most traditional bacterial, viral, and toxin agents.  However, unlike methods-based capabilities, DHS stated that these  require prior knowledge of an organism and the maintenance of agent‐ specific reagents. Further, agent-based capabilities do not cover a wide  array of potential threats, including genetically modified or de novo  agents, and have not been developed for known human pathogens,  especially those that may not be cultivable. Thus, according to DHS  responses, a methods-based approach will ultimately provide NBFAC  with capabilities not only for analyzing challenging samples but also with  a broader, more comprehensive bioforensics capability for characterizing  unique, engineered, or emerging biological agents.
Table 2 shows these two types of capabilities and the types of analyses  they could be used to perform on evidentiary samples. For example,  bacteriology involves culturing and deriving phenotypic information on an  agent to identify and characterize it.
However, toxinology, another agent-based capability, could involve  identifying and characterizing protein toxins such as ricin using an  immunoassay—such as ELISA, an enzyme-linked immunoabsorbant  assay. In addition, analytical chemistry, a methods-based capability,  could be used to characterize toxins by using mass spectrometry, which  also supports proteomics analysis. That is, both types of capability could  be involved in analyzing toxins. According to DHS responses to our  questions, use of each of its mass spectrometry methods function  independently and provide complementary information to confirm results  derived from immunoassays and biological activity assays for protein  toxins. Electron microscopy—a methods-based capability—involves  nonbiological analysis of evidence samples. For example, it could provide  elemental analysis of an agent. Transmission electron microscopy (TEM)  and Scanning Electron Microscopy (SEM) can be used to provide images  of nonspore forming bacteria and viruses, and castor bean products,  among others.
subsections: section_title: DHS Has Funded R&D for Bioforensics
paragraphs: DHS has solicited and funded R&D projects to enhance NBFAC’s  bioforensics capabilities—completion of which DHS anticipates will  extend beyond 2025. The R&D is related to areas in which DHS has  stated there are capability gaps, or it is linked to some of the program  responses listed in the 2014 Roadmap. It also reflects DHS’s shift toward  methods-based approaches, such as genomics and proteomics.
Using a BAA mechanism, DHS solicited research proposals for R&D  related to enhancing its bioforensics capabilities. To more clearly  describe the type of research sought, the BAAs specified not only broad  topic areas as well as technical topic areas—more specific, technical  details about the type of research being solicited. Subsequently, DHS  awarded about 36 contracts for solutions or products addressing areas  related to bioforensics. According to the FBI’s response to our  questions, it is involved in the process from start to finish, including  assisting in drafting the BAA, the proposal evaluation and selection  process, and meeting with DHS and the contractors throughout the  course of the contract.
Before being used in FBI casework, SOPs and other deliverables from  the funded research would have to make a transition to NBFAC  operations and potentially be accredited under ISO 17025, as  appropriate. In responding to our questions, DHS stated that both ISO  17025 accreditation and deliverables such as publications provide the  data necessary to support the general acceptance of a method within the  scientific community and to meet the Daubert standards for admissibility  of analysis for a federal prosecution. DHS officials explained that, to the  extent possible, they publish their research results so that NBFAC’s  bioforensics techniques can be peer-reviewed, validated, and supported  in court. Figure 2 shows the broad topic areas and the years in which  research was solicited through the BAAs.
Broad and technical topic areas:  Based on our review, both the broad  topic areas in figure 2 and the underlying technical topic areas in the  BAAs reflect the long-term, methods-based enhancements, and also the  enhancements to toxin analysis capabilities for the FBI’s current  casework needs sought by DHS. For example, the following technical  topics were included as part of the 2015 solicitation for Bioforensics  research: products to identify select agents including toxins, with high next generation and novel technologies to characterize biological  threat agents for source attribution, bacterial populations of select agents with critical knowledge gaps,  including C. botulinum and B. anthracis (North Africa, Middle East), high-confidence methods for metagenomics analysis of complex  biologicals in complex samples to support whole genome sequencing,  and  informatics and statistical tools.
DHS-funded R&D:  In line with the broad topic areas indicated in the  figure—bacterial population genetics, sequence-based approach to  bioforensics, and bioforensics research—we found that DHS-funded R&D  contracts include the following areas: population genetics for forensics, biological toxin identification, metagenomics sequence data, statistical confidence in evidentiary materials based on bacterial  forensic proteomics of virus production,
Bayesian taxonomic assignment for next-generation sequencing, and sequencing-based bioforensics analyses.
R&D supports DHS’s efforts to develop methods-based capabilities,  including sequencing methods to enable genomic analysis of any  organism in any sample, as well as bioinformatics methods for de novo  assembly, metagenomic classification, comparative analysis,  identification of genetic engineering signatures, and the inference of  biological function. For example, according to DHS’s responses to our  questions,
Population genetics: research into population genetics, a 5-year  timeline project, is published in the open scientific literature, in the  sequence data in GeneBank; NBFAC used the information to better  understand the genetic diversity of the organisms studied in that  project. According to documentation we reviewed, results from such  studies will refine understanding of the population genetics of certain  select agents to better calculate match statistics in a forensic  setting.
Biological toxin characterization: Regarding toxins, DHS has  funded contracts to develop SOPs for protein toxin characterization  using mass spectrometry, among other projects.
Metagenomics sequencing: DHS research into genetic issues is  ongoing. DHS is seeking a means for future use of metagenomics  analyses on complex samples. Regarding metagenomics, DHS plans  research into high-confidence metagenomics analysis of complex  biological samples, as well as developing statistical models and  software to identify the organisms in a complex sample and estimate  their relative abundance, including developing an existing system for  probabilistic reconstruction of the taxonomic structure present in a  metagenomic sample.
Bioforensic proteomics: DHS has also funded research on  proteomics—including proteomics of virus production—and analysis  of proteins and metabolites of unknown samples to complement  genetic characterization.
Ricin  Ricin, or Ricinus communis, is one of the most  poisonous naturally occurring substances.  Ricin is derived from the beans of the castor  plant. Ricin is toxic to cells and damages all  human organs. It is considered a select agent  (toxin). No antidote is available.
DHS and the FBI are Enhancing NBFAC’s Biological Toxin Analysis  Capabilities for Current Bioforensics Casework:  Based on our review  and DHS’s responses, DHS’s primary focus is on bioforensics capabilities  in the short term to address the FBI’s current casework needs. Such  casework has involved the FBI’s investigation of multiple biocrimes  involving the use of ricin, including a case in 2013 in which ricin was sent  to the U.S. President. NBFAC analyzed some of the samples in that case,  according to the FBI’s responses to our questions.
FBI casework carried out by NBFAC involves the FBI’s transporting  evidentiary samples to NBFAC, which (1) develops a sample analysis  plan (which could involve traditional as well as bioforensics analyses) for  FBI approval, (2) conducts analyses, and (3) reports the results to law  enforcement, which uses them to inform the bioforensics investigation.
Based on our review, for prosecution in a case involving ricin, the  scientific evidence may need to establish that the toxin is present in an  evidentiary sample. We found that a combination of analytical  capabilities may be used to confirm this, with each detecting a specific  target. For example, to confirm the presence of ricin in a sample, antibody  tests, such as ELISA, and mass spectrometry can be used for detecting  the presence of ricin, and examining the protein, respectively. Added to  them can be cell-free translation assays, another type of antibody test,  which also detects ricin. We also found that NBFAC’s capabilities for  analyzing ricin toxins initially  included all the independent capabilities  above, with the exception of an accredited mass spectrometry capability  for characterizing ricin and other toxins. Based on our review, we found  that NBFAC had contracted with a laboratory to examine protein toxins by  mass spectrometry when it did not yet have that capability. Doing so,  according to their responses, resulted in a 2—3 day delay, and the  laboratory was also not accredited under ISO 17025. As a result, the FBI  further responded that it requested that DHS develop an in-house ISO  17025 accredited toxin analysis capability at NBFAC. The FBI provided  the equipment and funding for this transition to NBFAC.
Enhancing Genomics and Proteomics Is a Long-Term Effort:  According to a DHS official, DHS is continuing to enhance both genomics  and proteomics capabilities, which is expected to provide a  complementary capability that will link proteomic analysis to  metagenomics analysis of complex samples, thereby providing additional  information about an agent. Further, according to this official, genomics  and mass spectrometry will support developing metagenomics and  proteomics. Based on our review, some of the ways in which  metagenomics capabilities may be used are as follows:
Metagenomics:  It allows sampling of the genomes of microbes  without culturing them; rather, the DNA is directly isolated from the  sample before genome sequencing. A DHS official stated that DHS  plans to provide comprehensive metagenomics analysis of complex  evidentiary samples. These types of samples may contain both  microbial and human DNA as well as mixtures that derive from  possible processing steps (growth media, etc.), which could provide  links to a possible source.
In the context of metagenomics, according to a DHS official, “complex  samples may be from any environment and can be mixtures of many  types of organisms. The simplest of metagenomics samples may be  viruses in the tissue culture which contains the genomes of two  organisms, the cell line, and the virus. The most complex  metagenomic samples are from soil samples. Soil samples may  contain an organism of interest, at low concentration, but also will  likely have DNA and other biological materials from things such as  plants, animals, fungi, bacteria and viruses. The ability for the forensic  laboratory to collect metagenomic data and analyze it relies on the  development of tools for metagenomics.” In this regard, according to  an expert who we contacted, metagenomics or the evaluation of  environmental samples for genetic information is a task that may not  give DHS the returns from its investment. It is a very time consuming  technique and should probably be left to academia and/or industry.  Once these methods are developed, DHS would be able to apply the  most applicable techniques, according to this expert.
Figure 3 illustrates the possible composition of a complex sample.
Metagenomics analysis of a complex sample could reveal the  presence of DNA and other types of material at different percentages,  including eukaryotic nucleic acids. However, because evaluating  metagenomic sequence data is based on relative abundances, large  amounts of data may be generated. Interpreting these data and their  meaning in terms of an agent’s source will be necessary.
Based on our review, some of the ways in which proteomics capabilities  may be used are as follows:
Proteomics:  Proteomics is the study of proteomes. A proteome is a  set of proteins produced in an organism, system, or biological  context. In response to our questions, DHS advised us that it plans  to establish a proteomics capability for NBFAC sometime in the future.  Mass spectrometry is being used for proteomics analysis and is able  to provide information indicative of a particular protein.
While proteomics does not replace genomic analysis, it may provide  additional information if the microbial DNA is too damaged for  analysis, according to an expert who attended our April meeting. In  addition, according to this expert, there are differences in microbes  between those naturally occurring and those grown in laboratories,  including differences in growth patterns. Proteins express themselves  based on different food sources. Consequently, according to this  expert, analyzing the microbes to determine the growth medium used  could be useful for bioforensics. Further, protein profiles have the  potential to provide information on the environment that the  microorganism has experienced. For example, cultivation might  provide information about the skills of the people who grew the  organisms. Thus, proteomics provides a different level of  discrimination from that of genomics. Finally, according to another  expert who we contacted, proteomics analysis should become a  valuable tool for bioforensics and may rival genetic information when  methods have matured.
Based on DHS’s responses to our questions, achieving a genomics and  proteomics capability will also require (1) a bioinformatics and a statistical  framework for inference and analysis of unknowns in microbial isolates,  and (2) significantly expanded genome databases and an understanding  of the underlying determinants of various pathogenic traits. DHS  responded that both of these are currently funding priorities. In this  regard, DHS stated that NBFAC continues to expand a major genomics  capability that includes multiple, complementary sequencing platforms and advanced bioinformatics within high-performance computing  environment. In its responses, DHS termed this approach as “agent- agnostic” as the analytical procedures require no knowledge of which  agent might be present in a sample. However, while DHS also stated that  it provides confidence estimates for aspects of its genome sequencing  and continues its incremental development of its genomics capability, it  also acknowledges the need for statistical frameworks. For example, it  stated that “the issues regarding statistical uncertainty require the  development of statistical frameworks to ensure that attribution signatures  are clearly defined and understood; that there is standardization,  validation and verification of the signatures; that relevant source  populations are fully characterized and understood; the limitations of  measurement tools are known, and the statistical methods being used are  appropriate for the signatures data.” Finally, DHS stated in its response to  our questions, that understanding and communicating of the uncertainty  is of particular significance when using metagenomics and proteomics on  complex sample types.
Other experts we interviewed also agreed that there is a need for more  flexible bioforensics capabilities. For example, an expert from our meeting  stated that currently, characterizing an agent is achieved by using  sequence data. Learning what can be exploited for this purpose is in its  early stages. In addition, a U.K. official we interviewed said that while a  priority list of organisms will still be needed for responding to emerging  pathogens and diseases or synthetic biological agents, now a more  agnostic or “horizon spanning approach” will be used. Nevertheless, not  all experts were in agreement that DHS should pursue metagenomics for  bioforensics purposes, at least not in the short-term. For example, in a  2016 independent assessment of DHS Bioforensics R&D program,  reviewers recommended a “more measured investment” in metagenomics  and expressed doubt that an operational metagenomics capability was  likely to be available at NBFAC in 5 years. Instead, they suggested that  DHS take a more proactive investment stance by following developments  in the field that were occurring elsewhere.
Completion of Capability Enhancements:   Based on our review of the  roadmaps that DHS provided to us regarding the bioforensics  enhancements, DHS estimates that most of the R&D tasks associated  with capability enhancements will be completed by 2025 or later, with  some exceptions. For example, in July 2016, a DHS official indicated that  DHS’s new mass spectrometry casework capability may be available after  it has been accredited under ISO 17025 over the next 12 months (in  2017). In addition, DHS has a 3-5 year focus for developing  metagenomics so that it can be used on casework. Completion of more  advanced enhancements will likely extend beyond 2025, according to  DHS responses to our questions, such as for genomics and proteomics,  areas that are evolving. Activities include establishing (1) integrated  processes within metagenomics analyses to facilitate high resolution  characterization of all agents and nucleic acids in complex samples; (2) a  bioinformatic and statistical framework for phenotypic inference and  analysis of “unknown unknown” microbial isolates; and (3) increased  capabilities to support large-scale proteomic analysis integrated with  inferential analyses. See appendix II for more details on the BAAs.
section_title: DHS is establishing an In- House Reference Collection
paragraphs: DHS is also taking actions to establish an in-house reference collection of  biological materials—the NBFAC BRC—which will provide materials for  comparative forensic analyses, assay development and evaluation, and  proficiency testing. According to DHS responses to our questions, the  BRC is a long-term storage site for materials acquired from other  institutions (government, academia, commercial and international  sources) and NBACC projects. Housed at NBACC, it includes select and  nonselect agent bacteria and viruses, toxins, and their near neighbors.
The BRC supports characterization of bacterial and viral agents by  determining phylogenetic relatedness of different bacterial and viral  isolates and enabling isolate-level characterizations, which according to  DHS, is important for isolates that have never been fully characterized or  sequenced.  DHS began obtaining a variety of biomaterials such as  select agents and toxins through subcontracts with government agencies,  which were stored in external laboratories. In fiscal year 2010, the new NBACC laboratory opened, after which the collection was consolidated  within the biocontainment facilities at NBACC and became available for  use as a reference material. DHS states that it is working with the FBI to  expand the number of strains of interest in the BRC. In addition, DHS  projects that develop information on biological organisms are published in  the open, peer-reviewed literature; sequence data are published in  GeneBank and are available to the larger community and NBFAC.
section_title: DHS is Developing a Capability to Identify and Characterize Engineered, Novel, and Unknown Agents
paragraphs: DHS’s actions also include developing incrementally a capability for  identifying and characterizing genetically engineered novel, and unknown  (emerging or synthetic) agents that uses methods-based capabilities. In  this regard, NBFAC has developed a genomics capability that DHS  asserts can be used to infer genetic engineering from DNA sequencing  and protein sequences. Genetic engineering involves inserting a foreign  sequence of genetic codes into an existing sequence of genetic codes in  a target organism with a view to altering some of its functions. DHS states  it can identify genetic modifications by screening against genes of interest  (for example, virulence factors or antibiotic resistance genes), comparing  genome alignments, and comparing regions with unusual sequence  composition to those typically found in nature.
In the past, restriction enzymes have been used to cut DNA and insert  specific genes from a different organism to produce a desired effect (for  example, producing human insulin using bacterial cells), which results in  “scarring” at the restriction sites. Genome characterization and analysis  according to DHS’s and the FBI’s responses to our questions,  respectively, would be able to detect such scarring. However, gene  editing techniques are evolving and may be harder to detect. Clustered,  regularly interspaced, short palindromic repeats (CRISPR) Cas-9—is now  available to researchers. It engineers microbes by inserting genes,  although unlike previous methods the restriction sites may not be evident,  and the enzymes used will not cause scarring. Figure 4 is a simple  illustration of genetic engineering using restriction enzymes.
Experts from our meeting, as well as other experts we interviewed,  indicated that identifying  genetic engineering could be approached by  determining an agents’ virulence and then using capabilities such as  mass spectrometry to identify if elements exist that suggest modification.  Nevertheless, it was thought that a genetically engineered agent would  have some parts that remain unchanged, which would help to determine  its characteristics. For example, according to an expert at our meeting,  the focus should be on (1) identifying those introductions of additional  virulence or genetic elements into an organism, which can be done fairly  quickly, and then determining if there are other elements that suggest  somebody has modified the organism and (2) using methods like mass  spectrometry, microscopy, and other methods that can identify the means  of production or culture and dissemination or delivery of the organism. In  addition, according to a U.K. expert we interviewed, the core part of the  genome in an engineered agent would not be changed, and it also must  reproduce and metabolize. If it is an engineered virus, it would have some  similarities to other viruses, such as in how it attaches itself to a cell to  propagate its genome. So within its genome some signatures would be  available for comparison. Further, this expert stated that even if the  organism was a synthetic one and CRISPR Cas-9 had been used, he  would still look to see whether any scarring was present (see figure 4).
Regarding synthetic agents, DHS asserts that they can be analyzed  similarly to those that are genetically engineered, with the addition of  NBFAC’s inferential analysis capability, whose analysis will provide clues  to the functionality of a synthetic agent. DHS is developing a capability  that will allow NBFAC to characterize unique, novel agents, “unknowns”  (emerging or synthetic organisms) and “unknown, unknowns” (de novo  synthetic organisms). However, achieving this capability will also require  a bioinformatics and a statistical framework for inference and analysis of  unknowns in microbial isolates and expanded genome databases,  according to DHS. DHS indicates that it is developing a “multi-layered  inferential analysis capability” that would include establishing comparative  methods for the analysis of any DNA or protein sequence to identify such  things as peptides, restriction sites, and a statistical model that allows  confidence estimates to be placed on these analyses.
section_title: DHS Faces Numerous Challenges in Enhancing Its Bioforensics Capabilities
paragraphs: DHS faces numerous challenges as it attempts to enhance its  bioforensics capabilities. Our review of agency documentation and related  literature, and interviews with agency officials, scientists, and subject  matter experts at our meeting and elsewhere, as well as our prior work,  indicate that challenges must be overcome if DHS is to develop enhanced capabilities suitable for bioforensics. These include capabilities that not  only can be relied on for identifying and characterizing known agents but  also those that have been genetically engineered, or are unique, novel or  unknown (emerging or synthetic). Challenges that DHS faces include   (1) the ability to interpret and communicate results with defined statistical  confidence, (2) obtaining access to quality references and databases for  bioforensics analysis, and (3) the effect of the evolving threat landscape  on future casework needs. Further, their results must be also able to  stand up to court scrutiny.
subsections: section_title: DHS Faces Challenges in Its Ability to Interpret and Communicate Analyses Results with Statistical Confidence
paragraphs: DHS plans to develop advanced metagenomics and proteomics  capabilities. However, it is not clear to what extent or when DHS will be  able address key challenges related to enhancing its bioforensics  capabilities that include interpreting results from metagenomics and  proteomic analyses, with a defined statistical confidence, according to  both DHS and the FBI officials. Further, communicating the uncertainty  associated with the results will be particularly important when using these  capabilities on complex sample types.
Without a defined level of statistical confidence, the probative value of  inferences made from the results of such analyses may not be known.  In general, as the level of statistical confidence increases in these  inferences—signifying a higher degree of scientific certainty—the  probative value of the inference also increases. In figure 5, we have  extracted and expanded on one dimension of “the forensic continuum”  that has been used to represent the evaluation and analysis of a  bioforensics sample and its probative value. As indicated in the figure,  probative value depends on confidence in analysis and the interpretation  and meaning of evidence.
Investigative  leads or bioforensics data may rely on the use of  bioinformatics and data and inferences using those data. In this regard,  according to a DHS official, an issue DHS continues to struggle with is  how to interpret metagenomics analysis—whether it is possible to define  with certainty whether a piece of the genome of an agent is present— versus defining the error rates for each sequencing base call, which DHS  can do.
DHS solicitations for R&D reflect some of these challenges, including the  following extract from a related 2012 BAA solicitation regarding  interpreting the results of metagenomics analyses:  “Currently, it is difficult to assign confidence to the results of  metagenomic analyses. For example, in metagenomic  sequencing, what do a small number of reads that match a  particular organism say about the probability that the organism is  actually present in the sample? New methods are needed to  assess the likelihood that an organism is present in a  metagenomic sample and to provide confidence intervals on  abundance estimates.
Bioforensics R&D is looking to invest in the development and  application of mathematical models for (1) estimating the  likelihood of a genome being present in a metagenomic sample,  and (2) the most likely composition of a metagenomic sample  including a list of genomes and their relative abundance. The  system should go beyond metagenomic classification to provide a  statistically supported estimate of sample composition that could  be used in a biothreat agent detection context.”
Based on our review, we found that analysis of metagenomics data sets  will rely on advanced bioinformatics analyses that involve a large  statistical component. However, forensic casework may involve mixtures,  and separating these into individual components may be difficult—a  problem that may also apply to metagenomics. In this regard, challenges  involve the development and applications of appropriate bioinformatics  and data that will provide the ability to not only to describe the relative  abundance of sequence data but also to make inferences using those  data to provide either an investigative lead or to support attribution.  Efforts to achieve this are complex and will be conducted over multiple  years. For example, according to NBACC’s 2015 annual plan, “a  sequence-based, bioinformatics-driven genomics approach is a complex  endeavor that requires incremental implementation of critical technologies  over multiple years.”
Regarding proteomics, challenges remain in interpreting data. For  example, according to experts at our meeting, a quantitative measure for  proteomics needs to be available so that an informed decision can be  made. However, this is complicated by the lack of a framework for  expressing confidence in a result. Further, related to data analysis and  interpretation, for example, the potential for rapidly expanding protein  databases to result in false matches exists and the lack of standardized  approaches to proteomic data analysis is problematic.
section_title: DHS is Challenged by Data Quality and Accessing Agent Strains for the BRC
paragraphs: DHS must address several challenges related to its reference materials  that could affect NBFAC’s comparative analysis of evidentiary  samples.  According to DHS’s responses to our questions, these include access to  reference strains of interest, international agents, and ensuring the quality  of the data in the BRC. During our review we found that in contrast to  human DNA—a single species—the challenges for bioforensics involve a  multitude of species. Further, the quality of the data entered into a  particular database, including the metadata, and whether the database is  kept up to date could affect analysis if NBFAC uses that database. In  addition, ensuring that agents of interest are available for comparative  analyses is necessary.
Regarding the BRC specifically, not all strains are readily available and  obtaining agents internationally raises issues, according to DHS  responses. Further, not all researchers are willing to share their strains.  As a result, DHS is working with the FBI to develop an acquisition and  curation plan to expand the number of strains of interest in the BRC.   A DHS official stated that replacing agent-specific assays with DNA  sequencing methods will require DHS to have a comprehensive,  sophisticated database, which it currently does not have. Therefore,  ensuring the usefulness and quality of its reference collection and its  ability to obtain the strains of interest will continue to be a challenge for  DHS.
section_title: Experts Identified Two Key Challenges to Enhancing Bioforensics Capabilities
paragraphs: Experts at our meeting and others we interviewed identified two key  challenges associated with enhancing bioforensics capabilities:   (1) accessing and maintaining quality data on global microbial species in  databases and (2) implementing statistical frameworks and acceptable  communications of statistical analyses in court.
Reference databases and quality data:  Experts and officials in both the  United States and the United Kingdom whom we interviewed had differing  opinions about the challenges associated with obtaining access to global  microbial species and maintaining quality data for comparative analyses  of samples. For example, some indicated the need to establish and  maintain and a central database, whereas others considered it necessary  only to have the ability to access one that is relevant when an incident  occurs. Experts at our meeting also expressed reservations about  whether a centralized system is the best solution. They stated that it is  possible that a hybrid system in which each organization would own its  own dataset but would allow it to be searched by others might be a possible solution. Alternatively,  organizations could be required to send  their data to a central database in addition to storing it locally.
Questions have been raised about how reference data can be used  effectively for bioforensics. For example, regarding the use of population  genetics, it has been observed in the literature that a more useful  database for each pathogen would consist of a detailed record of human  and enzootic outbreaks noted through international outbreak surveillance  systems, and “representative” genetic sequences from each outbreak.  Another expert suggested that if necessary one could individually  forage  and collect organisms of interest in relevant areas or countries. Further,  according to these experts, much is known about Bacillus anthracis, but  other organisms like Burkholderia are much more challenging. To solve  this problem would involve first going back to close the gaps in the  reference databases and the population genetics.
Another challenge involved concerns about managing the quality of the  data entered into the database to ensure it meets quality standards.  Regarding the quality of such references for bioforensics, standards are  needed for data repositories and reference collections of pathogens and  other microorganisms, according to the experts at our meeting. Also,  because of the uncertainty about the reference data, the meeting’s  experts stated that the raw data should be maintained for further  analyses. In addition, according to these experts, questions about the  meaning of the data for these applications and the confidence value of  the data need to be resolved before focusing on them for bioforensics  purposes. According to a U.K. official we interviewed, the level of  uncertainty in matching microbes cannot be quantified, and attribution  depends on a reference set, which is incomplete for microbes. It could be  concluded that the microbe in question has the same DNA as that of a  microbe in the reference database, but not with certainty that it would not  match another microbe that is not in the reference database. Because of  the limits of using one approach, it is important to also use traditional  forensics to build an evidence base. In court, traditional forensics, in  addition to expert testimony on bioforensics, would therefore be used for  attribution.
Statistical frameworks and communicating  results:  As we reported in  2014, a statistical framework allows for statistically meaningful  comparative analyses; it is a set of concepts and organizing principles  that support the compilation and presentation of a set of statistics.  Experts at our meeting expressed the view that a gap that permeates  science, capabilities, and bioinformatics is the lack of a formulation or  framework for expressing confidence in genomics results as well as  similar challenges with non-genetic results. This is especially true with  mixed, metagenomics samples. Further, how to combine or communicate  uncertainties and error rates associated with the analytical and collection  processes needs greater clarity, according to the experts at our meeting.  They stated that to have a more robust statistical foundation, it is critical  to do enough experiments to assess the various contributions of these  sources of variability. NBFAC is moving into the realm of metagenomics,  which has problems with the statistical unknowns associated with it.  Metagenomic samples may contain mixtures as we stated previously. In  this regard, these experts stated that the problem of mixtures is an  opportunity for statistical methods to improve the results for different kinds  of evidence. For example, is the evidence confirmatory; is it consistent  with what’s in the database, and what are some possible alternatives that  could have given rise to the evidence? In addition, these experts stated  that challenges in a bioforensics context include the need for a  quantitative measure for genetics, proteomics, or other methods so that  an informed decision can be made.
Communicating results using statistical probabilities may not always be  acceptable by courts, despite the need for statistical frameworks to assist  in interpreting bioforensics analyses. Even if accepted, such statistical  information may not be understood. This issue is important because  statistics could play a large part in some types of analyses. For example,  using statistics when communicating the results of human DNA analysis  are generally accepted by courts in the United States. For interpretation  of bioforensic results, according to the experts at our meeting, the  question should be: What is the confidence you have achieved with the  data or information that you have? How that confidence is communicated  is important. It is forensic evidence, a piece of the puzzle. It adds value, but the confidence for it may be low, whereas the confidence for some  other evidence may be high. The level of uncertainty for that result should  also be indicated. Thus, bioforensic analyses of microbial DNA, and its  associated statistical elements, may have to overcome many obstacles  before they reach a similar level of acceptance in a U.S. federal court.
Other experts have acknowledged such challenges. For example,  regarding the U.S. legal system, issues may arise when new methods are  applied for the first time in bioforensics that have not undergone the depth  of scrutiny undergone by traditional forensic techniques. Also, reference  databases used for comparisons take time to develop. Our interviews  with U.K. scientists and government officials provided some insights into  issues associated with human DNA analysis results, which are generally  accepted by courts and that could have implications for bioforensics in  both the United Kingdom and the United States. For example, regarding  the communication of probability data, according to a U.K. official we  interviewed, while academics say there is some set level of probability to  achieve “beyond a reasonable doubt,” the court is concerned with the  baseline probability. However, this official stated that human DNA is the  only science in which the baseline probability data is considered  incontrovertible. Further, he said that in almost every other science, the  legal system would want assurances from expert witnesses regarding the  analysis results—not the numerical scientific results themselves.
In light of the discussion above, it is not clear how long it will take for the  results of metagenomics and proteomics analyses to be acceptable to  courts. Nevertheless, what is clear is that the ability to quantify statistical  uncertainty will require the use of comprehensive databases that contain  characteristics of signatures and information on the variations in the  population of the agent in question.
section_title: The Evolving Threat Landscape Raises Questions about the Bioforensics Capabilities That Will Be Needed
paragraphs: A long-term challenge facing NBFAC, according to DHS’s responses to  our questions, is the increasingly complex biological threat landscape:  New infectious disease agents emerge every year, and advances in  genetic engineering and “do it yourself” biology methods make the  nefarious use of enhanced and biological agents a possibility. DHS  further responded that as a result NBFAC must regularly establish new  methods and assays to support bioforensics casework that may involve  future threats. Further, we found it is still challenging to distinguish  between a natural and a deliberately released organism. However,  according to the experts at our meeting, when using “natural,” “accidental”  and “deliberate,” the issue could be more to do with the means by which  an agent is used, or about the characteristics of the agent itself.  Determining intent is likely to rely on information beyond the science  alone. In addition, while epidemiologic tools determine whether something  is unusual, what is also needed is a defined and validated tool that will  determine whether a microbe is unusual, made by humans, cultured, or  engineered.
Although DHS is developing capabilities to detect manipulated agents, it  faces several challenges related to the perceived potential for the creation  of agents that could cause harm accidentally or intentionally. To identify  and characterize novel synthetic agents, these challenges go beyond  identifying changes in an agent’s genomics (such as its antibiotic  resistance). DHS, in its responses to our questions, stated that identifying  more complex traits is more difficult because of the current scientific  understanding of how these processes work at the molecular level. DHS  also responded that both genetic engineering and NBFAC’s ability to infer  its intended effects require a deeper understanding of the physiology of  the biological agent as well as its interaction with a human host. Concerns  have also been raised about the potential for gain of function research to  result in manipulation of microbial agents with the potential for causing  harm. Such manipulations could involve, for example, agents or toxins  in which harmful consequences have been enhanced, such as making  them antibiotic resistant, more virulent, or more transmissible to  humans. The use of CRISPR Cas-9 also raises other issues because it  may be more difficult to detect than were previous gene editing approaches. However, not all agree that the risk of possible misuses of  biology is significant.
Regarding the use of genetically engineered agents to cause harm and  the likelihood  of this becoming a problem, we found differences in the  views of the experts we spoke to both here and in the United Kingdom.  Some said that it is difficult to create new pathogens so the use of  naturally-occurring microbes is of the greatest concern. While  acknowledging there are many technically possible misuses of biology,  they concluded that it is far more likely that minor modifications would be  made to existing organisms rather than the creation of new ones.
According to an expert we contacted, developing a new microbe with  novel pathogenic characteristics or antibiotic resistance significantly more  difficult than introducing these characteristics by gene manipulation.  Thus, one of the challenges DHS faces is to consider the risks in relation  to not only the bioforensics capabilities it needs but also its strategy for  addressing current and potential threats.
section_title: Conclusions
paragraphs: DHS has identified some bioforensics capability gaps since 2010 using an  informal, undocumented process but has not systematically identified the  gaps or performed a bioforensics capability gap analysis. In the absence  of a bioforensics gap analysis demonstrating the existence of gaps, it is  difficult to determine whether DHS has identified all its capability needs  and gaps. Identifying  gaps and prioritizing bioforensics capability needs  and gaps can help guide the proper allocation of resources to the highest  priority needs. Therefore, without a capability gap analysis and  documentation of the results of its process for identifying gaps, the  rationale for DHS’s resource allocations and its plans for future  enhancements to its existing capabilities are not clear.
section_title: Recommendation for Executive Action
paragraphs: We recommend that the Secretary of Homeland Security—in consultation  with the Federal Bureau of Investigation—conduct a formal bioforensics  capability gap analysis to identify scientific and technical gaps and needs  in bioforensics capabilities to help guide current and future bioforensics  investments and update its analysis periodically.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report for review and comment to DHS and the  FBI. DHS provided written comments, which are reproduced in appendix  IV. DHS concurred with our recommendation. The FBI did not provide  comments. Neither DHS nor the FBI provided technical comments.  In its  response, DHS described actions it plans to take to address the  recommendation. Specifically, according to DHS, S&T's Homeland  Security Advanced Research and Projects Agency's  Chemical and  Biological Defense (CBD) Division has initiated a formal, well- documented capability analysis of its Bioforensics R&D program. Further,  DHS stated that CBD will leverage this analysis to conduct a parallel  capability analysis of the Chemical Forensics and Attribution program that  addresses similar analytical and attribution needs for chemical threat  agents. DHS states that the CBD Division staff has prepared newly  updated Operational Requirements Documents and Strategic Plans  (Fiscal Years 2017-2021) for both programs, although we have not  reviewed these documents. According to DHS, the CBD Division initially  identified and compiled a number of bioforensics capability needs from a  review of external programs and meetings with end-users, such as the  FBI, and it is identifying and grouping additional needs under three areas  (science, technology and methods, and bioinformatics and data) through  reviews of documents, such as the National Research Council, Science  Needs for Microbial Forensics, 2014, and GAO’s report, Anthrax: Agency  Approaches to Validation and Statistical Analyses Could be Improved  (GAO-15-80), among others. According to DHS, the CBD Division is  conducting the formal capabilities analysis using methods and best  practices identified in the documents that include the DHS Instruction  Manual 107-01-001-01, DHS Manual for the Operation of Joint  Requirements Integration and Management System,  April 21, 2016;   DHS S&T "Requirements Development Guide" April 2008; and GAO’s  reports, Program Evaluation: Experienced Agencies Follow a Similar  Model for Prioritizing Research (GAO-11-176) and Chemical, Biological,  Radiological, and Nuclear Risk Assessments: DHS Should Establish  More Specific Guidance for Their Use (GAO-12-272). Finally, according  to DHS, the CBD Division is consolidating and prioritizing these needs to  ensure that they are in alignment and harmonized with current research  goals and strategic plans within DHS, S&T, Homeland Security Advanced  Research and Projects Agency, and the CBD Division. DHS plans to  complete these efforts by June 30, 2017, and states that the CBD  Division will ensure that the formal analysis is updated on an annual basis  and is used to guide current and future bioforensics investments.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date.  At that time, we will send copies of this report to the  Secretary of Homeland Security and the Director of the FBI, appropriate  congressional committees, and other interested parties. The report is also  available at no charge on GAO’s website at http://www.gao.gov.
If you and your staff have any questions about this report, please contact  Timothy M. Persons, Ph.D. at (202) 512-6412 or personst@gao.gov.  Contact points for our Office of Congressional Relations and Office of  Public Affairs may be found on the last page of this report. Key  contributors the report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: For this report, we evaluated (1) the extent to which DHS and the FBI  have identified gaps in their bioforensics capabilities since 2010,   (2) bioforensics needs experts have identified, and (3) any actions DHS  and the FBI have taken to enhance their bioforensics capabilities,  including those for characterizing a novel synthetic biological weapon,  and any challenges they have experienced in enhancing bioforensics  capabilities.
To determine the extent to which DHS and the FBI have identified gaps in  their bioforensics capabilities, we reviewed agency documents and  interviewed relevant agency officials about their efforts to identify such  gaps since 2010, which is when the Department of Justice closed the  FBI’s investigation into the 2010 anthrax case. We examined agency  planning documents, such as DHS’s Strategic Plan 2015-2019 and  NBFAC’s Bioforensics Roadmap for research, among others. We  reviewed DHS policy and guidance, such as DHS’s Joint Requirements  Integration and Management System, which formed the basis for the  criteria we used to compare and assess the extent to which DHS had  identified capability gaps or conducted a capability gap analysis of its  bioforensics capabilities. We also interviewed agency officials, including  those with DOD, to determine whether any gaps had been identified that  related to bioforensics and their interactions with DHS in this regard.
We developed a list of bioforensics needs that experts had identified. To  do this, we identified capabilities that might be needed for bioforensics  purposes from a 2014 NRC publication entitled Science Needs for  Microbial Forensics: Developing Initial International Research Priorities  and the 2009 National Research and Development Strategy for Microbial  Forensics from the National Science and Technology Council (NSTC).  We excluded capability needs in the literature that were not related to  science and technology development as these would have been beyond  our scope. We grouped the remaining capability needs into three broad  areas: (1) science, (2) technologies and methods, and (3) bioinformatics  and data. We then convened, with the assistance of the National  Academy of Sciences (NAS), a 2-day meeting of 16 experts to discuss  and update the capability needs we identified, including identifying issues  related to these needs. To identify the experts appropriate for the  meeting, we worked iteratively with NAS staff to identify and review  biographical information and relevant qualifications of experts, as well as  factors such as representation from academia, industry, and expertise in  a range of areas. The Board on Life Sciences of NAS solicited  nominations for the expert panel from its extensive contacts in the  microbial forensics area. From this initial list, NAS selected experts based  on their knowledge and expertise in forensics, microbiology, molecular  genetics, non-genetic methods, genetic engineering, bioinformatics,  statistics, and legal issues related to bioforensics. Once we came to  agreement with NAS on the final list of 16 experts for the meeting, these  experts were evaluated for any conflicts of interest. A conflict of interest  was considered to be any current financial or other interest that might  conflict with the service of an individual because it (1) could impair  objectivity and (2) could create an unfair competitive advantage for any  person or organization. We discussed internally all potential conflicts. The  experts were determined to be free of conflicts of interest, and the group  as a whole was judged to have no inappropriate biases. See appendix III  for a list of the experts. The meeting was recorded and transcribed to  ensure that we accurately captured the experts’ statements, and we  reviewed and analyzed the transcripts as a source of evidence. We  developed the session topics based on our researchable objectives and  issues that were identified in our audit work. The session topics were  gaps in the science underpinning bioforensics capabilities, gaps in  capabilities (technologies) and methods for attribution, and gaps in  bioinformatics, data and statistical Interpretation  of bioforensics. We  subsequently obtained their comments on the list of capability needs  identified during the April 2016 meeting to update and amend it based on  their input.
To determine the actions DHS and the FBI had taken to enhance their  bioforensics capabilities since 2010 and any challenges they  encountered, we reviewed agency documents, including planning  documents and research and development (R&D) efforts. We also  examined DHS’s actions to enhance NBFAC’s capabilities for the long  term as well as for the FBI’s casework. We reviewed DHS’s Broad Area  Announcements (BAA) and Open Broad Area Announcements (OBAA)  from 2008 to 2016. These are the mechanisms by which DHS solicits  research to develop its bioforensics capabilities. We obtained details on  contracted external R&D efforts. Deliverables included statistical models,  SOPs, and genetic sequences from external researchers. To determine  any challenges to enhancing bioforensics capabilities, we reviewed  agency documentation, including planning and contract documentation,  related literature, and our prior work on bioforensics. We interviewed  agency officials and scientists, including those at DHS, DOD, and the FBI  and obtained the opinions of experts in the United Kingdom, which  collaborates with DHS and the FBI on bioforensics-related issues, as well  as those in the United States regarding bioforensics-related challenges.  We also discussed potential challenges with experts present at our expert  meeting. We conducted site visits to national laboratories and academic  institutions conducting research on bioforensics-related issues, including  issues related to synthetic biology. These included discussions with DHS  contractors, scientists in academia, officials from the U.K. Home Office,  Public Health England at Porton Down, and scientists in academia  regarding challenges related to bioforensics capabilities. We also  interviewed some of the scientists involved in conducting research for  DHS.
We conducted this performance audit from July 2015 to January 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: DHS R&D to Enhance Bioforensics Capabilities, 2008 – 2016
subsections: section_title: Bacterial and viral bioforensics research and development
paragraphs: Sequence-based  approach to  bioforensics analysis  Whole genome  approach to microbial  forensics  Bioforensics research  R&D: Whole genome  approach to microbial  forensics  and the identification of adaptive mutations that can have forensic utility  Establishing match criteria for discriminating “difference” or “sameness” in sample  comparisons  Developing statistically rigorous sampling strategies to acquire spatially referenced  genetic information on reservoirs of these pathogens  Developing bioinformatics-based analytical tools for supporting hypotheses testing  regarding pathogen origin that go beyond current phylogeny-based inferential methods  and can meet forensic (legal) admissibility  Develop novel techniques to culture threat agents from complex environmental  samples  Improve dry collection and extraction strategies for forensic samples  Develop detection methods for rare variant detection in a bacterial sample using ultra- high throughput next generation sequencing technology  Understand dynamics of mobile elements in select agent bacteria  Develop forensic genotyping methods for select agent viruses   Develop novel applications of orthogonal methods to genetic characterization of  biological threat agent signatures and their sample matrices   Develop biased primer set design to amplify biological threat agents from complex  backgrounds  Production methods for ultraclean reagents  Sequence data error model for next-generation and single molecule sequencing  platforms  Taxonomic classification of metagenomic sequences  Develop and apply mathematical models for statistical confidence measurements in  metagenomic analysis  Develop a procedure to transport agents from BSL-3 to BSL-2 laboratories  Produce whole-genome sequencing to capture the global biodiversity of human, plant,  and animal pathogens (bacterial, viral, and fungal) in support of microbial forensics  analysis  Development and population of comparative genomic database with pathogen  sequence data at the National Center for Biotechnology Information Center  Products to identify select agents including C. botulinum toxins, with high confidence  Next generation and novel technologies to characterize biological threat agents (the  organism, the agent, or the sample matrix) for source attribution  Research on the bacterial populations of select agents with critical knowledge gaps,  including C. botulinum and B. anthracis (North Africa,  Middle East)
section_title: Year 2016
section_title: Appendix III: List of Experts
paragraphs: The names and affiliations of the experts who participated in the group  meeting held April 20-21, 2016, in Washington, D.C. are as follows:
Christopher Bidwell, J.D., Senior Fellow for Nonproliferation  Law and  Policy at the Federation of American Scientists.
Bruce Budowle, Ph.D., Professor, Executive Director of Institute of  Applied Genetics Molecular and Medical Genetics, University of North  Texas Health Science Center.
Rockne Harmon, J.D. Consultant, Instructor at U.C. Davis in the  Masters in Forensic Science program.
Dag Harmsen, MD, Ph.D., Professor, Head of Research, Center for  Oral and Maxillofacial  Surgery, Department of Periodontology,  University of Munster.
Molly Isbell, Ph.D., Director of Quality Assurance and Statistical  Sciences. Signature Science, LLC.
Dana Kadavy, Ph.D., Director of Biological Services, Signature  Science, LLC.
Karen Kafadar, Ph.D., Commonwealth Professor and Chair of  Statistics, University of Virginia.
Paul Keim, Ph.D., Regents’ Professor in Biology Cowden Endowed  Chair in Microbiology Northern Arizona University’s Microbial,  Genetics and Genomics Center. Northern Arizona University.
Jack Melling,  Ph.D. (via phone), Consultant.
Stephen S. Morse, Ph.D., Professor, Epidemiology, Founding Director  and Senior Resident Scientist, Center for Public Health Preparedness,  Columbia University,
Karen Nelson, Ph.D., President, The J. Craig Venter Institute.
David Relman, MD, Thomas C. and Joan M. Merigan Professor in  Medicine, and Microbiology  and Immunology, Co-Director of the  Center for International  Security and Cooperation, Stanford University,  and Chief of Infectious Diseases, the Veterans Affairs Palo Alto  Health Care System.
Tom Slezak, Ph.D., Associate Program Leader for Informatics for the  Global Security Program Efforts, Lawrence Livermore National  Laboratory.
Stephen Turner, Ph.D., Assistant Professor of Public Health Sciences,  University of Virginia School of Medicine.
Stephan Velsko, Ph.D., Senior Scientist and Associate Program  Leader Lawrence Livermore National Laboratory.
Karen Wahl, Ph.D., Chemist, Pacific Northwest National Laboratory.
The comments of most of these experts represented the views of the  experts themselves and not the agency, university, or company with  which they are affiliated.
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
paragraphs: Timothy M. Persons, (202) 512-6412 or personst@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, Sushil Sharma (Assistant  Director), Pille Anvelt, James Ashley, Hazel Bailey, Amy Bowser, Caitlin  Dardenne, Jack Melling, Jeff Mohr, Penny Pickett, Amber Sinclair, Maria  Stattel, Elaine Vaurio, and Elizabeth Wood made key contributions to this  report.